`i^dZddlZddlmZmZddlZddlmZm Z ddl m Z m Z m Z ddlmZmZmZmZmZmZmZmZmZmZmZmZddlmZmZdd lmZej e!Z" dde#de#de#de#de#de#de#fdZ$GddeZ%dgZ&dS)z#Image processor class for GLM-4.1V.N)OptionalUnion)BaseImageProcessor BatchFeature)convert_to_rgbresizeto_channel_dimension_format) OPENAI_CLIP_MEANOPENAI_CLIP_STDChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imagemake_flat_list_of_imagesto_numpy_array valid_imagesvalidate_preprocess_arguments) TensorTypelogging) VideoInput1 num_framesheightwidthtemporal_factorfactor min_pixels max_pixelscx||krtd|d|||ks||krtd|d|d|t||t||z dkr1tdt||t||z t||z |z}t||z |z}t||z |z} | |z|z|krtt j||z|z|z } t|t j|| z |z |z}t|t j|| z |z |z}nc| |z|z|krWt j|||z|zz } t j|| z|z |z}t j|| z|z |z}||fS)Nzt:z% must be larger than temporal_factor:zheight:z or width:z must be larger than factor:z4absolute aspect ratio must be smaller than 200, got ) ValueErrormaxminroundmathsqrtfloorceil) rrr r!r"r#r$h_barw_bart_barbetas /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/glm4v/image_processing_glm4v.py smart_resizer41sO##`j````aaa %&..`6``U``X^``aaa VU  c&%00 03 6 6 l3vuCUCUX[\bdiXjXjCj l l    &6/ " "V +E %&. ! !F *E *. / // AE u}uz))y*v-5CDDFDJv}v'=>>GHHFDJut|f'<==FGG   + +yzF':U'BCDD &4-&011F: %$,/0069 %<c&.eZdZdZddgZddejdddddddddf d ed ee e e fd ed ed e e e fdedee e ee fdee e ee fdede de de ddffd Zddddddddddddejdfde eefd eed ee e e fd eed eed ee deedee e ee fdee e ee fdee dee dee deedeedee e effdZddddddddddddddejdfdedeed eed ee e e fd eed eed ee deedee e ee fdee e ee fdee dee dee deedee e efdeedee e eff"dZd!de de fd ZxZS)"Glm4vImageProcessora Constructs a GLM-4V image processor that dynamically resizes images based on the original images. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image's (height, width) dimensions. size (`Dict[str, int]` *optional*, defaults to `{"shortest_edge": 112 * 112, "longest_edge": 28 * 28 * 15000}`): Size of the image's `(height, width)` dimensions after resizing. Can be overridden by the `size` parameter in the `preprocess` method. Available options are: - `{"height": int, "width": int}`: The image will be resized to the exact size `(height, width)`. Do NOT keep the aspect ratio. - `{"shortest_edge": int, "longest_edge": int}`: The image will be resized to a maximum size respecting the aspect ratio and keeping the shortest edge less or equal to `shortest_edge` and the longest edge less or equal to `longest_edge`. - `{"max_height": int, "max_width": int}`: The image will be resized to the maximum size respecting the aspect ratio and keeping the height less or equal to `max_height` and the width less or equal to `max_width`. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`): Resampling filter to use when resizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image by the specified scale `rescale_factor`. rescale_factor (`int` or `float`, *optional*, defaults to `1/255`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `[0.48145466, 0.4578275, 0.40821073]`): Mean to use if normalizing the image. This is a float or list of floats for each channel in the image. image_std (`float` or `List[float]`, *optional*, defaults to `[0.26862954, 0.26130258, 0.27577711]`): Standard deviation to use if normalizing the image. This is a float or list of floats for each channel in the image. do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. patch_size (`int`, *optional*, defaults to 14): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to 2): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to 2): The merge size of the vision encoder to llm encoder. pixel_valuesimage_grid_thwTNgp?r do_resizesizeresample do_rescalerescale_factor do_normalize image_mean image_stddo_convert_rgb patch_sizetemporal_patch_size merge_sizereturnc  Ftjdi| |d|vsd|vrtd|ddd}||_||_||_||_||_||_||nt|_ ||nt|_ | |_ | |_| |_| |_dS)N shortest_edge longest_edge:size must contain 'shortest_edge' and 'longest_edge' keys.rqrIrJ)super__init__r'r<r;r=r>r?r@r rAr rBrDrErFrC)selfr;r<r=r>r?r@rArBrCrDrErFkwargs __class__s r3rPzGlm4vImageProcessor.__init__|s ""6"""  !**DT&/&;$#6 $,r5images data_formatinput_data_formatc t|}| r d|D}d|D}|r/t|drtd|t |d}t |d|\}}||}}g}|D]}|r=t | ||| | | z|d|d \}}t|||f|| }|r|||| }|r| ||| | }t||| }| |tj |}|tjkr|dddd}|jd| zdkrVtj|dtj| |jd| zz d}tj||gd}|jd}|jd| z}|| z|| z}}||| ||| z| | || z| | }|ddddddddd }|||z|z|| z| z| z}||||ffS)a~ Preprocess an image or batch of images. Copy of the `preprocess` method from `CLIPImageProcessor`. Args: images (`ImageInput`): Image or batch of images to preprocess. Expects pixel values ranging from 0 to 255. If pixel values range from 0 to 1, set `do_rescale=False`. vision_info (`List[Dict]`, *optional*): Optional list of dictionaries containing additional information about vision inputs. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. `shortest_edge` and `longest_edge` keys must be present. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the `PILImageResampling` enums. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Scale factor to use if rescaling the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Mean to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Standard deviation to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image. patch_size (`int`, *optional*, defaults to `self.patch_size`): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to `self.temporal_patch_size`): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to `self.merge_size`): The merge size of the vision encoder to llm encoder. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. c,g|]}t|SrN)r.0images r3 z3Glm4vImageProcessor._preprocess..s @@@nU++@@@r5c,g|]}t|SrN)rrYs r3r\z3Glm4vImageProcessor._preprocess..s <<r?r@rArBrDrErFrCrUrVrr resized_height resized_widthprocessed_imagesr[patchesrepeatschannelgrid_tgrid_hgrid_wflatten_patchess r3 _preprocesszGlm4vImageProcessor._preprocesss/z*&11  A@@@@@F=vay I I &vay>OPPP (.  + +E 0<2!$7% 2#O4#N3111-  ?(fw g U.Te ff jiSd'0{VghhhE  # #E * * * *(+,, */ / /''1a33G = 1 1Q 6 6i BJ')< a@PSf@f)gnoGngw%7a@@@G-"q!%88':5} 7R//    j   j     ##Aq!Q1aA>>!// VOf $g0C&Cj&PS]&]   888r5videosreturn_tensorsc`||n|j}|d|vsd|vrtd|ddd}||n|j}||n|j}||n|j}||n|j}||n|j}| | n|j} | | n|j} | | n|j } | | n|j } | | n|j } ||n|j }|$| |}t|}|t|stdt!||| | ||| i}|gg}}|D]S}||||||||| | | | | ||| \}}||||Tt)j|}t)j|}|||d t/|| S) a Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If passing in images with pixel values between 0 and 1, set `do_rescale=False`. videos (`VideoInput`): Video to preprocess. Expects a single or batch of videos with pixel values ranging from 0 to 255. If passing in videos with pixel values between 0 and 1, set `do_rescale=False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image. size (`Dict[str, int]`, *optional*, defaults to `self.size`): Size of the image after resizing. Shortest edge of the image is resized to size["shortest_edge"], with the longest edge resized to keep the input aspect ratio. resample (`int`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only has an effect if `do_resize` is set to `True`. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): Rescale factor to rescale the image by if `do_rescale` is set to `True`. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`): Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`. image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`): Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to `True`. The max pixels of the image to resize the image. patch_size (`int`, *optional*, defaults to `self.patch_size`): The spatial patch size of the vision encoder. temporal_patch_size (`int`, *optional*, defaults to `self.temporal_patch_size`): The temporal patch size of the vision encoder. merge_size (`int`, *optional*, defaults to `self.merge_size`): The merge size of the vision encoder to llm encoder. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. NrIrJrKrrLrMzkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.)r?r@rArBr;r<r=)r;r<r=r>r?r@rArBrDrErFrUrCrV)r8r9)data tensor_type)r<r'r;r=r>r?r@rArBrDrErFrC fetch_imagesrrrrextendrorprqupdater)rQrTrr;r<r=r>r?r@rArBrDrErFrCrrUrVrr8vision_grid_thwsr[r|r9s r3 preprocesszGlm4vImageProcessor.preprocess(stZ'ttTY  !BBBBr5rr c &|d|j}|d|j}|dddd}||z}t|j||||d|d|j \}} ||z| |z} } | | zS) a A utility that returns number of image patches for a given image size. Args: height (`int`): Height of the input image. width (`int`): Width of the input image. images_kwargs (`dict`, *optional*) Any kwargs to override defaults of the image processor. Returns: `int`: Number of image patches per image. rDrFr<rrLrMrIrJ)rrr r"r#r$r!)getrDrFr4rE) rQrr images_kwargsrDrFr<r"ryrzrrs r3get_number_of_image_patchesz/Glm4vImageProcessor.get_number_of_image_patchess#&&|T_EE "&&|T_EE   9Ve)f)fggj((4/O,N+ 4) ) ) % (:5} 7Rr5)N)__name__ __module__ __qualname____doc__model_input_namesrBICUBICboolrdictstrintrfloatlistrPr FIRSTrrrrrr __classcell__)rSs@r3r7r7Rsw%%N()9:)-'9'A,3!:>9=##$"-"-"-tCH~&"-% "-  "- c5j) "-"-U5$u+#567"-E%e"456"-"-"-!"-"- "-"-"-"-"-"-N%))-15%)*.'+:>9=$(-1$()-2B2HDH!F9F9j*,-F9D>F9tCH~& F9 -. F9 TN F9!F9tnF9U5$u+#567F9E%e"456F9SMF9&c]F9SMF9!F9./F9 $E#/?*?$@A!F9F9F9F9V(,$()-15%)*.'+:>9=$(-1$()-;?2B2HDH%NCNCNC$NCD> NC tCH~& NC -. NCTNNC!NCtnNCU5$u+#567NCE%e"456NCSMNC&c]NCSMNC!NC !sJ!78!NC"./#NC$$E#/?*?$@A%NCNCNCNC`#cr5r7)rrrr)'rr+typingrrnumpyrpimage_processing_utilsrrimage_transformsrr r image_utilsr r r rrrrrrrrrutilsrr video_utilsr get_loggerrrkrr4r7__all__rNr5r3rs*) """"""""FFFFFFFF                            )(((((((%%%%%%  H % %0       BCCCCC,CCCL ! !r5