`i{~dZddlZddlmZmZddlZddlmZm Z m Z ddl m Z m Z mZddlmZmZmZmZmZmZmZmZmZmZddlmZmZmZmZmZm Z m!Z!erddl"Z"e j#e$Z%d ee&e&ee&eefd e&e&efd Z'Gd d e Z(GddeZ)dgZ*dS)zImage processor class for Fuyu.N)OptionalUnion)BaseImageProcessor BatchFeature get_size_dict)padresizeto_channel_dimension_format) ChannelDimension ImageInputPILImageResamplingget_image_sizeinfer_channel_dimension_formatis_scaled_imageis_valid_imagemake_list_of_imagesto_numpy_arrayvalidate_preprocess_arguments) TensorTypefilter_out_non_signature_kwargsis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsimagesreturnct|r|ggSt|trtd|Dr|St|tr d|DSt d)Nc3@K|]}t|tVdSN) isinstancelist.0images /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/fuyu/image_processing_fuyu.py z.make_list_of_list_of_images..@s,'T'TE 5$(?(?'T'T'T'T'T'Tc,g|]}t|S)rr$s r' z/make_list_of_list_of_images..Ds!???u#E**???r)zHimages must be a list of list of images or a list of images or an image.)rr"r#all ValueError)rs r'make_list_of_list_of_imagesr/:sfz&$C'T'TV'T'T'T$T$T &$@?????? _ ` ``r)cDeZdZdZddeeeeffdZd dZ dS) FuyuBatchFeaturez BatchFeature class for Fuyu image processor and processor. The outputs dictionary from the processors contains a mix of tensors and lists of tensors. N tensor_typec||S||\fdfd|D]|\}t|tr-t|dtrfd|D|<Gt|trfd|D|<n||<}|S)a5 Convert the inner content to tensors. Args: tensor_type (`str` or [`~utils.TensorType`], *optional*): The type of tensors to use. If `str`, should be one of the values of the enum [`~utils.TensorType`]. If `None`, no modification is done. N)r2c4|r|S|Sr!r+)elem as_tensor is_tensors r'_convert_tensorz._convert_tensor^s&y  9T?? "r)cn |S#dkrtdtdxYw)Noverflowing_valueszKUnable to create tensor returning overflowing values of different lengths. zUnable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.)r.)r5r8keys r'_safe_convert_tensorzAFuyuBatchFeature.convert_to_tensors.._safe_convert_tensorcsT &t,,, ...$%rsss Xs &4rc,g|]}fd|DS)c&g|] }|Sr+r+r%r5r<s r'r,zBFuyuBatchFeature.convert_to_tensors...rs%KKKT22488KKKr)r+)r%elemsr<s r'r,z7FuyuBatchFeature.convert_to_tensors..rs/___PUKKKKUKKK___r)c&g|] }|Sr+r+r?s r'r,z7FuyuBatchFeature.convert_to_tensors..us%JJJD11$77JJJr))_get_is_as_tensor_fnsitemsr"r#)selfr2valuer8r<r6r7r;s @@@@@r'convert_to_tensorsz#FuyuBatchFeature.convert_to_tensorsPs  K#99k9RR 9 # # # # # #      **,, 8 8JC%&& 8:eAh+E+E 8____Y^___S E4(( 8JJJJEJJJS 1077S  r)rrc t|dgddl i}d tdkrtd}t |rn\t |t s$t|st |tr| n tdt |d  fd | D]\}}t |trIt |dtr.g}|D]#}| fd|D$|||<ct |tr fd |D||< |||<||_ |S) a  Send all values to device by calling `v.to(*args, **kwargs)` (PyTorch only). This should support casting in different `dtypes` and sending the `BatchFeature` to a different `device`. Args: args (`Tuple`): Will be passed to the `to(...)` function of the tensors. kwargs (`Dict`, *optional*): Will be passed to the `to(...)` function of the tensors. Returns: [`BatchFeature`]: The same instance after modification. torchrNdevicez*Attempting to cast a BatchFeature to type z. This is not supported.crj|r |jiS|S|S)N)rI)is_floating_pointto)r5argsrIkwargsrHs r'_toz FuyuBatchFeature.to.._tosN&u&t,, 0tw////!wwfw---Kr)c&g|] }|Sr+r+r%r5rOs r'r,z'FuyuBatchFeature.to..s!!>!>!>##d))!>!>!>r)c&g|] }|Sr+r+rQs r'r,z'FuyuBatchFeature.to..s!777Tss4yy777r))rrHgetlenrr"strrintr.rCr#appenddata) rDrMrNnew_dataargkvnew_vr@rOrIrHs `` @@@r'rLzFuyuBatchFeature.to{s $ *** H%% >c$ii!mmq'Cc"" rC%% r)=)= rCQTAUAU r!!pcRUhh!p!p!pqqq        JJLL % %DAq!T"" %z!A$'='= %@@ELL!>!>!>!>!>!>!>????# At$$ %7777Q777 !c!ff   r)r!)rr) __name__ __module__ __qualname____doc__rrrUrrFrLr+r)r'r1r1Isa ))huS*_7M.N))))V888888r)r1c!>eZdZdZgdZddejdddddddddf d ed ee e e fd ed ed e de dede e ee fde e ee fdede dee e e fffd Zejddfdejd e e e fd edee e efdee e efdejf dZ d-dejd e e e fde de dee e efdee e efdejfdZeddddddddddddejddfd eed ee e e fd eed eed ee dee deedee dee deedee dee e e fdee e efdee e efdeefdZd.d e d!e dee e e fde fd"Zd.dd#dee e e fdd#fd$Z d.d%d#d&d#d'd#d(d#d)e d*e d+edee e e fdefd,ZxZS)/FuyuImageProcessora This class should handle the image processing part before the main FuyuForCausalLM. In particular, it should handle: - Processing Images: Taking a batch of images as input. If the images are variable-sized, it resizes them based on the desired patch dimensions. The image output is always img_h, img_w of (1080, 1920) Then, it patches up these images using the patchify_image function. - Creating Image Input IDs: For each patch, a placeholder ID is given to identify where these patches belong in a token sequence. For variable-sized images, each line of patches is terminated with a newline ID. - Image Patch Indices: For each image patch, the code maintains an index where these patches should be inserted in a token stream. Args: do_resize (`bool`, *optional*, defaults to `True`): Whether to resize the image to `size`. size (`dict[str, int]`, *optional*, defaults to `{"height": 1080, "width": 1920}`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `Resampling.BILINEAR`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. do_pad (`bool`, *optional*, defaults to `True`): Whether to pad the image to `size`. padding_value (`float`, *optional*, defaults to 1.0): The value to pad the image with. padding_mode (`str`, *optional*, defaults to `"constant"`): The padding mode to use when padding the image. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. image_mean (`float`, *optional*, defaults to 0.5): The mean to use when normalizing the image. image_std (`float`, *optional*, defaults to 0.5): The standard deviation to use when normalizing the image. do_rescale (`bool`, *optional*, defaults to `True`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `1 / 255`): The factor to use when rescaling the image. patch_size (`dict[str, int]`, *optional*, defaults to `{"height": 30, "width": 30}`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. rimage_input_ids image_patchesimage_patch_indices_per_batch#image_patch_indices_per_subsequenceTN?constantg?gp? do_resizesizeresampledo_pad padding_value padding_mode do_normalize image_mean image_std do_rescalerescale_factor patch_sizec  tjdi| ||_||nddd|_||_||_||_||_||_||_ | |_ | |_ | |_ | | nddd|_ dS)Ni8i)heightwidthr+)super__init__rkrlrmrnrorprqrrrsrtrurv)rDrkrlrmrnrorprqrrrsrtrurvrN __class__s r'r|zFuyuImageProcessor.__init__s ""6"""" ,DDTD2Q2Q    *(($"$,(2(>**r\^D_D_r)r& data_formatinput_data_formatrc t||\}}|d|d} } || kr|| kr|S| |z } | |z } t| | } t|| z}t|| z}td|||f|||d|}|S)a Resize an image to `(size["height"], size["width"])`. Args: image (`np.ndarray`): Image to resize. size (`dict[str, int]`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `PILImageResampling.BILINEAR`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Returns: `np.ndarray`: The resized image. rxry)r&rlrmr~rr+)rminrVr )rDr&rlrmr~rrN image_height image_width target_height target_widthheight_scale_factorwidth_scale_factoroptimal_scale_factor new_height new_width scaled_images r'r zFuyuImageProcessor.resize sF%35:K$L$L! k&*8nd7m| , & &<=+H+HL+l:)K7"#68JKK(<<==  &::;;  i(#/      r)modeconstant_valuesct||\}}|d|d} } d} d} | |z } | |z }t|| | f| |ff||||}|S)a Pad an image to `(size["height"], size["width"])`. Args: image (`np.ndarray`): Image to pad. size (`dict[str, int]`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. data_format (`ChannelDimension` or `str`, *optional*): The data format of the output image. If unset, the same format as the input image is used. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. rxryr)paddingrrr~r)rr )rDr&rlrrr~rrrrr padding_top padding_leftpadding_bottom padding_right padded_images r' pad_imagezFuyuImageProcessor.pad_imageDs,%35:K$L$L! k&*8nd7m|   &5${2  !>2\=4QR+#/    r)return_tensorsc p ||nj}nj||nj}||nj}| | nj}  nj ||nj}  nj  nj nj nj | | nj}  nj | | nj } t|tr(td|Drtdt!|}t#| | ||d|D}d|Ddd}| r)t%|rt&dt+|fd |D}t-|rfd |D}fd |D}d |D}d |D}dt/||D}|rfd|D}| r fd|D}|r fd|D}fd|D}||||d}t1||S)a Utility function to preprocess the images and extract necessary information about original formats. Args: images (`ImageInput`): Images to preprocess. Expects a single image, a list or images or a list of lists of images. Pixel values range from 0 to 255, or between 0 and 1 if `do_rescale` is `False`. do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the image to `size`. size (`dict[str, int]`, *optional*, defaults to `self.size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output image. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): `PILImageResampling` filter to use when resizing the image e.g. `PILImageResampling.BILINEAR`. do_pad (`bool`, *optional*, defaults to `self.do_pad`): Whether to pad the image to `size`. padding_value (`float`, *optional*, defaults to `self.padding_value`): The value to pad the image with. padding_mode (`str`, *optional*, defaults to `self.padding_mode`): The padding mode to use when padding the image. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. image_mean (`float`, *optional*, defaults to `self.image_mean`): The mean to use when normalizing the image. image_std (`float`, *optional*, defaults to `self.image_std`): The standard deviation to use when normalizing the image. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the image. rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`): The factor to use when rescaling the image. patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format of the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. Nc3fK|],}t|tot|dkV-dS)N)r"r#rT)r%r5s r'r(z0FuyuImageProcessor.preprocess..s=+i+iZ^JtT,B,B,Us4yyTU~+i+i+i+i+i+ir)z:Multiple images for a single sample are not yet supported.)rtrurqrrrsrkrlrmc&g|]}d|DS)c,g|]}t|Sr+)rr$s r'r,z..s CCC5..CCCr)r+r%rs r'r,z1FuyuImageProcessor.preprocess..s'___CCFCCC___r)cg|]}||Sr+r+rs r'r,z1FuyuImageProcessor.preprocess..sKKK&FKvKKKr)rzIt looks like you are trying to rescale already rescaled images. If the input images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again.cBg|]}|t|dSr) channel_dimrr%rrs r'r,z1FuyuImageProcessor.preprocess..sA   IOci 6!92C D D D   r)c0g|]}fd|DS)c@g|]}|S))rlr)r )r%r&rrDrls r'r,z..s-hhhX]UARSShhhr)r+)r%rrrDrls r'r,z1FuyuImageProcessor.preprocess..sEihhhhhaghhhr)cBg|]}|t|dSrrrs r'r,z1FuyuImageProcessor.preprocess..s2uuuTZntu~fQi=NOOOuuur)c g|] }|dg Srr+r% image_sizes r'r,z1FuyuImageProcessor.preprocess..s!P!P!Pj:a=/!P!P!Pr)c g|] }|dg S)r+rs r'r,z1FuyuImageProcessor.preprocess..s O O OZ*Q- O O Or)c8g|]\}}|d|dz gSrr+)r% original_size resized_sizes r'r,z1FuyuImageProcessor.preprocess..s=   + |!_}Q/ / 0   r)c4g|]}fd|DS)c Dg|]}|S))rlrrr)r)r%r&rrprorDrls r'r,z..sM   NN!)(5*; #   r)r+)r%rrrprorDrls r'r,z1FuyuImageProcessor.preprocess..sb           "(      r)c0g|]}fd|DS)c@g|]}|S))scaler)rescale)r%r&rrurDs r'r,z..s-tttdie>M^__tttr)r+)r%rrrurDs r'r,z1FuyuImageProcessor.preprocess..sEutttttmstttr)c2g|]}fd|DS)cBg|]}|S))meanstdr) normalize)r%r&rrrsrrDs r'r,z.. s>NN5zy\mNnnr)r+)r%rrrrsrrDs r'r,z1FuyuImageProcessor.preprocess.. s[  !'r)c.g|]}fd|DS)c2g|]}t|Sr+)r )r%r&r~rs r'r,z..s(hhhX],UKARSShhhr)r+)r%rr~rs r'r,z1FuyuImageProcessor.preprocess..sBihhhhaghhhr))rimage_unpadded_heightsimage_unpadded_widthsimage_scale_factors)rXr2)rkrlrmrnrtrurqrrrsrorprvr"r#anyr.r/rrlogger warning_oncerrzipr1)rDrrkrlrmrnrorprqrrrsrtrurvr~rr batch_imagesfirst_image_in_listoriginal_image_sizes image_sizesrrrrXs` ` `` `` ` `` r' preprocesszFuyuImageProcessor.preprocessjsL"+!6IIDN 'ttTY'388!-4;#-#9ZZt +9+E4K^'3'?||TEV #-#9ZZt !*!6IIDN )6)B HZ '3'?||TEV #-#9ZZt +9+E4K^#-#9ZZt fd # # [+i+ibh+i+i+i(i(i [YZZ Z26:: %!)%! `_R^___ LKLKKKANqQ  /*=>>    s     $ >?R S S     S_   T""  *L vuuu^juuu !P!PK!P!P!P O O; O O O  /23G/U/U             +   L  *L   + L  "*L #&<%:#6     T~FFFFr)rrc||n|j}|jd|jd}}||zdkrtd|d|||zdkrtd|d|||z}||z}||z}|S)a Calculate number of patches required to encode an image. Args: image_height (`int`): Height of the image. image_width (`int`): Width of the image. patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. Nrxryrz image_height=z must be divisible by z image_width=)rvr.) rDrrrv patch_height patch_widthnum_patches_per_dim_hnum_patches_per_dim_w num_patchess r'get_num_patchesz"FuyuImageProcessor.get_num_patches s$.#9ZZt $(OH$=tw?Wk , &! + +S SS\SSTT T  $ ) )Q QQKQQRR R , < +{ :+.CC r)z torch.Tensorct|dg||n|j}|d|d}}|j\}}}}|d||}|d||} | } | ||d||} | dddd d } | |d||z|z} | S) a| Convert an image into a tensor of patches. Args: image (`torch.Tensor`): Image to convert. Shape: [batch, channels, height, width] patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the patches. rHNrxryrrrr)rrvshapeunfold contiguousviewpermutereshape) rDr&rvrr batch_sizechannels_unfolded_along_heightpatchess r'patchify_imagez!FuyuImageProcessor.patchify_image9s $ ***#-#9ZZt $.x$8*W:Mk &+[" Ha % Q l K K'..q+{KK$$&&,,z8R{SS//!Q1a00//*b(\2IK2WXXr) image_input image_presentimage_unpadded_himage_unpadded_wimage_placeholder_idimage_newline_idvariable_sizedc t|dg||n|j}|d|d} } g} g} g} t|jdD]{}g}g}t|jdD].}|||fr|||f}|jd|jd}}|rut |t j|||f| z | z}t |t j|||f| z | z}|ddd|d|f}||}}|||}tj |g|tj |j }| | d d}||jdksJ|rz|d || z}tj |jddg|tj |j }tj||gd }|d }| |g|||||tjgtj |j 0| || |}g}g}| D] }d}g}g} |D]}!|!|k}"tj|"}tj|tj|!j |!}#tj|!d }$tj|!d }%tj|"d d}&|#|z|$|&<|#|%|&<||$| |%||z }|||| t3| | | ||dS)aProcess images for model input. In particular, variable-sized images are handled here. Args: image_input (`torch.Tensor` of shape [batch_size, subsequence_size, num_channels, height, width]): Tensor of images padded to model input size. image_present (`torch.Tensor` of shape [batch_size, subsequence_size, num_images]): Tensor of 1s and 0s indicating whether an image is present. image_unpadded_h (`torch.Tensor` of shape [batch_size, subsequence_size]): Tensor of unpadded image heights. image_unpadded_w (`torch.Tensor` of shape [batch_size, subsequence_size]): Tensor of unpadded image widths. image_placeholder_id (int): The id of the image placeholder token. Comes from an associated tokenizer. image_newline_id (int): The id of the image newline token. Comes from an associated tokenizer. variable_sized (bool): Whether to process images as variable-sized. patch_size (`dict[str, int]`, *optional*, defaults to `self.patch_size`): Size of the patches. rHNrxryrrr)rr)dtyperI)r&r)dimT)as_tuplerd)rX)rrvrangerrmathceilrrHfullint32rIr unsqueezesqueezercatrWtensor count_nonzeroarangeint64type_as full_likenonzeror1)'rDrrrrrrrrvrrrbatch_image_patchesbatch_image_input_ids batch_indexrerf subseq_indexr&rrnew_hnew_wrtensor_of_image_idsr newline_idsrgrhsample_image_input_ids index_offsetper_batch_indicesper_subsequence_indicessubseq_image_input_ids patches_maskindicesindices_in_stream_per_batch!indices_in_stream_per_subsequence patches_indss' r'preprocess_with_tokenizer_infoz1FuyuImageProcessor.preprocess_with_tokenizer_infoSs> $ ***#-#9ZZt $.x$8*W:Mk ,.8::< !21!566/ 6/ 6K OM %k&7&: ; ;) k) k  l!:;(k' \(ABE05 A A+L% A!$( I&6{L7P&QT`&`aadpp!!!$' I&6{L7P&QT_&_``cnn!!!&aaa%%&7 8495k "&"6"6L^i"6"j"jK*/*$ ';5;WbWi+++'#118J8J1KKSSTUVVG&'-*:::::% N.A.I.I"k]hNh.i.i+&+j06q91=,"'+#.#5 ''' /4i9Lk8Z`a.b.b.b+.A.I.I".M.M+MM5'***#**+>???!((1111#**5<%+VaVh+i+i+ijjjj ! ( ( 9 9 9  & &} 5 5 5 5CE%HJ+&; P P "L " &( #*@ , ,&59MM #1,?? ,{%+NdNkllltt* /4o>TVX.Y.Y+49ODZ\^4_4_1$}\DIII!L BDH 88z838n8% 8 eC)9$9:; 8 $E#/?*?$@A 8 8888|!$>BDH$$z$38n$ $  $ eC)9$9:; $$E#/?*?$@A$ $$$$L%$&&%))-15!%)-&*'+&*%)%)*./3>N>TDH/3#sGsGD>sGtCH~& sG -. sG  sG sGsmsGtnsGUOsGE?sGTNsG!sGT#s(^,sGeC)9$9:;sG $E#/?*?$@A!sG"!,#sGsGsG'&sGjCcxX\]`be]eXfOgsv2NcSVh@XdrF04B B #B &B ) B ) B " B B B T#s(^,B  B B B B B B B B r)rc)+rartypingrrnumpyrimage_processing_utilsrrrimage_transformsr r r image_utilsr r rrrrrrrrutilsrrrrrrrrH get_loggerr^rr#r/r1rc__all__r+r)r'rsP&% """"""""UUUUUUUUUU                        LLL  H % % a $tJ'($z*:JF G a $z  a a a ajjjjj|jjjZ_ _ _ _ _ +_ _ _ D r)