`iWMdZddlZddlZddlmZmZddlZddlm Z ddl m Z m Z ddl mZmZmZmZddlmZmZmZmZmZmZmZdd lmZmZmZmZdd lm Z erddl!Z!dd l"m#Z#m$Z$m%Z%erddl&Z&ej'e(Z)d Z*d Z+ d$de,de-de,de,de-de-de-de-dee.dee,de#j#fdZ/ d%dej0de,d eee,e1ffd!Z2Gd"d#e Z3d#gZ4dS)&z%Image processor class for Pix2Struct.N)OptionalUnion)hf_hub_download)BaseImageProcessor BatchFeature)convert_to_rgb normalizeto_channel_dimension_format to_pil_image)ChannelDimension ImageInputget_image_sizeinfer_channel_dimension_formatmake_flat_list_of_imagesto_numpy_array valid_images) TensorTypeis_torch_availableis_vision_availablelogging)requires_backends)Image ImageDraw ImageFontzybelkada/fontsc<ttdg|d}tjj|||f||f}||d|d||d}| ddddd|d|z|d|z|d|z|z}|dS) a Utility function to extract patches from a given image tensor. Returns a tensor of shape (1, `rows`, `columns`, `num_channels`x `patch_height` x `patch_width`). Args: image_tensor (torch.Tensor): The image tensor to extract patches from. patch_height (int): The height of the patches to extract. patch_width (int): The width of the patches to extract. torchr)strider) rtorch_extract_patches unsqueezernn functionalunfoldreshapesizepermute) image_tensor patch_height patch_widthpatchess /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/pix2struct/image_processing_pix2struct.pyr#r#4s+gY777))!,,Lh!(( k7R\hju[v(wwGool//22L4E4Ea4H4H,XceghhGooaAq!,,44! ,! +!|+k9G   Q  $blackwhitetext text_size text_colorbackground_color left_padding right_padding top_paddingbottom_padding font_bytes font_pathreturnc zttdtjd} | |} d| } || t j|} n| | } nttd} tj | d| } tj tjd d |}|d | | \}}}}||z|z}||z|z}tjd ||f|}tj |}|||f| || |S)a Render text. This script is entirely adapted from the original script that can be found here: https://github.com/google-research/pix2struct/blob/main/pix2struct/preprocessing/preprocessing_utils.py Args: text (`str`, *optional*, defaults to ): Text to render. text_size (`int`, *optional*, defaults to 36): Size of the text. text_color (`str`, *optional*, defaults to `"black"`): Color of the text. background_color (`str`, *optional*, defaults to `"white"`): Color of the background. left_padding (`int`, *optional*, defaults to 5): Padding on the left. right_padding (`int`, *optional*, defaults to 5): Padding on the right. top_padding (`int`, *optional*, defaults to 5): Padding on the top. bottom_padding (`int`, *optional*, defaults to 5): Padding on the bottom. font_bytes (`bytes`, *optional*): Bytes of the font to use. If `None`, the default font will be used. font_path (`str`, *optional*): Path to the font to use. If `None`, the default font will be used. visionP)width)r5 Nz Arial.TTFzUTF-8)encodingr)RGB)rrrr)xyr5fillfont)r render_texttextwrap TextWrapperwrapjoinioBytesIOrDEFAULT_FONT_PATHrtruetyperDrawrnewtextbboxr5)r5r6r7r8r9r:r;r<r=r>wrapperlines wrapped_textrJ temp_draw_ text_width text_height image_width image_heightimagedraws r/rKrKOsKLk8,,,",,,G LLdL # #E99U##L)"3z*%%  0+>>  dW9 E E EDuy8HIIJJI$-$6$6v|T$R$R!Aq*k|+m;K,~=L Iek<8:J K KE >% DII, ,>JL/9|?Q3QRSS %)Z:K-K!LgVVI OOL''4E(FGGPPP OOELL)Z!899A?P;QRRRy))I%i004D4III/ ;K;PQQ r0ceZdZdZddgZ dded ed eeee fd e d ed df fd Z dde j d e d edee eefd e j f dZ dde j dee eefdee eefd e j fdZddddddejdfdedeedeed eed ee d eeee fdee eefdedee eefd efdZxZS)Pix2StructImageProcessoraj Constructs a Pix2Struct image processor. Args: do_convert_rgb (`bool`, *optional*, defaults to `True`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `True`): Whether to normalize the image. Can be overridden by the `do_normalize` parameter in the `preprocess` method. According to Pix2Struct paper and code, the image is normalized with its own mean and standard deviation. patch_size (`dict[str, int]`, *optional*, defaults to `{"height": 16, "width": 16}`): The patch size to use for the image. According to Pix2Struct paper and code, the patch size is 16x16. max_patches (`int`, *optional*, defaults to 2048): The maximum number of patches to extract from the image as per the [Pix2Struct paper](https://huggingface.co/papers/2210.03347). is_vqa (`bool`, *optional*, defaults to `False`): Whether or not the image processor is for the VQA task. If `True` and `header_text` is passed in, text is rendered onto the input images. flattened_patchesattention_maskTNFdo_convert_rgb do_normalize patch_size max_patchesis_vqar?c tjdi|||nddd|_||_||_||_||_dS)N)rhrC)super__init__ryrxrwrzr{)selfrwrxryrzr{rl __class__s r/rz!Pix2StructImageProcessor.__init__s^ ""6"""(2(>**r\^D_D_(,& r0r`rcc 6t|jdt|tj|}t j|}|d|d}}t|tj\}} tj |||z z|| z z} tttj | |z|z |d} tttj | | z|z |d} t| |zd} t| |zd}t j j|d| |fddd d}t%|||}|j}|d}|d }|d }|||z|g}t j||dgd|||zdg}t j|d|g|d||zdg}|dz }|dz }|t j}|t j}t j|||gd }t j j|ddd|||zz g}t9|}|S) a Extract flattened patches from an image. Args: image (`np.ndarray`): Image to extract flattened patches from. max_patches (`int`): Maximum number of patches to extract. patch_size (`dict`): Dictionary containing the patch height and width. Returns: result (`np.ndarray`): A sequence of `max_patches` flattened patches. rrhrCrrbilinearFT)r)mode align_corners antialiasr"rr )rextract_flattened_patchesr r FIRSTr from_numpyrmathsqrtrfminfloorr%r& interpolater$squeezer#shaper(arangerepeattofloat32catpadfloatr)rr`rzryrcrlr,r-r_r^scalenum_feasible_rowsnum_feasible_colsresized_height resized_widthr. patches_shaperowscolumnsdepthrow_idscol_idsresults r/rz2Pix2StructImageProcessor.extract_flattened_patchess. $8'BBB,E3C3IK\]] ''$.x$8*W:Mk $25:J:P$Q$Q! k + )DEWbIbcddDJu|/Cl/R$S$SU` a acdeeDJu{/B[/P$Q$QS^ _ _abcc.=qAA- ;Q?? #// OOA   -0 0  '!** (|[II Q"a //4'>5"9::,t$$,,dAY77>>q'JJRRTX[bTbdeSfgg,w''//G ==DDT1MMUUW[^eWeghVijj 1 1 **U]++**U]++GWg6;;$((!Q;$QX.;Y1Z[[aacc'' r0 data_formatc V|jtjkr|tj}tj|}tj|}t|dtj tj |j z }t|f||||d|S)a Normalize an image. image = (image - image_mean) / image_std. The image std is to mimic the tensorflow implementation of the `per_image_standardization`: https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization Args: image (`np.ndarray`): Image to normalize. data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format for the output image. If unset, the channel dimension format of the input image is used. input_data_format (`str` or `ChannelDimension`, *optional*): The channel dimension format of the input image. If not provided, it will be inferred. g?)meanstdrrc) dtypenpuint8astyperrrrfrrprodrr )rr`rrcrlrradjusted_stddevs r/r z"Pix2StructImageProcessor.normalize5s, ;"( " "LL,,Ewu~~fUmmc3275;3G3G)H)H#HII  #/       r0images header_textreturn_tensorsc   ||nj}||nj}njnjj} | dt dt|}t|st d|r d|D}d|D} t|d | rt d| d d| d dttrgt|zfd t|D}|r fd |D} fd |D}d|D} t|| d|} | S)a Preprocess an image or batch of images. The processor first computes the maximum possible number of aspect-ratio preserving patches of size `patch_size` that can be extracted from the image. It then pads the image with zeros to make the image respect the constraint of `max_patches`. Before extracting the patches the images are standardized following the tensorflow implementation of `per_image_standardization` (https://www.tensorflow.org/api_docs/python/tf/image/per_image_standardization). Args: images (`ImageInput`): Image to preprocess. Expects a single or batch of images. header_text (`Union[list[str], str]`, *optional*): Text to render as a header. Only has an effect if `image_processor.is_vqa` is `True`. do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`): Whether to convert the image to RGB. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the image. max_patches (`int`, *optional*, defaults to `self.max_patches`): Maximum number of patches to extract. patch_size (`dict`, *optional*, defaults to `self.patch_size`): Dictionary containing the patch height and width. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. Can be one of: - Unset: Return a list of `np.ndarray`. - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`. - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`. - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`. - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input image. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input image. If unset, the channel dimension format is inferred from the input image. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: image in (height, width) format. Nrz8data_format is not an accepted input as the outputs are zkInvalid image type. Must be of type PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray.c,g|]}t|Sr~)r .0r`s r/ z7Pix2StructImageProcessor.preprocess..s @@@nU++@@@r0c,g|]}t|Sr~)rrs r/rz7Pix2StructImageProcessor.preprocess..s <<cHg|]\}}t||S))r=r>)re)rir`r=r>rs r/rz7Pix2StructImageProcessor.preprocess..sCAue[^ V_```r0c>g|]}|S))r`rc)r )rr`rcrs r/rz7Pix2StructImageProcessor.preprocess..s+kkk[`dnn5DUnVVkkkr0cBg|]}|S))r`rzryrc)r)rr`rcrzryrs r/rz7Pix2StructImageProcessor.preprocess..sJ     * *_p +     r0c|g|]9}|ddktj:S)r )axisr)sumrrrrs r/rz7Pix2StructImageProcessor.preprocess..s;\\\EEII2I..!3;;BJGG\\\r0)rtru)data tensor_type)rxrwryrzr{get ValueErrorrrrpop isinstancestrlen enumerater)rrrrwrxrzryrrrcrlr{attention_masksencoded_outputsr=r>s` ` `` ` @@r/ preprocessz#Pix2StructImageProcessor.preprocess\s9j(4'?||TEV +9+E4K^#-#9ZZt %0%vay I I   " !QRRRL$77J ;55I+s++ :*mc&kk9  )& 1 1F  lkkkkkdjkkkF           ]\U[\\\&'-QQ_m   r0)TTNrvFN)NN)__name__ __module__ __qualname____doc__model_input_namesboolrdictrrgrrndarrayrr rr rrrr __classcell__)rs@r/rsrsst(-.>? $!/3 T#s(^,      *EI OOzOO O $E#/?*?$@A O OOOOh?CDH % % z% eC)9$9:;% $E#/?*?$@A %  % % % % T&*)-'+%)/3;?(8(>DHqqqc]q! q tn q c] qT#s(^,q!sJ!78q&q$E#/?*?$@Aq qqqqqqqqr0rs) r1r2r3r4r4r4r4NNr)5rrPrtypingrrnumpyrhuggingface_hubrimage_processing_utilsrrimage_transformsr r r r image_utilsr rrrrrrutilsrrrrutils.import_utilsrrLPILrrrr get_loggerrloggerrRr#rrgbytesrKrChildProcessErrorrers__all__r~r0r/rs,+ """"""""++++++FFFFFFFFddddddddddddRQQQQQQQQQQQ3333330OOO//////////LLL  H % %$   :#"&#@@ @@@ @  @  @@@@}@ [@@@@Jbf'' :'"'7?cK\F\@]7^''''TPPPPP1PPPf & &r0