`iK(ddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZmZddlmZerddlZGd d e d Zd ZdedDdedDzZdZGdde ZdgZdS))OptionalUnion) BatchFeature) ImageInputmake_flat_list_of_images)MultiModalDataProcessingKwargsProcessorMixinUnpack) AddedTokenPreTokenizedInput TextInput)is_torch_availableNc(eZdZddidddddidZd S) ColPaliProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaults/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/colpali/processing_colpali.pyrr$sB y ,"  +D1  IIIr"rF)totalzcg|] }d|dd S)z4>r!.0is r# r*2s"555Aq555r"icg|] }d|dd S)z3r&r!r's r#r*r*2s"8]8]8]Q8]8]8]r"c ||z|z||dS)aZ Builds a string from the input prompt and image tokens. For example, for the call: build_string_from_input( prompt="Prefix str" bos_token="", image_seq_len=3, image_token="", ) The output will be: "Initial str" Args: prompt (`list[Union[str, ImageInput]]`): The input prompt. bos_token (`str`): The beginning of sentence token. image_seq_len (`int`): The length of the image sequence. image_token (`str`): The image token. num_images (`int`): Number of images in the prompt.  r!prompt bos_token image_seq_len image_token num_imagess r#build_string_from_inputr55s'&M)J6 M M6 M M MMr"c eZdZdZddgZdZdZ d d ed effd Z d!d e e d e e e ee ee fdeedefdZd"dZedefdZ d"d e e deedefdZd e e ee fdeedefdZ d#de dedfde dedfdede dde defddf dZxZS)$ColPaliProcessora Constructs a ColPali processor which wraps a PaliGemmaProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColPaliProcessor`] offers all the functionalities of [`PaliGemmaProcessor`]. See the [`~PaliGemmaProcessor.__call__`] for more information. Args: image_processor ([`SiglipImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`LlamaTokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. visual_prompt_prefix (`str`, *optional*, defaults to `"Describe the image."`): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*, defaults to `"Question: "`): A prefix to be used for the query. image_processor tokenizer)SiglipImageProcessorSiglipImageProcessorFast)GemmaTokenizerGemmaTokenizerFastNDescribe the image. Question: visual_prompt_prefix query_prefixct|||t|dstd|j|_t|ds]t t dd}d|gi}|||t |_ t |_ n|j |_ |j |_ | td|_ d|_||_||_dS) N) chat_templateimage_seq_lengthz;Image processor is missing an `image_seq_length` attribute.r3FT) normalizedspecialadditional_special_tokens)super__init__hasattr ValueErrorrDr IMAGE_TOKENadd_special_tokensconvert_tokens_to_idsimage_token_idr3 add_tokens EXTRA_TOKENS add_bos_token add_eos_tokenr@rA) selfr8r9rCr@rAr3 tokens_to_add __class__s r#rIzColPaliProcessor.__init__ds  )=QQQ(:;; \Z[[ [ / @y-00 5$[UDQQQK8;-HM  ( ( 7 7 7"+"A"A+"N"ND *D  "+":D (4D \***"' "' $8!(r"imagestextkwargsreturnc jtfdjji|}|ddd}|du}||t d||t d|'j|}t|}j gt|z} d|D}fdt| |D} j|fi|d d } |d d d|dd xxj z cc<j| fd d i|d} i| d | i} |r=| d| ddkd}| d|it!| S|t#|t$r|g}n?t#|t&rt#|dt$st d| jdz}g}|D]4}jjjz|z|zdz}||5|d d d|dd <j|fd d i|d}|SdS)a Main method to prepare for the model either (1) one or several texts, either (2) one or several image(s). This method is a custom wrapper around the PaliGemmaProcessor's [`~PaliGemmaProcessor.__call__`] method adapted for the ColPali model. It cannot process both text and images at the same time. When preparing the text(s), this method forwards the `text` and `kwargs` arguments to LlamaTokenizerFast's [`~LlamaTokenizerFast.__call__`]. When preparing the image(s), this method forwards the `images` and `kwargs` arguments to SiglipImageProcessor's [`~SiglipImageProcessor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timec8g|]}|dS)RGB)convert)r(images r#r*z-ColPaliProcessor.__call__..s$???uemmE**???r"c g|]Q\}}t|jjjtt |t rt|ndRS)r/)r5r9r1rDrL isinstancelistlen)r(r0 image_listrTs r#r*z-ColPaliProcessor.__call__..so   'FJ(!"n6"&"7 +2>F-f55F23c&kkAI?????F    +.i*@*@   M04/YY-:XYYZhiL]+// dCCOm,\:::d>SS:::#T^&+ .F CVB^\BBK$ 7 ,88@P9QUV9VX\]]""Hf#5666[111 1  $$$ Ov t,, ODGS1I1I O !MNNN~6;%'K * *043DDuLvUX\\""5))))9F}9U9Y9YZfhj9k9kM- ( 6($.&+ .K  - r"c i}|C|jgt|z}dgt|z}|||dtdi|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (list[list[str]], *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nrc)num_image_tokensnum_image_patchesr!)rDrfrxr )rT image_sizesrY vision_datarrs r#_get_num_multimodal_tokensz+ColPaliProcessor._get_num_multimodal_tokensso  " $ 56[9I9II !"c+&6&6 6    4D[lmm n n n,, ,,,r"c|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r9 pad_token)rTs r#rzz)ColPaliProcessor.query_augmentation_token s~''r"c |jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. rWr!r)rTrWrYs r#process_imageszColPaliProcessor.process_imagess"Bt}55F5f555r"c |jdd|i|S)a Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColPaliProcessor's [`ColPaliProcessor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). rXr!r)rTrXrYs r#process_queriesz ColPaliProcessor.process_queries7s"@t}11$1&111r"r,cpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec 8t|dkrtdt|dkrtd|dj|djkrtd|dj|djkrtd| |dj}g}t dt||D]:}g}t jjj ||||zdd} t dt||D]} t jjj || | |zdd} | t j d | |  d d d | t j|d || usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT) batch_first padding_valuez bnd,csd->bcnsr)dimrc)rfrKdevicedtyperangetorchnnutilsrnn pad_sequencer{einsummaxsumcatto) rTrrrrrscoresr) batch_scores batch_queriesjbatch_passagess r#score_retrievalz ColPaliProcessor.score_retrievalYs*@  A % %233 3 ! " "a ' '344 4 A  %);A)>)E E ENOO O A  $(:1(=(C C CLMM M  +A.4L%'q#.//<< ] ]A/1L!HN.;; Q^!34$VW<M1c"455zBB  !&!3!@!@&q1z>'9:\]"A""##L-PPTTYZT[[\]^bbghbii MM%)La888;;LIILL][[ \ \ \ \yQ''''r")NNNr>r?)NNNN)N)r,Nr)rrr__doc__ attributesimage_processor_classtokenizer_classryrIrrrrrrer rrrrpropertyrzrrintr __classcell__)rVs@r#r7r7KsU($[1JP>O$9( )) " )  ))))))@(,^b uu$uI0$y/4HYCZZ[u /0 u uuuun----$(#(((X((,!6!6$!6/0!6  !6!6!6!6F 2ItI./ 2/0 2  2 2 2 2L0449 >(>(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(>(>(>(>(>(>(>(r"r7)typingrrfeature_extraction_utilsr image_utilsrrprocessing_utilsr r r r tokenization_utils_baser rrrrrrrLrrQr5r7__all__r!r"r#rs.#"""""""444444????????XXXXXXXXXXXXOOOOOOOOOO''''''LLL     -U     55t5558]8]RWRWX[R\R\8]8]8]] NNN,L(L(L(L(L(~L(L(L(^  r"