`i)MddlmZmZddlmZddlmZmZddlm Z m Z m Z m Z ddl mZmZddlmZerddlZGd d e d ZGd de ZdgZdS))OptionalUnion) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)is_torch_availableNc(eZdZddidddddidZd S) ColQwen2ProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaults/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/colqwen2/processing_colqwen2.pyrr#sB y ,"  +D1  IIIr!rF)totalc eZdZdZddgZdZdZ ddeedeeffd Z d d ee d e e e ee ee fd eed efdZd!dZed efdZ d!d ee d eed efdZd e e ee fd eed efdZ d"de dedfde dedfdededde defd df dZedZxZS)#ColQwen2Processora Constructs a ColQwen2 processor which wraps a Qwen2VLProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColQwen2Processor`] offers all the functionalities of [`Qwen2VLProcessor`]. See the [`~Qwen2VLProcessor.__call__`] for more information. Args: image_processor ([`Qwen2VLImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`Qwen2TokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. visual_prompt_prefix (`str`, *optional*): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*): A prefix to be used for the query. image_processor tokenizerAutoImageProcessor)Qwen2TokenizerQwen2TokenizerFastNvisual_prompt_prefix query_prefixc t|||t|dsdn|j|_t|dsdn|j|_|d}||_|d}||_dS)N) chat_template image_tokenz <|image_pad|> video_tokenz <|video_pad|>zf<|im_start|>user <|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>zQuery: )super__init__hasattrr/r0r+r,)selfr&r'r.r+r,kwargs __class__s r"r2zColQwen2Processor.__init__Hs )=QQQ29)]2S2Sn??YbYn29)]2S2Sn??YbYn  '$M $8!  $L(r!imagestextr5returnc &|jtfd|jji|}|ddd}|du}||t d||t d|t |r|g}nt|trt |drnZt|tr6t|dtrt |ddst d|j gt|z} |j dd |i|d } | d } | |j j d z} d} tt| D]}|j| |vrW| ||jd | | | zzd| |<| dz } |j| |vW| |d |j| |<|j| fddi|d}t#i|| }|d dddf|d ddd fz}tt%j|d|}t$jjj|d|d<|r=|d|ddkd}|d|i|S|t|t6r|g}n?t|trt|dt6st d| |jdz}g}|D]$}|j|z|z}||%|j|fddi|d}|SdS)a Main method to prepare for the model either (1) one or several texts, either (2) one or several image(s). This method is a custom wrapper around the Qwen2VLProcessor's [`~Qwen2VLProcessor.__call__`] method adapted for the ColQwen2 model. It cannot process both text and images at the same time. When preparing the the text(s), this method forwards the `text` and `kwargs` arguments to Qwen2TokenizerFast's [`~Qwen2TokenizerFast.__call__`]. When preparing the the image(s), this method forwards the `images` and `kwargs` arguments to Qwen2VLImageProcessor's [`~Qwen2VLImageProcessor.__call__`]. Please refer to the doctsring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsrsuffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagesr7rimage_grid_thwz<|placeholder|>return_token_type_idsF)data pixel_valuesT) batch_first input_idstoken_type_idsilabelsz*Text must be a string or a list of strings r ) _merge_kwargsrr' init_kwargspop ValueErrorr isinstancelistr+lenr& merge_sizeranger/replaceprodrtorchsplittolistnnutilsrnn pad_sequence masked_fillupdatestrquery_augmentation_tokenr,append)r4r7r8audiovideosr5 output_kwargsr<r@ texts_doc image_inputsr= merge_lengthindexi text_inputs return_dataoffsetsrBrF texts_queryqueryaugmented_query batch_querys r"__call__zColQwen2Processor.__call__]shZ+* #  "&."<   }-11(DAA &d 2 A s9~~..]]A*il::'0|';'; ,.?>RWCXC]C]C_C_coCo.prs(( !   *il:: $-Q<#7#78I4K[#\#\IaLL($.&+ .K ',K{,Kl,KLLLK""23AAAqD9KHX[_`>`bfgg""Hf#5666   $$$ Ov t,, ODGS1I1I O !MNNN~6;%'K 4 4"&"3e";f"D""?3333($.&+ .K  + r!c @i}|tjdi|ddp jjfd|D}fd|D}|||dt di|S)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. NrrOc8g|]}jjg|RSr )r&get_number_of_image_patches).0 image_sizerr4s r" z@ColQwen2Processor._get_num_multimodal_tokens..sE!!!A$@\*\m\\\!!!r!c g|] }|dzz S)r>r )rr num_patchesrOs r"rtz@ColQwen2Processor._get_num_multimodal_tokens..s"ddd; A !=dddr!)num_image_tokensnum_image_patchesr )rrgetr[r&rOr )r4 image_sizesr5 vision_datarxrwrrOs` @@r"_get_num_multimodal_tokensz,ColQwen2Processor._get_num_multimodal_tokenss  "3=AA/SUVVM   ( ( (&**<>>a$BVBaJ!!!!!"-!!!  edddRcddd    4D[lmm n n n,, ,,,r!c|jjS)z Return the query augmentation token. Query augmentation buffers are used as reasoning buffers during inference. )r' pad_token)r4s r"r]z*ColQwen2Processor.query_augmentation_tokens~''r!c |jdd|i|S)a Prepare for the model one or several image(s). This method is a wrapper around the `__call__` method of the ColQwen2Processor's [`ColQwen2Processor.__call__`]. This method forwards the `images` and `kwargs` arguments to the image processor. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. r7r rn)r4r7r5s r"process_imagesz ColQwen2Processor.process_imagess"Bt}55F5f555r!c |jdd|i|S)a Prepare for the model one or several texts. This method is a wrapper around the `__call__` method of the ColQwen2Processor's [`ColQwen2Processor.__call__`]. This method forwards the `text` and `kwargs` arguments to the tokenizer. Args: text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). r8r r)r4r8r5s r"process_queriesz!ColQwen2Processor.process_queries(s"@t}11$1&111r!cpuquery_embeddingsz torch.Tensorpassage_embeddings batch_size output_dtypez torch.dtype output_devicez torch.devicec 8t|dkrtdt|dkrtd|dj|djkrtd|dj|djkrtd| |dj}g}t dt||D]:}g}t jjj ||||zdd} t dt||D]} t jjj || | |zdd} | t j d | |  d d d | t j|d || usually obtained by padding the list of tensors. Args: query_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Query embeddings. passage_embeddings (`Union[torch.Tensor, list[torch.Tensor]`): Passage embeddings. batch_size (`int`, *optional*, defaults to 128): Batch size for computing scores. output_dtype (`torch.dtype`, *optional*, defaults to `torch.float32`): The dtype of the output tensor. If `None`, the dtype of the input embeddings is used. output_device (`torch.device` or `str`, *optional*, defaults to "cpu"): The device of the output tensor. Returns: `torch.Tensor`: A tensor of shape `(n_queries, n_passages)` containing the scores. The score tensor is saved on the "cpu" device. rzNo queries providedzNo passages providedz/Queries and passages must be on the same devicez-Queries and passages must have the same dtypeNT)rC padding_valuez bnd,csd->bcnsr)dimr>r?)rNrKdevicedtyperPrSrVrWrXrYr^einsummaxsumcatto) r4rrrrrscoresrf batch_scores batch_queriesjbatch_passagess r"score_retrievalz!ColQwen2Processor.score_retrievalJs*@  A % %233 3 ! " "a ' '344 4 A  %);A)>)E E ENOO O A  $(:1(=(C C CLMM M  +A.4L%'q#.//<< ] ]A/1L!HN.;; Q^!34$VW<M1c"455zBB  !&!3!@!@&q1z>'9:\]"A""##L-PPTTYZT[[\]^bbghbii MM%)La888;;LIILL][[ \ \ \ \yQ''''r!cT|jj}|jj}d|D}||zS)Ncg|]}|dv| S))pixel_values_videosvideo_grid_thwr )rrnames r"rtz7ColQwen2Processor.model_input_names..s*' ' ' DHqO.2&* )) 'sm ) sm )))))).(,^b CC$CI0$y/4HYCZZ[C 01 C CCCCJ----4(#(((X((,!6!6$!601!6  !6!6!6!6F 2ItI./ 201 2  2 2 2 2L0449 >(>(^0D DE>(".$~2F"FG>( >( }- >( ^S01 >( >(>(>(>(@ C CX C C C C Cr!r%)typingrrfeature_extraction_utilsr image_utilsrrprocessing_utilsr r r r tokenization_utils_baser rrWrrSrr%__all__r r!r"rs.,#"""""""44444455555555XXXXXXXXXXXXCCCCCCCC''''''LLL     .e    dCdCdCdCdCdCdCdCN  r!