`iF`dZddlmZmZddlZddlmZddlmZddlm Z ddl m Z dd l m Z dd lmZdd lmZmZmZd d lmZejeZeGdde ZedGddeZedGddee ZgdZdS)zPyTorch Fuyu model.)OptionalUnionN)nn)Cache)GenerationMixin)CausalLMOutputWithPast)PreTrainedModel) AutoModel)auto_docstringcan_return_tuplelogging) FuyuConfigc@eZdZUeed<dZdZdZdZdZ dZ gZ dZ dZ dS)FuyuPreTrainedModelconfigfuyuTpast_key_valuesc|jj}t|tjrJ|jjd||j |jj dSdSt|tj rS|jjd||j -|jj|j  dSdSdS)Ng)meanstd) rinitializer_range isinstancerLinearweightdatanormal_biaszero_ Embedding padding_idx)selfmodulers z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/fuyu/modeling_fuyu.py _init_weightsz!FuyuPreTrainedModel._init_weights.sk+ fbi ( ( ? M  & &CS & 9 9 9{&  &&((((('&  - - ? M  & &CS & 9 9 9!- "6#56<<>>>>> ? ?--N)__name__ __module__ __qualname__r__annotations__base_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_no_split_modules_skip_keys_device_placementr&r'r%rr"s` &*#"&N"3 ? ? ? ? ?r'rzt The Fuyu model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introceZdZddiZdeffd ZdZdZdZdZ d e j d e e j d e j d e j fd Z de jfdZde jde jde jfdZe ddee jdee j dee j dee j dee jdeedee jdeedeedeedeed eeeffdZxZS) FuyuModelzlanguage_model.modellanguage_modelrc^t||j|_|jj|_t j|j|_tj |j |j z|j z|j |_d|_|dS)NF)super__init__ pad_token_idr" text_config vocab_sizer from_configr8rr patch_size num_channels hidden_sizevision_embed_tokensgradient_checkpointing post_initr#r __class__s r%r;zFuyuModel.__init__Bs    !. ,7'3F4FGG#%9   1 1F4G GI[$ $  ',# r'c4|jSN)r8get_input_embeddingsr#s r%rJzFuyuModel.get_input_embeddingsOs"77999r'c:|j|dSrI)r8set_input_embeddingsr#values r%rMzFuyuModel.set_input_embeddingsRs 0077777r'c||_dSrIr8r#decoders r% set_decoderzFuyuModel.set_decoderUs%r'c|jSrIrQrKs r% get_decoderzFuyuModel.get_decoderXs ""r'word_embeddingscontinuous_embeddingsimage_patch_input_indicesreturnc R|jdt|ks-tdt|d|jd|}t |jdD]}t j||dkdd}|||}|jd||jdkr)td||jd|jd|d ||||j|||f<|S) aThis function places the continuous_embeddings into the word_embeddings at the locations indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous embeddings. Args: word_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Tensor of word embeddings. continuous_embeddings (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Tensor of continuous embeddings. The length of the list is the batch size. Each entry is shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative indices in image_patch_input_indices for that batch element. image_patch_input_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Tensor of indices of the image patches in the input_ids tensor. rz7Batch sizes must match! Got len(continuous_embeddings)=z and word_embeddings.shape[0]=T)as_tuplezGNumber of continuous embeddings continuous_embeddings[batch_idx].shape=zA does not match number of continuous token ids src_indices.shape=z in batch element .) shapelen ValueErrorclonerangetorchnonzerotodevice)r#rWrXrYoutput_embeddings batch_idx dst_indices src_indicess r%gather_continuous_embeddingsz&FuyuModel.gather_continuous_embeddings[s{( %a(C0E,F,FFFls3H/I/IllQ`QfghQill ,11334Q788  I -(A)(LPQ(Q\`aaabcdK4I>{KK #&;I&F&LQ&OOO i7LY7W7]ii6A6Gii\eiii9Ni8XYd8e8h8h!(99 i4 5 5! r' pixel_valuesc $fd|D}|S)a$ Encodes images into continuous embeddings that can be forwarded to the language model. Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. cg|]L}|jjjdMS)r)rCrerdtypesqueeze).0patchr#s r% z0FuyuModel.get_image_features..s[     $ $UXXd.F.M.S%T%T U U ] ]^_ ` `   r'r4)r#rlkwargspatch_embeddingss` r%get_image_featureszFuyuModel.get_image_featuress4    %    r' input_ids inputs_embedsimage_featuresc\|e||tj|jjtj|jk}|d}n||jjk}|}| d | |j}|j d|j dz}|| | krtd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. Nrorfrrz6Image features and image tokens do not match: tokens: z , features )rJrctensorrimage_token_idlongrfallsum unsqueeze expand_asrer^numelr`)r#rwrxryspecial_image_maskn_image_tokensn_image_featuress r%get_placeholder_maskzFuyuModel.get_placeholder_masks/  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11/99"==GG VVYYZgZnoo)/2^5I!5LL + , 2 2 4 48L8L8N8N N Nvvvdtvv "!r'N image_patchesimage_patches_indicesattention_mask position_idsr use_cacheoutput_attentionsoutput_hidden_states return_dictc .| | n |jj} | | n |jj} ||n |jj}| | n |jj} ||t d| |j\} }n| |j\} }}nt d|b||jn|j}||nd}tj |||ztj |}| d}|"|j |}|w||}tj|d|j|j}||||}|||}|j d ||||| | || d| }|S) a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. NzDYou cannot specify both input_ids and inputs_embeds at the same timez4You have to specify either input_is or inputs_embedsrr{)dim)rxry)rxrrrrrrrr4)rrrruse_return_dictr`r^rfget_seq_lengthrcarangerrr8rJrvcatrerormasked_scatter)r#rwrrrrrrxrrrrrt batch_size seq_length_rfpast_key_values_lengthruroutputss r%forwardzFuyuModel.forwards02C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "%._ "J  &(5(; %J AASTT T  )2)>Y%%MDXFIXId_%C%C%E%E%Ejk " <& 5K(KSXS]flL(11!44L  FD/DDFFyQQM  $#66}EE $y)9qAAADD]EY[h[noo !%!:!:GW";"" *889KM]^^M%$%  ')%+/!5#      r') NNNNNNNNNNN)r(r)r*_checkpoint_conversion_mappingrr;rJrMrTrVrcTensorlistrk FloatTensorrv LongTensorrr rrboolrtupler r __classcell__rGs@r%r7r7:sH '=>N%O" z      :::888&&&###*!*! $EL1*!$)< *!  *!*!*!*!X  u/@     ")":?:K"]b]n""""015048<1537+/59$(,0/3&*GGE,-G  - G ( 5 G !. Gu/0G"%G  12GD>G$D>G'tnGd^G u,, -GGG^GGGGGr'r7zz Fuyu Model with a language modeling head on top for causal language model conditioned on image patches and text. c!eZdZddddZdgZdeffd ZdZd Zd Z d Z e e dde ejde ejde ejde ejde ejde ede ejde ede ejde ede ede ede edeeeffdZ dfd ZxZS) FuyuForCausalLMzmodel.language_modelzmodel.vision_embed_tokenslm_head)z^language_model.modelz^vision_embed_tokensz^language_model.lm_headzlm_head.weightrct|t||_t j|jj|jjd|_ | dS)NF)r) r:r;r7modelrrr=rBr>rrErFs r%r;zFuyuForCausalLM.__init__se    v&& y!3!?ASA^ejkkk  r'c4|jSrI)rrJrKs r%rJz$FuyuForCausalLM.get_input_embeddings sz..000r'c:|j|dSrI)rrMrNs r%rMz$FuyuForCausalLM.set_input_embeddingss ''.....r'c:|j|dSrI)rrTrRs r%rTzFuyuForCausalLM.set_decoders w'''''r'c4|jSrI)rrVrKs r%rVzFuyuForCausalLM.get_decodersz%%'''r'Nrrwrrrrrrxrlabelsrrrlogits_to_keeprZc | | n |jj} | | n |jj} ||n |jj}| | n |jj} ||||||||| | |d }|d}t | trt| dn| }| |dd|ddf}d}| |j d|| |jj j d|}t|||j|j|jS)a@ image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Examples: ```python >>> from transformers import FuyuProcessor, FuyuForCausalLM >>> from PIL import Image >>> import requests >>> processor = FuyuProcessor.from_pretrained("adept/fuyu-8b") >>> model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b") >>> url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" >>> image = Image.open(requests.get(url, stream=True).raw) >>> prompt = "Generate a coco-style caption.\n" >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> generated_ids = model.generate(**inputs, max_new_tokens=7) >>> generation_text = processor.batch_decode(generated_ids[:, -7:], skip_special_tokens=True) >>> print(generation_text[0]) A blue bus parked on the side of a road. ```NT) rwrrrxrrrrrrrr)logitsrr>)lossrr hidden_states attentionsr4)rrrrrrrintslicer loss_functionr=r>r rrr)r#rwrrrrrrxrrrrrrrtrr slice_indicesrrs r%rzFuyuForCausalLM.forwards_l2C1N--TXT_Tq$8$D $+Jj "+!6IIDKSV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD&#3!/)     r'c vtj|f||||||d|} |ddkr d| d<d| d<| S)N)rrrxrrcache_positionrrr)r:prepare_inputs_for_generation) r#rwrrrxrrrrt model_inputsrGs r%rz-FuyuForCausalLM.prepare_inputs_for_generationwss=uww<   +)''"7)       !  ! !48L0 1,0L )r') NNNNNNNNNNNNr)NNNNNN)r(r)r*r_tied_weights_keysrr;rJrMrTrVr r rrcrrrrrrrrr rrrrs@r%rrs#"8 ;#,&&" ++z 111///((((((15048<1537+/59$()-,0/3&*()\ \ E,-\   - \ ( 5 \ !. \ u/0\ "%\   12\ D>\ &\ $D>\ 'tn\ d^\ ! \ " u,, -#\ \ \ ^\ B"r'r)rrr7)__doc__typingrrrcr cache_utilsr generationrmodeling_outputsr modeling_utilsr models.auto.modeling_autor utilsr r rconfiguration_fuyur get_loggerr(loggerrr7r__all__r4r'r%rs""""""""  ))))))666666------222222>>>>>>>>>>******  H % %?????/???. vvvvv#vv  vr WWWWW)?WW  Wt B A Ar'