`iDdZddlZddlmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZdd lmZmZmZmZdd lmZmZddlmZmZm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+e%j,e-Z.ee#dGdde"Z/Gdde j0Z1Gdde j0Z2Gdde j0Z3de2iZ4Gdde j0Z5Gd d!e j0Z6Gd"d#e j0Z7Gd$d%eZ8Gd&d'e j0Z9e#Gd(d)eZ:Gd*d+e j0Z;Gd,d-e j0Z< dKd/e j0d0ej=d1ej=d2ej=d3eej=d4e>d5e>fd6Z?Gd7d8e j0Z@Gd9d:eZAGd;de j0ZCe#d?Gd@dAe:ZDGdBdCe j0ZEe#dDGdEdFe:ZFe#dGGdHdIe:eZGgdJZHdS)LzPyTorch GIT model.N) dataclass)CallableOptionalUnion)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPoolingCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringcan_return_tuplelogging torch_int)deprecate_kwarg) GitConfigGitVisionConfigz} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejdfed<dZ ee ejdfed<dS)GitVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__r$rtorch FloatTensor__annotations__r%r&tupler'x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/git/modeling_git.pyr#r#4s 15L(5,-44459x 12999=AM8E%"3S"89:AAA:>Ju0#567>>>>>r1r#c eZdZdZfdZ d deejdeejdeejde d ej f d Z xZ S) GitEmbeddingsz;Construct the embeddings from word and position embeddings.cttj|j|j|j|_tj|j|j|_ tj |j|j |_ tj |j |_t|dd|_|dt%j|jdddS) N) padding_idxepsposition_embedding_typeabsolute position_idsrF persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutgetattrr9register_bufferr,arangeexpandselfconfig __class__s r2rAzGitEmbeddings.__init__Js !|F,=v?Q_e_rsss#%<0NPVPb#c#c f&8f>STTTz&"<== '.v7PR\']']$  EL)GHHOOPWXXej      r1Nr input_idsr; inputs_embedspast_key_values_lengthreturnc~||}n|dd}|d}||jdd|||zf}|||}n|}|jdkr||}||z }||}||}|S)Nr=rr:)sizer;rFr9rHrIrM) rSrVr;rWrX input_shape seq_length embeddingsrHs r2forwardzGitEmbeddings.forwardYs  #..**KK',,..ss3K ^  ,QQQ0FVlIl0l-lmL  --i88JJ&J  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r1)NNNr) r(r)r*r+rArr, LongTensorr-intTensorr_ __classcell__rUs@r2r4r4GsEE      "153759&' E,-u/0  12  !$   r1r4ceZdZdfd Zeddd ddejd eejd eejdee d ee d ee d e ejfdZ xZ S)GitSelfAttentionNc,t|j|jzdkr0t |ds t d|jd|jd||_|(td|j j d|j|_t|j|jz |_ |j|j z|_ t|jj|jjz dzd z|_|j|xj|jzc_t'j|j|j |_t'j|j|j |_t'j|j|j |_t'j|j|_|pt7|d d |_|jd ks |jd kr8|j|_t'jd|jzd z |j |_dSdS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()zInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.rr9r: relative_keyrelative_key_query) r@rArDnum_attention_headshasattr ValueError layer_idxlogger warning_oncerUr(raattention_head_size all_head_size vision_config image_size patch_sizeimage_patch_tokensnum_image_with_embeddingrLinearquerykeyvaluerKattention_probs_dropout_probrMrNr9rGrBdistance_embeddingrSrTr9rprUs r2rAzGitSelfAttention.__init__xs%    : :a ? ?PVXhHiHi ?8F$688 4888 #     ,!8,,,    $*#= #&v'9F( ' -zC C $  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # # #>r=qr1past_key_valuepast_key_values4.58new_nameversionFr&attention_mask head_maskoutput_attentionspixel_values_presentrYc 8|j\}}} |||d|j|jdd} |r|jnd} |||d|j|jdd} |||d|j|jdd} || | dddd| dddf| dddd| dddf|j \}}tj | ddddd| ddf|gd} tj | ddddd| ddf|gd} tj | | dd}|jdks |jdkrt| jd| jd}}|>tj|dz tj|j dd}n:tj|tj|j dd}tj|tj|j dd}||z }|||jzdz }|| j }|jdkrtjd | |}||z}n?|jdkr4tjd | |}tjd | |}||z|z}|t1j|jz }|||z}t4j|d}||}|||z}tj || }|dddd }| dd|j!fz}||}||fS)Nr=rrjrdimrkrldtypedevicerzbhld,lrd->bhlrzbhrd,lrd->bhlrr)"shaper{viewrmrs transposerxr|r}updaterpr,catmatmulr9tensorlongrrPrrGtoreinsummathsqrtr functionalsoftmaxrMpermute contiguousr[rt)rSr&rrrrr batch_sizer]_ query_layercutoff key_layer value_layerkey_layer_pastvalue_layer_pastattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r2r_zGitSelfAttention.forwardsR%2$7! J JJ} % % T*b$":D/E/E!!!QQQ*+[AAAvww9I-JDN00 ,N, 9QQQ7F7AAA-=#>"OUVWWWI)[AAAww1A%BDT$U[\]]]K!< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q'2'8';Y_Q=O*L*!&j1nEJWdWk!l!l!l!q!q"""'l%*UbUi!j!j!j!o!oprtu!v!v"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|S"S%**+BCC o--r1NNNNNFF)r(r)r*rArr,rbrr-r boolr/r_rcrds@r2rfrfws u u u u u uD_%0A6RRR7;15+/,1/4R.R.|R.!!23R.E-. R. "% R. $D> R.'tnR. u| R.R.R.SRR.R.R.R.R.r1rfcPeZdZfdZdejdejdejfdZxZS) GitSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr7) r@rArrzrDdenserIrJrKrLrMrRs r2rAzGitSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r1r& input_tensorrYc||}||}|||z}|SNrrMrIrSr&rs r2r_zGitSelfOutput.forward@ =11  ]33 }|'CDD r1r(r)r*rAr,rbr_rcrds@r2rri>>>>> U\RWR^r1reagerceZdZdfd ZdZeddd dd ejd eej d eej dee d ee d ee de ejfdZ xZS) GitAttentionNctt|j||||_t ||_t|_dS)N)r9rp) r@rAGIT_SELF_ATTENTION_CLASSES_attn_implementationrSroutputset pruned_headsrs r2rAzGitAttention.__init__sb .v/JK ,Cy    $F++ EEr1ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr)lenrrSrmrsrrr{r|r}rrrtunion)rSheadsindexs r2 prune_headszGitAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r1rrrrFr&rrrrrYcp|||||||\}}|||} | |fSr)rSr) rSr&rrrrr attn_outputself_attn_weightsattention_outputs r2r_zGitAttention.forward sQ*.      * * & & ;;{MBB!222r1rr)r(r)r*rArrr,rbrr-r rr/r_rcrds@r2rrs"""""";;;$_%0A6RRR7;15+/,1/433|3!!233E-. 3 "% 3 $D> 3'tn3 u| 333SR33333r1rcBeZdZfdZdejdejfdZxZS)GitIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r@rArrzrDintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrRs r2rAzGitIntermediate.__init__8sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r1r&rYcZ||}||}|Sr)rrrSr&s r2r_zGitIntermediate.forward@s, =11 00?? r1rrds@r2rr7s^99999U\elr1rcPeZdZfdZdejdejdejfdZxZS) GitOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r@rArrzrrDrrIrJrKrLrMrRs r2rAzGitOutput.__init__Hsf Yv79KLL f&8f>STTTz&"<== r1r&rrYc||}||}|||z}|Srrrs r2r_zGitOutput.forwardNrr1rrds@r2rrGrr1rceZdZdfd Zeddd ddejd eejd eejdee d ee d ee d e ejfdZ dZ xZS)GitLayerNct|j|_d|_t |||_t ||_t||_ dS)Nr)rp) r@rAchunk_size_feed_forward seq_len_dimr attentionr intermediaterr)rSrTrprUs r2rAzGitLayer.__init__Vsc '-'E$%f BBB+F33'' r1rrrrFr&rrrrrYc|||||||\}}t|j|j|j|} | |fS)N)rrr)rrfeed_forward_chunkrr) rSr&rrrrrrself_attention_weights layer_outputs r2r_zGitLayer.forward^sh48>>   /+!5 4B4 4 001  #T%A4CSUe  333r1c\||}|||}|Sr)rr)rSrintermediate_outputrs r2rzGitLayer.feed_forward_chunkws2"//0@AA{{#68HII r1rr)r(r)r*rArr,rbrr-r rr/r_rrcrds@r2rrUs((((((_%0A6RRR7;15+/,1/444|4!!234E-. 4 "% 4 $D> 4'tn4 u| 444SR40r1rc6eZdZfdZ ddejdeejdeejdeee e e ejfd ee d ee d ee d ee d ee dee eje ffdZ xZS) GitEncoderct|_tjfdt jD|_d|_dS)Nc0g|]}t|Sr0)r).0irTs r2 z'GitEncoder.__init__..s##a#a#aAHVQ$7$7#a#a#ar1F) r@rArTr ModuleListrangenum_hidden_layerslayergradient_checkpointingrRs `r2rAzGitEncoder.__init__~s`  ]#a#a#a#avG_A`A`#a#a#abb &+###r1NFTr&rrr use_cacheroutput_hidden_statesr return_dictrYc |jr%|jr|rtdd}|r|t |j}|rdnd} |rdnd} t |jD]?\} } |r| |fz} ||| nd}| ||||||}|d}|r | |dfz} @|r| |fz} | std||| | fDSt||| | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rTr0rrc3K|]}||V dSrr0)rvs r2 z%GitEncoder.forward..s4   =!===  r1r%rr&r') rtrainingrqrrr rT enumeraterr/r)rSr&rrrrrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r2r_zGitEncoder.forwards  & "4= " "##p"  ?0*$+>>>O"6@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO(L!$ M*!,M  P&9]1=M > |> !!23> E-. > "%uU5;L5M/N(N"OP > D> > $D>> 'tn> 'tn> d^>  uU\"$;; <> > > > > > > > r1rc(eZdZUeed<dZdZdZdS)GitPreTrainedModelrTgitTct|trtj|jd|jjtj|jj |jjtj|j j |jjt|tj rT|j j d|jj|j |j j dSdSt|tjr_|j j d|jj|j+|j j |jdSdSt|tjr?|j j |j j ddSdS)zInitialize the weights)meanstd)rNg?)rGitVisionEmbeddingsrinitnormal_class_embeddingrTinitializer_rangepatch_embeddingweightposition_embeddingrzdatabiaszero_rBr6rIfill_)rSmodules r2 _init_weightsz GitPreTrainedModel._init_weightss f1 2 2 a GOOF2$+B_O ` ` ` GOOF29t{?\O ] ] ] GOOF5<$+B_O ` ` ` fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r1N)r(r)r*rr.base_model_prefixsupports_gradient_checkpointingr%r0r1r2rrs= &*#*****r1rcveZdZdeffd ZdejdededejfdZd d ej dejfd Z xZ S) rrTczt||_|j|_|j|_|j|_tjtj |j|_ tj |j |j|j|jd|_|j|jzdz|_|jdz|_tj|j|j|_|dtj|jdddS)NF) in_channels out_channels kernel_sizestrider!rjrr;r<r>)r@rArTrD embed_dimrvrwr Parameterr,randnrConv2d num_channelsr num_patches num_positionsrBrrOrPrQrRs r2rAzGitVisionEmbeddings.__init__s   + + +!|EK,G,GHH!y+?    !Ot>1D!-1"$,t/A4>"R"R ^U\$:L-M-M-T-TU\-]-]jopppppr1r^heightwidthrYc|jddz }|jjd}|jddz }tjs&||kr ||kr||jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} | d| | | }| dddd}tj || | fdd }| dddddd| }t j||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr=g?rrjbicubicF)r[mode align_cornersr)rrr unsqueezer,jit is_tracingr;rwrreshaperrr interpolaterr) rSr^r5r6r3rr4class_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r2interpolate_pos_encodingz,GitVisionEmbeddings.interpolate_pos_encodings!&q)A- !4;EEaHH*03a7 y##%% >+*F*F6UZ??**4+<== =,QQQU3,QQQU3r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr1F pixel_valuesc <|j\}}}}|s<||jks ||jkr&td|d|d|jd|jd |jjj}|||}|ddd}|j |dd} tj | |gd } |r| | | ||z} n| ||jz} | S) NzInput image size (*z) doesn't match model ().rrjrr=r)rrvrorrrrflattenrrrQr,rrErr;) rSrFrErrr5r6 target_dtype patch_embeds class_embedsr^s r2r_zGitVisionEmbeddings.forward!sD'3'9$ Avu' Vt-F-F%SWSbJbJbqVqqeqqDOqq^b^mqqq +28 ++LOO,O,O,OPP #++A..88A>> +22:q"EE Y l;CCC # Q#d&C&CJPVX]&^&^^JJ#d&=&=d>O&P&PPJr1F) r(r)r*r rAr,rbrarEr-r_rcrds@r2rrsqqqqqqq,'D5<'D'DUX'D]b]i'D'D'D'DRE$5Z_Zfr1rcBeZdZfdZdejdejfdZxZS) GitVisionMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) r@rArTr r activation_fnrrzrDrfc1fc2rRs r2rAzGitVisionMLP.__init__5sf  #F$569V/1IJJ9V5v7IJJr1r&rYc||}||}||}|Sr)rSrRrTrs r2r_zGitVisionMLP.forward<s=// **=99 // r1rrds@r2rPrP4scKKKKKU\elr1rPrr$r{r|r}rscalingrMc tj||dd|z}|||z}tj|dtj|j}tj |||j }tj||} | dd } | |fS)Nr=r)rr)pr rrj) r,rrrrrfloat32rrrMr r) r$r{r|r}rrVrMkwargs attn_weightsrs r2eager_attention_forwardr\Ds<s}}R'<'<==GL!#n4 =((2U](SSVVW\WbccL=((6?([[L,|U33K''1--88::K  $$r1ceZdZdZfdZ d dejdeejdeejdeed e ejeejff d Z xZ S) GitVisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ |j |_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: rIgF)r@rArTrDr.rm num_headshead_dimroscaleattention_dropoutrM is_causalrrzk_projv_projq_projout_projrRs r2rAzGitVisionAttention.__init__^s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( / i?? i?? i??  $.$.AA r1NFr&rcausal_attention_maskrrYc n|j\}}}||}||} ||} ||||j|jdd}| |||j|jdd} | |||j|jdd} |jj dkr||||z}n||}n |du|_ t} |jj dkrt|jj } | ||| | ||j |j |jsdn|j\} } | |||} || } |sd} | | fS)z#Input shape: Batch x Time x Channelrrjflash_attention_2Nrr)rdrVrM)rrgrerfrr`rarrTrrdr\rrbr rMr>rrh)rSr&rrirrr]r.querieskeysvaluesattention_interfacerr[s r2r_zGitVisionAttention.forwardrs-:,?) J ++m,,{{=))]++,,z:t~t}UU__`acdeeyyZOOYYZ[]^__ZT^T]SS]]^_abcc ; +/B B B).C.O!/2G!G&2!62$>DN(? ; +w 6 6"9$+:Z"[ $7$7     nJ#}>CC$, % % % ! \"))*j)LLWWYY mmK00   LL((r1)NNF) r(r)r*r+rAr,rbrrr/r_rcrds@r2r^r^[sGGBBBBB.268<,1 /)/)|/)!./) ( 5 /) $D> /) u|Xel33 4 /)/)/)/)/)/)/)/)r1r^c eZdZdeffd Z d dejdejdejdeede ej f d Z xZ S) GitVisionEncoderLayerrTcDt|j|_t ||_t j|j|j|_ t||_ t j|j|j|_ dSr) r@rArDr.r^ self_attnrrIrJ layer_norm1rPmlp layer_norm2rRs r2rAzGitVisionEncoderLayer.__init__s ++F33<F & u ! &&&&&&&&r1rqceZdZdZdeffd Ze d deej deej dee dee d ee d e e e ff d ZxZS) GitVisionEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`GitVisionEncoderLayer`]. Args: config: GitVisionConfig rTct|_tjfdt jD|_d|_dS)Nc.g|]}tSr0)rqrrrTs r2rz-GitVisionEncoder.__init__..s"$l$l$lq%:6%B%B$l$l$lr1F) r@rArTrrrrlayersrrRs `r2rAzGitVisionEncoder.__init__sa  m$l$l$l$lERXRjLkLk$l$l$lmm &+###r1NrrirrrrYc@||n |jj}||n |jj}||n |jj}|rdnd}|rdnd}|} t |jD]2\} } |r|| fz}| | |||} | d} |r || dfz}3|r|| fz}t | ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr0)rrrr%r&r')rTrruse_return_dictr rr) rSrWrrirrrencoder_statesall_attentionsr&idx encoder_layerrs r2r_zGitVisionEncoder.forwards N2C1N--TXT_Tq$8$D $+Jj &1%>N+>Vd    r1)NNNNN)r(r)r*r+r rArrr,rbrrr/rr_rcrds@r2r{r{s,,,,,,, 268<,0/3&*D D !.D  ( 5 D $D> D 'tn D d^D  uo% &D D D D D D D D r1r{ceZdZdeffd Ze d deejdee dee dee d ee d e e e ff d Z xZS) GitVisionTransformerrTc4t||_|j}t ||_t j||j|_ t||_ t j||j|_ dSr) r@rArTrDrr^rrIrJ pre_layrnormr{encoderpost_layernorm)rSrTr.rUs r2rAzGitVisionTransformer.__init__1s  & -f55L8MNNN'//  l9&:OPPPr1NFrFrrrErrYc||n |jj}||n |jj}||n |jj}|t d|||}||}|||||}|d}||}|s|f|ddzSt||j |j S)Nz You have to specify pixel_valuesrE)rWrrrrrr) rTrrrror^rrrrr&r') rSrFrrrErr&encoder_outputsr%s r2r_zGitVisionTransformer.forward;s2C1N--TXT_Tq$8$D $+Jj &1%%'/!""*== =/)7&1    r1NNNFN)r(r)r*r rArrr,r-rrr/rr_rcrds@r2rr/sQQQQQQQ59,0/338&* & & u01& $D>& 'tn & #+4. & d^ &  uo% && & & ^& & & & & r1rzY The vision model from CLIP, used in GIT, without any head or projection on top. ceZdZUeed<dZdeffd ZdejfdZ e d de e j de ed e ed ed e edeeeff d ZxZS)GitVisionModelrTrFct|t||_|dSr)r@rAr vision_model post_initrRs r2rAzGitVisionModel.__init__osA    088 r1rYc$|jjjSr)rr^rrSs r2get_input_embeddingsz#GitVisionModel.get_input_embeddingsus +;;r1NFrrrErcV||n |jj}||||||S)a{ Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, GitVisionModel >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = GitVisionModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```N)rFrrrEr)rTrr)rSrFrrrErs r2r_zGitVisionModel.forwardxsC8&1%# 'tn # #' # d^ #  uo% &# # # ^# # # # # r1rcHeZdZdeffd ZdejdejfdZxZS) GitProjectionrTct||_tjtj|jj|jtj|j|jj |_ dSr) r@rArTr SequentialrzrurDrIrJvisual_projectionrRs r2rAzGitProjection.__init__sm  !# If*68J K K L+1E1T U U U" " r1r^rYc,||Sr)r)rSr^s r2r_zGitProjection.forwards%%j111r1) r(r)r*rrAr,rbr_rcrds@r2rrsj y      2%,25<22222222r1rzy The bare GIT Model transformer consisting of a CLIP image encoder and text decoder outputting raw hidden-states ceZdZfdZdZdZdZdedej dej dej fd Z dd Z e dd eej deej deej deej deej deej deeeeejfdeedeedeededeedeeej effdZxZS)GitModelct|_t|_t j|_t|_ t|_ j 7tjfdtj D|_|dS)Nc3|K|]6}tjtjddjjV7dS)rN)rr/r,zerosrurDr~s r2rz$GitModel.__init__..sU;; U[Av/C/OPPQQ;;;;;;r1)r@rArTr4r^rru image_encoderrrrrryr ParameterListrimg_temporal_embeddingrrRs `r2rAzGitModel.__init__s     '//+F,@AA!&)) !.v!6!6  * 6*,*:;;;;v>??;;;++D ' r1c|jjSrr^rFrs r2rzGitModel.get_input_embeddingss ..r1c||j_dSrr)rSr}s r2set_input_embeddingszGitModel.set_input_embeddingss*/'''r1c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrrr)rSheads_to_prunerrs r2 _prune_headszGitModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Cr1r[rrrYctjtj||||d}||dkt d}|S)Nrrr)diagonal-inf)r,triuones masked_fillfloat)rSr[rrmasks r2_generate_future_maskzGitModel._generate_future_masksNz%*T4eLLLWXYYY 5==99 r1Ncp|jd}|jd}|j}|j} tj||f|| } tj|||zft d|j| } tj||f| |j} |dkr7tj|jd|jd|zf| |j}tj| | fd} tj| || fd}tj| |fddddf}|/tj|jd|jdfd|}|jtj krtd tj ||j }t d||<| |jd||z||z|zf}| }|ddddd|f}|dddddf}||z|ddddd|f<|dddddddf}|S) NrrrrrrF) fill_valuerz1Memory key padding mask must be a boolean tensor.r)rrrr,rfullrrrrro zeros_likerQclone)rStgtmemorytgt_maskrXmemory_key_padding_masknum_tgt num_memoryrrtop_left top_right bottom_leftleftrightfull_attention_maskzero_negative_infinity origin_leftrs r2create_attention_maskzGitModel.create_attention_masks)A,\!_  ; J7eTTTJ #99 : &MM:    k j !?   "A % %{"HN1$58N$NOH y(K0a888 9hkk%&8&89qAAA#iu 1===dAAAgF " *&+j&,q/6>> from transformers import AutoProcessor, AutoModel >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base") >>> model = AutoModel.from_pretrained("microsoft/git-base") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "this is an image of two cats" >>> inputs = processor(images=image, text=text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer=z5You have to specify either input_ids or inputs_embedsrrrrz#pixel_values must be of rank 4 or 5)rVr;rWrXrjr)rrrrX)tgt_len)rrrrrrrrr )%rTrrrrro%warn_if_padding_and_no_attention_maskr[rr get_seq_length get_head_maskrndimrr%rrrappendr,rrr^rrrrepeatrrr rrrrr&r')rSrVrr;rFrrWrrrrrErr\r]rXprojected_visual_featuresvisual_features frame_idxvisual_features_frameembedding_outputr&rcombined_attention_maskexpanded_attn_maskrsequence_outputs r2r_zGitModel.forward sJ2C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U ^ "#  &"/5996..000$3355 #&&y$+2OPP $(!  # A%%"&"4"4 ;S#5### "a''"$!&|'9!',>$QQQ 111aaa%78Sk-?--'**T-H-SS)#**+@AAAA#()O"C"C"C!!FGGG(,(>(>(O(O %??%'#9 +   % ,(- !'*A/?/Ea/HI&,'.))) %%>$D$D  ! !! $ $(A(F(Fq(I(I I1a% % !  #<>N"OUVWWW --j:J:PRbRijj#'"<"< ,#9 #=# #   %"< 0 6 B"""b!()) &))%7111?U>U>V>VXYXYXY8Y%Z""'111{1~o.?.?+a.ARAR(RSSSWiiSSS,, 2+/!5#!-T!9'   *!, <#%(;; ;&-+;)7&1     r1r) NNNNNNNNNNFN)r(r)r*rArrrrar,rrrbrrrrrr listr-rr/rr_rcrds@r2rrs &///000CCC#ek5<\a\h 0#0#0#0#d-115/3/3,004KO$(,0/3).&*c c EL)c !.c u|, c u|, c EL) c   -c "%tE4E/F(F"GHc D>c $D>c 'tnc #'c d^c  uU\"$>> ?c c c ^c c c c c r1rz` GIT Model with a `language modeling` head on top for autoregressive language modeling. c eZdZdgZfdZdZdZe ddee j dee j d ee j d ee j d ee j d ee j d ee j dee e e e j fdeedeedeededeede ee j effdZ ddZxZS)GitForCausalLMz output.weightct|t||_t j|j|j|_| dSr) r@rArrrrzrDrCrrrRs r2rAzGitForCausalLM.__init__s[    F##i 2F4EFF  r1c|jSrrrs r2get_output_embeddingsz$GitForCausalLM.get_output_embeddingss {r1c||_dSrr)rSnew_embeddingss r2set_output_embeddingsz$GitForCausalLM.set_output_embeddingss $ r1NFrVrr;rFrrWlabelsrrrrrErrYc | | n |jj} |d} ||||||||| | | | |  }|d}||}d}||jjjdjjj}|dd|dddf }|ddddf }|j | d|jj | dfd|jj i|}| s|f|ddz}||f|zn|St|||j|j|jS) a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` Examples: Image captioning example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> import requests >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-coco") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-coco") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> generated_caption = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> print(generated_caption) two cats sleeping on a pink blanket next to remotes. ``` Visual question answering (VQA) example: ```python >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> from huggingface_hub import hf_hub_download >>> from PIL import Image >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa") >>> file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset") >>> image = Image.open(file_path).convert("RGB") >>> pixel_values = processor(images=image, return_tensors="pt").pixel_values >>> question = "what does the front of the bus say at the top?" >>> input_ids = processor(text=question, add_special_tokens=False).input_ids >>> input_ids = [processor.tokenizer.cls_token_id] + input_ids >>> input_ids = torch.tensor(input_ids).unsqueeze(0) >>> generated_ids = model.generate(pixel_values=pixel_values, input_ids=input_ids, max_length=50) >>> print(processor.batch_decode(generated_ids, skip_special_tokens=True)) ['what does the front of the bus say at the top? special'] ``` Video captioning example: ```python >>> import av >>> import numpy as np >>> from PIL import Image >>> from huggingface_hub import hf_hub_download >>> from transformers import AutoProcessor, AutoModelForCausalLM >>> processor = AutoProcessor.from_pretrained("microsoft/git-base-vatex") >>> model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-vatex") >>> # set seed for reproducibility >>> np.random.seed(45) >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> def sample_frame_indices(clip_len, frame_sample_rate, seg_len): ... ''' ... Sample a given number of frame indices from the video. ... Args: ... clip_len (`int`): Total number of frames to sample. ... frame_sample_rate (`int`): Sample every n-th frame. ... seg_len (`int`): Maximum allowed index of sample's last frame. ... Returns: ... indices (`list[int]`): List of sampled frame indices ... ''' ... converted_len = int(clip_len * frame_sample_rate) ... end_idx = np.random.randint(converted_len, seg_len) ... start_idx = end_idx - converted_len ... indices = np.linspace(start_idx, end_idx, num=clip_len) ... indices = np.clip(indices, start_idx, end_idx - 1).astype(np.int64) ... return indices >>> # load video >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample frames >>> num_frames = model.config.num_image_with_embedding >>> indices = sample_frame_indices( ... clip_len=num_frames, frame_sample_rate=4, seg_len=container.streams.video[0].frames ... ) >>> frames = read_video_pyav(container, indices) >>> pixel_values = processor(images=list(frames), return_tensors="pt").pixel_values >>> generated_ids = model.generate(pixel_values=pixel_values, max_length=50) >>> print("Generated caption:", processor.batch_decode(generated_ids, skip_special_tokens=True)) Generated caption: ['a woman is sitting at a table and she is talking about the food she is holding.'] ``` NF) rr;rFrrWrrrrrErrr=rrC)losslogitsrr&r')rTrrrrrrrSrxr loss_functionrrCrrr&r')rSrVrr;rFrrWrrrrrrErrZryrrrnum_image_tokensshifted_logitsrs r2r_zGitForCausalLM.forwardsj&1%IIKKNAAAqrrE]--//F%4%##B (>?? B ;1 D FY,F)-)9TGf$$v E%#3!/)     r1c N|F|}|jd|kr|}n|jddz }|dd|df}|j}|||}|||d||d} |D]\} } | | vr| | | <| S)NrrF)rVrrFrr)rrnew_onesgetr) rSrVrrrrZ past_lengthremove_prefix_lengthr\ model_inputsr|r}s r2prepare_inputs_for_generationz,GitForCausalLM.prepare_inputs_for_generations  &)88::Kq!K//'2$$(1q'9A'=$!!!!%9%:%:":;I o  !&// <A $D>A 'tnA #'A d^A uU\"$:: ;!A A A ^A HOS$$$$$$$$r1r)rrrr)r)Ir+r dataclassesrtypingrrrr,r activationsr cache_utilsr r generationr modeling_attn_mask_utilsr modeling_layersrmodeling_outputsrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrrrutils.deprecationrconfiguration_gitrr get_loggerr(rqr#rr4rfrrrrrrrrrrPrbrr\r^rqr{rrrrr__all__r0r1r2r s  !!!!!!,,,,,,,,,, !!!!!!........))))))BBBBBB999999 GFFFFFFFllllllllll10000099999999  H % %  ? ? ? ? ?; ? ?   ?-----BI---`v.v.v.v.v.ryv.v.v.t     BI     /3/3/3/3/329/3/3/3f     bi           %%%%%)%%%PE E E E E E E E P********6PPPPP")PPPf     29   .%% I% <% % < % U\* %  %%%%%.F)F)F)F)F)F)F)F)T/////6///fT T T T T ryT T T n3 3 3 3 3 293 3 3 l 2 2 2 2 2 '2 2  2 j 2 2 2 2 2BI 2 2 2 ~ ~ ~ ~ ~ !~ ~  ~ B zzzzz'zz  zz Q P Pr1