`iddlZddlmZmZddlZddlmZmZmZddlm Z ddl m Z ddl m Z mZmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZddl m!Z!ddl"m#Z#ej$e%Z&Gddej'Z(Gddej'Z)Gddej'Z*Gddej'Z+Gddej'Z,Gddej'Z-GddeZ.Gdd ej'Z/Gd!d"ej'Z0Gd#d$ej'Z1Gd%d&ej'Z2Gd'd(ej'Z3Gd)d*eZ4Gd+d,e4Z5Gd-d.e4eZ6gd/Z7dS)0N)OptionalUnion)Tensordevicenn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentions!CausalLMOutputWithCrossAttentions)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging)deprecate_kwarg)BlipTextConfigc eZdZdZfdZ d deejdeejdeejde d ej f d Z xZ S) BlipTextEmbeddingsz;Construct the embeddings from word and position embeddings.c*ttj|j|j|j|_tj|j|j|_ tj |j|j |_ tj |j |_|dt!j|jddt'|dd|_||_dS) N) padding_idxeps position_ids)rF) persistentposition_embedding_typeabsolute)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandgetattrr$configselfr9 __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/blip/modeling_blip_text.pyr'zBlipTextEmbeddings.__init__/s !|F,=v?Q_e_rsss#%<0NPVPb#c#c f&8f>STTTz&"<==   EL)GHHOOPWXXej    (/v7PR\']']$ Nr input_idsr! inputs_embedspast_key_values_lengthreturnc|||}n|dd}|d}||jdd|||zf}|||}|}|jdkr||}||z }||}||}|S)Nr"rr%)sizer!r,r$r.r/r3) r;r?r!r@rA input_shape seq_length embeddingsr.s r=forwardzBlipTextEmbeddings.forwardAs  #..**KK',,..ss3K ^  ,QQQ0FVlIl0l-lmL   00;;M"  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r>)NNNr) __name__ __module__ __qualname____doc__r'rr5 LongTensor FloatTensorintrrH __classcell__r<s@r=rr,sEE(153759&' E,-u/0  12  !$   r>rcBeZdZdfd ZdZdZdZdZeddd  dd e j d e e j de e j de e j de e j de e de ede e j dee j fdZxZS)BlipTextSelfAttentionNct||_|j|jzdkr.t |dst d|j|jfz|j|_t|j|jz |_|j|jz|_ ||_ tj |j|j |_ |rItj |j|j |_tj |j|j |_nHtj |j|j |_tj |j|j |_tj|j|_t)|dd|_|jdks |jdkr8|j|_tjd|jzd z |j|_dSdS) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)r$r% relative_keyrelative_key_queryr)r&r'r9r*num_attention_headshasattr ValueErrorrOattention_head_size all_head_size layer_idxrLinearqueryencoder_hidden_sizekeyvaluer1attention_probs_dropout_probr3r8r$r-r(distance_embeddingr;r9is_cross_attentionr^r<s r=r'zBlipTextSelfAttention.__init__as    : :a ? ?PVXhHiHi ?^%v'ABC  $*#= #&v'9F 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # # #>r=qr>c||_dSNattn_gradients)r;rks r=save_attn_gradientsz)BlipTextSelfAttention.save_attn_gradients}s,r>c|jSrirjr;s r=get_attn_gradientsz(BlipTextSelfAttention.get_attn_gradientss ""r>c||_dSri attention_map)r;rrs r=save_attention_mapz(BlipTextSelfAttention.save_attention_maps*r>c|jSrirqrns r=get_attention_mapz'BlipTextSelfAttention.get_attention_maps !!r>past_key_valuepast_key_values4.58new_nameversionF hidden_statesattention_mask head_maskencoder_hidden_statesencoder_attention_maskoutput_attentionscache_positionrBc F|j\} } } ||| d|j|jdd} |du} | r|n|}d}|Ht |tr1|j |j }| r|j }n |j }n|}| r|n|}| r3|1|r/|j |j j}|j |j j}n||| d|j|jdd}||| d|j|jdd}|N| s|nd}||||j d|i\}}| r$t |trd|j|j <t'j| |dd}|jdks |jd kr4|d} t'j| t&j|j dd}t'j| t&j|j dd}||z }|||jzdz }|| j }|jdkrt'jd | |}||z}n?|jd kr4t'jd | |}t'jd ||}||z|z}|t?j |jz }||||jz}tCj"d|}|#|}|||z}t'j||}|$dddd%}|dd|j&fz}|j|}||fS)Nr"rrXFrTrVrW)dtyperrzbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr )'shaper`viewrYr\ transpose isinstancer is_updatedgetr^cross_attention_cacheself_attention_cachelayerskeysvaluesrbrcupdater5matmulr$rDr6longrrer-toreinsummathsqrtrSoftmaxr3permute contiguousr])r;r|r}r~rrrwrr batch_sizerF_ query_layerrgrcurr_past_key_valuecurrent_states key_layer value_layerattention_scoresposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probsattention_probs_dropped context_layernew_context_layer_shapes r=rHzBlipTextSelfAttention.forwardsf%2$7! J JJ} % % T*b$":D3EY//>  &/+>?? 6,7;;DNKK %O*9*O''*9*N''&5#2DW..-  F/"=*"=+24>BGI-4T^DKKK((j"d&>@XYY1a   >**j"d&>@XYY1a  *7I!St)<)C)C{DN=M~<^**& ;&F*_FY*Z*ZFAEO.t~>!< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q&++--a0J"\*EJ}OcdddiijlnoppN"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|>#',,"?"?  &= &I # %S"S* *,CD o--r>riNNNNNFN)rIrJrKr'rlrorsrurr5rrrNr booltuplerHrPrQs@r=rSrS`sruuuuuu8---###+++"""_%0A6RRR7;15=A>B+/,115h.h.|h.!!23h.E-. h. ((9: h. !)): ; h."%h.$D>h.!.h. u| h.h.h.SRh.h.h.h.h.r>rScPeZdZfdZdejdejdejfdZxZS)BlipTextSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr) r&r'rr_r*denser/r0r1r2r3r:s r=r'zBlipTextSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r>r| input_tensorrBc||}||}|||z}|Srirr3r/r;r|rs r=rHzBlipTextSelfOutput.forward@ =11  ]33 }|'CDD r>rIrJrKr'r5rrHrPrQs@r=rri>>>>> U\RWR^r>rceZdZdfd ZdZeddd dd ejd eej d eej d eej dee d ee deejde ejfdZ xZS)BlipTextAttentionFNctt||||_t ||_t |_dS)Nr^)r&r'rSr;routputset pruned_headsrfs r=r'zBlipTextAttention.__init__sR )&2DPYZZZ (00 EEr>ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr)lenrr;rYr\rrr`rbrcrrr]union)r;headsindexs r= prune_headszBlipTextAttention.prune_heads s u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r>rvrwrxryr|r}r~rrrrBc ||||||||}||d|} | f|ddz} | S)Nr}r~rrwrrrr)r;r) r;r|r}r~rrwrr self_outputsattention_outputoutputss r=rHzBlipTextAttention.forwardsfyy )"7+/)!   ;;|A FF#% QRR(88r>)FN)NNNNFN)rIrJrKr'rrr5rrrNr rrrHrPrQs@r=rrs"""""" ;;;$_%0A6RRR7;15=A+/,115|!!23E-.  ((9:  "% $D>!. u| SRr>rcBeZdZfdZdejdejfdZxZS)BlipTextIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSri) r&r'rr_r*intermediate_sizerr hidden_actstrr intermediate_act_fnr:s r=r'zBlipTextIntermediate.__init__9sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r>r|rBcZ||}||}|Sri)rrr;r|s r=rHzBlipTextIntermediate.forwardAs, =11 00?? r>rrQs@r=rr8s^99999U\elr>rcPeZdZfdZdejdejdejfdZxZS)BlipTextOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r&r'rr_rr*rr/r0r1r2r3r:s r=r'zBlipTextOutput.__init__Isf Yv79KLL f&8f>STTTz&"<== r>r|rrBc||}||}|||z}|Srirrs r=rHzBlipTextOutput.forwardOrr>rrQs@r=rrHrr>rc.eZdZfdZeddd ddejd eejd eejd eejd eejdee d ee deejde ejfdZ dZ xZS) BlipTextLayerc`t||_|j|_d|_t |||_||_|jjr!t ||jj||_ t||_ t||_ dS)Nrr)rgr^)r&r'r9chunk_size_feed_forward seq_len_dimr attention layer_num is_decodercrossattentionr intermediaterr)r;r9rr<s r=r'zBlipTextLayer.__init__Ws  '-'E$*6YGGG" ; ! "34;+AY###D 188$V,, r>rvrwrxryNFr|r}r~rrrrrBc |||||||} | d} | dd} |1|| ||||||} | d} | | ddz} t|j|j|j| } | f| zS)N)r}r~rrwrrrr)rrrfeed_forward_chunkrr)r;r|r}r~rrrwrrself_attention_outputsrrcross_attention_outputs layer_outputs r=rHzBlipTextLayer.forwardes"& )/+) "0" " 2!4(, ,&*&9&9 5#&; /"3-':'' # 7q9  7 ;;G0  #T%A4CSUe  ((r>c\||}|||}|Sri)rr)r;rintermediate_outputrs r=rz BlipTextLayer.feed_forward_chunks2"//0@AA{{#68HII r>r)rIrJrKr'rr5rrrNr rrrHrrPrQs@r=rrVs) - - - - -_%0A6RRR7;15=A>B+/,115%)%)|%)!!23%)E-. %) ((9: %) !)): ; %)"%%)$D>%)!.%) u| %)%)%)SR%)Nr>rcFeZdZfdZ ddejdeejdeejdeejd eejd eed ee d ee d ee dee deejde e eje ffdZ xZS)BlipTextEncoderct|_tjfdt jD|_d|_dS)Nc0g|]}t|S)r).0ir9s r= z,BlipTextEncoder.__init__..s##f#f#fM&!$<$<#f#f#fr>F) r&r'r9r ModuleListrangenum_hidden_layerslayergradient_checkpointingr:s `r=r'zBlipTextEncoder.__init__s`  ]#f#f#f#feFLdFeFe#f#f#fgg &+###r>NFTr|r}r~rrrw use_cacheroutput_hidden_states return_dictrrBc N|jr%|jr|rtdd}|rt |t r/tdtj|}nqt |tr$t|t|j }n8|6tt|j t|j }| rdnd} |rdnd} |r|dnd}t|j j D]Y}|j |}| r| |fz} |||nd}||||||||| }|d}|r| |dfz} | ||dfz}Z| r| |fz} | st d ||| | |fDSt||| | | S) NzZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.)r9rrrrXc3K|]}||V dSrir)rvs r= z*BlipTextEncoder.forward..s4  =!===  r>)last_hidden_staterwr| attentionscross_attentions)rtrainingloggerwarningrr warning_oncer from_legacy_cacher r9rrrr)r;r|r}r~rrrwrrrrrall_hidden_statesall_self_attentionsall_cross_attentionsr layer_modulelayer_head_mask layer_outputss r=rHzBlipTextEncoder.forwardsp  & "4= " "p"  /511 ##` #6"G"X"XO\:: "5o|[_[fGgGgGg"h"h ("5  444l$+6V6V6V###7@BBD$5?bb4%6f;P;\rrbft{455 V VA:a=L# I$58H$H!.7.CillO(L%&!  M*!,M  V&9]1=M) NNNNNNFFTN)rIrJrKr'r5rrrNr rrrrrHrPrQs@r=rrsK,,,,,7;15=A>B+/$(,1/4&*15W W |W !!23W E-. W ((9: W !)): ; W "%W D>W $D>W 'tnW d^W !.W  uU\"$MM NW W W W W W W W r>rcBeZdZfdZdejdejfdZxZS)BlipTextPoolercttj|j|j|_tj|_dSri)r&r'rr_r*rTanh activationr:s r=r'zBlipTextPooler.__init__sC Yv163EFF '))r>r|rBcr|dddf}||}||}|S)Nr)rr)r;r|first_token_tensor pooled_outputs r=rHzBlipTextPooler.forwards@+111a40 #566  66 r>rrQs@r=rrs^$$$$$ U\elr>rcBeZdZfdZdejdejfdZxZS)BlipTextPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) r&r'rr_r*rrrrr transform_act_fnr/r0r:s r=r'z(BlipTextPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr>r|rBc||}||}||}|Sri)rrr/rs r=rHz'BlipTextPredictionHeadTransform.forwards= =11 --m<< }55 r>rrQs@r=rrscUUUUUU\elr>rc*eZdZfdZdZdZxZS)BlipTextLMPredictionHeadc>tt||_t j|j|jd|_t j tj |j|_ |j |j_ dS)NF)bias) r&r'r transformrr_r*r)decoder Parameterr5zerosr"r:s r=r'z!BlipTextLMPredictionHead.__init__sz 8@@y!3V5FUSSS LV->!?!?@@ !I r>c(|j|j_dSri)r"r$rns r= _tie_weightsz%BlipTextLMPredictionHead._tie_weights&s I r>cZ||}||}|Sri)r#r$rs r=rHz BlipTextLMPredictionHead.forward)s*}55  ]33 r>)rIrJrKr'r(rHrPrQs@r=r r sV & & & & &&&&r>r cBeZdZfdZdejdejfdZxZS)BlipTextOnlyMLMHeadcptt||_dSri)r&r'r  predictionsr:s r=r'zBlipTextOnlyMLMHead.__init__1s/ 3F;;r>sequence_outputrBc0||}|Sri)r-)r;r.prediction_scoress r=rHzBlipTextOnlyMLMHead.forward5s ,,_==  r>rrQs@r=r+r+0s^<<<<<!u|! !!!!!!!!r>r+c,eZdZUdZeed<dZgZdZdS)BlipTextPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r9bertct|tjtjfr,|jjd|jjnWt|tj r=|j j |jj dt|tjr'|j "|j j dSdSdS)zInitialize the weightsg)meanstd?N) rrr_r(weightdatanormal_r9initializer_ranger/r"zero_fill_)r;modules r= _init_weightsz%BlipTextPreTrainedModel._init_weightsEs fry",7 8 8 * M  & &CT[5R & S S S S  - - * K  " " $ $ $ M  $ $S ) ) ) fbi ( ( %V[-D K  " " $ $ $ $ $ % %-D-Dr>N) rIrJrKrLr__annotations__base_model_prefix_no_split_modulesr?rr>r=r2r2;sI  % % % % %r>r2c#eZdZdZdfd ZdZdZdZdede e d e d e d ef d Z ddeejdeejdeejdeejdeejdeejdeejdeejdeedee dee dee dee d ee deejd ee ejeff dZxZS) BlipTextModela& The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in [Attention is all you need](https://huggingface.co/papers/1706.03762) by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. argument and `is_decoder` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. Tct|||_t||_t ||_|rt|nd|_| dSri) r&r'r9rrGrencoderrpooler post_init)r;r9add_pooling_layerr<s r=r'zBlipTextModel.__init__\so     ,V44&v.. 0AKnV,,,t  r>c|jjSrirGr,rns r=get_input_embeddingsz"BlipTextModel.get_input_embeddingsfs ..r>c||j_dSrirK)r;rcs r=set_input_embeddingsz"BlipTextModel.set_input_embeddingsis*/'''r>c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrFrrr)r;heads_to_prunerrs r= _prune_headszBlipTextModel._prune_headsmsU +0022 C CLE5 L u % / ; ;E B B B B C Cr>r}rErrrBc|dkr|dddddddf}n=|dkr |r|\}}tj||}|ddddf||d|ddddfk} | |j} | jd|jdkrP|jd| jdz } tjtj||| f|| j| gd} | dddddddf|ddddddfz}n,|ddddddf}ntd |d |jd ||j }d |z dz}|S)a= Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r NrXrr)rrr")axisz!Wrong shape for input_ids (shape z) or attention_mask (shape )rr7g) rr5r6repeatrrrcatonesr[) r;r}rErrextended_attention_maskrrFseq_ids causal_maskprefix_seq_lens r=get_extended_attention_maskz)BlipTextModel.get_extended_attention_maskus&     1 $ $&4QQQaaa]&C # #    ! !Q & & K)4& J,z&AAA%dD!!!m4;;J TUVVZabfhihihikoboZpp )nn^-ABB $Q'.*>q*AAA%3%9!%<{?PQR?S%SN"')!J!+Z HQW_j_p(   ###K+6aaaqqq!!!m*D~VWVWVWY]_cefefefVfGg*g''*8D$9I*J''sKss\j\psss #:"<"<4:"<"N"N#&)@#@H"L&&r>NFr?r!r~r@encoder_embedsrrrwrrrrrc | | n jj} | | n jj} | | n jj} |r| | n jj} nd} ||t d|7|||}|\}}|j}ne|)|dd}|\}}|j}n:|)|dd}|\}}|j}nt dd}| Bt| ts| ddj dn| }|,tj|||zf|}||||}|t|t"r|d\}}}n|\}}}||f}t|t"rfd|D}nF|,tj|| }|}n|}nd}|jj}||||| }n|}|||||| | | | | | }|d}j|nd}| s||f|d dzSt1|||j|j|j|j S)a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. past_key_values (`Cache`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NFzDYou cannot specify both input_ids and inputs_embeds at the same timer"zGYou have to specify either input_ids or inputs_embeds or encoder_embedsrrc:g|]}|Sr)invert_attention_mask)rmaskr;s r=rz)BlipTextModel.forward..s(2w2w2wX\43M3Md3S3S2w2w2wr>rT)r?r!r@rA) r}r~rrrwrrrrrr)r pooler_outputrwr|rr)r9rruse_return_dictrr[%warn_if_padding_and_no_attention_maskrDrrr rget_seq_lengthr5rYrr^listrb get_head_maskrrGrFrGrrwr|rr)r;r?r}r!r~r@r_rrrwrrrrrrrErrFrrArZencoder_batch_sizeencoder_sequence_lengthrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputsr.rs` r=rHzBlipTextModel.forwardsH2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**K%0 "J %FF  &',,..ss3K%0 "J ")FF  '(--//4K%0 "J #*FFfgg g!"  &"/5996"1%+B//$3355 #  !"ZZBX5X(YZZ]]^deeN150P0P K1 1  ! ,/66 ^AVWXAYA^A^A`A`>"$;QQAVA[A[A]A]>"$;Q$68O#P 0$77 e2w2w2w2w`v2w2w2w//'/).4HQW)X)X)X&262L2LMc2d2d//262L2LMc2d2d//.2 +&&y$+2OPP  !##)+'= /     . ,, 2"7#B+/!5#)'   *!,8< 8O O444UY  J#]3oabb6II I;-'+;)7&1,=     r>)T)NNNNNNNNNNNNNFN)rIrJrKrLr'rLrNrRrrrOrrr^rr5r rrrHrPrQs@r=rDrDSs ///000CCC<'$<'38:<'GM<'[_<' <'<'<'<'@-115/3,004158<9=+/$(,0/3&*%*15!P P EL)P !.P u|, P EL) P   - P !.P  ( 5P !) 6P "%P D>P $D>P 'tnP d^P TNP !.!P " uU\"$PP Q#P P P P P P P P r>rDc' eZdZddgZfdZdZdZdZdZ d d e e j d e e j de e j de e j de e j de e j de e j de e j de e de e de e de e de e de e de e de ede e j deee j eff$dZd!fd ZxZS)"BlipTextLMHeadModelzcls.predictions.decoder.weightzcls.predictions.decoder.biasct|t|d|_t ||_|j|_dS)NF)rI)r&r'rDr3r+clslabel_smoothingr:s r=r'zBlipTextLMHeadModel.__init__JsO    !&EBBB &v..%5r>c4|jSri)r3rLrns r=rLz(BlipTextLMHeadModel.get_input_embeddingsQsy--///r>c:|j|dSri)r3rNr;new_embeddingss r=rNz(BlipTextLMHeadModel.set_input_embeddingsTs &&~66666r>c$|jjjSri)rsr-r$rns r=get_output_embeddingsz)BlipTextLMHeadModel.get_output_embeddingsWsx#++r>cT||jj_|j|jj_dSri)rsr-r$r"rws r=set_output_embeddingsz)BlipTextLMHeadModel.set_output_embeddingsZs%'5$$2$7!!!r>NFTr5r?r}r!r~r@rrlabelsrwrrrr return_logitsr reductionrrBcb| | n |jj} |d} ||||||||| | | | | ||}|d}||}|r#|ddddddfSd}||ddddddf}|ddddf|j}t||j}|| d|jj | d}|dkr<| | dd d}| s|f|d dz}||f|zn|St|||j|j|j|j S) a encoder_hidden_states (`torch.FloatTensor`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. encoder_attention_mask (`torch.FloatTensor`, *optional*): Mask to avoid performing attention on the padding token indices of the encoder input. This mask is used in the cross-attention if the model is configured as a decoder. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. labels (`torch.LongTensor`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels n `[0, ..., config.vocab_size]` past_key_values (`Cache`, *optional*): Contains precomputed key and value hidden states of the attention blocks. Can be used to speed up decoding. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). NF) r}r!r~r@rrrwrrrrrrrr"r)rrtnonerX)losslogitsrwr|rr)r9rer3rsrrrrrtrr)rDsumrrwr|rr)r;r?r}r!r~r@rrr}rwrrrrr~rrrrr.r0lm_lossshifted_prediction_scoresloss_fctrs r=rHzBlipTextLMHeadModel.forward^s T&1%c Ltj|f||d|}d|d<|S)N)rwr}Tr)r&prepare_inputs_for_generation)r;r?rwr} model_kwargs model_inputsr<s r=rz1BlipTextLMHeadModel.prepare_inputs_for_generationsL=uww<  +)     &* \"r>)NNNNNNNNNNNNNFTr5N)NN)rIrJrK_tied_weights_keysr'rLrNrzr|rr5rr rrrrrrHrrPrQs@r=rqrqGs':Z $D>Z 'tnZ d^Z  ~Z TN!Z "C=#Z $!.%Z & uU\"$EE F'Z Z Z Z x          r>rq)rDrqr2)8rtypingrrr5rrrtorch.nnr activationsr cache_utilsr r r generationrmodeling_layersrmodeling_outputsrrrmodeling_utilsr pytorch_utilsrrrutilsrutils.deprecationrconfiguration_blipr get_loggerrIrModulerrSrrrrrrrrr r+r2rDrq__all__rr>r=rs-" """""""" $$$$$$$$$$%%%%%%!!!!!!CCCCCCCCCC))))))999999 .-----llllllllll000000......  H % %00000000hR.R.R.R.R.BIR.R.R.l        ///// ///f     29        RY   :::::.:::|^ ^ ^ ^ ^ bi^ ^ ^ D     RY    bi$ry0!!!!!")!!!%%%%%o%%%0p p p p p +p p p h~~~~~1?~~~B N M Mr>