`iL&dZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m Z dd l mZmZdd lmZmZmZdd lmZdd lmZmZmZmZmZdd lmZddlmZm Z m!Z!ddl"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*m+Z+e$j,e-Z.dZ/ee#dGddeZ0ee#dGddeZ1Gdde j2Z3Gdde j2Z4Gdd e j2Z5Gd!d"e j2Z6Gd#d$e j2Z7Gd%d&e j2Z8Gd'd(e j2Z9Gd)d*e j2Z:Gd+d,e j2Z;Gd-d.e j2ZGd2d3e j2Z?Gd4d5eZ@Gd6d7e j2ZAGd8d9e j2ZBdWd:ZCe#Gd;deDZEe#d?Gd@dAeDZFe#dBGdCdDeDZGGdEdFe j2ZHGdGdHe j2ZIGdIdJe j2ZJe#dKGdLdMeDZKe#dNGdOdPeDZLGdQdRe j2ZMe#dSGdTdUeDZNgdVZOdS)XzPyTorch BridgeTower ModelN) OrderedDict) dataclass)OptionalUnion)nn)CrossEntropyLoss)ACT2FNQuickGELUActivation)Cache DynamicCacheEncoderDecoderCache)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions,BaseModelOutputWithPoolingAndCrossAttentionsMaskedLMOutput ModelOutputSequenceClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging torch_int)deprecate_kwarg)BridgeTowerConfigBridgeTowerTextConfigBridgeTowerVisionConfigRobertaTokenizerz. Output type of [`BridgeTowerModel`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)BridgeTowerModelOutputa text_features (`torch.FloatTensor` of shape `(batch_size, text_sequence_length, hidden_size)`): Sequence of hidden-states at the text output of the last layer of the model. image_features (`torch.FloatTensor` of shape `(batch_size, image_sequence_length, hidden_size)`): Sequence of hidden-states at the image output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size x 2)`): Concatenation of last layer hidden-state of the first token of the text and image sequence (classification token), respectively, after further processing through layers used for auxiliary pretraining tasks. N text_featuresimage_features pooler_output hidden_states attentions)__name__ __module__ __qualname____doc__r%rtorch FloatTensor__annotations__r&r'r(tupler)/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/bridgetower/modeling_bridgetower.pyr$r$0s26M8E-.55526NHU./66615M8E-.5558 Output type of ['BridgeTowerForContrastiveLearning'] cLeZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed <dS) BridgeTowerContrastiveOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Image-text contrastive loss. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). text_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. image_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. cross_embeds (`torch.FloatTensor)`, *optional*, returned when model is initialized with `with_projection=True`): The text-image cross-modal embeddings obtained by applying the projection layer to the pooler_output. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Nlosslogits text_embeds image_embeds cross_embedsr(r))r*r+r,r-r7rr.r/r0r8r9r1r:r;r(r)r2r3r4r6r6Hs )-D(5$ %,,,*.FHU& '...6:K% 123:::7;L(5!234;;;7;L(5!234;;;8PTV>VWW L!39NOOO = RYv'96;MPQ;QRRS0223ry);a)?ASTTU    L!39NOOO r3 hidden_stateattention_maskc|&|tj|j}|j&|j|j|jnd|_||||d|j|dS)NdtypedeviceF) need_weightsrRkey_padding_maskr)tor.boolr\rRr[rJ)rTrWrXs r4 attentionz&BridgeTowerResidualAttention.attentionzs  %+..UZ H[.\\N~) N  L$6|?R  S S S  yy   n+      r3Nc|||||z}||}|jD] }||}||z}|SN)rarMrQrPvalues)rTrWrXresidual_statelayers r4forwardz$BridgeTowerResidualAttention.forwardst%tyy7N7NP^(_(__yy00 X__&& / /E 5..LL% 4 r3rc) r*r+r,rGr.Tensorrarrg __classcell__rVs@r4r=r=hs" el EL    "EL(5z3BridgeTowerTransformer.__init__..s"aaa!-f55aaar3rc.g|]}tSr2rorps r4rsz3BridgeTowerTransformer.__init__..s"]]]!-f55]]]r3) rFrGrInum_hidden_layersremove_last_layerr ModuleListrange resblocks stop_gradientrSs `r4rGzBridgeTowerTransformer.__init__s !-!'!9  # ]aaaauTE[^_E_?`?`aaaDNN ]]]]]uTE[?\?\]]]DN$1r3NrWrXcg}|jD]R}|||}|jr(||=||S|Src)ryrzappenddetach)rTrWrXr(blocks r4rgzBridgeTowerTransformer.forwardst ^ 3 3E 5~>>L! 3$$\%8%8%:%:;;;;$$\2222r3rc) r*r+r,rGr.rhrrgrirjs@r4rlrlsh 2 2 2 2 2EL(51D!-1"$,t/A4>"R"R ^U\$:L-M-M-T-TU\-]-]jopppppr3 embeddingsheightwidthreturnc|jddz }|jjd}|jddz }tjs&||kr ||kr||jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} | d| | | }| dddd}tj || | fdd }| dddddd| }t j||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr?r rbicubicF)sizemode align_cornersdim)shaperweight unsqueezer.jit is_tracingrrrreshapepermuter functional interpolateviewcat) rTrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r4interpolate_pos_encodingz4BridgeTowerVisionEmbeddings.interpolate_pos_encodings!&q)A- !4;EEaHH*03a7 y##%% >+*F*F6UZ??**4+<== =,QQQU3,QQQU3r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr3F pixel_valuesc <|j\}}}}|s<||jks ||jkr&td|d|d|jd|jd |jjj}|||}|ddd}|j |dd} tj | |gd } |r| | | ||z} n| ||jz} | S) NzInput image size (*z) doesn't match model (z).r[rrrr)rr ValueErrorrrr[r_flatten transposerrr.rrrr) rTrr batch_sizerrrr target_dtype patch_embeds class_embedsrs r4rgz#BridgeTowerVisionEmbeddings.forwardsD'3'9$ Avu' Vt-F-F%SWSbJbJbqVqqeqqDOqq^b^mqqq +28 ++LOO,O,O,OPP #++A..88A>> +22:q"EE Y l;CCC # Q#d&C&CJPVX]&^&^^JJ#d&=&=d>O&P&PPJr3F) r*r+r,r rGr.rhintrr/rgrirjs@r4rrsq6qqqqqq,'D5<'D'DUX'D]b]i'D'D'D'DRE$5Z_Zfr3rcpeZdZfdZ d dejdefdZ d dejdefdZdejfdZ xZ S) BridgeTowerVisionTransformerctt|_t jjj|_t|_ t jjj|_ j |_ j s9t j fdtjD|_dSdS)Nr@cPg|]"}tjjj#S)r@)rrKrIrLrps r4rsz9BridgeTowerVisionTransformer.__init__.. s/vvvQRf0f6KLLLvvvr3)rFrGrrrrKrIrLln_prerl transformerln_postshare_layernormrwrxru ln_separaterSs `r4rGz%BridgeTowerVisionTransformer.__init__s 5f==l6#56;PQQQ 1&99|F$6F>-2K"KLL L  !\ 1 1>>-$2D"DG`"`aa a  !] 2 2>>-1ty="AD]`d`iDi"ijj j%&b9M&b&b&bcc cr3r*r+r,rGrgrirjs@r4rr<sS f f f f fdddddddr3rcPeZdZfdZdejdejdejfdZxZS)BridgeTowerSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr@) rFrGrrOrIdenserKrLDropouthidden_dropout_probdropoutrSs r4rGzBridgeTowerSelfOutput.__init__Wsf Yv163EFF f&8f>STTTz&"<== r3r( input_tensorrc||}||}|||z}|SrcrrrKrTr(rs r4rgzBridgeTowerSelfOutput.forward]@ =11  ]33 }|'CDD r3r*r+r,rGr.rhrgrirjs@r4rrVi>>>>> U\RWR^r3rcBeZdZfdZdejdejfdZxZS)BridgeTowerIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSrc) rFrGrrOrIintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrSs r4rGz BridgeTowerIntermediate.__init__fsn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r3r(rcZ||}||}|Src)rrrTr(s r4rgzBridgeTowerIntermediate.forwardns, =11 00?? r3rrjs@r4rres^99999U\elr3rcPeZdZfdZdejdejdejfdZxZS)BridgeTowerOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) rFrGrrOrrIrrKrLrrrrSs r4rGzBridgeTowerOutput.__init__vsf Yv79KLL f&8f>STTTz&"<== r3r(rrc||}||}|||z}|Srcrrs r4rgzBridgeTowerOutput.forward|rr3rrjs@r4rrurr3rcBeZdZfdZdejdejfdZxZS)BridgeTowerPoolercttj|j|j|_tj|_dSrc)rFrGrrOrIrTanh activationrSs r4rGzBridgeTowerPooler.__init__sC Yv163EFF '))r3r(rcr|dddf}||}||}|S)Nr)rr )rTr(first_token_tensor pooled_outputs r4rgzBridgeTowerPooler.forwards@+111a40 #566  66 r3rrjs@r4r r s^$$$$$ U\elr3r ceZdZdfd Zeddd ddejd eejd eejd eejdee d ee d eejde ejfdZ xZ S)BridgeTowerSelfAttentionNcRt|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj|j|_|pt#|dd|_|jdks |jd kr6|j|_tjd |jzd z |j|_|j|_||_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()position_embedding_typeabsolute relative_keyrelative_key_queryrr)rFrGrInum_attention_headshasattrrrattention_head_size all_head_sizerrOquerykeyvaluerattention_probs_dropout_probrgetattrrmax_position_embeddingsrdistance_embedding is_decoder layer_idxrTrUrr%rVs r4rGz!BridgeTowerSelfAttention.__init__s    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #&v'9F( ' -zC C $  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # +"r3past_key_valuepast_key_values4.58new_nameversionFr(rX head_maskencoder_hidden_statesoutput_attentionscache_positionrc|j\}} } ||} | |d|j|jdd} d} |du} |Ht |tr1|j |j } | r|j }n |j }n|}| r|n|}| r3|1| r/|j |j j}|j |j j}n||}||d|j|jdd}||}||d|j|jdd}|N| s|nd}||||j d|i\}}| r$t |trd|j|j <t'j| |dd}|jdks |jd krt| jd|jd}}|>t'j|dz t&j|j dd}n:t'j|t&j|j dd}t'j|t&j|j dd}||z }|||jzdz }|| j }|jdkrt'jd | |}||z}n?|jd kr4t'jd | |}t'jd ||}||z|z}|t?j |jz }|||z}tBj"#|d}|$|}|||z}t'j||}|%dddd&}|'dd|j(fz}||}||fS)NrrrFr0TrrrZrzbhld,lrd->bhlrzbhrd,lrd->bhlrrrr ))rrrrrrrr is_updatedgetr%cross_attention_cacheself_attention_cachelayerskeysrdrrupdater.matmulrrlongr\rr#r"r_r[einsummathsqrtrrsoftmaxrr contiguousrr)rTr(rXr-r.r(r/r0r seq_lengthrr query_layerr3is_cross_attentioncurr_past_key_valuecurrent_states key_layer value_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r4rgz BridgeTowerSelfAttention.forwards%2$7! Jjj// !&&z2t7OQUQijjtt q   2$>  &/+>?? 6,7;;DNKK %O*9*O''*9*N''&5#2DW..-  F/"=*"=+24>BGI-4T^DKKK00I!z2t7OQUQijjtt1I**^44K%**B 8$:Ri1oo *7I!St)<)C)C{DN=M~<^**& ;&F*_FY*Z*ZFAEO.t~>!< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q'2'8';Y_Q=O*L*!&j1nEJWdWk!l!l!l!q!q"""'l%*UbUi!j!j!j!o!oprtu!v!v"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|S"S%**+BCC o--r3NNNNNNFN)r*r+r,rGrr.rhrr/r r`r1rgrirjs@r4rrs######6_%0A6RRR7;15=A+/,115e.e.|e.!!23e.E-. e. ((9: e. "% e.$D>e.!.e. u| e.e.e.SRe.e.e.e.e.r3reagerceZdZdfd ZdZeddd dd ejd eej d eej d eej dee d ee deejde ejfdZ xZS)BridgeTowerAttentionNctt|j||||_t ||_t|_dS)Nrr%) rFrG#BRIDGE_TOWER_SELF_ATTENTION_CLASSES_attn_implementationrTroutputset pruned_headsr&s r4rGzBridgeTowerAttention.__init__ sc 78ST $;    ,F33 EEr3ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr)lenrrTrrr`rrrrr^rrunion)rTheadsindexs r4 prune_headsz BridgeTowerAttention.prune_heads*s u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r3r'r(r)r*Fr(rXr-r.r/r0rc ||||||||}||d|} | f|ddz} | S)NrXr-r.r(r/r0rr)rTr^) rTr(rXr-r.r(r/r0 self_outputsattention_outputoutputss r4rgzBridgeTowerAttention.forward<sfyy )"7+/)!   ;;|A FF#% QRR(88r3rUrV)r*r+r,rGrfrr.rhrr/r r`r1rgrirjs@r4rYrYs"""""";;;$_%0A6RRR7;15=A+/,115|!!23E-.  ((9:  "% $D>!. u| SRr3rYcbeZdZd fd Zeddd d dZd ZxZS) BridgeTowerBertCrossLayerNcFt|j|_d|_t |||_|j|_|j|_t |||_t||_ t||_ dS)Nrr%) rFrGchunk_size_feed_forward seq_len_dimrYrar$add_cross_attentioncrossattentionr intermediaterr^rTrUr%rVs r4rGz"BridgeTowerBertCrossLayer.__init__Vs '-'E$-f JJJ +#)#= 26YOOO3F;;'// r3r'r(r)r*Fc |||d|d} | d} | dd} || ||||||} | d} | | ddz} t|j|j|j| } | f| z} | S)N)rXr-r/r(rrrh)rarsrfeed_forward_chunkrprq)rTr(r.rXr-encoder_attention_maskr(r/r0self_attention_outputsrjrkcross_attention_outputs layer_outputs r4rgz!BridgeTowerBertCrossLayer.forwardas"& )/ "0" " 2!4),"&"5"5 1"7+/)#6# # 3153ABB770  #T%A4CSUe   /G+r3c\||}|||}|Srcrtr^rTrjintermediate_outputr{s r4rwz,BridgeTowerBertCrossLayer.feed_forward_chunk2"//0@AA{{#68HII r3rcrV)r*r+r,rGrrgrwrirjs@r4rmrmUs 0 0 0 0 0 0_%0A6RRR #+++SR+Zr3rmc0eZdZdfd Zeddd ddejd eejd eejd eejd eejdee d ee deejde ejfdZ dZ xZS)BridgeTowerTextLayerNct|j|_d|_t |||_|j|_|j|_|jr0|jst|dt |d||_ t||_ t||_ dS)Nrroz> should be used as a decoder model if cross attention is addedrr[)rFrGrprqrYrar$rrrrsrrtrr^rus r4rGzBridgeTowerTextLayer.__init__s '-'E$-f JJJ +#)#=  # x? j D!h!h!hiii"6vWamv"w"w"wD 3F;;'// r3r'r(r)r*Fr(rXr-r.rxr/r0rc |||||||} | d} |jr | dd} n | dd} |jrV|Tt|dstd|d|| ||||||} | d} | | ddz} t |j|j|j| } | f| zS) N)rXr-r/r(r0rrrrsz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`rh) rar$rrrsrrwrprq)rTr(rXr-r.rxr(r/r0ryrjrkrzr{s r4rgzBridgeTowerTextLayer.forwards9"& )/+) "0" " 2!4 ? 1,QrT2GG,QRR0G ? >4@4!122  DdDDD '+&9&9 5#&; /"3-':'' # 7q9  7" ==G0  #T%A4CSUe  ((r3c\||}|||}|Srcr}r~s r4rwz'BridgeTowerTextLayer.feed_forward_chunkrr3rc)NNNNNFN)r*r+r,rGrr.rhrr/r r`r1rgrwrirjs@r4rrs. 0 0 0 0 0 0_%0A6RRR7;15=A>B+/,1152)2)|2)!!232)E-. 2) ((9: 2) !)): ; 2)"%2)$D>2)!.2) u| 2)2)2)SR2)hr3rcHeZdZdfd Z ddejdeejdeejdeejd eejd eed ee d ee d ee dee deejde e eje ffdZ xZS)BridgeTowerTextEncoderNct|_tjfdt jD|_d|_dS)Nc2g|]}t|Sro)r)rqirUs r4rsz3BridgeTowerTextEncoder.__init__..s' ` ` `1 !&A 6 6 6 ` ` `r3F) rFrGrUrrwrxrurfgradient_checkpointingrus ` r4rGzBridgeTowerTextEncoder.__init__sf  ] ` ` ` `fF^@_@_ ` ` `   ',###r3FTr(rXr-r.rxr( use_cacher/output_hidden_states return_dictr0rc | rdnd} |rdnd} |r|jjrdnd}|jr%|jr|rtdd}|rD|jjr8|6tt|jt|j}|rO|jjrCt|tr.tdtj |}t|j D]Z\}}| r| |fz} |||nd}||||||||| }|d}|r$| |dfz} |jjr ||d fz}[| r| |fz} | std ||| | |fDSt||| | | S) Nr2zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rUzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.)rxr(r/r0rrrc3K|]}||V dSrcr2rqvs r4 z1BridgeTowerTextEncoder.forward..(s4  =!===  r3)last_hidden_stater(r(r)cross_attentions)rUrrrtraininglogger warning_oncer$rr rr1from_legacy_cache enumeraterfr)rTr(rXr-r.rxr(rr/rrr0all_hidden_statesall_self_attentionsall_cross_attentionsr layer_modulelayer_head_mask layer_outputss r4rgzBridgeTowerTextEncoder.forwardsc#7@BBD$5?bb4%6d4;;Zdrr`d  & "4= " "##p"  v/ vO4K1,dk2R2R2RT`hlhsTtTtTtuuO  U/ UJPU4V4V U   \    2COTTO(44 V VOA|# I$58H$H!.7.CillO(L%'= /"3-   M*!,M  V&9]1=MB+/$(,1/4&*15P P |P !!23P E-. P ((9: P !)): ; P "%P D>P $D>P 'tnP d^P !.P  uU\"$MM NP P P P P P P P r3rc2eZdZdZfdZ ddZdZxZS)BridgeTowerTextEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. clttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_t#|dd|_|dt)j|jdd|d t)j|jt(j d|j|_tj|j|j|j|_ dS) N) padding_idxr@rrrrFrtoken_type_idsr)rFrGrr vocab_sizerI pad_token_idword_embeddingsr"position_embeddingstype_vocab_sizetoken_type_embeddingsrKrLrrrr!rrr.rrzerosrrr;rrSs r4rGz"BridgeTowerTextEmbeddings.__init__Cs} !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<== '.v7PR\']']$  EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg    ".#%<  *F,>DL\$ $ $    r3Nrc|.|t||j|}n||}||}n|dd}|d}|mt |dr2|jddd|f}||d|} | }n+tj|tj |j j }|| |}| |} || z} |jdkr||} | | z } || } || } | S)NrrrrrZr)"create_position_ids_from_input_idsr&create_position_ids_from_inputs_embedsrrrrr.rr;rr\rrrrrKr) rT input_idsrr inputs_embedspast_key_values_length input_shaperAbuffered_token_type_ids buffered_token_type_ids_expandedrrrs r4rgz!BridgeTowerTextEmbeddings.forward\s{  $A)TM]_uvv #JJ=YY  #..**KK',,..ss3K ^  !t-.. m*.*=aaa*n*M'3J3Q3QR]^_R`bl3m3m0!A!&[ SWSdSk!l!l!l   00;;M $ : :> J J"%::  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r3c|dd}|d}tj|jdz||jzdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrrrZr)rr.rrr;r\rr)rTrrsequence_lengthrs r4rz@BridgeTowerTextEmbeddings.create_position_ids_from_inputs_embedss$((**3B3/ %a.|  q /D4D"Dq"HPUPZcpcw   %%a((// <<|jj|jjdn>t|t<r)|jj|jj t|tjtBfr'|j"|jjdSdSdS)Ngr)stdgg?)meanrr)"rUinitializer_factorrrrIrurryrinitnormal_rJin_proj_weight in_proj_biasdatazero_out_projrrPrBrErrrrOrrrKrfill_!BridgeTowerForContrastiveLearning logit_scalelogit_scale_init_valueBridgeTowerMLMHead)rTrrproj_stdattn_stdfc_stdr~s r4 _init_weightsz(BridgeTowerPreTrainedModel._init_weightssk, f: ; ; N /51t{?\;\ae:efH{.4H$+11d:F+5 M M 9x#~NNN ',22444 3 :3OOO  56C<HHH 0 7X^LLLL GOOF-=8c>O R R R GOOF-@GXX[^O \ \ \ \ BIr| D E E N M  & &CTCZ & @ @ @ @  - - N K  " " $ $ $ M  $ $S ) ) ) )  A B B N   # ) )$+*L M M M fry*<= > > %6;CZ K  " " $ $ $ $ $ % %CZCZr3N) r*r+r,rr0base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placementrModulerr2r3r4rrs] %&+#35ST"3%BI%%%%%%r3rcHeZdZUeed<fdZedZddZxZ S)BridgeTowerVisionModelrUcrt|t||_dSrc)rFrGrvisualrSs r4rGzBridgeTowerVisionModel.__init__s.    26:: r3c8|jjjjjSrc)rrrrr[rTs r4r[zBridgeTowerVisionModel.dtypes{%5<BBr3NFc`|||j||Src)rtyper[)rTimage image_maskrs r4rgzBridgeTowerVisionModel.forwards'{{5::dj11:?WXXXr3)NF) r*r+r,r r0rGpropertyr[rgrirjs@r4rrs ####;;;;;CCXCYYYYYYYYr3ra0 The model can behave as an encoder (with only self-attention) as well as a decoder, in which case a layer of cross-attention is added between the self-attention layers, following the architecture described in *Attention is all you need*_ by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin. To behave as an decoder the model needs to be initialized with the `is_decoder` argument of the configuration set to `True`. To be used in a Seq2Seq model, the model needs to initialized with both `is_decoder` argument and `add_cross_attention` set to `True`; an `encoder_hidden_states` is then expected as an input to the forward pass. .. _*Attention is all you need*: https://huggingface.co/papers/1706.03762 c"eZdZUeed<dfd ZdZdZdZe dde e j d e e j d e e j d e e j d e e j d e e j de e j de e j de e de ede ede ede ede e j deee j effdZxZS)BridgeTowerTextModelrUTct|||_t||_t ||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) rFrGrUrrrencoderr pooler post_init)rTrUadd_pooling_layerrVs r4rGzBridgeTowerTextModel.__init__st     3F;;-f55 3DN'///$  r3c|jjSrcrrrs r4get_input_embeddingsz)BridgeTowerTextModel.get_input_embeddingss ..r3c||j_dSrcrrTrs r4set_input_embeddingsz)BridgeTowerTextModel.set_input_embeddingss*/'''r3c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrfrarf)rTheads_to_prunerfrds r4 _prune_headsz!BridgeTowerTextModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Cr3NrrXrrr-rr.rxr(rr/rrr0rc | | n |jj} | | n |jj} | | n |jj} |jjr| | n |jj} nd} ||t d|+||||}n.||dd}nt d|\}}||j n|j }d}| Bt| ts| ddj dn| }|tj|||zf|}|gt!|jdr1|jjddd|f}|||}|}n!tj|tj| }|||}|jjrL|J|\}}}||f}|tj||}||}nd}|||jj}|||||| }||||||| | | | | | }|d}|j||nd}| s||f|d dzSt9|||j|j|j|j S)NFzDYou cannot specify both input_ids and inputs_embeds at the same timerz5You have to specify either input_ids or inputs_embedsrr2r\rrZ)rrrrr) rXr-r.rxr(rr/rrr0r)rr'r(r(r)r)!rUr/ruse_return_dictr$rr%warn_if_padding_and_no_attention_maskrr\rr rget_seq_lengthr.onesrrrrrr;get_extended_attention_maskinvert_attention_mask get_head_maskrurrrr(r(r)r) rTrrXrrr-rr.rxr(rr/rrr0rrrAr\rrrextended_attention_maskencoder_batch_sizeencoder_sequence_lengthrrencoder_hidden_shapeencoder_extended_attention_maskembedding_outputencoder_outputssequence_outputrs r4rgzBridgeTowerTextModel.forwards{$2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T!"  &"/5996"1%+B//$3355 #  !"Z*jCY6Y)ZdjkkkN  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z150P0PQ_al0m0m ; ! 3&;&G=R=W=W=Y=Y :  7$68O#P %-).4HQW)X)X)X&.2.H.HI_.`.` + +.2 +&&y$+2OPP ??%)'#9 +  ,, 2"7#B+/!5#)'   *!,8< 8O O444UY  J#]3oabb6II I;-'+;)7&1,=     r3)T)NNNNNNNNNNNNNN)r*r+r,rr0rGrrrrrr.rhr r`rr1rrgrirjs@r4rrs "!!! ///000CCC-11515/3,0048<9=+/$(,0/3&*15s s EL)s !.s !. s u|, s EL) s   -s  ( 5s !) 6s "%s D>s $D>s 'tns d^s !.s uU\"$PP Q!s s s ^s s s s s r3rzv The bare BridgeTower Model transformer outputting BridgeTowerModelOutput object without any specific head on c"eZdZfdZdZdZe ddeej deej deej d eej d eej d eej d eej d eej dee dee dee dee deej de de eejeffdZdZxZS)BridgeTowerModelct|_jjjrIt jjj|_ t jjj|_ npt j fdtj D|_ t j fdtj D|_ t jdj|_t!|_t%|_jsejr^|jjjD]L}|jjjjj|j_|jjjjj|j_Mt j fdtj D|_t j fdtj D|_t=|_t=|_ t j!jj"|_#t j!jj"|_$j%r)tM|_'tM|_(ntt j fdtj dz D|_'t j fd tj dz D|_(|)dS) NcNg|]!}tjjj"Sr2rrOrI)rqrrrU text_configs r4rsz-BridgeTowerModel.__init__..s+qqqA;2F4FGGqqqr3cNg|]!}tjjj"Sr2r )rqrrrU vision_configs r4rsz-BridgeTowerModel.__init__..s+sssa=4f6HIIsssr3rc2g|]}t|Srrmrqrrs r4rsz-BridgeTowerModel.__init__..' j j jQ &{a @ @ @ j j jr3c2g|]}t|Srrrs r4rsz-BridgeTowerModel.__init__..rr3r@c.g|]}tSr2rrps r4rsz-BridgeTowerModel.__init__.."[[[!%f--[[[r3rc.g|]}tSr2rrps r4rsz-BridgeTowerModel.__init__..rr3)*rFrGrUrr$share_cross_modal_transformer_layersrrOrIcross_modal_text_transformcross_modal_image_transformrwrxrurrr vision_modelr text_modelr"init_layernorm_from_vision_encoderrcross_modal_ln_separaterrrrcross_modal_image_layerscross_modal_text_layersr cross_modal_image_poolercross_modal_text_poolerrKrLcross_modal_text_layernormcross_modal_image_layernormshare_link_tower_layersrcross_modal_text_link_towercross_modal_image_link_towerr)rTrUrrrrVs ` @@r4rGzBridgeTowerModel.__init__s     , (  6 .0i 8OQWQc.d.dD +/1y9RTZTf/g/gD , ,.0mqqqqqQVW]WoQpQpqqq//D +02}sssssSXY_YqSrSrsss00D ,&(\!V5G%H%H"2=AA.{;;, J1Z J'.F J J!%!2!9!A!H!M #07?DI (* j j j j%PVPhJiJi j j j) ) %(*} j j j j%PVPhJiJi j j j( ( $ ):&(A(A%'8'@'@$+-,v7IvOd*e*e*e'+-<8JPVPe+f+f+f(  ) /CF/K/KD ,0DV0L0LD - -/1}[[[[uV=UXY=Y7Z7Z[[[00D ,13 [[[[uV=UXY=Y7Z7Z[[[11D - r3c4|jSrc)rrrs r4rz%BridgeTowerModel.get_input_embeddingss33555r3c:|j|dSrc)rrrs r4rz%BridgeTowerModel.set_input_embeddingss ,,U33333r3NFrrXrr pixel_maskr-rr:image_token_type_idxr/rrlabelsrrc | | n |jj} | | n |jj} | rdnd}| rdnd}| rdnd}| rdnd}| rdnd}||td| | n |jj} | r| nd} |}|j|}| r||fz }|&tj |tj |j }|j || |j }t|jjj|jjz dz}|jjjd|D]}|||d}| r||fz }|?|jj||jj|}n|ddd }| r||fz }|jjjjd|D]}||}| r||fz }|jj||jj}||}|tjdtj |j |}|||z}| |}|tj!d | tj |j |}||z}|"|}tj |d|dftj |j }|j || |j }|j#d||||| } | d}!|j$d||||| }"|"d}#| r||!|#ffz }| r|| d|"dffz }d}$tK|t|jjjD]h}%|jjj|%||d}|jjjj|%||jj}| |jj||z}|j&|$}&|j'|$}'|&|||z|!|}(|'||#|})|j#|$dz|(|)||| } | d}!|j$|$dz|)|(||| }"|"d}#|$dz }$| r||fz }||fz }||!|#ffz }| r|| d|"dffz }j|!|#}+}*|(|*|+},| r|||f}| stSd |*|+|,||fDStU|*|+|,|| S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. output_hidden_states (`bool`, *optional*): If set to `True`, hidden states are returned as a list containing the hidden states of text, image, and cross-modal components respectively. i.e. `(hidden_states_text, hidden_states_image, hidden_states_cross_modal)` where each element is a list of the hidden states of the corresponding modality. `hidden_states_txt/img` are a list of tensors corresponding to unimodal hidden states and `hidden_states_cross_modal` is a list of tuples containing `cross_modal_text_hidden_states` and `cross_modal_image_hidden_states` of each brdige layer. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base") >>> model = BridgeTowerModel.from_pretrained("BridgeTower/bridgetower-base") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> outputs.keys() odict_keys(['text_features', 'image_features', 'pooler_output']) ```Nr2zYBridgeTowerModel does not use `inputs_embeds`. Make sure to pass in `input_ids` instead.r)rrZrrrr)rXrxr/c3K|]}||V dSrcr2rs r4rz+BridgeTowerModel.forward..s0= ===r3)r%r&r'r(r))+rUr/rrrrrrr.rr;r\rr_rbrrfrurrrrr[rrryrrrr expand_asr%rfullr&r"r!rxr(r)get_cls_featuresr1r$)-rTrrXrrr,r-rr:r-r/rrr.rall_hidden_states_textall_hidden_states_imageall_hidden_states_crossrrrr9extend_text_masks split_indexrfr~image_embeds_with_lncross_modal_texttext_token_type_embeddingsimage_token_type_embeddingscross_modal_imageextend_image_maskslayer_outputs_textcross_text_featureslayer_outputs_imagecross_image_featureslink_layer_indexrtext_link_towerimage_link_towercross_text_features_cross_image_features_r%r& cls_featuress- r4rgzBridgeTowerModel.forwardsj2C1N--TXT_Tq$8$D $+Jj (??BK# 9&;.8&  ,3??!!$"3"9::Um@LL (//1a88L  7 #  6 #&-9CL[LQ ; ;E 5..L# ;'L?:'#07DD\EVEVW[WhWnEoEopp ::;GG%)%?%? KI4D E E E& & )$ % % # ::;KNh;hii#??@TUU&*&@&@ Jt1IL\ ] ] ]' ' )( ) ) $ 46QQ <<=QRRZ  # #A & &(9(>(>q(A(A B*#   "_HHU_UdUdUfUfggjj    =T9!<  ,#5/    13>d;A>  -#4/     315  V #)<>R(S'U U #  V %7%:?OPO#@AQR $3?// <NQR>R!S$%0'9"3 """ #5Q"7 "U$"?@PST@T"U%$1'8"3 ### $7q#9  ! # Z&;.8&'L?:''-@BV,W+YY'  Z#);A)>@STU@V(W'YY#)<=Q~ ,,]NKK  k!79PRi j  'GXZmn  &')&+*     r3c||}||}tj||gdS)Nrr)r$r#r.r)rTr%r&cls_features_textcls_features_images r4r4z!BridgeTowerModel.get_cls_featuressF 88GG!::>JJy+-?@bIIIIr3)NNNNNNNNNNNNNF)r*r+r,rGrrrrr. LongTensorr/rr`rr1rhr$rgr4rirjs@r4r r s 66666p666444156:594815155948.2,0/3&*-1).j j E,-j !!23j !!12 j u01 j U-. j E-.j   12j u01j 'smj $D>j 'tnj d^j )*j #'j uU\"$:: ;!j j j ^j XJJJJJJJr3r c$eZdZfdZdZxZS)"BridgeTowerPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) rFrGrrOrIrrrrr transform_act_fnrKrLrSs r4rGz+BridgeTowerPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr3c||}||}||}|Src)rrQrKrs r4rgz*BridgeTowerPredictionHeadTransform.forwards= =11 --m<< }55 r3rrjs@r4rOrOsLUUUUUr3rOc&eZdZdfd ZdZxZS)rNc^t||_t||_t j|j|jj d|_ t j tj |jj |_|||j _dSdS)NF)r)rFrGrUrO transformrrOrIrrdecoderrr.rrr)rTrUrrVs r4rGzBridgeTowerMLMHead.__init__s  ;FCCy!3V5G5RY^___ LV-?-J!K!KLL  "(DL     r3cj||}|||jz}|Src)rUrVr)rTx mlm_scores r4rgzBridgeTowerMLMHead.forwards1NN1%% LL++di7 r3rcrrjs@r4rrsL))))))r3rc$eZdZfdZdZxZS)BridgeTowerITMHeadc|ttj|d|_dSNrrFrGrrOfc)rTrIrVs r4rGzBridgeTowerITMHead.__init__s0 )K++r3c0||}|Srcr_)rTrX itm_scores r4rgzBridgeTowerITMHead.forwardsGGAJJ r3rrjs@r4r[r[sG,,,,,r3r[z\ BridgeTower Model with a language modeling head on top as done during pretraining. ceZdZdgZfdZdZdZe ddee j dee j dee j d ee j d ee j d ee j d ee j d ee j dee dee dee dee j de eee j ffdZxZS)BridgeTowerForMaskedLMzmlm_score.decoder.weightct|t||_t ||_|dSrc)rFrGr rrrYrrSs r4rGzBridgeTowerForMaskedLM.__init__sR    +F33+F33 r3c|jjSrcrYrVrs r4get_output_embeddingsz,BridgeTowerForMaskedLM.get_output_embeddingss ~%%r3c||j_dSrcrg)rTnew_embeddingss r4set_output_embeddingsz,BridgeTowerForMaskedLM.set_output_embeddingss!/r3NrrXrrr,r-rr:r/rrr.rc | | n |jj} |||||||||| | |  } || r| jn| d}d}| jt }| |j} ||d|jj j | d}| st|}||f|zn|St||| j | jS)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM >>> from PIL import Image >>> import requests >>> url = "http://images.cocodataset.org/val2017/000000360943.jpg" >>> image = Image.open(requests.get(url, stream=True).raw).convert("RGB") >>> text = "a looking out of the window" >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist()) >>> print(results) .a cat looking out of the window. ```N rXrrr,r-rr:r/rrrrr7r8r(r))rUrrrYr%rr_r\rrrr1rr(r))rTrrXrrr,r-rr:r/rrr.rk mlm_logitsmasked_lm_lossloss_fctr^s r4rgzBridgeTowerForMaskedLM.forwards1d&1%Q 'tnQ d^Q )*Q  ~uU%677 8Q Q Q ^Q Q Q Q Q r3rdz BridgeTower Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text matching. ceZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee deejde e e ejffdZ xZS)#BridgeTowerForImageAndTextRetrievalct|t||_t |jdz|_|dSr])rFrGr rr[rIrbrrSs r4rGz,BridgeTowerForImageAndTextRetrieval.__init__UsZ    +F33+F,>,BCC r3NrrXrrr,r-rr:r/rrr.rc | | n |jj} |||||||||| | |  } | r| jn| d}||}d}| 4t }| |j} ||| }| st|}||f|zn|St||| j | j S)a  image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-base-itm-mlm") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, 1].item() ```Nrmrrn) rUrrr'rbrr_r\r1rr(r))rTrrXrrr,r-rr:r/rrr.rkr'r8itm_lossrqr^s r4rgz+BridgeTowerForImageAndTextRetrieval.forward_s \&1%Q 'tnQ d^Q )*Q  'u/@)AA BQ Q Q ^Q Q Q Q Q r3ruc$eZdZfdZdZxZS)BridgeTowerContrastiveHeadc|ttj|||_dSrcr^)rTrI embed_sizerVs r4rGz#BridgeTowerContrastiveHead.__init__s0 )K44r3c0||}|Srcra)rTrXs r4rgz"BridgeTowerContrastiveHead.forwards GGAJJr3rrjs@r4rzrzsG55555r3rzzl BridgeTower Model with a image-text contrastive head on top computing image-text contrastive loss. ceZdZfdZe ddeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee dee de e e ejffdZ xZS)rct|t||_t |j|j|_t |j|j|_t |jdz|j|_ tj tj |jj|_|dSr])rFrGr rrzrIcontrastive_hidden_size itc_text_headitc_image_headitc_cross_modal_headrrr.rrUrrrrSs r4rGz*BridgeTowerForContrastiveLearning.__init__s    +F3378JFLjkk89KVMkll$>v?QTU?UW]Wu$v$v!< T[5W(X(XYY r3NTrrXrrr,r-rr:r/rr return_lossrc "| | n |jj} |||||||||| d|  } | r| jn| d}| r| jn| d\}}}|d}|d}|jjj|}|jtj ddtj |jjj j |}|j||z}t j||ddd ddfdd }t j||ddd ddfdd |j }t j||dd |j }tj|||gd }|j|j }tj|||z}tj|||z}tj|||z}d}| rtjt;||j }t j||}t j||}t j||}||z|zdz }| s||||f| ddz} ||f| zn| St?|||||| j| j S)a image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> from transformers import BridgeTowerProcessor, BridgeTowerForContrastiveLearning >>> import requests >>> from PIL import Image >>> import torch >>> image_urls = [ ... "https://farm4.staticflickr.com/3395/3428278415_81c3e27f15_z.jpg", ... "http://images.cocodataset.org/val2017/000000039769.jpg", ... ] >>> texts = ["two dogs in a car", "two cats sleeping on a couch"] >>> images = [Image.open(requests.get(url, stream=True).raw) for url in image_urls] >>> processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> model = BridgeTowerForContrastiveLearning.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-itc") >>> inputs = processor(images, texts, padding=True, return_tensors="pt") >>> loss = model(**inputs, return_loss=True).loss >>> inputs = processor(images, texts[::-1], padding=True, return_tensors="pt") >>> loss_swapped = model(**inputs, return_loss=True).loss >>> print("Loss", round(loss.item(), 4)) Loss 0.0019 >>> print("Loss with swapped images", round(loss_swapped.item(), 4)) Loss with swapped images 2.126 ```NTrmrr rr0rrZr)rprr2rg@)r7r8r9r:r;r(r))!rUrrr'r(rrrrr.r3r;rr\r2rrr normalizerrr_rrrexpr:trrb cross_entropyr6r))!rTrrXrrr,r-rr:r/rrrrkr'hidden_states_txthidden_states_imghidden_states_cross_modalr9r:r=r;r8rlogits_text_to_imagelogits_text_to_crosslogits_image_to_crossitc_lossr.text_to_image_losstext_to_cross_lossimage_to_cross_lossr^s! r4rgz)BridgeTowerForContrastiveLearning.forwardsj&1%>;K;K L L{ Z  ]\#f++fmDDDF!#!SU["\"\ *-??BUUY\\H Nk<FQRQSQSTF-5-AXK&((v M+#%%!/)    r3) NNNNNNNNNTNN)r*r+r,rGrrr.rMr/r`rr6r1rgrirjs@r4rrs|     156:594815155948,0/3&*&*x x E,-x !!23x !!12 x u01 x U-. x E-.x   12x u01x $D>x 'tnx d^x d^x  +U53D-EE Fx x x ^x x x x x r3r)rrurdr r)r)Pr-r= collectionsr dataclassesrtypingrrr.rtorch.nnr activationsr r cache_utilsr r rmodeling_layersrmodeling_outputsrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrutils.deprecationrconfiguration_bridgetowerrrr get_loggerr*r_TOKENIZER_FOR_DOCr$r6rr=rlrrrrrrr rr\rYrmrrrrrrrr rOrr[rdrurzr__all__r2r3r4rs=  ######!!!!!!"""""""" %%%%%%66666666CCCCCCCCCC999999.-----llllllllll7777777777000000hhhhhhhhhh  H % %'  :::::[::   :$  :::::;::   :4)))))29)))XRY6PPPPP")PPPf7"7"7"7"7"297"7"7"tddddd29ddd4     BI        bi                    B.B.B.B.B.ryB.B.B.L %'# 3333329333l===== ===@GGGGG5GGGVY Y Y Y Y RYY Y Y zV=V=V=V=V= V=V=V=t 4 4 4 4 %%%%%%%%D Y Y Y Y Y7 Y Y Y    U U U U U 5U U   U p oJoJoJoJoJ1oJoJ  oJf "          d d d d d 7d d  d N ] ] ] ] ] *D] ]  ] @ G G G G G (BG G  G T   r3