`i ddlZddlZddlmZddlmZmZmZddlZddlm Z m Z ddl m Z ddl mZmZddlmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2m3Z3ddl4m5Z5m6Z6dZ7Gdde j8Z9dZ:dZ;Gdde j8Z< dade j8dej d ej d!ej d"eej d#e=d$e=d%eej d&e'e,fd'Z>Gd(d)e j8Z?Gd*d+e j8Z@Gd,d-e j8ZAd.ZBGd/d0e j8ZCGd1d2e j8ZDGd3d4eZEGd5d6e j8ZFGd7d8e j8ZGe-Gd9d:e$ZHGd;de j8ZJGd?d@e j8ZKGdAdBe j8ZLee-GdCdDeZMGdEdFe j8ZNGdGdHe j8ZOedIGdJdKe j8ZPGdLdMe j8ZQGdNdOe j8ZRdPZSdbdQZTdRej dSeUdTej fdUZVGdVdWe j8ZWGdXdYeZXe-GdZd[e$ZYGd\d]eYZZGd^d_eYeZ[gd`Z\dS)cN) dataclass)CallableOptionalUnion)Tensornn)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsBaseModelOutputWithPast,BaseModelOutputWithPoolingAndCrossAttentionsCausalLMOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSModuleUtilsMixinPreTrainedModelget_parameter_dtype)Unpack) find_pruneable_heads_and_indicesprune_linear_layer)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)OutputRecordercheck_model_inputs) EvollaConfig SaProtConfigc||}tj|d||z}||zS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor r%dim)neinttorchcumsumtype_aslong) input_ids padding_idxmaskincremental_indicess ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/evolla/modeling_evolla.py"create_position_ids_from_input_idsr65s` << $ $ ( ( * *D,t333;;DAADH  # # % % 33c8eZdZdZfdZ ddZdZxZS)EvollaSaProtEmbeddingszV Same as BertEmbeddings with a tiny tweak for positional embeddings indexing. cttj|j|j|j|_|jr&tj |j|j |_ nd|_ tj |j |_t|dd|_|dt%j|jdd|j|_|jdkr+tj|j|j|j|_|j|_|j|_d|_dS) N)r2epsposition_embedding_typeabsolute position_ids)r%F persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsemb_layer_norm_before LayerNormlayer_norm_eps layer_normDropouthidden_dropout_probdropoutgetattrr=register_bufferr-arangemax_position_embeddingsexpandr2position_embeddings token_dropout mask_token_idr?selfconfig __class__s r5rDzEvollaSaProtEmbeddings.__init__JsB !|F,=v?Q_e_rsss  ' # l6+=6CXYYYDOO"DOz&"<== '.v7PR\']']$  EL)GHHOOPWXXej    ".  ': 5 5')|.0BPTP`(((D $$1#1 r7Nc|-|t||j}n||}|||}|}|jr||||jkdd}d}||dn |j d}||jkd |z }|d|z zd|z ddddfz |j }|j dkr||} || z}|j||}|0||dz |j }|S)Nr@gQ?r%r>)r6r2&create_position_ids_from_inputs_embedsrIrW masked_fillrX unsqueezesumshapefloattodtyper=rVrM) rZr1attention_maskr? inputs_embeds embeddingsmask_ratio_train src_lengthsmask_ratio_observedrVs r5forwardzEvollaSaProtEmbeddings.forwardcs  $A)TM]^^ #JJ=YY   00;;M#    )"7#//d>P1P0[0[\^0_0_adeeJ) 4B4N.,,R000T]TcdeTfK#,0B#B"G"G"K"K"Q"Q"S"SVa"a $,<(<=EXAXZ[Z[Z[]acgZg@hhll J  ': 5 5"&":":<"H"H #&99J ? &44J  %$~'?'?'C'CCGG HXYYJr7c|dd}|d}tj|jdz||jzdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor Nr@r%rfdevicer)sizer-rSr2r0rprarU)rZrh input_shapesequence_lengthr?s r5r_z=EvollaSaProtEmbeddings.create_position_ids_from_inputs_embedss$((**3B3/ %a.|  q /D4D"Dq"HPUPZcpcw   %%a((// <<.E.Q.Q#*D  QW]3AHEEEMMdm\\AK4=11E)UEN33366qx@@C"wwyytQQQ)9:D "wwyytQQQ)9:D !111r7qkreturnc||d\|_|_t||j|j|jt||j|j|jfS)Nr)rr)rrrrrerf)rZrrs r5rmz#EvollaSaProtRotaryEmbedding.forwards-1-H-HZ\-H-]-]*$* %Q(8$:J K K N NUVU\ N ] ] $Q(8$:J K K N NUVU\ N ] ]  r7)r|) rurvrwrxr-r__annotations__r,rDrtuplermryrzs@r5rrs l C      2222   %, 5u|A[;\        r7rr^modulequerykeyvaluergscalingrP head_maskkwargsc tj||dd|z} t|dr.|jdvr$|jd} tj| tj| j dd} tj| tj| j dd} | | z } | | |j zdz }| |j }|jd krtjd ||}n<|jd kr1tjd ||}tjd ||}||z}| |z} |$|ddddddd|jd f}| |z} tj| dtj |j } tj| ||j} || |z} tj| |}|dd}|| fS)Nr|r r= relative_keyrelative_key_queryror@r%rrzbhld,lrd->bhlrrzbhrd,lrd->bhlrr)r*rf)ptraining)r-matmul transposehasattrr=rcrSr0rpviewdistance_embeddingrTrerfeinsumr functionalsoftmaxfloat32rPr contiguous)rrrrrgrrPrr attn_weights seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_key causal_mask attn_outputs r5eager_attention_forwardrsI<s}}Q':':;;gELv011?f6TY77[^ j > !$QQQ111o " o%=> #k1 =((2U](SSVVW\WbccL=((6?([[L#i/ ,|U33K''1--88::K  $$r7ceZdZd fd Z d dejdeejdeejdeejdeejd ee d e ejfd Z xZ S)EvollaSaProtSelfAttentionNFct||_|j|jzdkr0t |ds t d|jd|jd|j|_t|j|jz |_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_|j|_|pt#|dd|_d|_|jdks |jd kr7|j|_tjd |jzd z |j|_n%|jd krt/|j |_|j|_||_d|_|jo| |_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()r=r>rrr|r%rotaryr)r)rCrDr[rGnum_attention_headsr ValueErrorr,attention_head_size all_head_sizerLinearrrrattention_probs_dropout_probrPrQr=rotary_embeddingsrTrErr is_decoder layer_idxr is_causal)rZr[r=ris_cross_attentionr\s r5rDz"EvollaSaProtSelfAttention.__init__s    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #&v'9F( ' -zC C $"&  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # #  )X 5 5%@TE]%^%^%^D " +" C1C-Cr7 hidden_statesrgrencoder_hidden_statesencoder_attention_maskrrc |jdd\}}||d|jf} ||| dd} |du} | r|n|} | r|n|}|| | dd} || | dd}| |jdzz} |jdkr|| | \} } t}|j j dkrE|jdvr%td|j j d |jd t|j j }||| | ||f|jsd n|j|j|d |\}}|||d}||fS) Nr@r%r|reagerrzESM z attention does not support z^ embeddings. Set attention explicitly to 'eager' with `model.set_attn_implementation('eager')`r^)rPrr)rcrrrrrrr=rrr[_attn_implementationrrrrPrreshaper)rZrrgrrrr batch_sizer hidden_shape query_layerrcurrent_states key_layer value_layerattention_interfacerrs r5rmz!EvollaSaProtSelfAttention.forward3s"/!4SbS!9 J"JD4LM jj//44\BBLLQPQRR 2$>2DW..-3EY//>HH^,,11,??II!QOO jj0055lCCMMaQRSS "D$CC$,L % %  % % ! \"))*j"EEPPRR L((r7)NNFrt) rurvrwrDr-rr FloatTensorrrrrmryrzs@r5rrs D D D D D DJ7;15=A>B 3)3)|3)!!233)E-. 3) ((9: 3) !)): ; 3)+,3) u| 3)3)3)3)3)3)3)3)r7rc$eZdZfdZdZxZS)EvollaSaProtSelfOutputcttj|j|j|_tj|j|_dSN) rCrDrrrGdenserNrOrPrYs r5rDzEvollaSaProtSelfOutput.__init__jsJ Yv163EFF z&"<== r7cd||}||}||z}|SrrrPrZr input_tensors r5rmzEvollaSaProtSelfOutput.forwardo4 =11  ]33 % 4 r7rurvrwrDrmryrzs@r5rriG>>>>> r7rcHeZdZdfd ZdZ ddeefdZxZS) EvollaSaProtAttentionNFctt||||_t ||_t |_tj |j |j |_ dS)N)rrr;) rCrDrrZroutputset pruned_headsrrKrGrL)rZr[rrr\s r5rDzEvollaSaProtAttention.__init__wsk -f ^pqqq ,V44 EEf&8f>STTTr7ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrr%r))lenrrZrrrrrrrrrrunion)rZheadsindexs r5 prune_headsz!EvollaSaProtAttention.prune_heads~s u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r7rc ||}|j|f||||d|\}} |||}|S)Nrgrrr)rKrZr) rZrrgrrrrhidden_states_lnr_s r5rmzEvollaSaProtAttention.forwardsk >>-88"  )"7#9       Qkk+}== r7)NFrt) rurvrwrDrrrrmryrzs@r5rrvsUUUUUU;;;*"# +,r7rcf|dzdtj|tjdz zzS)zz This is the gelu implementation from the original EVOLLA_SA_PROT repo. Using F.gelu yields subtly wrong results. g?rg@)r-erfmathsqrt)rs r5gelurs/ s7cEIa$)C..&8999 ::r7cBeZdZfdZdejdejfdZxZS)EvollaSaProtIntermediatecttj|j|j|_dSr)rCrDrrrGintermediate_sizerrYs r5rDz!EvollaSaProtIntermediate.__init__s6 Yv163KLL r7rrcN||}t|}|Sr)rr)rZrs r5rmz EvollaSaProtIntermediate.forwards& =11 ]++ r7rurvrwrDr-rrmryrzs@r5rrscMMMMMU\elr7rc$eZdZfdZdZxZS)EvollaSaProtOutputcttj|j|j|_tj|j|_ dSr) rCrDrrr rGrrNrOrPrYs r5rDzEvollaSaProtOutput.__init__sJ Yv79KLL z&"<== r7cd||}||}||z}|Srrrs r5rmzEvollaSaProtOutput.forwardrr7rrzs@r5rrrr7rcFeZdZfdZ ddeefdZdZxZS)EvollaSaProtLayerct|j|_d|_t ||_|j|_|j|_|jr/|jst|dt |d|_ t||_ t||_ tj|j|j|_dS)Nr%z> should be used as a decoder model if cross attention is addedT)rr;)rCrDchunk_size_feed_forward seq_len_dimr attentionradd_cross_attention RuntimeErrorcrossattentionr intermediaterrrrKrGrLrYs r5rDzEvollaSaProtLayer.__init__s '-'E$.v66 +#)#=  # Y? l"d#j#j#jkkk"7SW"X"X"XD 4V<<(00 f&8f>STTTr7Nrc |j|f||d|}|jr8|6t|dstd|d|j|f||||d|}||}|S)N)rgrrz'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r)rrrAttributeErrorrfeed_forward_chunk) rZrrgrrrrattention_output layer_outputs r5rmzEvollaSaProtLayer.forwards*4>  )      ? 4@4!122 $`d``` 3t2  -#&;'=        ../?@@ r7c||}||}|||}|Sr)rKrr)rZrattention_output_lnintermediate_outputrs r5rz$EvollaSaProtLayer.feed_forward_chunksE"nn-=>>"//0CDD{{#68HII r7rt) rurvrwrDrrrmrryrzs@r5rrs U U U U U$"# !!+,!!!!Fr7rcPeZdZfdZe ddeefdZxZS)EvollaSaProtEncoderct|_tjfdt jD|_tjj j |_ d|_ dS)Nc.g|]}tS)r).0rr[s r5 z0EvollaSaProtEncoder.__init__..s"#g#g#g!$5f$=$=#g#g#gr7r;F) rCrDr[r ModuleListrangenum_hidden_layerslayerrKrGrLemb_layer_norm_aftergradient_checkpointingrYs `r5rDzEvollaSaProtEncoder.__init__s}  ]#g#g#g#guVMeGfGf#g#g#ghh $&L1CI^$_$_$_!&+###r7Nrc t|jD]\}}|||nd} ||f|| ||d|} |jr||}t|S)Nr)last_hidden_state) enumerater-r.r) rZrrgrrrri layer_modulelayer_head_masks r5rmzEvollaSaProtEncoder.forwards )44  OA|.7.CillO(L-)&;'=   MM  $ E 55mDDM1MRRRRr7rt) rurvrwrDr!rrrmryrzs@r5r$r$s,,,,,"# SS+,SSSSSSSSr7r$cBeZdZfdZdejdejfdZxZS)EvollaSaProtPoolercttj|j|j|_tj|_dSr)rCrDrrrGrTanh activationrYs r5rDzEvollaSaProtPooler.__init__#sC Yv163EFF '))r7rrcr|dddf}||}||}|S)Nr)rr:)rZrfirst_token_tensor pooled_outputs r5rmzEvollaSaProtPooler.forward(s@+111a40 #566  66 r7r rzs@r5r7r7"s^$$$$$ U\elr7r7creZdZUeed<dgZdZdZdZe e e ddge e ddgdZ d Z d S) EvollaSaProtPreTrainedModelr[rTr%r)r layer_namer)r attentionscross_attentionscv|jj}t|tjrJ|jjd||j |jj dSdSt|tj rU|jjd||j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weightsr^meanstdNr)r[initializer_range isinstancerrweightdatanormal_biaszero_rEr2rKfill_)rZrrFs r5 _init_weightsz)EvollaSaProtPreTrainedModel._init_weightsAs'k+ fbi ( ( * M  & &CS & 9 9 9{&  &&((((('&  - - * M  & &CS & 9 9 9!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r7N)rurvrwr'r_no_split_modules_supports_flash_attn_supports_sdpa_supports_attention_backendrr#r_can_record_outputsrOr'r7r5r?r?1s ,-N"&+%~&?qU`aaab N4AJZ [ [ [  * * * * *r7r?c eZdZdeffd ZdZdZdZe dde e j de e j d e e e j effd Z dde d e ed e e jd e e jd e f dZxZS)EvollaSaProtProteinEncoderr[ct|t||_t ||_dSr)rCrDr9rir$encoderrYs r5rDz#EvollaSaProtProteinEncoder.__init__Rs=    088*622 r7c|jjSrrirIrZs r5get_input_embeddingsz/EvollaSaProtProteinEncoder.get_input_embeddingsWs ..r7c||j_dSrrZrZrs r5set_input_embeddingsz/EvollaSaProtProteinEncoder.set_input_embeddingsZs*/'''r7c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrXr-rr)rZheads_to_pruner-rs r5 _prune_headsz'EvollaSaProtProteinEncoder._prune_heads]sU +0022 C CLE5 L u % / ; ;E B B B B C Cr7Nr1rgrcR|}|\}}|j}|tj||f|}|||}|||}|||} | d} t| | j| j | j S)Nrr1rg)rgr)r1rrArB) rqrpr-onesriget_extended_attention_maskrXrrrArB) rZr1rgrrrrrprhextended_attention_maskencoder_outputssequence_outputs r5rmz"EvollaSaProtProteinEncoder.forwardes  nn&& !, J!  !"Z*j)A6RRRN)N[[ "&"B"B>S^"_"_,,}E\,]])!,;-)7&1,=     r7rrrprfc6|t|}|dkr |jjs|t jdt |dkr|dddddddf}ng|dkr4|jjrtj|||}n,|ddddddf}ntd|d|j d| |}d |z tj |jz}|S) a Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`Tuple[int]`): The shape of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. Nr|zNThe `device` argument is deprecated and will be removed in v5 of Transformers.r z!Wrong shape for input_ids (shape z) or attention_mask (shape rrr)rr*r[rwarningswarn FutureWarningr*create_extended_attention_mask_for_decoderrrcrer-finfomin)rZrgrrrprfrhs r5rgz6EvollaSaProtProteinEncoder.get_extended_attention_mask~sa& ='--E""$$))dk.D)! dfs     1 $ $&4QQQaaa]&C # #    ! !Q & &{% K*:*e++''+9D$9I*J''sKss\j\psss #:"<"<5"<"I"I#&)@#@EKPUDVDVDZ"Z&&r7rNN)rurvrwr'rDr\r_rcr$rr-rrrrrmr,rprfrgryrzs@r5rVrVQs>3|333333 ///000CCC26  EL) !.  uU\"$PP Q     8*.'+ 6'6'6'3Z6'& 6'  $ 6'  6'6'6'6'6'6'6'6'r7rVc&eZdZdfd ZdZxZS)!EvollaSequenceCompressorAttention@ct|dz|_||_||z}t j||_t j||_t j||d|_ t j||dzd|_ t j||d|_ dS)NrFrLr|) rCrDscalerrrK norm_media norm_latentsrto_qto_kvto_out)rZr*dim_headr inner_dimr\s r5rDz*EvollaSequenceCompressorAttention.__init__s t^  u$ ,s++L--Ic95999 YsIM>>> i 3U;;; r7c ||}||}|j}||}t j||fd}||dd\}}|| d| d|d dddd}|| d| d|d dddd}|| d| d|d dddd}||j z}t j || dd} | | dd z } | j\} } } } t j| | |j}|d d d d d d f}|d d d d d d f}||z}| d|z d } | d}t j ||}| dddd}|| d| dd}||S) z Args: x (torch.Tensor): image features shape (b, n1, D) latent (torch.Tensor): latent features shape (b, n2, D); n2: num of latent tokens rr)r|r@rr%r Tr*keepdimNg)rzr{rr|r-r~r}r}rrqpermuteryrramaxdetachrcrfrerpr`boolrrr~)rZrlatentsr3hrkv_inputrvsimbsnhskdokdrfmask_expones_expattnouts r5rmz)EvollaSequenceCompressorAttention.forwards OOA  ##G,, J IIg  9a\r222zz(##)) 2*  1 FF166!99affQiiB / / 7 71a C C FF166!99affQiiB / / 7 71a C C FF166!99affQiiB / / 7 71a C C  Nl1akk"b1122CHHTH2299;;;9BSz"c""%%dk224qqq()aaaD()("ooq4xoo//66{{r{""l4##kk!Q1%%kk#((1++sxx{{B77{{3r7)rurvrrzs@r5rtrtsL < < < < < <) ) ) ) ) ) ) r7rtc&eZdZdfd ZdZxZS)EvollaFeedForwardc>tt||z}tj||_tj||d|_tj|_ tj||d|_ dSNFrx) rCrDr,rrKnormrfc1GELUr:fc2)rZr*multrr\s r5rDzEvollaFeedForward.__init__sz d OO L%% 9S)%888'))9Y%888r7c |||||Sr)rr:rr)rZrs r5rmzEvollaFeedForward.forwards6xx1(>(>??@@@r7)rrrzs@r5rrsS999999AAAAAAAr7rc*eZdZdeffd ZdZxZS)!EvollaSequenceCompressorResamplerr[c pt|jj}|j|_t jtj |j|d|_ t j g|_ t|jD]^}|j t j t!||j|jt'||jg_t j|j|_t j||j|_dS)NT) requires_grad)r*rr)r*r)rCrDprotein_encoder_configrGresampler_num_latents num_latentsr Parameterr-randnrr*layersr+resampler_depthappendrtresampler_dim_headresampler_headsrresampler_ff_multrKrrprotein_projector)rZr[protein_repr_dimrr\s r5rDz*EvollaSequenceCompressorResampler.__init__s !8D!7|EK0@BR$S$Scghhh mB'' v-..  A K   9 06;T\b\r*.>VE]^^^     L!344 !#+;V=O!P!Pr7cN|jd}|j\}}tj||j|j}tj||fd}tj||jj}|jd|dddz}||j }|j D]#\} } | ||||z}| ||z}$| |} | | S)Nrr%r)r@) rcr-rfrrerpr~rrrfrrr) rZembedsr3brr latent_maskrfrrfftransformed_features r5rmz)EvollaSequenceCompressorResampler.forwards LO AjT%56699$+FF y$ ,!444z!}} 344,t$tyyQ':'::**V\**  , ,HD"d67D11G;GbkkG+GG"44W==yy,---r7)rurvrwr&rDrmryrzs@r5rrsZQ|QQQQQQ*.......r7rceZdZUdZeejed<dZeejed<dZ ee ejdfed<dZ ee ejdfed<dS)EvollaProteinEncoderModelOutputNsequence_compressor_outputr1.rrA) rurvrwrrr-rrr1rrrAr'r7r5rr)s?C): ;BBB59x 12999=AM8E%"3S"89:AAA:>Ju0#567>>>>>r7rcXeZdZdeffd ZedejdejfdZ xZ S)EvollaProteinEncoderr[ctt|j|_t ||_dS)Nr[)rCrDrVrmodelrsequence_compressor_resamplerrYs r5rDzEvollaProteinEncoder.__init__3sH /v7TUUU -NV\-]-]-]***r7r1rgc |||}|j}|||}t||jS)Nre)rr1)rr1rr)rZr1rgrprotein_outputprotein_embeds sequence_reprs r5rmzEvollaProteinEncoder.forward8sTiWW'9::>>ZZ .'4,>    r7) rurvrwr&rDr!r- LongTensorrrmryrzs@r5rr2s^|^^^^^^  !1 5CT        r7rceZdZ d deedeedeeffd ZdZeddd  d d ZxZ S)#EvollaSequenceAlignerCrossAttentionNprotein_encoder_dimstructure_encoder_dimmsa_encoder_dimct|j|_|j|_|jdz|_t |j|jz |_|j|jz|_|j}|j }|j }tj |j|j|_ |?tj ||j|_tj ||j|_nd|_d|_|?tj ||j|_tj ||j|_nd|_d|_|?tj ||j|_tj ||j|_nd|_d|_t)|j|_tj||_tj |j|j||_t3|j||_tjt9jdg|_tjt9jdg|_dS)Nrrxr^) rCrDrGrryr,rr$aligner_attention_probs_dropout_probaligner_enable_biasaligner_ffn_multrrr key_protein value_protein key_structurevalue_structurekey_msa value_msa EvollaRMSNormattention_normrNrPout_projrrrr-tensorgate_attentiongate_ffw) rZr[rrrr enable_biasffn_multr\s r5rDz,EvollaSequenceAlignerCrossAttention.__init__Es  !-#)#= -t3 #&t'7$:R'R#S#S !58PP'-'R$0 *Yt/1CDD  *!y)PQQD !#+>@R!S!SD  #D !%D  ,!#+@$BT!U!UD #%9-BDDV#W#WD !%D #'D  &9_d6HIIDLY8JKKDNNDL!DN+D,<==z">??  $"2D4D;WWW #D$4h?? l5<+>+>?? U\3%%8%899 r7c |||g} d| D} | stdtj| d} ||} || } |jG|j@||}||} ||} nd} d} |jG|j @||}||} | |}nd} d}|j G|j @||}| |}| |}nd}d}| | |g}d|D}tj|d}| ||g}d|D}tj|d}| dd|j |jfz}| j|d d dd } | dd|j |jfz}|j|d d dd }| dd|j |jfz}|j|d d dd }| |jz} |Stj| d | d|j}|ddddddf| ddddddfz}tj| |dd }||dd z }|d|z tj|jj}t;jd|}tj||}|d d dd }| dd |j fz}|j|}|!|}|S)z query_states: text key_value_states: protein query_states: [bs, query_seq_len, dim] key_value_states: [bs, kv_seq_len, dim] query_attn_mask: [bs, query_seq_len] kv_attn_mask: [bs, kv_seq_len] cg|]}||Srr'r(rs r5r)zGEvollaSequenceAlignerCrossAttention.cross_attention..sAAAa1====r7z=At least one modality should be provided for cross attention.r%r)Ncg|]}||Srr'rs r5r)zGEvollaSequenceAlignerCrossAttention.cross_attention..s;;;1Q]Q]]]r7cg|]}||Srr'rs r5r)zGEvollaSequenceAlignerCrossAttention.cross_attention..s???Qqr7r@rr|r rTr)"rr-r~rrrrrerrrrrqrrrrryrfrprrrrr`rrprfrqrSoftmaxrrr)rZ query_statesprotein_key_value_statesstructure_key_value_statesmsa_key_value_statesquery_attn_maskprotein_kv_attn_maskstructure_kv_attn_maskmsa_kv_attn_mask kv_attn_maskrkey_layer_proteinvalue_layer_proteinkey_layer_structurevalue_layer_structure key_layer_msavalue_layer_msarrnew_query_layer_shapenew_key_layer_shapenew_value_layer_shapergrattention_scoresattention_probs context_layernew_context_layer_shapes r5cross_attentionz3EvollaSequenceAlignerCrossAttention.cross_attentionxs*-.DFVW AA<AAA  ^\]] ]y1555 )),77 jj--   'D,>,J'?'B'B<'P'P $ $ 0 01I J J "&"4"45M"N"N   $ "&    )d.B.N)C)F)F|)T)T &"&"4"45O"P"P $($8$89S$T$T ! !"& $( ! < #(B#7#:#:<#H#H  LL)=>>M"nn-ABBOO M"O&(;]K ;; ;;; IiQ/// *,A?S ??+??? i 333 + 0 0 2 23B3 7  $  $; ! 'k&(=>FFq!QPQRR 'nn..ss3  $  $7  #IN$78@@Aq!LL + 0 0 2 23B3 7  $  $; ! 'k&(=>FFq!QPQRR !DJ.   "#j):):1)=)=|?P?PQR?S?STTWWXdXkllO(D!!!T)9:\!!!TSWYZYZYZJZ=[[|K1D1DR1L1LMM #l&7&7B&7&M&M&T&T&V&VV '33   % % ' '\5G)H)H)L  -"*,,,-=>> _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S* *,CD  m44 r7past_key_valuepast_key_values4.58new_nameversionc 8|q|j\} }}|ctj| || j| || fjz|j}nd}|q|j\} }}|ctj| || j| || fjz|j}nd}|q|j\} }}|ctj| || j| || fjz|j}nd}|}||s,||s||rv|}|||||||||}tj |j |z}||z}|}| |tj |j z}||z}|S)N)rq)rrrrrrrr) rcr-rfrerprUTanyrtanhrrr)rZrprotein_kv_statesstructure_kv_states msa_kv_statesrrrrprotein_batch_maskstructure_batch_maskmsa_batch_maskrrprotein_kv_seq_lenr*structure_kv_seq_lenmsa_kv_seq_lenrresiduals r5rmz+EvollaSequenceAlignerCrossAttention.forwards`  (*;*A 'B"C#+Jr#56699:L:STT(//6H"5M/NNPQ"&-..% $(  *,?,E )B$c%-Jr#788;;22556H6OPP$++."1E+FFHI"])**! $ $  */C/G/G/I/I *#/4J4N4N4P4P/).>.B.B.D.D)$H 00*):+>%2 /%9'=!11  M"Jt':;;mKM$}4M$H GGM22UZ 5N5NNM$}4Mr7)NNNNNNNNNN) rurvrwrr,rDrr"rmryrzs@r5rrDs.2/3)- 1:1:&c]1: (} 1: "# 1:1:1:1:1:1:fnnn`_%0A6RRR"#!GGGSRGGGGGr7rRMSNormc,eZdZdfd ZdZdZxZS)rư>cttjt j||_||_dS)z< EvollaRMSNorm is equivalent to T5LayerNorm N)rCrDrrr-rfrIvariance_epsilon)rZrGr<r\s r5rDzEvollaRMSNorm.__init__5sD l5:k#:#:;; #r7c|j}|tj}|ddd}|tj||jzz}|j||zS)Nr|r@T)r) rfrer-rpowrErsqrtrrI)rZr input_dtypevariances r5rmzEvollaRMSNorm.forward=s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r7cHt|jjd|jS)Nz, eps=)rrIrcrr[s r5 extra_reprzEvollaRMSNorm.extra_reprDs& )**II$2GIIIr7)r)rurvrwrDrmrryrzs@r5rr3sb$$$$$$;;;JJJJJJJr7rc|eZdZUejed<ddeffd Zeje dZ xZ S)EvollaRotaryEmbeddingrNr[ctt|drSt|jt r9|jd|jd|_nd|_|j|_ |j|_ ||_ t|j|_ | |j |\}|_|d|d|j|_dS)N rope_scaling rope_typetypedefaultrFrA)rCrDrrHrdictgetrrTmax_seq_len_cachedoriginal_max_seq_lenr[r rope_init_fnattention_scalingrRroriginal_inv_freq)rZr[rprr\s r5rDzEvollaRotaryEmbedding.__init__Ks  6> * * 'z&:Mt/T/T '#044[&BUBYBYZ`BaBabbDNN&DN"("@$*$B! /?+/+<+>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk ^ U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;sBE++E/2E/r) rurvrwr-rrr&rDno_gradrrmryrzs@r5rrHsl//|//////"U]__ < <_ < < < <