`i7cdZddlZddlZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZdd l m!Z!ddl"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-e&r ddl.m/Z/ddl0m1Z1e)j2e3Z4Gddej5Z6Gddej5Z7Gddej5Z8Gddej5Z9Gddej5Z:Gddej5Z;Gd d!ej5Z<Gd"d#eZ=Gd$d%ej5Z>e%Gd&d'e!Z?Gd(d)e?Z@e%Gd*d+e?ZAe%d,-Gd.d/e?eZBe%Gd0d1e?ZCe%d2-Gd3d4e?ZDe%Gd5d6e?ZEe%Gd7d8e?ZFgd9ZGdS):zPyTorch UMT5 model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringis_torch_flex_attn_availableis_torch_fx_proxyis_torchdynamo_compilinglogging)deprecate_kwarg) UMT5Config) BlockMask)make_flex_block_causal_maskc&eZdZdfd ZdZxZS) UMT5LayerNormư>cttjt j||_||_dS)ze Construct a layernorm module in the UMT5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/umt5/modeling_umt5.pyr*zUMT5LayerNorm.__init__?sD l5:k#:#:;; #ch|tjddd}|tj||jzz}|jjtj tj fvr||jj}|j|zS)NT)keepdim) tor,float32powmeanrsqrtr/r.dtypefloat16bfloat16)r0 hidden_statesvariances r4forwardzUMT5LayerNorm.forwardGs !##EM2266q99>>r4>PP% Ht?T4T(U(UU  ;  ? ? ?),,T[->??M{]**r5)r')__name__ __module__ __qualname__r*rD __classcell__r3s@r4r&r&>sL$$$$$$ + + + + + + +r5r&c*eZdZdeffd ZdZxZS)UMT5DenseActDenseconfigcJttj|j|jd|_tj|j|jd|_tj|j |_ t|j |_ dSNFbias)r)r*rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr0rLr3s r4r*zUMT5DenseActDense.__init__Ysx )FNFKeDDD)FKeDDDz&"566 &-.r5c||}||}||}t|jjt jr]|j|jjjkrC|jjjt j kr$| |jjj}||}|SN) rTrZrX isinstancerUr.r,Tensorr?int8r:r0rBs r4rDzUMT5DenseActDense.forward`s .. //  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r5rErFrGr"r*rDrHrIs@r4rKrKXsS/z//////       r5rKc*eZdZdeffd ZdZxZS)UMT5DenseGatedActDenserLcttj|j|jd|_tj|j|jd|_tj|j|jd|_tj |j |_ t|j |_dSrN)r)r*rrQrRrSwi_0wi_1rUrVrWrXr rYrZr[s r4r*zUMT5DenseGatedActDense.__init__ps IfnfkFFF IfnfkFFF )FKeDDDz&"566 &-.r5c|||}||}||z}||}t |jjtjr]|j |jjj krC|jjj tj kr$| |jjj }||}|Sr]) rZrfrgrXr^rUr.r,r_r?r`r:)r0rB hidden_gelu hidden_linears r4rDzUMT5DenseGatedActDense.forwardxshhtyy7788  -00 #m3  ]33 tw~u| 4 4 C#tw~';;;$ 22),,TW^-ABBM .. r5rbrIs@r4rdrdosS/z//////r5rdc*eZdZdeffd ZdZxZS) UMT5LayerFFrLc$t|jrt||_nt ||_t |j|j|_ tj |j |_ dS)Nr2)r)r* is_gated_actrdDenseReluDenserKr&rRlayer_norm_epsilon layer_normrrVrWrXr[s r4r*zUMT5LayerFF.__init__sx    <"8"@"@D  "3F";";D 'F+B,,,   4<eDDD4<eDDD4<eDDD4>4>FFq!QPQRRr5c~d}|j}|j}|jsC|dz}||dktj|zz }t j|}n(t j|t j| }|dz}||k}t j | |z tj ||z z }|||z z}||tjz}t j|t j ||dz }|t j |||z }|S)a Adapted from Mesh Tensorflow: https://github.com/tensorflow/mesh/blob/0cb87fe07da627bf0b7e60475d59f95ed6b5be3d/mesh_tensorflow/transformer/transformer_layers.py#L593 Translate relative position to a bucket number for relative attention. The relative position is defined as memory_position - query_position, i.e. the distance in tokens from the attending position to the attended-to position. If bidirectional=False, then positive relative positions are invalid. We use smaller buckets for small absolute relative_position and larger buckets for larger absolute relative_positions. All relative positions >=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr7r!)r{r|ryr:r,longabsmin zeros_likelogfloatmath full_likewhere) r0relative_positionrelative_buckets num_buckets max_distance max_exactis_small log_ratiorelative_position_if_larges r4_relative_position_bucketz'UMT5Attention._relative_position_bucketsP*9 ;  c A K !2Q!6 : :5: F F T T  % *; < <  !&+NO`>a>a!b!b b  1$ $y0I/5577)CDDtxP\_hPhGiGii y!89 %.ej1I1I%I"%*Y &8RT_bcTc(d(d& & " EK2CE_```r5c||jjj}|,tj|tj|dddf}n |dddf}tj|tj|dddf}||z }||}||} | gdd} | S)z%Compute binned relative position biasN)r?device)r7rr!r) rr.rr,arangerrr unsqueeze) r0 query_length key_lengthrcache_positioncontext_positionmemory_positionrrelative_position_bucketvaluess r4 compute_biaszUMT5Attention.compute_biass >18?F  !$|L SYZZZ[\[\[\^b[bc  -aaag6 ,zFSSSTXZ[Z[Z[T[\+.>>#'#A#ABS#T#T --.FGG **44Q77 r5past_key_valuepast_key_values4.58new_nameversionrBencoder_hidden_statesattention_masklayer_head_maskrc|jdd\}}|du} ||} | |d|j|jdd} d} |Ft |tr1|j |j } | r|j } n |j } n|} | r|n|} | r3|1| r/| j |j j}| j |j j}n|| }|| }||d|j|jdd}||d|j|jdd}|N| s|nd}| |||j d|i\}}| r$t |trd|j|j <t'j| |dd}|||zn|}|jd}|js+t'jd|j||f|j|j }n3||||j| }|dddd| dddf}|$|ddddddd|jdf}||z}|jrUt'j|jd}d |t;|j<|dd|f}n|}||z }t>j !|"d #|}t>j $||j$|j% }|||z}t'j||}|dd&}|||d}|'|}||fS)Nr7r8r!FrTr )rr?)rrrdim)ptraining)(shaperrrr~ transposer^r is_updatedgetrwcross_attention_cacheself_attention_cachelayerskeysrrrupdater,matmulget_seq_lengthrzzerosrr?rrr-listboolr functionalsoftmaxrtype_asrXr contiguousr)r0rBrrrrr batch_size seq_lengthis_cross_attention query_statesrcurr_past_key_valuecurrent_states key_states value_statesscoresreal_seq_lengthr position_bias causal_maskmaskposition_bias_masked attn_weights attn_outputs r4rDzUMT5Attention.forwardsH"/!4RaR!8 J3$>vvm,, #((RtG^__iijkmnoo   &:oGZ+[+[ &(377GGJ! K&5&K##&5&J##"1 2DW..-  F/"=*"=,3DNCHJ.5dnELLL//J66.11L#RtG^__iijkmnooJ',,ZT\4KbccmmnoqrssL*7I!St+>+E+E dn?OQ_>`,,( L&F*_FY*Z*ZFAEO.t~>lJ,@,@A,F,FGGL[Kf*'E'E'G'GGGlv%b) / A!KDL*j9&-W]WcMM!--FMR`.M*!!!QQQ aaa*?@M  %(AAAqqq2HJ4DR4H2H)HIK)K7M   1:m1!455D,-Dd'(( )#0DIIKK#@ #0 &&},,V\\^^,DDLLVTT },,\T\TXTa,bb   &'/9Ll<>> !++Aq11<<>> !&&z:rBB ff[)) L((r5)FN)NNNNNNN)rErFrG__doc__rintr*r,r_rrrr r rDrHrIs@r4rvrvsX""XVY]"""""":%, - - - ^    _%0A6RRR9=+/152615])])|]) ( 5])"% ]) !. ]) "%,/ ])!.])])])SR])])])])])r5rvcjeZdZd deeffd Zeddd d dZxZS) UMT5LayerSelfAttentionNrwctt|d||_t |j|j|_tj |j |_ dS)NTrzrwrn) r)r*rv SelfAttentionr&rRrqrrrrVrWrXr0rLrwr3s r4r*zUMT5LayerSelfAttention.__init__gsb *6t_hiii'F >Gr5r]) NNNNNNFFNrrIs@r4rrs//(3-//////_%0A6RRR"##';;;SR;;;;;r5rcLeZdZdZdeffd ZdejdejfdZxZ S)UMT5ClassificationHeadz-Head for sentence-level classification tasks.rLcttj|j|j|_tj|j|_tj|j|j |_ dS)N)r) r)r*rrQrRdenserVclassifier_dropoutrX num_labelsout_projr[s r4r*zUMT5ClassificationHead.__init__sc Yv~v~>> zF$=>>>  &.&2CDD r5rBrc||}||}tj|}||}||}|Sr])rXrr,tanhrras r4rDzUMT5ClassificationHead.forwards[ ]33  =11  =11  ]33  m44 r5) rErFrGrr"r*r,r_rDrHrIs@r4rrsw77EzEEEEEE U\elr5rcTeZdZUeed<dZdZdZdgZdgZ e dZ dZ dZ d S) UMT5PreTrainedModelrL transformerTrrUcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r,tensorrr)r0r input_mask dummy_inputss r4r#z UMT5PreTrainedModel.dummy_inputs s=L.. \*-- !*"&0  r5c |jj}t|tr$|jj|dzd St|ttttfr|j jj d|dzt|dr5|jjs)|jjj d|dzt|dr[|jjj d||jjdzz|jjjd Sd St|t(r`t|drN|jjj d|dz|jjjd Sd St|t,r|jjj d||jjdzzt|jdr/|jj#|jjj|jjj d||jjdzzt|jdr1|jj'|jjjd Sd Sd St|t2r|jjj d||jjdzzt|jdr/|jj#|jjj|jjj d||jjdzzt|jdr1|jj'|jjjd Sd Sd St|t:rt|jjj d||jjdzzt|jdr/|jj#|jjj|jjj d||jjdzzt|jdr/|jj#|jjj|jjj d||jjdzzt|jdr1|jj'|jjjd Sd Sd St|t@r|jj}|jj!}|jj"}|j#jj d|||zdzz|j$jj d||dzz|j%jj d||dzz|j&jj d|||zdzz|j'r0|j(jj d||dzzd Sd Sd S) zInitialize the weights?)r=stdlm_head qa_outputs classifierrPN))rLinitializer_factorr^r&r.datafill_ UMT5ModelUMT5ForConditionalGenerationUMT5EncoderModelUMT5ForQuestionAnsweringsharednormal_hasattrtie_word_embeddingsr(r)rRrPzero_UMT5ForTokenClassificationr+rrrrKrTrUrSrdrfrgrvr}rrrrrrzr)r0modulefactorrRr~rs r4 _init_weightsz!UMT5PreTrainedModel._init_weightssY/ fm , ,= o M  $ $Vc\ 2 2 2 2 2  , (    ; o M % - -3FSL - I I Ivy)) O$+2Q O%*22#2NNNv|,, 4!(-553Ft{ObgkNkDl5mmm!&+1133333 4 4 : ; ;* ov|,, 4!(-553FSL5QQQ!&+1133333 4 4 6 7 7& o L  $ , ,#6dkFY^bEb;c , d d dv|V,, /1B1N !&,,... O " ' / /SfI\aeHe>f / g g gv// 2FO4H4T$)//11111 2 24T4T  1 2 2 o I  ! ) )s4;CV[_B_8` ) a a avy&)) ,fin.H #))+++ I  ! ) )s4;CSX\B\8] ) ^ ^ ^vy&)) ,fin.H #))+++++ , ,.H.H  6 7 7 o K  # + +&T[EX]aDa:b + c c cv{F++ . 0@0L  %++--- K  # + +&T[EX]aDa:b + c c cv{F++ . 0@0L  %++--- I  ! ) )s4;CSX\B\8] ) ^ ^ ^vy&)) ,fin.H #))+++++ , ,.H.H  . . ok)G!%!1 k+G HO ( (cv'L^B^cgAg7h ( i i i HO ( (cv$7O ( P P P HO ( (cv$7O ( P P P HO ( (cv'L^B^cgAg7h ( i i i1 o.5:BBQW\chl[lQmBnnnnn o o o or5c|jj}|jj}|tdt |rHt j|jdddz|}t j||dddfgd}nD| |j}|dddf |dddf<||d<|td| |d k||S) Nzself.model.config.decoder_start_token_id has to be defined. In UMT5 it is usually set to the pad_token_id. See UMT5 docs for more information.r8)r!.rr!).rz1self.model.config.pad_token_id has to be defined.) rLdecoder_start_token_id pad_token_id ValueErrorrr,fullrcat new_zerosclone masked_fill_)r0rr>r?shifted_input_idss r4 _shift_rightz UMT5PreTrainedModel._shift_rightVs!%!C{/ ! )6  Y ' ' ? % 9?3B3+?$+FH^ _ _  % +Q*RXZ [ [ [   ) 3 3IO D D )238)<)B)B)D)D c122g &(> f %  PQQ Q&&'8D'@,OOO  r5N)rErFrGr"__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyr#r;rGr5r4rrs %&*#!$ !F X@o@o@oD!!!!!r5rc eZdZdfd ZdZ ddZ ddeejdfdejd ejd e d e f d Z e dejd e de dejd ejde f dZxZS) UMT5StackNct||_j|_t jfdt jD|_tj j |_ t j j|_d|_|dS)Nc2g|]}t|S)r)r).0irLs r4 z&UMT5Stack.__init__..ws&#e#e#eqIf$B$B$B#e#e#er5rnF)r)r* embed_tokensryrrrange num_layersblockr&rRrqfinal_layer_normrVrWrXgradient_checkpointing post_init)r0rLrWr3s ` r4r*zUMT5Stack.__init__ss    ( +]#e#e#e#eERXRcLdLd#e#e#eff -fn&B[ \ \ \z&"566 ',# r5c||_dSr])rWr0new_embeddingss r4set_input_embeddingszUMT5Stack.set_input_embeddingss*r5c | | n |jj} | | n |jj} | | n |jj} | | n |jj} |#|!|jrdnd}t d|d|d|1|}|d|d}n@||dd}n!|jrdnd}t d|d|d |j r%|j r| rt d d } |+|j t d | |}|\}}| d ur|jst d|d|jr]| rZ|X|jjr7tt!|jt!|j}nt!|j}n |jsd}||nd}| t%j|||z|j} |/t+s!||z}t%j|||j}|jr6|||| t1|tr|jn|| }nT|P|ddddddf}||j}d|z t%j|jjz}nd}|jrQ|O|\}}}||f}|t%j||j}||}nd}|||jj }|||jj }| rdnd}| rdnd}| r |jrdnd}|!|}tE|j#D][\}}||} ||}!| r||fz}|||||| |!|| | |  }"|"d}| r||"dfz }|jr ||"dfz }\|$|}|!|}| r||fz}| stKd|||||fDStM|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer8zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz$UMT5Stack.forward..s4  =!===  r5)last_hidden_staterrB attentionscross_attentions)'rLr r output_hidden_statesuse_return_dictryr@rrr\rrrrWis_encoder_decoderr r rr,rrrr-_update_causal_maskr^rr:r?rrinvert_attention_mask get_head_maskrYrX enumeraterZr[tupler)#r0rrrrre head_maskcross_attn_head_maskrr r rl return_dictrerr_msg_prefix input_shaperrpast_key_values_lengthmask_seq_lengthrencoder_batch_sizeencoder_sequence_length_encoder_hidden_shapeencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrBrU layer_modulerr  layer_outputss# r4rDzUMT5Stack.forwards "+!6IIDK+/?BZZNw>wwwww  "#..**K!r;r?;;II  &',,..ss3KK+/?BZZNu>uuXfuuuvv v  & "4= " "##p"   ( !_``` --i88M!, J   ? j !hT!h!h!hiii ? # G_4;1G&9$DK888,dk:Z:Z:Z''OO'3$+&F&F&FO ##OETE`!?!?!A!A!Afg  !"\&(>(KTaThN  !*B*D*D !4zAO"Z OML`aaaN ? 22o/BCC%44$!KK '(D$)9:K%..}/B.CCK, M|jjd vr0|s.t j|j} tj| | } | S) Nflash_attention_2r&flex_attentionrFsdpa)rery is_trainingr!r8)sequence_length target_lengthr?rr)cudaxpunpu)rL_attn_implementationrr^r,r_r$ris_compileabler_ignore_causal_mask_sdparr?rget_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positionrtyperr_unmask_unattended) r0rrrrr past_seen_tokensusing_compilable_cacher?rrr min_dtypes r4rozUMT5Stack._update_causal_mask.s ; +/B B B)~/D.I.I.K.K)%%4 ; +/? ? ?.%,77 M!<^!L!L! ! @O?Z?99;;;`aCRC^!?!?di ; +v 5 5>T 5]n 5%>*'7 M   t"&,Q/ ! +??AAMMnel;;<$R((%7!; PP +')#)!, Q   K , 6 6*%*.DDD%E  E**.I0CKQZ[[Kr5rrr?rc ||dkr|}nMtj|j}tj||f|||j}|dkrtj|d}|tj||j|ddkz}|ddddddf |ddd}|| }|j d} |ddddddd| f|ddddddf |jz} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer?rr!)diagonalrfr8r)rr,rrrArtriurreshapeexpandrDrr: masked_fill) rrrr?rrkwargsrr mask_length padding_masks r4rz?UMT5Stack._prepare_4d_causal_attention_mask_with_cache_positionrs>  %.*<*<*>*>!*C*C(KK E**.I* -0Ye\j\qK!###jqAAA 5< n>STTTWeWmWmnprsWtWtt tK%dD!!!QQQ&67>>z1bRTUUK))//11 ,226 *111aaaL[L+@ANSTSTSTVZ\`bcbcbcScDdDgDg&EE   ,q0 5@AAAqqq,;,AV5W5c5c )66 AAAqqq!!!\k\12r5r]) NNNNNNNNNNNNN)F)rErFrGr*rarDrr,r_r rro staticmethodrr?rrHrIs@r4rQrQrsU      +++ "#!!i i i i d#( BBelK78BlB B  B  BBBBH4 444{ 4  4  444\44444r5rQc&BeZdZUdZdZeed<ddgZfdZdZ dZ d Z d Z d Z e dd eejdeejdeejdeejdeejdeejdeejdeeeejdeedeejdeejdeedeedeedeedeejdeeejeff"dZxZS) r/ao Examples: ```python >>> from transformers import UMT5Model, AutoTokenizer >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> noisy_text = "UN Offizier sagt, dass weiter werden muss in Syrien." >>> label = " verhandelt" >>> inputs = tokenizer(inputs, return_tensors="pt") >>> labels = tokenizer(label=label, return_tensors="pt") >>> outputs = model(input_ids=inputs["input_ids"], decoder_input_ids=labels["input_ids"]) >>> hidden_states = outputs.last_hidden_state ```umt5rLencoder.embed_tokens.weightdecoder.embed_tokens.weightct|tj|j|j|_tj|}d|_ d|_ d|_ t||j|_ tj|}d|_ d|_ |j|_t||j|_|dSNFT)r)r*rr vocab_sizerRr3copydeepcopyryr tie_encoder_decoderrQencodernum_decoder_layersrYdecoderr]r0rLencoder_configdecoder_configr3s r4r*zUMT5Model.__init__s    l6#4fnEE v..$)!#( -2* == v..$(!-2*$*$=! ==  r5c|jSr]r3r0s r4get_input_embeddingszUMT5Model.get_input_embeddings {r5c|||_|j||j|dSr]r3rrarr_s r4razUMT5Model.set_input_embeddings;$  )).999 )).99999r5c|jjrL||jj|j||jj|jdSdSr]rLr6_tie_or_clone_weightsrrWr3rrs r4 _tie_weightszUMT5Model._tie_weights\ ; * O  & &t|'@$+ N N N  & &t|'@$+ N N N N N O Or5c|jSr]rrs r4 get_encoderzUMT5Model.get_encoder |r5c|D]/\}}|jj|j|0dS) Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrr attention prune_headsr0heads_to_prunerheadss r4 _prune_headszUMT5Model._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C Cr5Nrrrr rtdecoder_head_maskruencoder_outputsrredecoder_inputs_embedsr r rlrvrrc*| | n |jj} ||n |jj}||||| || ||}ne|rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}|||| | ||||| | ||| }|s||zSt|j |j |j |j |j |j |j |j S) a+ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. Example: ```python >>> from transformers import AutoTokenizer, UMT5Model >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5Model.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for UMT5Model. >>> # This is not needed for torch's UMT5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nrrrertr rlrvrr!r7rirBrj rrrerrrrtrur r rlrvr)rirdecoder_hidden_statesdecoder_attentionsrkencoder_last_hidden_staterencoder_attentions)rLr rmrr^rlenrrrirrBrjrk)r0rrrr rtrrurrrerr r rlrvrrBdecoder_outputss r4rDzUMT5Model.forwardsb"+!6IIDKD $D>D 'tnD d^!D "!!12#D $ uU&');; <%D D D ^D D D D D r5r/z< UMT5 Model with a `language modeling` head on top. ) custom_introc(beZdZdZdZgdZfdZdZdZdZ dZ e dd e e jd e e jd e e jd e e jde e jde e jde e jde eee jde ede e jde e jde e jde ede ede ede ede e jdeee jeff$dZde jfdZxZS)r0a Examples: ```python >>> from transformers import UMT5ForConditionalGeneration, AutoTokenizer >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> summary = "Weiter Verhandlung in Syrien." >>> inputs = tokenizer(article, text_target=summary, return_tensors="pt") >>> outputs = model(**inputs) >>> loss = outputs.loss ```r)rrzlm_head.weightc4t||j|_t j|j|j|_tj |}d|_ d|_ d|_ t||j|_tj |}d|_ d|_ |j|_t||j|_t j|j|jd|_|dS)NFTrO)r)r*rR model_dimrrrr3rrryr rrQrrrYrrQr(r]rs r4r*z%UMT5ForConditionalGeneration.__init__s    l6#4fnEE v..$)!#( -2* == v..$(!-2*$*$=! == y1BOOO  r5c|jSr]rrs r4rz1UMT5ForConditionalGeneration.get_input_embeddingsrr5c|||_|j||j|dSr]rr_s r4raz1UMT5ForConditionalGeneration.set_input_embeddingsrr5c|jjrL||jj|j||jj|jdSdSr]rrs r4rz)UMT5ForConditionalGeneration._tie_weightsrr5c|jSr]rrs r4rz(UMT5ForConditionalGeneration.get_encoderrr5Nrrrr rtrrurrrerlabelsr r rlrvrrc| | n |jj} ||n |jj}||||| ||||}ne|rct |t sNt |dt |dkr|dndt |dkr|dnd}|d}| || || }|||| | ||||| |||| }|d}|jj r ||j dzz}| |}d}| ptd }| |j} ||d |d | d }|s|f|ddz|z}||f|zn|St#|||j|j|j|j|j|j|j S) aK input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, UMT5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5ForConditionalGeneration.from_pretrained("google/umt5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer("Studies have shown that good for you", return_tensors="pt").input_ids >>> outputs = model.generate(input_ids) >>> tokenizer.decode(outputs[0], skip_special_tokens=True) ```Nrrr!r7rrr*r= ignore_indexr8 losslogitsrrrrkrrr)rLr rmrr^rrrGrr6rr(rr:rrrrrrBrjrkri)r0rrrr rtrrurrrerrr r rlrvrrBrsequence_output lm_logitsrloss_fctoutputs r4rDz$UMT5ForConditionalGeneration.forwardsoj"+!6IIDK_ $D>_ 'tn!_ "d^#_ $!!12%_ & uU&'8 9'_ _ _ ^_ D)EL))))))))r5r0c4eZdZdZdZdgZfdZdZdZdZ dZ d Z e dd e ejd e ejd e ejde ejde ede ede edeeejeffdZxZS)r1a Examples: ```python >>> from transformers import UMT5EncoderModel, AutoTokenizer >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> article = "UN Offizier sagt, dass weiter verhandelt werden muss in Syrien." >>> input_ids = tokenizer(article, return_tensors="pt").input_ids >>> outputs = model(input_ids) >>> hidden_state = outputs.last_hidden_state ```rrc2t|tj|j|j|_tj|}d|_ d|_ t||j|_ | dSNF)r)r*rrrrRr3rrr rnrQrr])r0rLrr3s r4r*zUMT5EncoderModel.__init__ys}    l6#4fnEE v..#( ,1) ==  r5c|jSr]rrs r4rz%UMT5EncoderModel.get_input_embeddingsrr5cH||_|j|dSr])r3rrar_s r4raz%UMT5EncoderModel.set_input_embeddingss%$  )).99999r5cl|jjr'||jj|jdSdSr])rLr6rrrWr3rs r4rzUMT5EncoderModel._tie_weightss? ; * O  & &t|'@$+ N N N N N O Or5c|jSr]rrs r4rzUMT5EncoderModel.get_encoderrr5c|D]:\}}|jj|jdj|;dS)rrN)rrrZrrrrs r4rzUMT5EncoderModel._prune_headss] +0022 P PLE5 L u % +A . < H H O O O O P Pr5Nrrrtrer rlrvrc ^||n |jj}||||||||}|S)aQ input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). Example: ```python >>> from transformers import AutoTokenizer, UMT5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google/umt5-small") >>> model = UMT5EncoderModel.from_pretrained("google/umt5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```Nr)rLrmr) r0rrrtrer rlrvrs r4rDzUMT5EncoderModel.forwardsMF&1% -'tn-d^- uU&'8 9---^-----r5r1z UMT5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c$eZdZdgZddgZdeffd Ze ddee j dee j d ee j d ee j d ee j d ee j d ee j dee e j dee j dee j dee j deedeedeedeedeeeff dZxZS)UMT5ForSequenceClassificationFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weightrrrLct|t||_t ||_|d|_dSr)r)r*r/rrclassification_headr]model_parallelr[s r4r*z&UMT5ForSequenceClassification.__init__s[    $V,,#9&#A#A  #r5Nrrrr rtrrurrerrr r rlrvrcp||n |jj}| d} || td|jj|(| &|t d||}|||||||||| | | | ||}|d}||jj  |j }ttj|ddkrt d|j\}}}||ddf|d |ddd ddf}||}d}| | |j } |jjp|jjdkr d |j_nS|jjdkr7| jtjks| jtjkr d |j_n d |j_|jjd kr\t1}|jjdkr1||| }n||| }n|jjd krLt5}||d |jj| d }n*|jjd krt7}||| }|s|f|ddz}||f|zn|St9|||j|j|j|j |j!|j"|j# S)as input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrr rtrrurrerr r rlrvrr!z7All examples must have the same number of tokens.r8 regressionsingle_label_classificationmulti_label_classificationr)$rLrmNotImplementedErrorr3rEr@rGreq eos_token_idr:rrr,unique_consecutivesumrrr problem_typerr?rrrsqueezerrrrrrrkrrr)r0rrrr rtrrurrerrr r rlrvrreos_maskrr}r1sentence_representationrrrrs r4rDz%UMT5ForSequenceClassification.forwards|&1%Kbdd   $)>)F  U !% 1 1) < < "" )/#9/!5+'"7/!5##   "!*<< 899<<_=STT u' Q88 9 9A = =VWW W%4%:" A{"1(AAA+">"C"CJPRT_"`"`abababdfhihihiai"j))*ABB  YYv}--F{'/;)Q../;DK,,[+a//V\UZ5O5OSYS_chclSlSl/LDK,,/KDK,{'<77"99;)Q..#8FNN$4$4fnn6F6FGGDD#8FF33DD)-JJJ+--x B 0F G GUWYY)-III,..x// FY,F)-)9TGf$$v E.#3")"?&9$5&-&G")"?&9    r5)NNNNNNNNNNNNNNN)rErFrG"_keys_to_ignore_on_load_unexpectedrr"r*rrr,rr_rrrrrsrrDrHrIs@r4rrs+s)s&79VW$z$$$$$$15158<=A,0487;=A59=A-1$(,0/3&*!P P E,-P !.P $E$45 P !))9 : P EL) P $EL1P 'u|4P "$u'8"9:P   12P  ((9:P )*P D>P $D>P 'tnP d^!P " u55 6#P P P ^P P P P P r5rc6eZdZdgZdgZdeffd Ze ddee j dee j dee j d ee j d ee j d ee d ee d ee de e e j effdZxZS)r8rz'transformer.encoder.embed_tokens.weightrLc6t||j|_t||_t j|j|_t j |j |j|_ | dSr]) r)r*rr1rrrVrrXrQr1r+r]r[s r4r*z#UMT5ForTokenClassification.__init__sz     ++F33z&";<< )F$68IJJ r5Nrrrtrerr rlrvrc ||n |jj}||||||||} | d} || } || } d} |Ft } | | d|j|d} |s| | ddf}| | f|zn|St| | | j | j S)aB input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. N)rrtrer rlrvrr8r7)rrrBrj) rLrmrrXr+rrrrrBrj)r0rrrtrerr rlrvrrBrrrrs r4rDz"UMT5ForTokenClassification.forwards6&1%7 'tn7 d^7  uU\"$99 :7 7 7 ^7 7 7 7 7 r5r8c&2eZdZddgZfdZdZdZdZdZe dd e e j d e e j d e e j d e e jd e e j de e j de e jde eee jde e j de e j de e j de e j de ede ede ede edeee j eff"dZxZS)r2rrcHt||j|_t j|j|j|_tj |}d|_ d|_ d|_ t||j|_tj |}d|_ d|_ |j|_t||j|_|j|_t j|j|j|_|dSr)r)r*rRrrrrr3rrryr rrQrrrYrrrQr)r]rs r4r*z!UMT5ForQuestionAnswering.__init__s    l6#4fnEE v..$)!#( -2* == v..$(!-2*$*$=! ==  +)FNF4EFF r5c|jSr]rrs r4rz-UMT5ForQuestionAnswering.get_input_embeddingsrr5c|||_|j||j|dSr]rr_s r4raz-UMT5ForQuestionAnswering.set_input_embeddingsrr5c|jjrL||jj|j||jj|jdSdSr]rrs r4rz%UMT5ForQuestionAnswering._tie_weightsrr5c|jSr]rrs r4rz$UMT5ForQuestionAnswering.get_encoderrr5Nrrrr rtrrurstart_positions end_positionsrerr r rlrvrc4||n |jj}| | n |jj} | | d} |(| &|td||}| | n |jj} ||n |jj}||||| ||||}ne|rct |tsNt|dt|dkr|dndt|dkr|dnd}|d}| ||| d||||| ||| }|d}| |}| dd \}}| d  }| d  }d}| | t| dkr-| d |j} t| dkr-| d |j} |d}| d|} | d|} t%| }||| }||| }||zdz }|s||f|ddz|z}||f|zn|St'||||j|j|j|j|j|j|j S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. UMT5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [UMT5 Training](./umt5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) UMT5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [UMT5 Training](./umt5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. decoder_head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. NFrrrr!r7r) rrrerrrrtrur r rlrvr8rr) r start_logits end_logitsrrrrkrrr)rLrmr r@rGrr^rrrr)splitrrrr:rrrrrrBrjrkri)r0rrrr rtrrurrr rerr r rlrvrBrrrr"r# total_loss ignored_indexr start_lossend_lossrs r4rDz UMT5ForQuestionAnswering.forwardsx&1%)F  U !% 1 1) < < !*!6IIDKZ $D>Z 'tn!Z "d^#Z $ uU&')LL M%Z Z Z ^Z Z Z Z Z r5r2)r1r0r2rr8r/r)Hrrrtypingrrr,rtorch.nnrrr activationsr cache_utilsr r r generationrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrrrutils.deprecationr configuration_umt5r"!torch.nn.attention.flex_attentionr#integrations.flex_attentionr$ get_loggerrErModuler&rKrdrlrvrrrrrrQr/r0r1rr8r2__all__rOr5r4r:s """""""" AAAAAAAAAA!!!!!!CCCCCCCCCC))))))>>>>>>999999.-----100000****** !!K;;;;;;JJJJJJ  H % %+++++BI+++4 .RY<")$D)D)D)D)D)BID)D)D)NRY:bi>GGGGG*GGGVRY$o!o!o!o!o!/o!o!o!dvvvvv#vvvr K K K K K #K K K \ c)c)c)c)c)#6c)c)  c)Liiiii*iiiX ` ` ` ` ` $7` `  ` FI I I I I !4I I I XJ J J J J 2J J J Z   r5