`iyLdZddlZddlmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZmZmZdd lmZmZmZdd lmZdd lmZmZd dlmZmZmZmZmZd dl m!Z!d dl"m#Z#ddl$m%Z%erddl&m'Z'm(Z(GddeZ)eGddeZ*GddeZ+GddeZ,eGdde*Z-edGd d!e*e Z.Gd"d#eZ/Gd$d%e!Z0Gd&d'eZ1gd(Z2dS))zPyTorch PLBART model.N)OptionalUnion)nn)CrossEntropyLoss)Cache)GenerationMixin)AttentionMaskConverter_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)BaseModelOutputSeq2SeqLMOutputSeq2SeqModelOutput)PreTrainedModel)auto_docstringis_torch_flex_attn_available)BartClassificationHead BartDecoder BartEncoderBartForCausalLMBartScaledWordEmbedding)'BigBirdPegasusForSequenceClassification)shift_tokens_right) PLBartConfig) BlockMaskmake_flex_block_causal_maskceZdZdS)PLBartScaledWordEmbeddingN__name__ __module__ __qualname__}/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/plbart/modular_plbart.pyr r 6Dr&r c jeZdZUeed<dZdZddgZdZdZ dZ de e j dfde j fd Zdee e j d fd e j d e j d efdZede j dedede jd e j def dZde e j dfde e j dfde jde j fdZdS)PLBartPreTrainedModelconfigmodelTPLBartDecoderLayerPLBartEncoderLayerattention_maskN inputs_embedsc*||jjdkr d|vr|nd}nw|jjdkrt||j}nQ|jjdkr,t |t jrt|d}nt||j}|S)Nflash_attention_2rsdpaflex_attentionF) is_causal r+_attn_implementationr dtype isinstancetorchTensorrr )selfr/r0s r'_update_full_maskz'PLBartPreTrainedModel._update_full_maskEs  %{/3FFF343F3FD1V;;"E^UbUh!i!i15EEEnel;;b%@[`%a%a%aN"|<|jjd vr.tj|j} tj| | } | S) Nr4rr)sizedevicer2gFr3)r0past_key_values_length is_training)sequence_length target_lengthr8r? batch_size)cudaxpunpu)r+r7r9r:r;ronesshaperCanyget_seq_lengthis_compileabler _ignore_causal_mask_sdpatrainingr8get_max_cache_shape5_prepare_4d_causal_attention_mask_with_cache_positiontypefinfomin_unmask_unattended) r<r/r>r?r@past_seen_tokensusing_compilable_cacher8rGrH causal_mask min_dtypes r'_update_causal_maskz)PLBartPreTrainedModel._update_causal_mask\s0 ; +/? ? ?.%,77 !<^!L!L '!<J*03\5G5JK-4"" " ! ; +/B B B)~/D.I.I.K.K)%%4 @O?Z?99;;;`aCRC^!?!?di ; +v 5 5>T 5%>*'7 M   t"&,Q/ ! +??AAMMnel;;<$R((%7!; PP +')#)!, Q   K , 6 6*%*.DDD  E**.I0CKQZ[[Kr&rGrHr8rIc ||dkr|}nMtj|j}tj||f|||j}|dkrtj|d}|tj||j|ddkz}|ddddddf |ddd}|| }|j d} |ddddddd| f|ddddddf |jz} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. N) fill_valuer8rCr)diagonalrCrFr)dimr:rWrXfullrCtriuarangereshapeexpandclonerNto masked_fill) r/rGrHr8r?rIkwargsr\r] mask_length padding_masks r'rUzKPLBartPreTrainedModel._prepare_4d_causal_attention_mask_with_cache_positions>  %.*<*<*>*>!*C*C(KK E**.I* -0Ye\j\qK!###jqAAA 5< n>STTTWeWmWmnprsWtWtt tK%dD!!!QQQ&67>>z1bRTUUK))//11 ,226 *111aaaL[L+@ANSTSTSTVZ\`bcbcbcScDdDgDg&EE   ,q0 5@AAAqqq,;,AV5W5c5c )66 AAAqqq!!!\k\12r&encoder_hidden_statesencoder_attention_mask input_shapec\|||jjdkr d|vr|nd}n|jjdkrt||j|d}n`|jjdkr3t |t jrt||dd}nt||j|d}|S) Nr2rr3rF)tgt_lenr4F) query_lengthr5r6)r<rprqrrr0s r'_update_cross_attn_maskz-PLBartPreTrainedModel._update_cross_attn_masks ! ,1G1S{/3FFFCDH^C^C^)?)?dh&&1V;;*M*!''O***&& 15EEE4elCC-H.%0_"'...**D*M,?UW***&&%r&)r"r#r$r__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attnrr:r;r=rrr^ staticmethodintr8rUSizervr%r&r'r*r*:s &*#-/CDNelD01|.J u|['@!ABJlJ J  JJJJX4 444{ 4  4  444\4n!&$U\4%78!&!&elD&8 9!&Z !& | !&!&!&!&!&!&r&r*ceZdZdS) PLBartEncoderNr!r%r&r'rrr(r&rceZdZdS) PLBartDecoderNr!r%r&r'rr r(r&rc&"eZdZddgZdeffd ZdZdZdZdZ e dd e e j d e e j d e e j d e e jde e jde e j de e jde ee jde ede e jde e jde ede ede ede ede e j deee jeff"dZxZS) PLBartModelencoder.embed_tokens.weightdecoder.embed_tokens.weightr+ctt||j|j}}|jrt j|jnd}t||j|||_ t||j |_ t||j |_ |dS)Ng?) embed_scale)super__init__ pad_token_id vocab_sizescale_embeddingmathsqrtd_modelr sharedrencoderrdecoder init_weights)r<r+ padding_idxrr __class__s r'rzPLBartModel.__init__s    "("5v7HZ 393IRdi///s / FNKepqqq $VT[99 $VT[99  r&c|jSN)rr<s r'get_input_embeddingsz PLBartModel.get_input_embeddingss {r&cX||_|j|j_|j|j_dSr)rr embed_tokensr)r<values r'set_input_embeddingsz PLBartModel.set_input_embeddings s' $(K !$(K !!!r&c|jjrL||jj|j||jj|jdSdSr)r+tie_word_embeddings_tie_or_clone_weightsrrrrrs r' _tie_weightszPLBartModel._tie_weights%s\ ; * O  & &t|'@$+ N N N  & &t|'@$+ N N N N N O Or&c|jSr)rrs r' get_encoderzPLBartModel.get_encoder*s |r&N input_idsr/decoder_input_idsdecoder_attention_mask head_maskdecoder_head_maskcross_attn_head_maskencoder_outputsr@r0decoder_inputs_embeds use_cacheoutput_attentionsoutput_hidden_states return_dictr?returnc| | n |jj} ||n |jj}| | n |jj} ||n |jj}|| t ||jj}|||||| | ||}ne|rct|tsNt|dt|dkr|dndt|dkr|dnd}| |||d|||| | | | ||| }|s||zSt|j |j|j|j|j|j |j|jS) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. N)rr/rr0rrrrrr)last_hidden_state hidden_states attentions) rr/rprqrrr@r0rrrrr?)rr@decoder_hidden_statesdecoder_attentionscross_attentionsencoder_last_hidden_staterpencoder_attentions)r+rrruse_return_dictrrrr9r lenrrrr@rrr)r<rr/rrrrrrr@r0rrrrrr?decoder_outputss r'forwardzPLBartModel.forward-sb2C1N--TXT_Tq$8$D $+Jj "+!6IIDK)F 29dk>V W W   ""ll#-#+"3%9'+OO O_!M!M -"1!"4474H4H14L4Loa00RV14_1E1E1I1I?1--tO,,'1"1!"4#1'!5+//!5#)'    5"_4 4!-?+;"1"?.9,=&5&G"1"?.9    r&)NNNNNNNNNNNNNNNN)r"r#r$_tied_weights_keysrrrrrrrrr: LongTensorr;list FloatTensorrboolrtuplerr __classcell__rs@r'rr s(79VW |      000 OOO 15598<9=,08<7;=A+/59=A$(,0/3&*59#k k E,-k !!12k $E$45 k !) 6 k EL) k $E$45k 'u|4k "$u'8"9:k "%k   12k  ((9:k D>k $D>k 'tnk d^!k "!!12#k $ uU\"$66 7%k k k ^k k k k k r&rzv The PLBART Model with a language modeling head. Can be used for code-to-text, text-to-code and code-to-code. ) custom_introc(eZdZdZdgZgdZdeffd ZdZdZ d#d e d e e d e d e jffd Zd e d dfdZe d$de ejde ejde ejde ejde ejde ejde ejde eejde ede ejde ejde ejde e de e de e de e d e ejd eeejeff$d!Zdejfd"ZxZS)%PLBartForConditionalGenerationr,final_logits_bias)rrzlm_head.weightr+clt|t||_|dt jd|jjjftj |j |jjjd|_ | dS)NrrF)bias)rrrr,register_bufferr:zerosrnum_embeddingsrLinearrlm_headr)r<r+rs r'rz'PLBartForConditionalGeneration.__init__s     ((  0%+q$*BSBb>c2d2deeey1B1QX]^^^  r&c4|jSr)r,rrs r'rz*PLBartForConditionalGeneration.get_encoderz%%'''r&c4|jSr)r, get_decoderrs r'rz*PLBartForConditionalGeneration.get_decoderrr&NTnew_num_tokenspad_to_multiple_of mean_resizingrct|||}||jjd|S)Nr)rresize_token_embeddings_resize_final_logits_biasweightrN)r<rrrnew_embeddingsrs r'rz6PLBartForConditionalGeneration.resize_token_embeddingssG88I[]jkk &&~'<'B1'EFFFr&c|jjd}||kr|jddd|f}nBtjd||z f|jj}tj|j|gd}|d|dS)NrFrrc)rdr)rrNr:rrCcatr)r<rold_num_tokensnew_bias extra_biass r'rz8PLBartForConditionalGeneration._resize_final_logits_biass/5b9 ^ + +-aaa..@AHHa.)H%IRVRhRopppJy$"8*!E1MMMH 0(;;;;;r&rr/rrrrrrr@r0rlabelsrrrrr?cn||n |jj}| || t| |jj}|||||||||| | | | ||||}||d}||j|jz}d}| Kt}|| d|jj | d}|s|f|ddz}||f|zn|St|||j |j|j|j|j|j|j S)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example Mask-filling: ```python >>> from transformers import AutoTokenizer, PLBartForConditionalGeneration >>> model = PLBartForConditionalGeneration.from_pretrained("uclanlp/plbart-base") >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> # en_XX is the language symbol id for English >>> TXT = " Is 0 the Fibonacci number ? en_XX" >>> input_ids = tokenizer([TXT], add_special_tokens=False, return_tensors="pt").input_ids >>> logits = model(input_ids).logits >>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item() >>> probs = logits[0, masked_index].softmax(dim=0) >>> values, predictions = probs.topk(5) >>> tokenizer.decode(predictions).split() ['first', 'same', 'highest', 'result', 'number'] ``` N)r/rrrrrrr@r0rrrrrr?rrFr) losslogitsr@rrrrrpr)r+rrrr,rrrkrCrviewrrr@rrrrrpr)r<rr/rrrrrrr@r0rrrrrrr?outputs lm_logitsmasked_lm_lossloss_fctoutputs r'rz&PLBartForConditionalGeneration.forwardsV&1%x $D>x 'tn!x "d^#x $!!12%x & uU\"O3 4'x x x ^x tDELDDDDDDDDr&rceZdZdS)PLBartClassificationHeadNr!r%r&r'rrCr(r&rceZdZfdZxZS)PLBartForSequenceClassificationc :tjdi|dS)a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`] or [`PLBartMultiTokenizer`] depending on the checkpoint. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) PLBart uses a specific language id token as the starting token for `decoder_input_ids` generation that varies according to source and target language, *e.g.* 50003 for *en_XX*, and 50001 for *java*. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). For translation and summarization training, `decoder_input_ids` should be provided. If no `decoder_input_ids` is provided, the model will create this tensor by shifting the `input_ids` to the right for denoising pre-training following the paper. decoder_attention_mask (: obj:*torch.LongTensor* of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. cross_attn_head_mask (: obj:*torch.Tensor* of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules in the decoder. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nr%rr super_kwargsrs r'rz'PLBartForSequenceClassification.forwardHs(B '','''''r&)r"r#r$rrrs@r'rrGs8!(!(!(!(!(!(!(!(!(r&rc.eZdZefdZxZS)PLBartForCausalLMc :tjdi|dS)a cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, PLBartForCausalLM >>> tokenizer = AutoTokenizer.from_pretrained("uclanlp/plbart-base") >>> model = PLBartForCausalLM.from_pretrained("uclanlp/plbart-base", add_cross_attention=False) >>> assert model.config.is_decoder, f"{model.__class__} has to be configured as a decoder." >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> logits = outputs.logits >>> expected_shape = [1, inputs.input_ids.shape[-1], model.config.vocab_size] >>> list(logits.shape) == expected_shape True ```Nr%rrs r'rzPLBartForCausalLM.forwardms': '','''''r&)r"r#r$rrrrs@r'rrlsB((((^(((((r&r)rrrrr*)3__doc__rtypingrrr:rtorch.nnr cache_utilsr generationr modeling_attn_mask_utilsr r r modeling_outputsr rrmodeling_utilsrutilsrrbart.modeling_bartrrrrr(bigbird_pegasus.modeling_bigbird_pegasusrmbart.modeling_mbartrconfiguration_plbartrintegrations.flex_attentionrrr r*rrrrrrr__all__r%r&r'r s """""""" %%%%%% ))))))  .-----AAAAAAAA_^^^^^555555...... !!VUUUUUUUU     7   G&G&G&G&G&OG&G&G&T     K        K   K K K K K 'K K K \ _D_D_D_D_D%:O_D_D  _DD     5   "("("("("(&M"("("(J((((((((B   r&