`i~dZddlmZmZddlZddlZddlmZddlm Z m Z m Z ddl m Z mZddlmZdd lmZmZmZdd lmZdd lmZmZmZdd lmZmZd dlmZej e!Z"dZ#dZ$d#dZ%Gddej&Z'dZ(Gddej&Z)eGddeZ*eGdde*Z+edGdde*eZ,edGd d!e*Z-gd"Z.dS)$zPyTorch CTRL model.)OptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)Cache DynamicCache)GenerationMixin)BaseModelOutputWithPastCausalLMOutputWithPastSequenceClassifierOutput)PreTrainedModel)Conv1D find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging) CTRLConfigcNdtjdd|dzz|z z }||zS)Nri')torchpow)posi d_model_size angle_ratess z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/ctrl/modeling_ctrl.py angle_defnr!(s0eiQ!V 'DEEEK  cttj|tj|dtj|tj|d|}tj|dddddf}tj|dddddf}tj||gd}|S)Ndtyperrrdim) r!rarangeint64to unsqueezesincoscat)positionrr% angle_radssinescosines pos_encodings r positional_encodingr5-s XU[11144U;;EEaHH \55588??II!LLJ IjADqD) * *Ei 111add7+,,G9eW-2666L r"c tj||dddd}|jd}|t j|z }|A|d|d} } ||| | z | d| fdzz }|||z}tj|d} || |z} tj| |} | | fS) Nrrr rr&gr')rmatmulpermuteshapenpsqrtsizesoftmax) qkvmaskattention_mask head_mask matmul_qkdkscaled_attention_logitsndnsattention_weightsoutputs r scaled_dot_product_attentionrL<s Q !Q1 5 566I B'"'"++5 (--b113J3O3OPR3S3SB4R" crc(9#:T#AA!"9N"J &=2FFF- 9 \+Q / /F $ $$r"c@eZdZdfd ZdZdZ ddZxZS) MultiHeadAttentionNct||_||_||_t ||jz |_tj|||_ tj|||_ tj|||_ tj|||_ t|_dSN)super__init__ num_headsr layer_idxintdepthrLinearWqWkWvdenseset pruned_heads)selfrrSrT __class__s r rRzMultiHeadAttention.__init__Ws "("677 )L,77)L,77)L,77Y|\:: EEr"c|j|jz}t|dkrdSt||j||j\}}t |j||_t |j||_t |j||_t |j |d|_ |jt|z |_||jz|_|j ||_dS)Nrrr') rrSlenrr]rrXrYrZr[union)r^headsattention_head_sizeindexs r prune_headszMultiHeadAttention.prune_headsfs"/4>A u::?? F7t~Obdhduvv u%TWe44$TWe44$TWe44' EqAAA #e**4/$.@ -33E::r"ct||d|j|j}|gdS)Nr&rrrr )reshaperSrVr9)r^x batch_sizes r split_into_headsz#MultiHeadAttention.split_into_headsws3 IIj"dndj A Ayy&&&r"Fc B|jd} ||}||}||}||| }||| }||| }|"||||jd| i\}}t||||||} | dgd} | d}| | d|j }| |} | |fS)Nrcache_positionrhrr&) r:rXrYrZrlupdaterTrLr9rirr[)r^rAr@r?rB layer_pastrCrD use_cacheoutput_attentionsrnrkrKscaled_attentionattnoriginal_size_attentions r forwardzMultiHeadAttention.forward{sWQZ GGAJJ GGAJJ GGAJJ  ! !!Z 0 0  ! !!Z 0 0  ! !!Z 0 0  !$$Q4>eps) rQrRrNmulti_head_attentionrffnr LayerNorm layernorm1 layernorm2Dropoutdropout1dropout2)r^rrSrraterTr_s r rRzEncoderLayer.__init__s $6|YZc$d$d$d!2<EE,|>>>,|>>> 4((  4(( r"Fc V||} || | | ||||||| } | d} || } || z} || } || }||}| |z} | f| ddz}|S)NrprCrDrqrrrnrr)rrrrrr)r^rjrBrprCrDrqrrrnnormed attn_outputs attn_outputout1out2 ffn_outputoutputss r rvzEncoderLayer.forwards##00    !)/)1   #1o mmK00 ;t$$XXd^^ ]]:.. j 'L,,r")rNrw)rxryrzrRrvr{r|s@r rrse ) ) ) ) ) ) """"""""r"rc$eZdZUeed<dZdZdS)CTRLPreTrainedModelconfig transformerct|tjtfrT|jjd|jj|j |j j dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|j j |jjddSdS)zInitialize the weights.g)meanstdN?) isinstancerrWrweightdatanormal_rinitializer_rangebiaszero_ Embedding padding_idxrfill_)r^modules r _init_weightsz!CTRLPreTrainedModel._init_weightss. fry&1 2 2 * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r"N)rxryrzr__annotations__base_model_prefixrr"r rrs7 %*****r"rceZdZfdZdZdZdZe ddee j dee dee j d ee j d ee j d ee j d ee j d ee dee dee dee dee jdeee jeffdZxZS) CTRLModelc4tj|_j|_t j|jtj |_ tj j j|_tjj|_tjfdt'jD|_tjjj|_|dS)Nc `g|]*}tjjjj|+S)r)rn_embdn_headr resid_pdrop).0rrs r z&CTRLModel.__init__..sF   V]FM6:vGYefggg   r"r)rQrRrrn_layer num_layersr5 n_positionsrfloatr4rr vocab_sizewr embd_pdropdropout ModuleListrangehrlayer_norm_epsilon layernorm post_initr^rr_s `r rRzCTRLModel.__init__s    "M ./0BDDUW\Wbccf/??z&"344     v~..      fm9RSSS r"c|jSrPr)r^s r get_input_embeddingszCTRLModel.get_input_embeddingss v r"c||_dSrPr)r^new_embeddingss r set_input_embeddingszCTRLModel.set_input_embeddingss r"c|D]*\}}|j|j|+dS)zv Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} N)itemsrrrf)r^heads_to_prunelayerrcs r _prune_headszCTRLModel._prune_headssQ+0022 B BLE5 F5M . : :5 A A A A B Br"N input_idspast_key_valuesrCtoken_type_ids position_idsrD inputs_embedsrqrroutput_hidden_states return_dictrnreturnc P| | n |jj} ||n |jj}| | n |jj} | | n |jj} ||t d|T||||}|d|d}|j d}n;|*|dd}|j d}nt d||j n|j }|r|t|j}|rCt|tr.tdtj|}||nd}|@t%j||d|zt$j|}|d}||dkrt d ||d}|d d }||j }d |z t%j|jjz}|||jj}|N|d|d}||}|t;j|jz}nd}|||}|d}t%j t%j!||z||zd |}|t;j|jz}|j"||_"|j"|ddf}||z|z}|#|}| rdnd}| rdnd}tI|j%D]<\}}| r||fz}||||||||| | }|d}| r ||d fz }=|&|}| r||fz}| std||||fDStO||||S)aE input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) Example: ```python >>> from transformers import AutoTokenizer, CTRLModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 5, 1280] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer&rz5You have to specify either input_ids or inputs_embeds)rzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.)r%devicez$batch_size has to be defined and > 0rrr$rrrc3K|]}||V dSrPr)rrAs r z$CTRLModel.forward..s1bcbobobobobor")last_hidden_stater hidden_states attentions)(rrrrqruse_return_dict ValueError%warn_if_padding_and_no_attention_maskr=viewr:rr rtuplelogger warning_oncefrom_legacy_cacheget_seq_lengthrr)longr,r+r%finfomin get_head_maskrrr;r<rtriuonesr4r enumeraterrr )r^rrrCrrrDrrqrrrrrnkwargs input_shaperkr past_lengthtoken_type_embedsseq_lenrB pos_embedsrall_hidden_statesall_attentionsrrrs r rvzCTRLModel.forwardsb2C1N--TXT_Tq!*!6IIDKcdd d  "  6 6y. Q Q Q#..**K!r;r?;;I"+JJ  &',,..ss3K&,Q/JJTUU U%.%:!! @T  ?0*$+>>>O  NOU;; N   U    +<_MMO:I:Uo44666[\   < [_{5RZ_ZdmstttL'11!44L  %Q !GHHH+00R@@N ,55a88BB1EEN,..TZ.@@N!N2ek$*6M6M6QQN&&y$+2EFF  %+00[_EEN $~ 6 6  ):!;!; ;   !    FF9--Mb/z%*W{%:Gkd $D>d 'tnd d^d !.d  uU\"$;; <d d d ^d d d d d r"rz The CTRL Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introc eZdZdgZfdZe ddeejdee deej deejdeejd eej d eej d eejd ee d ee dee dee deej de eej effdZddZxZS)CTRLLMHeadModelzlm_head.weightct|t||_t j|j|jd|_| dS)NTr) rQrRrrrrWrrlm_headrrs r rRzCTRLLMHeadModel.__init__s`    $V,,y0AMMM  r"NrrrCrrrDrlabelsrqrrrrrnrc V| | n |jj} ||||||||| | | | |  }|d}||}d}||j||fd|jji|}| s|f|ddz}||f|zn|St |||j|j|j S)a  input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLLMHeadModel >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLLMHeadModel.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Wikipedia The llama is", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> sequence_ids = model.generate(inputs["input_ids"]) >>> sequences = tokenizer.batch_decode(sequence_ids) >>> sequences ['Wikipedia The llama is a member of the family Bovidae. It is native to the Andes of Peru,'] >>> outputs = model(**inputs, labels=inputs["input_ids"]) >>> round(outputs.loss.item(), 2) 9.21 >>> list(outputs.logits.shape) [1, 5, 246534] ```N) rrCrrrDrrqrrrrrnrrr)losslogitsrrr) rrrr loss_functionrrrrr)r^rrrCrrrDrrrqrrrrrnrtransformer_outputsr lm_logitsr rKs r rvzCTRLLMHeadModel.forwardsx&1%c $D>c 'tnc d^c !.c uU\"$:: ;!c c c ^c Jr"ra The CTRL Model transformer with a sequence classification head on top (linear layer). [`CTRLForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ceZdZfdZe ddeejdeedeej deejdeejdeej d eej d eejd ee d ee d ee dee de e ej effdZxZS)CTRLForSequenceClassificationct||j|_t||_t j|j|jd|_| dS)NFr) rQrR num_labelsrrrrWr classifierrrs r rRz&CTRLForSequenceClassification.__init__Vsi     +$V,,)FM4?OOO r"NrrrCrrrDrrrqrrrrrc  | | n |jj} ||||||||| | | |  } | d}||}||jdd\}}n|jdd\}}|jj|dkrt d|jjd}n|}||jjk|jtj }tj |jd|jtj }||z d}n)d}t|jjd |tj ||j |f}d}|Z|jjf|jdkr d |j_nN|jdkr7|jtjks|jtjkr d |j_n d |j_|jjd krWt-}|jdkr1|||}n|||}n|jjd krGt1}||d|j|d}n*|jjd krt5}|||}| s|f| ddz}||f|zn|St7||| j| jS)a2 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values[0].shape[-2]` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only input IDs that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example of single-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl") >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> import torch >>> torch.manual_seed(42) # doctest: +IGNORE_RESULT >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> labels = torch.tensor(1) >>> loss = model(**inputs, labels=labels).loss >>> round(loss.item(), 2) 0.93 ``` Example of multi-label classification: ```python >>> import torch >>> from transformers import AutoTokenizer, CTRLForSequenceClassification >>> tokenizer = AutoTokenizer.from_pretrained("Salesforce/ctrl") >>> model = CTRLForSequenceClassification.from_pretrained( ... "Salesforce/ctrl", problem_type="multi_label_classification" ... ) >>> # CTRL was trained with control codes as the first token >>> inputs = tokenizer("Opinion My dog is cute", return_tensors="pt") >>> assert inputs["input_ids"][0, 0].item() in tokenizer.control_codes.values() >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> predicted_class_id = logits.argmax().item() >>> model.config.id2label[predicted_class_id] 'LABEL_0' ``` ```python >>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)` >>> num_labels = len(model.config.id2label) >>> model = CTRLForSequenceClassification.from_pretrained("Salesforce/ctrl", num_labels=num_labels) >>> num_labels = len(model.config.id2label) >>> labels = torch.nn.functional.one_hot(torch.tensor([predicted_class_id]), num_classes=num_labels).to( ... torch.float ... ) >>> loss = model(**inputs, labels=labels).loss >>> loss.backward() # doctest: +IGNORE_RESULT ```N) rrCrrrDrrqrrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r&)rr%z will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`)r regressionsingle_label_classificationmulti_label_classification)r r rr)rrrrr: pad_token_idrr+rrint32r)argmaxrrr_rx problem_typerr%rrUrsqueezerrrrrr)r^rrrCrrrDrrrqrrrrr rr rksequence_lengthlast_non_pad_token non_pad_mask token_indices pooled_logitsr loss_fctrKs r rvz%CTRLForSequenceClassification.forward_sLP&1%!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x 2 22t G GUWYY)-III,..x v66 F#%(;ABB(??F)-)9TGf$$v E' -;*5     r"r)rxryrzrRrrrrr rrrrrrrvr{r|s@r rrJsw15+/6:59371559-1$(,0/3&*p p E,-p "%p !!23 p !!12 p u/0 p E-.p   12p )*p D>p $D>p 'tnp d^p  uU\"$<< =p p p ^p p p p p r"r)rrrrr)/__doc__typingrrnumpyr;rrtorch.nnrrr cache_utilsr r generationr modeling_outputsr rrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_ctrlr get_loggerrxrr!r5rLModulerNrrrrrr__all__rr"r r:s """""""" AAAAAAAAAA........))))))iiiiiiiiii------YYYYYYYYYY+*****  H % %   %%%%4DDDDDDDDN```/////29///d*****/****I I I I I #I I I X KKKKK)?KK  K\   { { { { { $7{ {   { | c b br"