`ibdZddlZddlmZmZddlZddlmZddlmZm Z m Z m Z ddlm Z ddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZdd lmZddlm Z m!Z!ddl"m#Z#ddl$m%Z%e!j&e'Z(d*dZ)Gddej*Z+Gddej*Z,GddeZ-e GddeZ.e Gdde.Z/e dGd d!e.eZ0e d"Gd#d$e.Z1e Gd%d&e.Z2e Gd'd(e.Z3gd)Z4dS)+zPyTorch MPT model.N)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLoss LayerNormMSELoss) functional)Cache DynamicCache)GenerationMixin)!_prepare_4d_causal_attention_mask)GradientCheckpointingLayer))BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsQuestionAnsweringModelOutput SequenceClassifierOutputWithPastTokenClassifierOutput)PreTrainedModel)auto_docstringlogging)deprecate_kwarg) MptConfigcjtjd|z dtj|ddd|}dt jt j|z}tjd|dztj|}|||z z}dtj d|z }|d|dd}||krAtj |ddddddf|ddddddfgdddd|df}||z}| dS) a Link to paper: https://huggingface.co/papers/2108.12409 - Alibi tensor is not causal as the original paper mentions, it relies on a translation invariance of softmax for quick implementation. This implementation has been copied from the alibi implementation of MPT source code that led to slightly different results than the Bloom alibi: https://huggingface.co/mosaicml/mpt-7b/blob/main/attention.py#L292 r)dtypedevice?N.dimr) torcharangeint32viewmathceillog2int64floatpowconcatsqueeze) num_headssequence_lengthalibi_bias_maxralibinum_heads_power_of_2baseslopess x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/mpt/modeling_mpt.pybuild_mpt_alibi_tensorr8-sH L_,au{6 R R R W WXY[\^_ap q qE $)I*>*> ? ?? </!35;v V V V \ \ ^ ^D >$88 9D 59Q%% %F [[0!Q 7 7Fy((vaaaAsl3VAAAsssCK5HIqQQQRSRSRSU_V_U_adRde FNE ==  c eZdZdZddedeeffd Zeddd  dd e j d e j dee d ee j d ee j f dZ xZ S) MptAttentionzzMulti-head self attention. Using torch or triton attention implementation enables user to also use additive bias. Nconfig layer_idxc*t|j|_|j|_|j|_|j|jz|_|jj|_|j)dtj |j|jz z |_|jj |_ |jj |_ tj|jd|jzd|_tj|j|jd|_||_dS)Nrr Fbias)super__init__ hidden_sizen_heads max_seq_lenmax_seq_lengthhead_dim attn_config softmax_scaler(sqrt attn_pdropattn_dropout_pclip_qkvrLinearWqkvout_projr=)selfr<r= __class__s r7rBzMptAttention.__init__Is !-~ $0(DL8 #/=   %!"TYt/?$,/N%O%O!OD $0;*3 Id.D4D0D5QQQ  $"2D4D5QQQ "r9past_key_valuepast_key_values4.58new_nameversion hidden_states position_biasattention_maskcache_positionc|jdd\}}||}|jr"||j |j}|dd\} } } | |||j|jdd} | |||j|jdd} | |||j|jdd} |$d|i} | | | |j | \} } tj | | dd|j z} ||n||z}|t|jdkr$t!d t|j| jd}t#d |d|z }t#d |d|z }|dd|d|df}| |z} |2| |tj| jj} t.j| d| j}t.j||j|j }tj || }|d ddd !||d}|"|}||fS) Nr )minmaxr r"rr\z6Expecting position_bias shape to be 3 dimensions, got rptraining)#shaperOrMclampchunkreshaperDrG transposeupdater=r$matmulrIget_seq_lengthlen ValueErrorr_size masked_fillfinforr^rr softmaxr,todropoutrLrdpermute contiguousr'rP)rQrYrZrTr[r\ batch_size seq_length mixed_qkv query_states key_states value_states cache_kwargsattention_scores query_length key_lengthposition_bias_query_indexposition_bias_key_index attn_weightscontext_states attn_outputs r7forwardzMptAttention.forwardYs-"/!4RaR!8 JIIm,, = O!T]N NNI1:1J1J. j,#++J DLRVR_``jjklnopp '' J dm\\ffghjkll #++J DLRVR_``jjklnopp  &,n=L'6'='=j,X\Xfht'u'u $J  < j6J6J2r6R6RSSVZVhh%4%>".9"$*T->"?"?r9FrYrZr[ layer_past use_cacheoutput_attentionsr\c||}|} ||||||\} } || | z}||}|} ||| } | | fS)N)rZr[rTr\)rrrrr) rQrYrZr[rrrr\layernorm_outputr attn_outputsrrs r7rzMptBlock.forwards ;;}55 &*YY ')&) &/& & " l// ==H ;;}55!*H55|##r9r)NFFN) rrrrrrrBr$rr boolrrrs@r7rrs@@y@Xc]@@@@@@2'+"'15"$"$|"$|"$ "$ UO "$  "$ "$!."$"$"$"$"$"$"$"$r9rc eZdZUeed<dZdZdgZdgZfdZ de j fdZ e ed d d d eeejejfd eeejejffdZxZS)MptPreTrainedModelr< transformerTrz lm_head.*.c:tj|i|dSr)rArB)rQinputskwargsrRs r7rBzMptPreTrainedModel.__init__s%&+F+++++r9modulect|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|trF|j|jj |jj ddSdS)zInitialize the weights.g)meanstdNr!) isinstancerrNweightdatanormal_r<initializer_ranger@zero_ Embedding padding_idxrfill_)rQrs r7 _init_weightsz MptPreTrainedModel._init_weightss/ fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  * * *{&  &&((( M  $ $S ) ) ) ) ) * *r9rSrTrUrVrc||ddj\}}||ztfd|DS)zw Converts the cache to the format expected by Mpt, i.e. to tuple(tuple([batch_size * num_heads, ...])) rc3K|]>}|d|dfV?dS)rrN)rh).0rbatch_size_times_num_headsrGrxs r7 z;MptPreTrainedModel._convert_to_mpt_cache..so  1 %%&@(JWW1 %%&@*hWW       r9)retuple)rTrwr0rrGrxs @@@r7_convert_to_mpt_cachez(MptPreTrainedModel._convert_to_mpt_cachest7Fa6H6K6Q3 Ix%/)%;"      .       r9)rrrr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_keys_to_ignore_on_load_missingrBrModuler staticmethodrrr$rrrrs@r7rrs %&*## '4o#,,,,,*BI****"_%0A6RRR uU\5<%?@A uU\5</0 1   SR\     r9rcZeZdZdeffd ZdZddZdejfdZ e dd e ej d e e d e ejd e ej d e ede ede ede ede ejdeeejdfeffdZxZS)MptModelr<ctj|_j|_t jj|j|_t j fdtj D|_ t|jj|_d|j_d|_|dS)Nc2g|]}t|S))r=)r)rir<s r7 z%MptModel.__init__..s&$c$c$cqXf%B%B%B$c$c$cr9rF)rArBrCrDr0rr vocab_sizewte ModuleListrangen_layersblocksrrnorm_fr@gradient_checkpointing post_initrQr<rRs `r7rBzMptModel.__init__s    !-< 143CDDm$c$c$c$cERXRaLbLb$c$c$cdd   0f6OPPP  &+# r9c|jSrr)rQs r7get_input_embeddingszMptModel.get_input_embeddings&s xr9rNc&t||||Sr)r8)rQr0r1r2rs r7r8zMptModel.build_mpt_alibi_tensor)s%i.RXYYYr9new_embeddingsc||_dSrrrQrs r7set_input_embeddingszMptModel.set_input_embeddings,s !r9 input_idsrTr[ inputs_embedsrroutput_hidden_states return_dictr\r.c ||n |jj}||n |jj}||n |jj}||n |jj}||t d| |j\} } n| |j\} } } nt d|jr%|jr|rt dd}|| |}|r|t|j}|rCt|tr.t dtj|}|}|rdnd}|rdnd}||nd }| |z}|t#j| |f|j }n||j}||j|jj|j }t1|| | f||}|}|jD]2}|r||fz}|||||||| }|d }|r ||d fz}3||}|r||fz}|std ||||fDSt9||||S) input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) NzDYou cannot specify both input_ids and inputs_embeds at the same timez5You have to specify either input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)r<zPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `DynamicCache` instead, e.g. `past_key_values=DynamicCache.from_legacy_cache(past_key_values)`.rr)rr[rrrZr\rc3K|]}||V dSrr)rvs r7rz#MptModel.forward..s1ghgtgtgtgtgtr9)last_hidden_staterTrY attentions)r<rrruse_return_dictrnrerrdlogger warning_oncerr rrfrom_legacy_cacherlr$onesrrsr8r0rErrrrr)rQrrTr[rrrrrr\rrwrx_rYall_self_attentionsall_hidden_statespast_key_values_lengthseq_length_with_pastr3 causal_maskblockoutputss r7rzMptModel.forward/s]62C1N--TXT_Tq$8$D $+Jj "+!6IIDKcdd d  "%._ "J  &(5(; %J AATUU U  & "4= " "##p"   HHY//M  ?0*$+>>>O  NOU;; N   U    +<_MMO% $5?bb4"6@BBDFUE`!?!?!A!A!Afg),BB  !"Z5I(JS`SghhhNN+..}/CDDN++DNDK t $D>t 'tnt d^t !.t  uU\3&')RR St t t ^t t t t t r9rz The MPT Model transformer with a language modeling head on top (linear layer with weights tied to the input embeddings). ) custom_introcjeZdZdgZdeffd ZdejfdZe dde ej de e d e ejd e ejd e ejd e e d e e de e de e de ejdeeejeffdZxZS)MptForCausalLMzlm_head.weightr<ct|t||_t j|j|jd|_| dSNFr?) rArBrrrrNrCrlm_headrrs r7rBzMptForCausalLM.__init__sa    #F++y!3V5FUSSS  r9rc||_dSr)rrs r7set_output_embeddingsz$MptForCausalLM.set_output_embeddingss % r9NrrTr[rlabelsrrrrr\rc | | n |jj} ||||||||| |  } | d} || }d}|5||j}|j||fd|jji| }| s|f| ddz}||f|zn|St||| j | j | j S)a\ input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for language modeling. Note that the labels **are shifted** inside the model, i.e. you can set `labels = input_ids` Indices are selected in `[-100, 0, ..., config.vocab_size]` All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` N)rTr[rrrrrr\rrrlosslogitsrTrYr) r<rrrrsr loss_functionrrrTrYr)rQrrTr[rrrrrrr\rtransformer_outputsrY lm_logitsrrs r7rzMptForCausalLM.forwards)@&1%F $D>F 'tnF d^F !.F  uU\"$EE FF F F ^F F F F F r9r a The MPT Model transformer with a sequence classification head on top (linear layer). [`MptForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-1) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). c2eZdZdeffd Ze ddeejdee deej deej deej d ee d ee d ee d ee d e e ej effdZxZS)MptForSequenceClassificationr<ct||j|_t||_t j|j|jd|_| dSr) rArB num_labelsrrrrNrCscorerrs r7rBz%MptForSequenceClassification.__init__sk     +#F++Yv163D5QQQ  r9NrrTr[rrrrrrrc | | n |jj} ||||||||| } | d} || } ||jd} n |jd} |jj| dkrt d|jjd}n|}||jjk| jtj }tj |jd| jtj }||z d}n)d}t|jjd| tj | | j |f}d}|.|jjf|jdkr d |j_nN|jdkr7|jtjks|jtjkr d |j_n d |j_|jjd krWt-}|jdkr1|||}nb|||}nU|jjd krt1}|||}n*|jjd krt3}|||}| s|f| ddz}||f|zn|St5||| j| j| j S)6 input_ids (`torch.LongTensor` of shape `(batch_size, input_ids_length)`): `input_ids_length` = `sequence_length` if `past_key_values` is `None` else `past_key_values.get_seq_length()` (`sequence_length` of input past key value states). Indices of input sequence tokens in the vocabulary. If `past_key_values` is used, only `input_ids` that do not have their past calculated should be passed as `input_ids`. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrTr[rrrrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r`)rrz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r regressionsingle_label_classificationmulti_label_classificationr)r<rrr re pad_token_idrnrsrr$r&r%argmaxrrrRr problem_typerrlongrr r/rrrrTrYr)rQrrTr[rrrrrrrrYrrwlast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r7rz$MptForSequenceClassification.forwards<&1%!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   {'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x v66)-III,..x v66 F#%(;ABB(??F)-)9TGf$$v E/ /?-;*5     r9r )rrrrrBrrr$r r rrrrrrrrs@r7rrs>y15+/1504)-$(,0/3&*d d E,-d "%d !. d   - d & d D>d $D>d 'tnd d^d  uU\"$DD Ed d d ^d d d d d r9rc2eZdZdeffd Ze ddeejdee deej deej deej d ee d ee d ee d ee d e e ej effdZxZS)MptForTokenClassificationr<ct||j|_t||_t |dr|j|j}n!t |dr|j|j}nd}tj ||_ tj |j |j|_ |dS)Nclassifier_dropoutrg?)rArBrrrhasattrr3rrrrtrNrC classifierr)rQr<r3rRs r7rBz"MptForTokenClassification.__init__s     +#F++ 6/ 0 0 %V5N5Z!'!:   V- . . %63H3T!'!6  !$ z"455 )F$68IJJ r9NrrTr[rrrrrrrc | | n |jj} ||||||||| } | d} || } || } d}|p|| j}|j\}}t}|| ||z|j | ||z}| s| f| ddz}||f|zn|St|| | j | j S)r"Nr#rr )rrrYr)r<rrrtr5rsrrerr'rrrYr)rQrrTr[rrrrrrdeprecated_argumentsrrYrrrwrxr/rs r7rz!MptForTokenClassification.forwardsL>&1%y"15+/1504)-$(,0/3&*B B E,-B "%B !. B   - B & B D>B $D>B 'tnB d^B  uU\"$99 :B B B ^B B B B B r9r1ceZdZfdZe d deejdeejdeejdeejdeejdee d ee d ee d e e e ffd Z xZS)MptForQuestionAnsweringct|t||_t j|jd|_|dS)Nr ) rArBrrrrNrC qa_outputsrrs r7rBz MptForQuestionAnswering.__init__sY    #F++)F$6:: r9Nrr[rstart_positions end_positionsrrrrc ||n |jj}|||||||} | d} || } | dd\} } | d} | d} d}||t|dkr|d}t|dkr|d}| d}| d|}| d|}t|}|| |}|| |}||zdz }|s| | f| ddz}||f|zn|St|| | | j | j S) rN)r[rrrrrrr`r") ignore_indexr )r start_logits end_logitsrYr)r<rrr;splitr/rvrmrorfrrrYr)rQrr[rr<r=rrrrsequence_outputrr@rA total_loss ignored_indexr/ start_lossend_lossrs r7rzMptForQuestionAnswering.forwards 2&1%E 'tnE d^E  u22 3E E E ^E E E E E r9r9)r rrrr1r9r)5rr(typingrrr$rtorch.nnrrrr r r cache_utilsr r generationrmodeling_attn_mask_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_utilsrutilsrrutils.deprecationrconfiguration_mptr get_loggerrrr8rr;rrrrr rr1r9__all__rr9r7rVs """""""" LLLLLLLLLLLL$$$$$$........))))))IIIIII999999.-----,,,,,,,,000000((((((  H % %.G)G)G)G)G)29G)G)G)TRY*7$7$7$7$7$)7$7$7$t- - - - - - - - `U U U U U !U U U p U U U U U 'U U  U p    o o o o o #5o o   o dU U U U U 2U U U pO O O O O 0O O O d   r9