`iP>ddlZddlmZddlZddlmZddlmZmZddlm Z m Z ddl m Z ddl mZdd lmZd d lmZd d lmZmZmZmZmZmZmZmZd d lmZddlmZej e!Z"dZ#dZ$GddeZ%dZ&Gddej'Z(Gdde(Z)Gdde(Z*e(e)e*dZ+GddeZ,GddeZ-Gd d!eZ.Gd"d#eZ/Gd$d%eZ0Gd&d'eZ1Gd(d)eZ2gd*Z3dS)+N)Optional)nn)Cache StaticCache)_flash_attention_forward!flash_attn_supports_top_left_mask)PreTrainedModel)logging)deprecate_kwarg)GemmaForCausalLM)LlamaDecoderLayerLlamaForQuestionAnsweringLlamaForSequenceClassificationLlamaForTokenClassification LlamaModelLlamaPreTrainedModelapply_rotary_pos_emb repeat_kv) MistralMLP)DiffLlamaConfigzkajuma/DiffLlama-0.3B-handcutrceZdZdS) DiffLlamaMLPN__name__ __module__ __qualname__/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/diffllama/modular_diffllama.pyrr2Dr!rc<ddtjd|zzz S)Ng?g333333?g333333ӿ)mathexp) layer_idxs r"lambda_init_fnr(6s! txy 0111 11r!cfeZdZdZddedeeffd Zeddd  dd e j d e e j e j fd ee j dee j dee dedee j de e j ee j ee e j ffdZxZS)DiffLlamaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperNconfigr'ct||_||_|(td|jjd|j|_|j |_ |j |_ t|d|j |j z|_ |j|_|j |jz|_|j|_|j|_d|_t'j|j |j |j z|j|_t'j|j |j|j z|j|_t'j|j |j|j z|j|_t'j|j |j z|j |j|_t5||_t'jt;jd|j|j f|_ t'jt;jd|j|j f|_!t'jt;jd|j|j f|_"t'jt;jd|j|j f|_#t'j$d|j z|j%d |_&dS) NzInstantiating z without passing a `layer_idx` is not recommended and will lead to errors during the forward call if caching is used. Please make sure to provide a `layer_idx` when creating this class.head_dimT)biasr)sizer F)epselementwise_affine)'super__init__r+r'logger warning_once __class__rattention_dropout hidden_sizenum_attention_heads num_headsgetattrr-num_key_value_headsnum_key_value_groupsmax_position_embeddings rope_theta is_causalrLinearattention_biasq_projk_projv_projo_projr( lambda_init Parametertorchnormallambda_std_dev lambda_q1 lambda_k1 lambda_q2 lambda_k2RMSNorm rms_norm_eps groupnormselfr+r'r6s r"r3zDiffLlamaAttention.__init__=sa  "     ,!8,,,    "(!9!-3 D4D4VWW #)#= $(Nd6N$N!'-'E$ +i 0$.4=2PW]Wlmmm i 0$2JT]2Zagavwww i 0$2JT]2Zagavwww i >@PW]Wlmmm ))44el1f6KSWS`Rb&c&c&cddel1f6KSWS`Rb&c&c&cddel1f6KSWS`Rb&c&c&cddel1f6KSWS`Rb&c&c&cddA $56;Nchiiir!past_key_valuepast_key_values4.58new_nameversionF hidden_statesposition_embeddingsattention_mask position_ids use_cachecache_positionreturnc |\} } } | } ||} ||}||}| | | |j|jdd} || | |j|jdd}|| | |j|jdd}|\}}t| |||\} }|&|||d}| |||j |\}}t||j }t||j }tjtj|ddd}|dddd}tj| |ddt'j|jz }|$|ddddddd|jdf}||z}t,j|dtj| j}t,j||j|j }tjtj |j!|j"zdtj| j}tjtj |j#|j$zdtj| j}||z |j%z}tj||}tj|dd\}}|||zz }d|j%z |&|z}|dd'}|(| | d}|)|}||fS) Nrr sincosr`dimrrgdtype)ptraining)*r/rCrDrEviewr:r- transposer<rupdater'rr=rIcatchunkrepeatmatmulr%sqrtshaper functionalsoftmaxfloat32torkdropoutr7rmr&sumrLrMrNrOrGrR contiguousreshaperF)rTr[r\r]r^rVr_r`kwargsbsz target_len_q_len query_states key_states value_statesrerd cache_kwargs attn_weights causal_masklambda_1lambda_2 lambda_full attn_output attn_output1 attn_output2s r"forwardzDiffLlamaAttention.forward_s+//11Z{{=11 [[// {{=11 #((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm &S#7 jRUWZ#[#[ j  &#&snUUL'6'='=j,X\Xfht'u'u $J z4+DEE  t/HII y\1!!D!D!D"MMM #**1aA66 |L*2F2Fq!2L2LMMPTPYZ^ZgPhPhh  %(AAAqqq2HJ4DR4H2H)HIK'+5L},,\r,WWZZ[g[mnn },,\T=S^b^k,ll 9UYt~'FBV[Vcdddeehh    9UYt~'FBV[Vcdddeehh    )D,<< l<>> %*[aQ%G%G%G" l"[<%?? 4++t~~k/J/JJ !++Aq11<<>> !))#ub99 kk+.. L((r!NNNNFN)rrr__doc__rrintr3r rITensortuple LongTensorrboolr __classcell__r6s@r"r*r*:sIGG j j j8C= j j j j j jD_%0A6RRR 2637+/59<)<)|<)#5<#=><)!. <) u/0 <) "% <)<)!!12<) u|Xel3XeEL>Q5RR S<)<)<)SR<)<)<)<)<)r!r*ceZdZdZfdZeddd dd ejd eejejfd e ej d e ej de e d e de ej deejdffdZ xZS)DiffLlamaFlashAttention2aN DiffLlama flash attention module. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only required change would be on the forward pass where it needs to correctly call the public API of flash attention and deal with padding tokens in case the input contains any of them. c`tj|i|t|_dSr)r2r3r _flash_attn_uses_top_left_mask)rTargsrr6s r"r3z!DiffLlamaFlashAttention2.__init__s6$)&))) /P.Q.Q+++r!rUrVrWrXNFr[r\r]r^r_r`rac t|trtd|\}} } ||} ||} ||} | || |j|j  dd} | || |j |j  dd} | || |j |j  dd} |4t d|| |\}}n|\}}t| | ||\} } |&|||d}|| | |j|\} } | dd} | dd} | dd} |jr|jnd}| j}| jjdkr | jjnd}|t.jkrt/jr=t5t.d rt/j|nt/j}n3t5|jd r |jj}n|jjj}t d |d | |} | |} | |} t/j!| dd \}}|"dddd}|"dddd}tG| | ||| ||tI|dd|j%|j& }tG| | ||| ||tI|dd|j%|j& }t/j'||gd }t/j!|dd \}}t/j(t/j)|j*|j+zdt.j | j}t/j(t/j)|j,|j-zdt.j | j}||z |j.z}|||zz }d|j.z |/|z}|0|| d1}|2|}|dfS)Nz`static` cache implementation is not compatible with `attn_implementation==flash_attention_2` make sure to use `sdpa` in the mean time, and open an issue at https://github.com/huggingface/transformersrr aYThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.46 `position_ids` will be removed and `position_embeddings` will be mandatory.rcmpscpuget_autocast_dtype_pre_quantization_dtypezThe input hidden states seems to be silently casted in float32, this might be related to the fact you have upcasted embedding or layer norm layers in float32. We will cast back the input in .rfsliding_window)r^r{ruse_top_left_maskr@rhrj)3 isinstancer ValueErrorr/rCrDrErnr:r-ror<r4r5 rotary_embrrpr'rmr7rkdevicetyperIryis_autocast_enabledhasattrrget_autocast_gpu_dtyper+rweightrzrrrsrr;rr@rqr&r|rLrMrNrOrGrRr~r}rF)rTr[r\r]r^rVr_r`rrrrrrrerdr dropout_rate input_dtype device_type target_dtype value_states1 value_states2rrrrrrs r"rz DiffLlamaFlashAttention2.forwards" o{ 3 3 }  &**,, UA{{=11 [[// {{=11 $((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm  &   G    |\BBHC*HC#7 jRUWZ#[#[ j  &#&snUUL'6'='=j,X\Xfht'u'u $J $--a33 ))!Q// #--a33 15Gt--C #( 2>2E2Je2S2Sl)..Y^ %- ' '(** 8u&:;;8E,[999577 &?@@ 8#{B #{17   $ $$$    (??<88L#|44J'??<88L',{<'J'J'J$ }%,,Q1a88 %,,Q1a88 /     % "4)94@@"An    0     % "4)94@@"An    i| <"EEE %*[aQ%G%G%G" l9UYt~'FBV[Vcdddeehh    9UYt~'FBV[Vcdddeehh    )D,<< "[<%?? 4++t~~k/J/JJ !))#ub99DDFF kk+.. D  r!r)rrrrr3r rIrrrrrrrrrs@r"rrs) RRRRR_%0A6RRR 6:37+/59B!B!|B!#5<#=>B!!!12 B! u/0 B! "% B!B!!!12B! u|T! "B!B!B!SRB!B!B!B!B!r!rc>eZdZdZeddd ddejd eejejfd eejd eej dee d e d eej deejeejeeejffdZ dS)DiffLlamaSdpaAttentiona DiffLlama attention module using torch.nn.functional.scaled_dot_product_attention. This module inherits from `DiffLlamaAttention` as the weights of the module stays untouched. The only changes are on the forward pass to adapt to SDPA API. rUrVrWrXNFr[r\r]r^r_r`rac f|\} } } ||} ||} ||}| | | |j|jdd} | | | |j|jdd} || | |j|jdd}|\}}t| | ||\} } |&|||d}| | ||j |\} }t| |j } t||j }tjtj|ddd}|dddd}|}||ddddddd| jdf}| jjdkr>|<| } | } |}|duo| dk}tjj| | |||jr|jnd| }tj|dd\}}tjtj|j|jzdtj  | j!}tjtj|j"|j#zdtj  | j!}||z |j$z}|||zz }d|j$z |%|z}|dd}|| | d}|&|}|dfS) Nrr rcrfrhricudar) attn_mask dropout_pr@rj)'r/rCrDrErnr:r-ror<rrpr'rr=rIrqrrrsrvrrr}rrwscaled_dot_product_attentionrmr7r&r|rLrMryrzrkrNrOrGrRrF)rTr[r\r]r^rVr_r`rrrrrrrrerdrrr@rrrrrrs r"rzDiffLlamaSdpaAttention.forward<s&**,, UA{{=11 [[// {{=11 #((eT^T]SS]]^_abcc __S%1I4=YYccdeghii #((eT5Mt}]]gghiklmm &S#7 jRUWZ#[#[ j  &#&snUUL'6'='=j,X\Xfht'u'u $J z4+DEE  t/HII y\1!!D!D!D"MMM #**1aA66 $  %%aaaAAA/E1A"1E/E&EFK   #v - -+2I'2244L#..00J'2244L 4'5EAI h)FF   !04 Fd,,3 G  &+[aQ%G%G%G" l9UYt~'FBV[Vcdddeehh    9UYt~'FBV[Vcdddeehh    )D,<< "[<%?? 4++t~~k/J/JJ !++Aq11<<>> !&&sE266 kk+.. D  r!r) rrrrr rIrrrrrrrr r!r"rr4s_%0A6RRR 2637+/59I!I!|I!#5<#=>I!!. I! u/0 I! "% I!I!!!12I! u|Xel3XeEL>Q5RR SI!I!I!SRI!I!I!r!r)eagerflash_attention_2sdpac(eZdZdedeffd ZxZS)DiffLlamaDecoderLayerr+r'ct||t|j|||_dS)N)r+r')r2r3DIFFLLAMA_ATTENTION_CLASSES_attn_implementation self_attnrSs r"r3zDiffLlamaDecoderLayer.__init__s? +++4V5PQY_ktuuur!)rrrrrr3rrs@r"rrsWvv3vvvvvvvvvvr!rceZdZdZdZdZdS)DiffLlamaPreTrainedModelFctj||t|tr|jjd|jj|j jd|jj|j jd|jj|j jd|jjdSdS)Nr) r _init_weightsrr*rLdatanormal_r+rKrMrNrO)rTmodules r"rz&DiffLlamaPreTrainedModel._init_weightss%dF333 f0 1 1 I   ! ) )!T[-G H H H   ! ) )!T[-G H H H   ! ) )!T[-G H H H   ! ) )!T[-G H H H H H  I Ir!N)rrr_supports_flex_attn_supports_attention_backendrr r!r"rrs4"'IIIIIr!rceZdZdS)DiffLlamaModelNrr r!r"rrr#r!rceZdZdS)DiffLlamaForCausalLMNrr r!r"rrr#r!rceZdZdS)"DiffLlamaForSequenceClassificationNrr r!r"rrr#r!rceZdZdS)DiffLlamaForQuestionAnsweringNrr r!r"rrr#r!rceZdZdS)DiffLlamaForTokenClassificationNrr r!r"rrr#r!r)rrrrrr)4r%typingrrIr cache_utilsrrmodeling_flash_attention_utilsrr modeling_utilsr utilsr utils.deprecationr gemma.modeling_gemmarllama.modeling_llamarrrrrrrrmistral.modeling_mistralrconfiguration_diffllamar get_loggerrr4_CHECKPOINT_FOR_DOC_CONFIG_FOR_DOCrr(Moduler*rrrrrrrrrr__all__r r!r"rs$  --------iiiiiiii------000000333333                    211111444444  H % %5#     :   222b)b)b)b)b)b)b)b)JR!R!R!R!R!1R!R!R!jR!R!R!R!R!/R!R!R!l 1 "vvvvv-vvv I I I I I3 I I I     Z        +        )G        $=        &A      r!