0`iSddlZddlmZmZmZddlZddlmZmZddl m Z ddZ ddZ Gd d ej jZ dd eeejfd eejd eefdZeZGddej jZ dd eeejfdeefdZGddej jZejZ dd eeejffdZGddejjZdS)N)OptionalTupleUnion) rearrangerepeat) apply_rotaryFc|s3|dd\}}tj| |fdS|ddddf|ddddf}}ttj| |fdddS)Nrdim.z... d two -> ... (d two))two)chunktorchcatrstack)x interleavedx1x2s v/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/vllm_flash_attn/layers/rotary.py rotate_halfr s \##By2#r++++3!8aQTT lBrc2YB7779SYZ[[[[c 2|jddz}||jdksJt||sdnd}t||sdnd}tj|dd|f|zt |dd|f||zz|d|dfgdS)z x: (batch_size, seqlen, nheads, headdim) cos, sin: (seqlen, rotary_dim / 2) or (batch_size, seqlen, rotary_dim / 2) r rz... d -> ... 1 (2 d)z... d -> ... 1 (d 2).Nr )shaperrrr)rcossinrro_dims rapply_rotary_emb_torchr s Yr]Q F QWR[ K[,,E[ \ \C K[,,E[ \ \C 9 3<3 QsGVG|_k!J!JS!P PRSTWY_Y`Y`T`Rab    rc eZdZe d deeejfdeejdeefdZ edZ dS) ApplyRotaryEmbFrNseqlen_offsets cu_seqlens max_seqlenc  t||||||||} t|tr||||||_n|||||d|_||_||_||_|s| n|S)N)r#r$r%rinplace)r isinstanceintsave_for_backwardr#rr'r%) ctxrrrrr'r#r$r%outs rforwardzApplyRotaryEmb.forward&s  )!!#     nc * * &  ! !#sJ 7 7 7!/C    ! !#sJ G G G!%C % #!(ssq(rc |j}| |j\}}}}n |j\}}}|js|js|}t ||||||j|j|jd }|dddddddfS)NT)r#r$r%rr' conjugate)r# saved_tensorsrr'cloner r%)r+dor#rrr$dxs rbackwardzApplyRotaryEmb.backwardGs+  !363D 0Cj..# Cj s{ B    )!~K    4tT4t;;rFFrNN) __name__ __module__ __qualname__ staticmethodrr)rTensorrr-r4rrr"r"%s 34-1$())c5</0)U\*)SM)))\)@<<\<<`s-8    3['>:z  rcveZdZe ddeeejfdeefdZedZ dS) ApplyRotaryEmbQKV_NFrr# num_heads_qc v|||r|dkr=|j\} } } } } | dksJ|ddddddf| | d| }n^|dksJ|J|jd|z dz}|jd|d|zzksJ|ddddd||zf}t |||||dn||n|}||n|}|dkr|dddddf|ddddd f}}no|dksJ|J|jd|z dz}|jd|d|zzksJ|ddddd|f|dddd|||zf}}t |||||d t |||||d |||||t |tr |||||||_n ||||||d|_||_ ||_ |S) Nrr Tr#rr'rr)rr') is_contiguousr rreshaper r*r(r)r#rrA)r+qkvrrcos_ksin_krr#rAbatchseqlenthreenheadsheaddimqk num_heads_kqks rr-zApplyRotaryEmbQKV_.forwards =U]s/@/@/B/B]wwyyA~~8; 5vufgzzzzAAArr]**5&"gFFwwyyA~~~~"..."y|k9a? y|{Q_'DDDDDAAA9 k 999: C^^b     !=CCeE =CCeEwwyyA~~111aaa7|SAAAq\1wwyyA~~~~"..."y|k9a? y|{Q_'DDDDD111aaa+-.AAAqqq+ VaHa:a4a0b1 Cn+W[ \ \ \ \ E5.k[_ ` ` ` `  ! !#sE5 9 9 9 nc * * &  ! !#sE5 9 9 9!/C    ! !#sE5. I I I!%C %% rc &|j}||j\}}}}}n |j\}}}}|||r|dkr t |ddddddfd}nr|dksJ|jJ|jd|jz dz}|jd|jd|zzksJ|ddddd|j|zf}t|||||jddn||n|}||n|}|dkr|dddddf|dddddf} } n|dksJ|jJ|jd|jz dz}|jd|jd|zzksJ|ddddd|jf} |dddd|j|j|zf} t| ||||jdd t| ||||jdd |dddddddfS) NrCrzb s t h d -> b s (t h) drETr#rr'r/rr)rr'r/) r#r0rGr rrArr r) r+dqkvr#rrrJrKdqkrRdqdks rr4zApplyRotaryEmbQKV_.backwards+  !585F 2CeUNN%(%6 "CeU =U]t/A/A/C/C]xxzzQQQQ2A2X0JKKxxzzQ222#z!}s>1D z!}!k/(IIIII111aaa!@3?[#@!@@A -O     !=CCeE =CCeExxzzQaaaAgQQQ1W BxxzzQ222#z!}s>1D z!}!k/(IIIII!!!QQQ 1#/ 112!!!QQQ#/K2O OOP O     O    T4tT4==rNNFrN r6r7r8r9rr)rr:r-r4r;rrr@r@s 34"&33c5</033Z333\3j9>9>\9>9>9>rr@rAc Dt||||||||S)a Arguments: qkv: (batch_size, seqlen, 3, nheads, headdim) or (batch_size, seqlen, num_heads_q + 2 * num_heads_k, headdim). If qkv has shape (batch_size, seqlen, num_heads_q + 2 * num_heads_k, headdim) (e.g. MQA / GQA), then num_heads_q must be provided. cos, sin: (seqlen, rotary_dim / 2) cos_k, sin_k: (seqlen, rotary_dim / 2), optional interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). seqlen_offsets: (batch_size,) or int. Each sequence in Q and K is shifted by this amount. Most commonly used in inference when we have KV cache. Return: qkv: (batch_size, seqlen, 3, nheads, headdim) or (batch_size, seqlen, num_heads_q + 2 * num_heads_k, headdim) rotary_dim must be <= headdim Apply rotary embedding *inplace* to the first rotary_dim of Q and K. )r@r=)rIrrrJrKrr#rAs rapply_rotary_emb_qkv_r^s-4  # # S#ue[.+  rc\eZdZeddeeejffdZedZ dS)ApplyRotaryEmbKV_Frr#c(|j\}}}} } |dksJ|dddddf} t| ||||dt|tr|||||_n||||d|_||_|S)NrrTrF)rr r(r)r*r#r) r+kvrrrr#rLrMrrOrPrTs rr-zApplyRotaryEmbKV_.forwards.0h+vsFGaxxxx qqq!!!QwK sCKY]     nc * * &  ! !#s + + +!/C    ! !#sN ; ; ;!%C % rc |j}| |j\}}}n |j\}}t|dddddf||||jdd|ddddfS)NrTrV)r#r0r r)r+dkvr#rrs rr4zApplyRotaryEmbKV_.backward*s+  !'*'8 $Cnn(HC 111aL  )    D$d**rNFrr\r;rrr`r`sgeCQVQ]L]F^\ ++\+++rr`c>t|||||S)aR Arguments: kv: (batch_size, seqlen, 2, nheads, headdim) cos, sin: (seqlen, rotary_dim / 2) interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). seqlen_offsets: (batch_size,) or int. Each sequence in Q and K is shifted by this amount. Most commonly used in inference when we have KV cache. Return: kv: (batch_size, seqlen, 2, nheads, headdim) rotary_dim must be <= headdim Apply rotary embedding *inplace* to the first rotary_dim of K. )r`r=)rbrrrr#s rapply_rotary_emb_kv_rg@s (  " "2sCn M MMrceZdZdZ ddeffd ZddZdd Z dd ej d e ej d e eej fde ede ede ej e ej ej fff dZ xZS)RotaryEmbeddinga5 The rotary position embeddings from RoFormer_ (Su et. al). A crucial insight from the method is that the query and keys are transformed by rotation matrices which depend on the relative positions. Other implementations are available in the Rotary Transformer repo_ and in GPT-NeoX_, GPT-NeoX was an inspiration .. _RoFormer: https://arxiv.org/abs/2104.09864 .. _repo: https://github.com/ZhuiyiTechnology/roformer .. _GPT-NeoX: https://github.com/EleutherAI/gpt-neox If scale_base is not None, this implements XPos (Sun et al., https://arxiv.org/abs/2212.10554). A recommended value for scale_base is 512: https://github.com/HazyResearch/flash-attention/issues/96 Reference: https://github.com/sunyt32/torchscale/blob/main/torchscale/component/xpos_relative_position.py @FNTr ct||_t||_||_||}|d|d||_||_ |/tj d|d|tj d|zzd |zz nd}|d |dd|_ d|_d|_d|_d|_dS) a interleaved: if True, rotate pairs of even and odd dimensions (GPT-J style) instead of 1st half and 2nd half (GPT-NeoX style). pos_idx_in_fp32: if True, the position indices [0.0, ..., seqlen - 1] are in fp32, otherwise they might be in lower precision. This option was added because previously (before 2023-07-02), when we construct the position indices, we use the dtype of self.inv_freq. In most cases this would be fp32, but if the model is trained in pure bf16 (not mixed precision), then self.inv_freq would be bf16, and the position indices are also in bf16. Because of the limited precision of bf16 (e.g. 1995.0 is rounded to 2000.0), the embeddings for some positions will coincide. To maintain compatibility with models previously trained in pure bf16, we add this option. inv_freqF) persistentNrrdevicedtypeg?gffffff?scale)super__init__r floatbasepos_idx_in_fp32_compute_inv_freqregister_bufferr scale_baserarangefloat32_seq_len_cached _cos_cached _sin_cached _cos_k_cached _sin_k_cached) selfr rurryrvrorlrq __class__s rrszRotaryEmbedding.__init__is. $KK .))&11 ZeDDD&$%\!S!F%- H H H3QT9 TY\_bYb c c  We>>> !!rc xd|jtjd|jd|tj|jz zz S)Ng?rrrn)rurrzr r{)rros rrwz!RotaryEmbedding._compute_inv_freqs? I Q!F%-PPPSWS[[ ]  rc||jksI|jB|jj|ks2|jj|ks"|jrH|jr0||_|jrZtj||tj }|j jtj kr| |}n0|j }n(tj|||j j}|j }tj ||}|j Ztj|||_tj|||_dStj||j j|j j|dzz |jz }|j |jt'|dz}tj||z||_tj||z||_tj||z ||_tj||z ||_dSdSdS)Nrn)ro)rprorzs -> s 1)r|r}rorptraining is_inferencervrrzr{rlrwouterrqrtorr~ryrrr) rrMrorptrlfreqspowerrqs r_update_cos_sin_cachez%RotaryEmbedding._update_cos_sin_cachesN T) ) )'&&00%.. /"&"2"?"?"A"A/$*D # )LemLLL =&%-77#55V5DDHH#}HHLdm>QRRR=K8,,Ez!#(9U#3#3#6#6u#=#= #(9U#3#3#6#6u#=#=   Ltz/? HYZZZk"O$ U\ ::iz>Z>ZZ$)Ie$4$4u$<#@#@#G#G $)Ie$4$4u$<#@#@#G#G &+i&6&6&>%B%B5%I%I"&+i&6&6&>%B%B5%I%I"""I/...rrrIrb seqlen_offsetr%rAreturnc |jd}|#|||j|jn:t |t r%|||z|j|j|[|j$t||j|j |j ||St||j|j |j |j |j ||S|}t||j|j |j d|}|j$t||j|j |j |}n#t||j |j |j |}||fS)a* qkv: (batch, seqlen, 3, nheads, headdim) or (batch, seqlen, num_heads_q + 2 * num_heads_k, headdim) if kv is none, else it's just q of shape (batch, seqlen, nheads, headdim). If qkv has shape (batch, seqlen, num_heads_q + 2 * num_heads_k, headdim) (e.g. MQA / GQA), then num_heads_q must be provided. kv: (batch, seqlen, 2, nheads, headdim) seqlen_offset: (batch_size,) or int. Each sequence in x is shifted by this amount. Most commonly used in inference when we have KV cache. If it's a tensor of shape (batch_size,), then to update the cos / sin cache, one should pass in max_seqlen, which will update the cos / sin cache up to that length. Apply rotary embedding *inplace* to qkv and / or kv. rNrn)rr#rAT)rr'r#)rr#)rrrorpr(r)rqr^r}r~rrrapply_rotary_emb_funcrg)rrIrbrr%rArMrSs rr-zRotaryEmbedding.forwards(1  !  & &z#*CI & V V V V  s + + c  & &v '=cjX[Xa & b b b :z!,$$ $ 0#0 + -$$&& $ 0#0 +    A%   ,, Az!)$$ $ 0#0 *&& $ 0#0 b5Lr)rjFNTN)N)NN)NrNN)r6r7r8__doc__r)rsrwrrr:rrrr- __classcell__)rs@rririWs:(+"+" +"+"+"+"+"+"Z    ,J,J,J,Jb&*23$(%) HH \H U\ "HS%,./ H SM H c] H u|U5<#=>> ?HHHHHHHHrri)Fr5r[re)mathtypingrrrreinopsrrops.triton.rotaryr rr autogradFunctionr"r)r:r>rr@r^r`r=rgnnModulerir;rrrse  )))))))))) $$$$$$$$,,,,,,\\\\    8<8<8<8<8q>q>q>q>0q>q>q>p  /0!%#u|+,#>"+"+"+"+"+/"+"+"+J)./0 NN #u|+, NNNN.{{{{{eho{{{{{r