0`iG ddlmZmZmZmZddlZddlmZ ddlm Z dZ dZ n#e $rZ ee Z dZ YdZ [ ndZ [ wwxYw ddlmZdZdZn#e $rZ ee ZdZYdZ [ ndZ [ wwxYw ddlmZdZdZn#e $rZ ee ZdZYdZ [ ndZ [ wwxYwd Zd d eeeeffd Zd d eeeeffd Zd d eeeeffd Zd ded efdZd ded eefdZdZejddddddddddddf dej deej deej deej deef dZ!dddddddddddddddddddeddddfdeeededefdZ" d!ddddZ# d!ddddZ$dS)")OptionalUnionTupleListN) _vllm_fa2_CTF) _vllm_fa3_C)_flash_attn_fwdreturncts ddtfStj|ddkrdSdS)NFzFA2 is unavaible due to: r)Fz=FA2 is only supported on devices with compute capability >= 8TN) FA2_AVAILABLEFA2_UNAVAILABLE_REASONtorchcudaget_device_capabilitydevices }/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/vllm_flash_attn/flash_attn_interface.py_is_fa2_supportedr(sQ KJ2HJJJJ z''//2Q66LL :cts ddtfStj|ddks)tj|ddkrdSdS)NFzFA3 is unavaible due to: r )FzFA4 is only supported on devices with compute capability == 10r) FA4_AVAILABLEFA4_UNAVAILABLE_REASONrrrrs r_is_fa4_supportedr#9sQ KJ2HJJJJ z''//2b88MM :r fa_versionc|dvs Jd||dkrt|dS|dkrt|dS|dkrt|dSdS)Nr Unsupported FA version: r rr'r(rrr#r$rs ris_fa_version_supportedr,As  " " "$Kz$K$K " " "Q ((++ q ((++ q ((++ rc|dvs Jd||dkrt|dS|dkrt|dS|dkrt|dSdS)Nr&r)r rr'r(r*r+s rfa_version_unsupported_reasonr.Js  " " "$Kz$K$K " " "Q ((++ q ((++ q ((++ rcd|-|ddkr|n|S)Nr)stride contiguous)xs rmaybe_contiguousr4Zs,]qxx||q/@/@1<<>>>aGrr0r0 cache_seqlens cu_seqlens_qcu_seqlens_k_new cache_leftpad page_sizect|}||}tjj|||||||||| d| d| | | ||d|d||||}|S)Nrr)r4ropsr get_scheduler_metadata) batch_size max_seqlen_q max_seqlen_k num_heads_q num_heads_kvheaddimr6 qkv_dtype headdim_vr7r8r9r:max_seqlen_k_newcausal window_size has_softcap num_splitspack_gqa sm_marginscheduler_metadatas rr=r=^s"%]33M .EEL, \7T]  A A!& rrHrJc| | Jd| | Jd| | Jd| |jddz} | d}n%t| dksJ| d | d f}d |||fD\}}}tj|}|dkr||||t d |t d |d krt dtjj|||||||n||d||||| | d| |d |d | |o| d k|d\}} n|dkr| Jdtjjj g|||dd||||dd||||ddddd|||| | |d |d | d||dd ||||R\}} }!}!ntd||r|| fn|S)aU dropout_p should be set to 0.0 during evaluation Supports multi-query and grouped-query attention (MQA/GQA) by passing in K, V with fewer heads than Q. Note that the number of heads in Q must be divisible by the number of heads in KV. For example, if Q has 6 heads and K, V have 2 heads, head 0, 1, 2 of Q will attention to head 0 of K, V, and head 3, 4, 5 of Q will attention to head 1 of K, V. If causal=True, the causal mask is aligned to the bottom right corner of the attention matrix. For example, if seqlen_q = 2 and seqlen_k = 5, the causal mask (1 = keep, 0 = masked out) is: 1 1 1 1 0 1 1 1 1 1 If seqlen_q = 5 and seqlen_k = 2, the causal mask is: 0 0 0 0 0 0 1 0 1 1 If the row of the mask is all zero, the output will be zero. If window_size != (-1, -1), implements sliding window local attention. Query at position i will only attend to keys between [i + seqlen_k - seqlen_q - window_size[0], i + seqlen_k - seqlen_q + window_size[1]] inclusive. Arguments: q: (total_q, nheads, headdim), where total_q = total number of query tokens in the batch. k: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. v: (total_k, nheads_k, headdim), where total_k = total number of key tokens in the batch. cu_seqlens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seqlens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. dropout_p: float. Dropout probability. softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(headdim). causal: bool. Whether to apply causal attention mask (e.g., for auto-regressive modeling). window_size: (left, right). If not (-1, -1), implements sliding window local attention. softcap: float. Anything > 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). Nz*cu_seqlens_k or seqused_k must be providedz>cu_seqlens_k and seqused_k cannot be provided at the same timez5seqused_k must be provided if block_table is providedr0r5r rrc,g|]}t|Sr4.0r3s r z*flash_attn_varlen_func..!666q""666rzHFA2 does not support scheduler_metadata, q_descale, k_descale, v_descalezFA2 does not support s_auxz#FA2 does not support num_splits > 1Fr'zAlibi is not supported in FA3Tr)) shapelenr empty_likeNotImplementedErrorr<r varlen_fwdr fwd ValueError)"qkvr?r7r@ cu_seqlens_k seqused_kq_v dropout_p softmax_scalerGrHsoftcap alibi_slopes deterministicreturn_attn_probs block_tablereturn_softmax_lseoutrM q_descale k_descale v_descalerJr$s_aux cp_world_sizecp_rankcp_tot_seqused_kreal_window_sizedummy_cu_seqlens_k softmax_lse_s" rflash_attn_varlen_funcrysl  #y'<'<4(='< <  9#4#4H$5#4 4  )"7"7?#8"7 7 - #;1$$$$'NKN;66Q1I666GAq!),77Q  )i.C%)*?)+  %&BCC C >>%&KLL L 90;; q!  #/"6  L           Q  Q    09q=  -  [[0 q##%D###!&!6!:" " " " " "  "   "  "  "  "  " "  " '"  "  "  "  " " "  " !" #,"  " !" " Q #" ""2!!4#" $ %" & '" ( )" * +" , -" . /" 0 1" 2 3" 4 5" 6 7" " " [!QQ<@J@@AAA!3 .orWrr)rXrr<r fwd_sparse)r_r`ra block_count block_offset column_count column_indexrerfrGrgrhrirjrlrmrws rsparse_attn_funcr9sf - 66Q1I666GAq!y,77    +i!m C""4 0 activates softcapping attention. alibi_slopes: (nheads,) or (batch_size, nheads), fp32. A bias of (-alibi_slope * |i + seqlen_k - seqlen_q - j|) is added to the attention score of query i and key j. deterministic: bool. Whether to use the deterministic implementation of the backward pass, which is slightly slower and uses more memory. The forward pass is always deterministic. return_attn_probs: bool. Whether to return the attention probabilities. This option is for testing only. The returned probabilities are not guaranteed to be correct (they might not have the right scaling). Return: out: (total, nheads, headdim). softmax_lse [optional, if return_softmax_lse=True]: (nheads, total_q_seqlen). The logsumexp of each row of the matrix QK^T * scaling (e.g., log of the softmax normalization factor). Nr0rPc,g|]}t|SrRrSrTs rrVz+sparse_attn_varlen_func..rWrFr)rXrr<rvarlen_fwd_sparse)r_r`rar}r~rrr7rbr?r@rerfrGrgrhrirjrlrmrws rsparse_attn_varlen_funcrs| - 66Q1I666GAq!y,>>      +i!m +C."4 rs0///////////  !MM SVVMMMMMM!MM SVVMMMMMM999999!MM SVVMMMMMM dHSM.A(BdHSM.A(BdHSM.A(B,,,t,,,,,,c,},,,, HHHn+//3,0#  ''<' 5<( ' u|, 'EL)'}''''b  '+  (  An=n=$s)$n=45n=89n=n=n=n=r  H=  #H=H=H=H=H=n  %Y=( +Y=Y=Y=Y=Y=Y=Y=s@ #? :? AA* A%%A*. A99B> BB