.`iJ&ddlmZmZddlZddlZddlZejjZ ej GddZ dee dffdZ d:dej d ej d eej d eej d ed e dddeedededeej deej deej deej deej deej deej ej ff"dZ d;dej dej dej dededeej deej deej ej ej ffdZ ddej dej d ej d!ej d"ej d#ed$ed1edeeded2ed'edeej ej ffd3Z d>d4ej d5ej d6ed7ed1edeeded2ed'edeej ej ffd8Z d>dej dej d!ej d"ej d#ed$ed7ed1edeeded2ed'edeej ej ffd9ZdS)?)OptionalTupleNceZdZUdZejGddZdZee d<dZ e ee d<dZ e e je d<dZe e je d <dS) FlashMLASchedMetazE A class that stores the tile scheduler metadata of FlashMLA ceZdZUeed<eed<eed<eed<eed<eed<eed<eeed<eeed <eeed <d S) FlashMLASchedMeta.Configbs_qh_qpage_block_sizeh_kcausalis_fp8_kvcachetopkextra_page_block_size extra_topkN)__name__ __module__ __qualname__int__annotations__boolr/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/third_party/flashmla/flash_mla_interface.pyConfigrs     sm'},,,SM!!!!!rrFhave_initializedNconfigtile_scheduler_metadata num_splits)rrr__doc__ dataclasses dataclassrrrrrrrtorchTensorr rrrrr s " " " " " " " "#d"""#FHV ###6:Xel3:::)-J&-----rrreturnc"tdfS)a Returns an empty instance of FlashMLASchedMeta. The actual scheduling metadata will be generated during the first invocation of flash_mla_with_kvcache. Arguments: This function does not need any arguments, but we keep *args and **kwargs to be compatible with the old interface. Return: A tuple. Due to historical reasons, we return a tuple of (FlashMLASchedMeta, None) now. Only the first element is useful. N)r)argskwargss rget_mla_metadatar*&s    $$rFqk_cache block_table cache_seqlens head_dim_vrr softmax_scalerrindices attn_sink extra_k_cacheextra_indices_in_kvcache topk_lengthextra_topk_lengthc|}| }t|ts Jd| Jd| |jdnd}| | jdnd}| | jdnd}||jddz}|jsv| |r Jdd|_t|jd |jd|jd |jd|jd || ||| |_ndd }|jJ|jj|jd ks Jd |z|jj|jdks Jd |z|jj|jd ks Jd|z|jj |jdks Jd|z|jj |jd ks Jd|z|jj |ks Jd|z|jj | ks Jd|z|jj |ks Jd|z|jj|ks Jd|z|jj|ks Jd|z|M|r Jd| s Jdt ||||| |j|j| | ||| \}}}}nU| | | | || Jd|| Jdt ||||||||j|j \}}}}||_||_||fS)a Arguments: q: (batch_size, seq_len_q, num_heads_q, head_dim). k_cache: (num_blocks, page_block_size, num_heads_k, head_dim). Different modes (including fp8/bf16, and sparsity) has different KV cache layouts. See comments below for details. The KV cache must be contiguously valid for sparse attention on sm100. Here "contiguously valid" means that every byte, from the very beginning of the KV cache, till the last byte in the KV cache, is valid memory address to visit (i.e. won't IMA). In other words, the KV cache could be a slice of a larger array, but cannot be a list of disjoint memory blocks. block_table: (batch_size, max_num_blocks_per_seq), torch.int32. Can be None when sparse attention is used. cache_seqlens: (batch_size), torch.int32. Can be None when sparse attention is used. head_dim_v: Head_dim of v. Must be 512 sched_meta: FlashMLASchedMeta, return by get_mla_metadata. You may reuse the same sched_meta across different invocations, but only when the tensor shapes and the values of cache_seqlens, topk_length, and extra_topk_length remain the same. num_splits_placeholder: must be "None" (to be compatible with the old interface). softmax_scale: float. The scaling of QK^T before applying softmax. Default to 1 / sqrt(head_dim_k). causal: bool. Whether to apply causal attention mask. Only valid for dense attention is_fp8_kvcache: bool. indices: (batch_size, seq_len_q, topk). KV indices when sparse attention is enabled. Pay attention that indices_in_kvcache[i][j][k] = (the index of the page block where token t resides) * block_size + (the offset of token t among the page block), where t is the k-th token of the j-th q-sequence in the i-th batch. attn_sink: Optional[torch.Tensor], (num_heads_q, ), torch.float32. If presented, the final output will be scaled by exp(lse) / (exp(lse) + exp(attn_sink)). Have no affect on the returned softmax_lse. +inf will cause the result to become 0. extra_k_cache and extra_indices_in_kvcache: If provided, will attend to these extra tokens in addition to those in k_cache and indices_in_kvcache. Their format requirements are the same as k_cache and indices_in_kvcache respectively. topk_length/extra_topk_length: (batch_size, ), torch.int32. If provided, only the leftmost topk_length indices will be processed. Useful when the actual topk for different queries are different so that we can save some computation, compared to masking. For DeepSeek V3, DeepSeek V3.1, and DeepSeek V3.2: head_dim should be 576 while head_dim_v should be 512. In FP8+sparse mode, each token's KV cache is 656 Bytes, structured as: - The shape of the tensor `k_cache` is (num_blocks, page_block_size, num_heads_k, head_dim), and num_heads_k must be 1. - First 512 bytes: The "quantized NoPE" part, containing 512 float8_e4m3 values. - Next 16 bytes: Scale factors, containing 4 float32 values. The first float32 is the scale for the first 128 float8_e4m3 values, the second for the next 128, and so on. - Last 128 bytes: The "RoPE" part, containing 64 bfloat16 values. This part is not quantized for accuracy. Return: out: (batch_size, seq_len_q, num_heads_q, head_dim_v). softmax_lse: (batch_size, num_heads_q, seq_len_q), torch.float32. z9tile_scheduler_metadata must be of type FlashMLASchedMetaNznum_splits must be Nonez[causal must be False when indices_in_kvcache is not None (i.e. sparse attention is enabled)Trz Your input arguments are inconsistent with sched_meta. Please make sure the input arguments are consistent across different invocations of flash_mla_with_kvcache on the same sched_meta.z0sched_meta.config.b must be equal to batch_size.z1sched_meta.config.s_q must be equal to seq_len_q.z3sched_meta.config.h_q must be equal to num_heads_q.zCsched_meta.config.page_block_size must be equal to page_block_size.z3sched_meta.config.h_k must be equal to num_heads_k.z1sched_meta.config.causal must be equal to causal.zAsched_meta.config.is_fp8_kvcache must be equal to is_fp8_kvcache.zKsched_meta.config.topk must be equal to the last dim of indices_in_kvcache.z^sched_meta.config.extra_page_block_size must be equal to the page_block_size of extra_k_cache.zWsched_meta.config.extra_topk must be equal to the last dim of extra_indices_in_kvcache.z5causal must be False when sparse attention is enabledzAAADGJTDTAAA %---/|@J0J--- 6:QQQQTtwATAQQQ +z999= s_kv sm_scale: float d_v: The dimension of value vectors. Can only be 512 attn_sink: optional, [h_q], float32. If attn_sink is provided, when computing output, output will be additionally multiplied by exp(lse) / (exp(lse) + exp(attn_sink)). +-inf in attn_sink will be handled normally (i.e., -inf has no effect, +inf will make corresponding output all zeros). This argument has no effect on lse and max_logits. topk_length: optional, [s_q], int32. If provided, the i-th q token will only attend to k tokens specified by indices[i, :, :topk_length[i]], ignoring later k/v tokens (even if provided in indices). In extremely rare cases (topk_length provided, there is a valid topk index between topk_length[i] ~ s_kv, and that topk index points to a k token containing NaN), operator output will contain NaN, so please avoid this situation. Returns: (output, max_logits, lse) Please refer to tests/ref.py for the precise definitions of these parameters. - output: [s_q, h_q, d_v], bfloat16 - max_logits: [s_q, h_q], float - lse: [s_q, h_q], float, log-sum-exp of attention scores )r>sparse_prefill_fwd)r+rKr1rLrMr2r5resultss rflash_mla_sparse_fwdrQs0@// 2w#y+G NrTkv cu_seqlens_qo cu_seqlens_kv max_seqlen_qo max_seqlen_kvrErF is_varlenc |j\} } }|j\}}}| rdnd}| |dz} |#tj| | ||j|j}|,tj| | |jtjj}tjdtj|j}t |||||||||| ||| ||fS)Nr9rr:devicedtypeir\r[) r=r$emptyr[r\float32Tuint8r>dense_prefill_fwd)r+rRrSrTrUrVrWrErFrr0rX qo_total_len num_qo_heads head_dim_qk kv_total_len num_kv_heads head_dim_vomask_mode_codeworkspace_buffers r_flash_attn_varlen_forwardrks/0g+L, ./g+L,  'QQaN#-  {k, k!(Z[Zabbb {k, QXU][[[]{#35;qxXXX$$      8Ordodqdkdvc|j\}}}|j\}}}||krtd|d|d| rdnd}||dz}| #tj||||j|j} | #tj||||j|j} | #tj||||j|j} |dzd zd z}|jddz }d}|d |z|z|z|zz }|d |z|z|zd zz }||kr|d |z|z||zzz }tj|tj|j }t|||||||||| | | |||| || | | fS) Nz1SM100 bwd doesn't support GQA now. num_qo_heads: z, num_kv_heads: .r9rr:rZr;r]) r= ValueErrorr$r^r[r\rar>dense_prefill_bwd)rlr+rRrSrErFrTrUrVrWrmrnrorr0rXrcrdrerfrgrhrimax_seqlen_qo_alignedbsworkspace_bytesrjs r_flash_attn_varlen_backwardrzs$/0g+L, ./g+L, |##z\zzkwzzz{{{ 'QQaN#-  z [|[YZY` a a a z [|[YZY` a a a z [|[YZY` a a a*Q.14q8  Q ! #BOq2v 55 D{RROq0025 DqHHO|##1|+l:kK>WXX{?%+ahWWW$$         #( r2:rceZdZ ddejdejdejdejdejd ed ed ed eed ede ejejffdZ dejdejfdZ dS)FlashAttnVarlenFuncFNTr+rRrSrTrUrVrWrr0rXr&c t||||||||| |  \} } ||||| | ||||_||_||_| |_| |_| | fSN)rr0rX)rksave_for_backwardrVrWrr0rX) ctxr+rRrSrTrUrVrWrr0rXrErFs rforwardzFlashAttnVarlenFunc.forwardJs. q! =-    S aAsC NNN)) )! Cxrrldlsec~|j\}}}}}}} t|||||||| |j|j|j|j|j \} } } | | | dddddddf Sr~) saved_tensorsrzrVrWrr0rX) rrlrr+rRrSrErFrTrUrmrnros rbackwardzFlashAttnVarlenFunc.backwardes :=:K71ac=-0 1ac =#*;S=N:S->m    B 2r4tT4tCCr)FNT) rrrr$r%rrrfloatrrrrrrr|r|Is)- < < <  |  |   u|U\) *6 D L Dl D D D D D Drr| dropout_p deterministicc `|dksJ| rJt|||||||| || SNrr|apply) r+rRrSrTrUrVrWrr0rrrXs rflash_attn_varlen_funcrusP        $ $ 1a}m] y  rqkv cu_seqlens max_seqlenrec |dksJ|rJt|ddddd|f|dddd||dzf|dddd|dzdf||||||| S)Nrr;r) rrrrerr0rrrXs r flash_attn_varlen_qkvpacked_funcrs        $ $ AAAqqq,;, QQQ;{Q+F%F!GQQQPQPQPQS^abSbScScMcIdJ J y  rc |dksJ| rJt||ddddd|f|dddd|df||||| || Srr) r+rKrTrUrVrWrerr0rrrXs rflash_attn_varlen_kvpacked_funcrs        $ $ 2aaaL[L !2aaaKLL&8#9}m] y  r) NNFFNNNNNN)rJNN)NNFNT)NNNFNT)rNFFT)typingrrr"r$vllm._flashmla_Cvllmops _flashmla_Cr>r#rr*r%rrrrIrQrkrzautogradFunctionr|rrrrrrrs"""""""" & ........8 % d "# % % % %.%) &*(,,07;*.04!xx |x \x%,'xEL) x  x / xxE?x xxel #x %xEL)x'u|4x%,'x   -!x" 5< %&#xxxx@(,*.## |# #\# # #  % #%,'# 5<u| 34####\#'"&%),, |, |, |,< , < ,  ,, %, , %, , ,E?,, 5< %&,,,,t"&!%!%%)!@@ @ |@ |@ | @  @  @<@<@@@ @ @ @ @E?@ !@" 5<u| 34#@@@@F)D)D)D)D)D%.1)D)D)Dh%) | | |<  <   E?  5< %&8%)      E?   5< %&8%) | <<     E?  5< %&r