)`iA}$8dZddlZddlmZmZmZmZmZmZddl Z ddl m Z ddl m Z mZmZddlmZddlmZmZmZmZmZmZmZdd lmZd Zd Zejd Zejd Z ejdZ!GddZ"e d&de j#de j#de j#de$de$de$de j#de j#de$de$dee j#dee%e j#fdee%e j#fd eee j#d!e&d"e'd#e j#f"d$Z(e d'de j#de j#de j#de$de$de$de j#de j#de$dee j#dee%e j#fdee%e j#fd eee j#d!e&d#e j#fd%Z)dS)(a3 Copyright (c) 2023 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)ListLiteralOptionalTupleUnionoverload)flashinfer_api)gen_batch_mla_modulegen_trtllm_gen_fmha_modulesetup_cubin_loader)gen_mla_module)MaskModecheck_shape_dtype_devicedetermine_mla_backenddevice_support_pdlget_compute_capabilityget_device_sm_countlog2e)xqa_mlacd|jdkrtd|j|jdkrtd|j|jdkrtd|j|jdkrtd|j|j\}}}|jd}|dkrtd |||ks|d krtd |d ||j\}} |jd} ||krtd |d|d || d| z zdkrtd| d| dS)Nz"Expected q_nope_pe.ndim == 3, got z&Expected ckv_kpe_cache.ndim == 3, got r zExpected kv_len.ndim == 1, got z#Expected page_table.ndim == 2, got z&Expected 128 heads for q_nope_pe, got @z;Expected head dim 576 for q_nope_pe and ckv_kpe_cache, got  and Expected batch size z$ for q_nope_pe and block_table, got rBs  zQF%*FFGGG}%%a(( !  Q(-QQRRR3TBRTTUUUsLlLLMMM2SAQSSTTTC3 N1 E e||sczz R R R5 R R   !%+ U,<= = =fTdff  > $.#3 y -  eseeCeeVcee  j( )Q . .cYccV`cc  Or0ct}|}t||SN)r build_and_loadr get_library_path)modops r.get_trtllm_gen_fmha_modulerE|s= $ & &C    Bs++--... Ir0cBtSr@)rrAr0r.get_mla_modulerHs    * * , ,,r0c@t|g|RSr@)r rA)backendargss r.get_batch_mla_modulerLs$  /$ / / / > > @ @@r0ceZdZdZe d'dejdedeejdeejd eejd eejd e d dfd Z e d(dejdejd ejd ejde de de de dede dej dej ded dfdZe d)dejdejdejdejdeejdeejdeddeejd eejd!eejd"ed ejfd#Ze d*dejdejdejdejdeejdeejded$deejd eejd!eejd"ed eejejffd%Ze d)dejdejdejdejdeejdeejdedeejd eejd!eejd"ed eejeejejfffd&ZdS)+BatchMLAPagedAttentionWrappera Wrapper class for MLA (`Multi-head Latent Attention `_) PagedAttention on DeepSeek models. This kernel can be used in decode, and incremental prefill and should be used together with `Matrix Absorption trick `_: where :math:`W_{UQ}` is absorbed with :math:`W_{UK}`, and :math:`W_{UV}` is absorbed with :math:`W_{O}`. For MLA attention without Matrix Absorption (``head_dim_qk=192`` and ``head_dim_vo=128``, which is used in prefilling self-attention stage), please use :class:`flashinfer.prefill.BatchPrefillWithRaggedKVCacheWrapper`. More information about The Paged KV-Cache layout in MLA is explained in our tutorial :ref:`MLA Page Layout `. For more details about the MLA computation, Matrix Absorption and FlashInfer's MLA implementation, please refer to our `blog post `_. Example ------- >>> import torch >>> import flashinfer >>> num_local_heads = 128 >>> batch_size = 114 >>> head_dim_ckv = 512 >>> head_dim_kpe = 64 >>> page_size = 1 >>> mla_wrapper = flashinfer.mla.BatchMLAPagedAttentionWrapper( ... torch.empty(128 * 1024 * 1024, dtype=torch.int8).to(0), ... backend="fa2" ... ) >>> q_indptr = torch.arange(0, batch_size + 1).to(0).int() # for decode, each query length is 1 >>> kv_lens = torch.full((batch_size,), 999, dtype=torch.int32).to(0) >>> kv_indptr = torch.arange(0, batch_size + 1).to(0).int() * 999 >>> kv_indices = torch.arange(0, batch_size * 999).to(0).int() >>> q_nope = torch.randn( ... batch_size * 1, num_local_heads, head_dim_ckv, dtype=torch.bfloat16, device="cuda" ... ) >>> q_pe = torch.zeros( ... batch_size * 1, num_local_heads, head_dim_kpe, dtype=torch.bfloat16, device="cuda" ... ) >>> ckv = torch.randn( ... batch_size * 999, 1, head_dim_ckv, dtype=torch.bfloat16, device="cuda" ... ) >>> kpe = torch.zeros( ... batch_size * 999, 1, head_dim_kpe, dtype=torch.bfloat16, device="cuda" ... ) >>> sm_scale = 1.0 / ((128 + 64) ** 0.5) # use head dimension before matrix absorption >>> mla_wrapper.plan( ... q_indptr, ... kv_indptr, ... kv_indices, ... kv_lens, ... num_local_heads, ... head_dim_ckv, ... head_dim_kpe, ... page_size, ... False, # causal ... sm_scale, ... q_nope.dtype, ... ckv.dtype, ... ) >>> o = mla_wrapper.run(q_nope, q_pe, ckv, kpe, return_lse=False) >>> o.shape torch.Size([114, 128, 512]) FNautofloat_workspace_bufferuse_cuda_graph qo_indptr kv_indptr kv_indices kv_len_arrrJreturnc||_|j|_|dkr ||_dStjdtj|j|_tj|jj|jjdd|_ ||_ ||_ ||_ ||_ ||_|dkrt|j|_dS||_dS) anConstructor for BatchMLAPagedAttentionWrapper. Parameters ---------- float_workspace_buffer : torch.Tensor The user reserved workspace buffer used to store intermediate attention results in split-k algorithm. The recommended size is 128MB, the device of the workspace buffer should be the same as the device of the input tensors. use_cuda_graph : bool, optional Whether to enable CUDA graph capture for the prefill kernels, if enabled, the auxiliary data structures will be stored in provided buffers. The ``batch_size`` cannot change during the lifecycle of this wrapper when CUDAGraph is enabled. qo_indptr_buf : Optional[torch.Tensor] The user reserved buffer to store the ``qo_indptr`` array, the size of the buffer should be ``[batch_size + 1]``. This argument is only effective when ``use_cuda_graph`` is ``True``. kv_indptr_buf : Optional[torch.Tensor] The user reserved buffer to store the ``kv_indptr`` array, the size of the buffer should be ``[batch_size + 1]``. This argument is only effective when ``use_cuda_graph`` is ``True``. kv_indices_buf : Optional[torch.Tensor] The user reserved buffer to store the ``kv_indices`` array. This argument is only effective when ``use_cuda_graph`` is ``True``. kv_len_arr_buf : Optional[torch.Tensor] The user reserved buffer to store the ``kv_len_arr`` array, the size of the buffer should be ``[batch_size]``. This argument is only effective when ``use_cuda_graph`` is ``True``. backend : str The implementation backend, could be ``auto``/``fa2`` or ``fa3``. Defaults to ``auto``. If set to ``auto``, the function will automatically choose the backend based on the device architecture and kernel availability. If ``cutlass`` is provided, the MLA kernels will be generated by CUTLASS and only float_workspace_buffer is required and other arguments are ignored. cutlassN)dtypedeviceTcpu)r[ pin_memoryr\rO)_float_workspace_bufferr\_backendtorchemptyuint8_int_workspace_bufferr"r[ _pin_memory_int_workspace_buffer_use_cuda_graph_qo_indptr_buf_kv_indptr_buf_kv_indices_buf_kv_len_arr_bufr)selfrPrQrRrSrTrUrJs r.__init__z&BatchMLAPagedAttentionWrapper.__init__sZ(>$,3 i  #DM F%*[ ek$+& & & "16  & ,,2 1 1 1 - .'')) f  1$+>>DMMM#DMMMr0 num_heads head_dim_ckv head_dim_kper;causalsm_scale q_data_type kv_data_type use_profilerc t|j| | | |j||| |_|d}|d}|d}|jr|j|d|j|d|j dt||d|j |dn||j d|_||j d|_||j d|_ ||j d|_ | |_ ||_| |_| |_|j|j|j|j|||||| |_dS)aPlan the MLA attention computation. Parameters ---------- qo_indptr : torch.IntTensor The indptr of the query/output tensor, shape: ``[batch_size + 1]``. For decoding attention, the length of each query is 1, and the content of the tensor should be ``[0, 1, 2, ..., batch_size]``. kv_indptr : torch.IntTensor The indptr of the paged kv-cache, shape: ``[batch_size + 1]``. kv_indices : torch.IntTensor The page indices of the paged kv-cache, shape: ``[kv_indptr[-1]]`` or larger. kv_len_arr : torch.IntTensor The query length of each request, shape: ``[batch_size]``. num_heads : int The number of heads in query/output tensor. head_dim_ckv : int The head dimension of compressed-kv. head_dim_kpe : int The head dimension for rope k-cache. page_size : int The page size of the paged kv-cache. causal : bool Whether to use causal attention. sm_scale : float The scale factor for softmax operation. q_data_type : torch.dtype The data type of the query tensor. kv_data_type : torch.dtype The data type of the kv-cache tensor. use_profiler : bool, optional Whether to enable intra-kernel profiler, default is False. r]T) non_blockingN)rLr`r[_cached_moduletorfrgcopy_rhrilenrjr\_causal _page_size _sm_scale _use_profilerplanr_rdre _plan_info)rkrRrSrTrUrmrnror;rprqrrrsrtqo_indptr_hostkv_indptr_hostkv_len_arr_hosts r.rz"BatchMLAPagedAttentionWrapper.plansd3 M    O      #e,,"e,,$--..   Q   % %id % C C C   % %id % C C C  !23z??!2 3 9 9*SW 9 X X X  & &z & E E E E"+,,t{,"N"ND "+,,t{,"N"ND #-==4=#P#PD #-==4=#P#PD  #!)-22  (  &  1         r0q_nopeq_pe ckv_cache kpe_cacheoutlse return_lseprofiler_bufferr%r&return_lse_base_on_ec dSr@rG rkrrrrrrrrr%r&rs r.runz!BatchMLAPagedAttentionWrapper.runrs sr0Tc dSr@rGrs r.rz!BatchMLAPagedAttentionWrapper.runs -0Cr0c |jdkr|rtd|tdt|_|t j|}n"t ||j|j|j dt j ||gd} t j ||gd} t| | | | t j dtj |j }|j|j||| | | | |S||jrtd |jd }|j}|j}|j}|rt(jjnt(jj}|j }|t j|}n"t ||j|j|j d|r`|/t j |jdd tj | }n/t ||jdd tj |j d |jr|fnd}|jj|j|j|j|||||j||||||| g|R|r||fn|S)aRun the MLA attention computation. Parameters ---------- q_nope : torch.Tensor The query tensor without rope, shape: ``[batch_size, num_heads, head_dim_ckv]``. q_pe : torch.Tensor The rope part of the query tensor, shape: ``[batch_size, num_heads, head_dim_kpe]``. ckv_cache : torch.Tensor The compressed kv-cache tensor (without rope), shape: ``[num_pages, page_size, head_dim_ckv]``. ``head_dim_ckv`` is 512 in DeepSeek v2/v3 models. kpe_cache : torch.Tensor The rope part of the kv-cache tensor, shape: ``[num_pages, page_size, head_dim_kpe]``. ``head_dim_kpe`` is 64 in DeepSeek v2/v3 models. out : Optional[torch.Tensor] The output tensor, if not provided, will be allocated internally. lse : Optional[torch.Tensor] The log-sum-exp of attention logits, if not provided, will be allocated internally. return_lse : bool, optional Whether to return the log-sum-exp value, default is False. profiler_buffer : Optional[torch.Tensor] The buffer to store the profiler data. kv_len : Optional[torch.Tensor] The query length of each request, shape: ``[batch_size]``. Required when ``backend`` is ``cutlass``. page_table : Optional[torch.Tensor] The page table of the paged kv-cache, shape: ``[batch_size, num_pages]``. Required when ``backend`` is ``cutlass``. rXz4return_lse does not support cutlass backend for now.Nz9profiler_buffer does not support cutlass backend for now.r)dimrrZz5Profiler is enabled, profiler_buffer must be providedr rrrG)r`r!rHrwra empty_likerr"r[r\catr/rbfloat32cutlass_mla_paged_attentionr_r~r|r}r{rCAUSALvalue NON_CAUSALrrdrri)rkrrrrrrrrr%r&rr#r$rmr;rqrp mask_moder\ profiler_argss r.rz!BatchMLAPagedAttentionWrapper.runsT =I % % Y !WXXX* O#1"2"2D {&v..(v|V]E 64.b999I!Iy)&<"EEEM M6: N N N+au}T[IIIC   ; ;,   J  "!  KLO O >-3RHO))9L9R  ;"6**CC $V\6<     {k&,rr"2%-PVWWW(bqb)5=&-/3.@H**b   (  & O             !    &(0SzzS0r0)FNNNNrO)F)NNFNNNF)NNTNNNF)__name__ __module__ __qualname____doc__r raTensorboolrstrrlintfloatr[rrrrrrrGr0r.rNrNs??B %,0,0-1-1D$D$ % D$D$EL) D$ EL) D$ U\* D$U\*D$D$ D$D$D$^D$L#X X <X <X L X L X  X X X X X X [X kX X  X X X ^X t'+&*%*26)--1%*    l <  <  el # el # EN "%,/ & U\* #     X '+&*$(26)--1%* 0 0  0l 0< 0 < 0 el # 0el # 0DM 0"%,/ 0& 0U\* 0# 0 u|U\) * 0 0 0X 0'+&* 26)--1%*s1s1 s1ls1< s1 < s1 el # s1el #s1s1"%,/s1&s1U\*s1#s1 u|U5<#=>> ?s1s1s1^s1s1s1r0rN?rOr5r6workspace_bufferr7r8r9 block_tablesseq_lens max_seq_lenr:r bmm1_scale bmm2_scalesinks enable_pdlrJrVc |dkr"t|jddkrdnd}t| tjr!| jtjksJ| tz} t| tjr| jtjksJ|dkrt|jddks*|jtjks|jtjkrtd|jd|j| td | d d kr%td | d t|||||||||| | | | |S|dkr|t|jn|}tj}t|j}| d }|dkr|dkrtd|t!|||||| ||}| :|jd d|fz}tj|tj|j} n1|j\}}}}t)| |||gtj|jd| d}| d }|dd }|| d ||||||||| | ddd|d| ||||z| d | Std|d)a Parameters ---------- query: [batch_size, q_len_per_request, num_heads, head_dim_qk], head_dim_qk = qk_nope_head_dim (kv_lora_rank) + qk_rope_head_dim, should be concated q_nope + q_rope; q_len_per_request is the MTP query length. kv_cache: [num_pages, page_size, head_dim_ckv + head_dim_kpe] or [num_pages, 1, page_size, head_dim_ckv + head_dim_kpe], should be concated ckv_cache + kpe_cache. Both 3D and 4D formats are supported for backward compatibility. workspace_buffer: [num_semaphores, 4], used for multi_block mode. Must be initialized to 0 for its first use. qk_nope_head_dim: qk_nope_head_dim, must be 128 kv_lora_rank: kv_lora_rank, must be 512 qk_rope_head_dim: qk_rope_head_dim, must be 64 sparse_mla_top_k: sparse MLA top k, must be 0 for non-sparse MLA. block_tables: page_table of kv cache, [batch_size, num_pages] seq_lens: query_len max_seq_len: max sequence length for kv_cache out: output tensor, if not provided, will be allocated internally bmm1_scale: fused scale for mla bmm1 input. when using trtllm-gen backend, it can be a torch.Tensor with dtype torch.float32. bmm2_scale: fused scale for mla bmm2 input. when using trtllm-gen backend, it can be a torch.Tensor with dtype torch.float32. sinks: additional value per head in the denominator of the softmax. backend : str = "auto" The implementation backend, could be ``auto``/``xqa`` or ``trtllm-gen``. Defaults to ``auto``. When set to ``auto``, the backend will be chosen based on the device architecture and kernel availability. For sm_100 and sm_103 (blackwell architecture), ``auto`` will choose ``trtllm-gen`` backend. For sm_120 (blackwell architecture), ``auto`` will choose ``xqa`` backend. Note ---- In MLA, the actual BMM1 and BMM2 scales applied would be fused as: bmm1_scale = q_scale * k_scale * sm_scale / (head_dim_qk ** 0.5) bmm2_scale = v_scale * o_scale or, bmm1_scale = torch.Tensor([q_scale * k_scale * sm_scale / (head_dim_qk ** 0.5)) bmm2_scale = torch.Tensor([v_scale * o_scale]) The two scale factors should be static constant for cuda graph capture. Either (bmm1_scale, bmm2_scale) or (bmm1_scale_log2_tensor, bmm2_scale_tensor) should be provided. For static constant scale factors, the scale factors should be provided as float. - (bmm1_scale, bmm2_scale) For on-device fused scale tensors, which could dynamically change, the scale factors should be provided as torch.Tensor. - (bmm1_scale_log2_tensor, bmm2_scale_tensor) - Currently, only fp8 tensor core operation supports this mode. When both are provided, the dynamic scale factor tensors will be used. rOr z trtllm-genxqa z7XQA MLA only supports fp8 operation on SM120 GPUs, got rNXQA MLA does not support sinksr 2XQA MLA only supports q_len_per_request == 1, got r3z(Supported block_size are 32 and 64, got rrZrzBackend z not supported)rr\ isinstancerarr[rr float8_e4m3fnr!size"xqa_batch_decode_with_kv_cache_mlarrEtrtllm_paged_attention_decoderr>r"rbbfloat16rflattennumel element_size)r5r6rr7r8r9rrrr:rrrrrrJrun_funcsm_countr- out_shape batch_size_ num_q_heads max_q_lens r.%trtllm_batch_decode_with_kv_cache_mlar s~&25<@@CrIILLu *el++(5=0000%' *el++15=0000% "5< 0 0 3r 9 9{e111~!444l%+ll\d\jll   =>> > ::a==A  TUZZPQ]]TT 2                 L 0:0B u| , , ,  .//M&u|44]]2&& "  r!1!1T TTUU U/            ; CRC(L?:I+iu~elSSSCC,1K )J; $[,7     ZZ]] JJqMM  a##                     " " $ $'7'D'D'F'F F  /   4 ;G;;;<<r"rbrrviewrcsqueezer4r)r5r6rr7r8r9rrrrrrrrrr-q_len_per_requestrrrr workspace_u8 semaphorescratch kv_cache_new seq_lens_news r.rrs-`6@5G#EL111ZJ"5<00Hr""J 1 A TAR T T    {e)))X^u?R-R-R fEK f fV^Vd f f    9:::+    H {K$ 6 k)5>%,OOO(- % A{A  l 3 N L     $((55L../I?,,-G##A&&0033L%%a((L    Jr0)rNrrNNrO)NrrNN)*r functoolstypingrrrrrrra api_loggingr jitr r r jit.mlarutilsrrrrrrrrrr/r>cacherErHrLrNrrrrrrrrGr0r.rs BBBBBBBBBBBBBBBB ''''''UUUUUUUUUU######   :777t  --- AAAx1x1x1x1x1x1x1x1v "&-0-0*.!m=m= <m=lm=lm= m=  m=  m=,m=lm=m=m= %, m=eU\)*m=eU\)*m= D& 'm=m= !m=" \#m=m=m=m=`#'-0-0*.pp <plplp p  p  p,plpp %, peU\)*peU\)*p D& 'pp \ppppppr0