)`iH,ldZddlZddlmZddlmZmZddlZddlm Z ddl m Z m Z ddl mZdd lmZmZmZmZmZejd ejd ejd ed edededejdefdZe d,dejdejdejdejdejdejdejdejded edeejdeeejfd eeejfd!ed"ed#eed$eed%eded&eejd'df*d(Zej d-d ejd ejd ed ededef d*Ze d.dejdejdejdejdejdejdejdejd edeeejfd eeejfd#eed$eed'dfd+ZdS)/a3 Copyright (c) 2024 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)SimpleNamespace)OptionalUnion)flashinfer_api)gen_xqa_modulegen_xqa_module_mla)filename_safe_dtype_map)get_device_sm_countregister_custom_opregister_fake_opget_compute_capabilitydevice_support_pdl input_dtypekv_cache_dtype page_sizehead_dimhead_group_ratiouse_sliding_window output_dtype q_seq_lenc- t|||||||| |dkrd}nd}tdt|dt|dt|d|d|d |d |d |d |d dtdt dt dt dt ttj fdtj dtdtj dttj dtj dtj dtj dt dtj dt dt ttj fdtj d tj d!td"t d#ttj d$df, fd% } tdt|dt|dt|d|d|d |d |d |d |dtdt dt dt dt ttj fdtj dtdtj dttj dtj dtj dtj dt dtj dt dt ttj fdtj d tj d!td"t d#ttj d$df,d&} t| 'S)(NrTFzflashinfer::xqa_input_ _kv_cache__output_ _page_size_ _head_dim__head_group_ratio__use_sliding_window__use_spec_dec__spec_q_seq_len_outputworkspace_buffer mutates_argsrun_sm90_fp8_mhasm_count num_kv_headssliding_win_sizeq_scaler" rcp_out_scaleqsinksk_cachev_cache page_table max_seq_lenseq_lens batch_sizekv_scale semaphoresr# enable_pdlrmaskreturnc*||||t|tjrdn|t|trdn|||||| | | | | |t|tjrdn|t|trdn||||||dSN?) xqa_wrapper isinstancetorchTensorfloat)r&r'r(r)r*r"r+r,r-r.r/r0r1r2r3r4r5r#r6rr7modules b/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/xqa.pyxqazget_xqa_module..xqa=s6     gu|44 ACC'w.. ;DDG         h 55 CCC8x// =DDX     /     cdSN)r&r'r(r)r*r"r+r,r-r.r/r0r1r2r3r4r5r#r6rr7s rB _fake_xqaz!get_xqa_module.._fake_xqars 4 rD)rC) rbuild_and_loadr r boolintrr@r>r?rr r) rrrrrrrr use_spec_decrCrHrAs @rBget_xqa_modulerM"sz  n 1}}   W!8!E W WQhiwQx W WCZ[gCh W Wu~ W WJR W Wfv W WL^ W Wnz W WLU W W3/ / / /  / uel*+ /  / /  </  %/ / / L/ / ,/ / u|+,!/ "L#/ $ ,%/ &'/ ()/ *u|$+/ , -/ / / / / / b W!8!E W WQhiwQx W WCZ[gCh W Wu~ W WJR W Wfv W WL^ W Wnz W WLU W W      uel*+      <  %   L  ,  u|+,! "L# $ ,% &' () *u|$+ , -    2     rDr;NHDr,r.r/r0r2r"r#r5r(r-r*r4r) kv_layoutr'r6r+r7r8c|t|j}||nt|j}|jd}|jd}|jd}||z}|jd}|| z}| dk}|j|jks Jd|jt jkr |jt jks Jdn|j|jks Jd|dkr,|d d}|d d}|jt jkr/tt jd dd krd }nd}tt jd ddvrtdt|j|j| ||||j|}|dkr| Jd| d}| ||||r| nd| |||| ||||||| |||||dS)aApply attention with paged KV cache using XQA kernel. Parameters ---------- q : torch.Tensor Query tensor with shape ``[batch_size, beam_width, num_q_heads, head_dim]`` if not using speculative decoding, or ``[batch_size, beam_width, q_seq_len, num_q_heads, head_dim]`` if using speculative decoding. ``q_seq_len`` is the number of speculative decoding tokens. Data type should be torch.float16 or torch.bfloat16. Now only beam_width 1 is supported. k_cache: torch.Tensor Paged K cache tensor with shape ``[num_pages, page_size, num_kv_heads, head_dim]`` if :attr:`kv_layout` is ``NHD``, or ``[num_pages, num_kv_heads, page_size, head_dim]`` if :attr:`kv_layout` is ``HND``. Data type should match query tensor or be torch.float8_e4m3fn, in which case xqa will run fp8 calculation. Should be the same data type as v_cache. v_cache: torch.Tensor Paged V cache tensor with shape ``[num_pages, page_size, num_kv_heads, head_dim]`` if :attr:`kv_layout` is ``NHD``, or ``[num_pages, num_kv_heads, page_size, head_dim]`` if :attr:`kv_layout` is ``HND``. Data type should match query tensor or be torch.float8_e4m3fn, in which case xqa will run fp8 calculation. Should be the same data type as k_cache. page_table : torch.Tensor Page table tensor with shape ``batch_size, nb_pages_per_seq``. Data type should be torch.int32. K and V share the same table. seq_lens : torch.Tensor Sequence lengths tensor with shape ``[batch_size, beam_width]``. Data type should be torch.uint32. output : torch.Tensor Output tensor with shape that matches the query tensor. Data type should match query tensor or kv tensor. This tensor will be modified in-place. workspace_buffer : torch.Tensor Workspace buffer for temporary computations. Data type should be torch.uint8. semaphores : torch.Tensor Semaphore buffer for synchronization. Data type should be torch.uint32. num_kv_heads : int Number of key-value heads in the attention mechanism. page_size : int Size of each page in the paged KV cache. Must be one of [16, 32, 64, 128]. sinks : Optional[torch.Tensor], default=None Attention sink values with shape ``[num_kv_heads, head_group_ratio]``. Data type should be torch.float32. If None, no attention sinks are used. q_scale : Union[float, torch.Tensor], default=1.0 Scale factor for query tensor. kv_scale : Union[float, torch.Tensor], default=1.0 Scale factor for KV cache. sliding_win_size : int, default=0 Sliding window size for attention. If 0, no sliding window is used. kv_layout : str, default="NHD" The layout of the KV cache. Can be either ``NHD`` or ``HND``. sm_count : Optional[int], default=None Number of streaming multiprocessors to use. If None, will be inferred from the device. enable_pdl : Optional[bool], default=None Whether to enable PDL (Persistent Data Loader) optimization. If None, will be set to True if hardware supports it. rcp_out_scale : float, default=1.0 Reciprocal of output scale factor. q_seq_len : int, default=1 Query sequence length. When > 1, enables speculative decoding mode. mask : Optional[torch.Tensor], default=None Causal attention mask for speculative decoding mode (when ``q_seq_len > 1``). Shape: ``[batch_size, q_seq_len, mask_size_per_row]`` where ``mask_size_per_row = ((q_seq_len + 31) // 32) * 2``. Data type should be torch.uint16 (bit-packed format, aligned to 32 bits). Note ---- The function automatically infers several parameters from tensor shapes: - batch_size from q.shape[0] - num_q_heads from q.shape[-2] - head_dim from q.shape[-1] - input_dtype from q.dtype - kv_cache_dtype from k.dtype - head_group_ratio from num_q_heads // num_kv_heads - max_seq_len from page_table.shape[-1] * page_size Nr&K and V cache must have the same dtypez'KV cache must be fp8 when output is fp8z)Output and query must have the same dtypeHNDcudadevice TF)rY z0XQA is only supported on SM90, SM100, SM120 GPUsrz)Mask is required for speculative decoding) r rXrshapedtyper> float8_e4m3fn transposer RuntimeErrorrMrC)r,r.r/r0r2r"r#r5r(rr-r*r4r)rOr'r6r+rr7r3 num_q_headsrrnum_pages_per_seqr1rr& xqa_modules rBrCrCspJ&qx00)5;Mah;W;WJJ'"+Kwr{H#l2#(,#i/K*A- =GM ) ) )+S ) ) ) |u***} 3333 54333|qw&&&(S&&&E##B++##B++  ,,, "5#>#> ? ? Ba G G el&999::1=[PPMNNN     J1}}!L  $ NN.5A   +rDFcdt||||||tdt|dt|d|d|d|d| dd td t t tjfd tjd tjd tjdtjdtjdtdtjdtdt t tjfdtjdtjdtddffd }tdt|dt|d|d|d|d| d td t t tjfd tjd tjd tjdtjdtjdtdtjdtdt t tjfdtjdtjdtddfd}t|S)Nzflashinfer::xqa_mla_input_rrrrrr!r$r'r*r"r,r.r/r0r1r2r3r4r5r#r6r8c|t|tjrdn|t|trdn||||||||| t| tjrdn| t| trdn| | | | dSr:)xqa_wrapper_mlar=r>r?r@)r'r*r"r,r.r/r0r1r2r3r4r5r#r6rAs rBxqa_mlaz#get_xqa_module_mla..xqa_mlaas(  gu|44 ACC'w.. ;DDG       h 55 CCC8x// =DDX   !     rDcdSrFrG)r'r*r"r,r.r/r0r1r2r3r4r5r#r6s rB _fake_xqa_mlaz)get_xqa_module_mla.._fake_xqa_mlas & rD)rg) r rIr r rKrr@r>r?rJr r) rrrrrrrgrirAs @rBget_xqa_module_mlarjOs)  n  u%<[%I u uUlm{U| u uJS u u_g u u{K u uas u u3! ! uel*+!  !  < !  !  ! L! ! ,! ! u|+,! L!  ,! !  ! ! ! ! ! ! F u%<[%I u uUlm{U| u uJS u u_g u u{K u uas u u  uel*+    <      L  ,  u|+, L  ,       $    rDc | t|j} | | nt|j} |jd} |jd}d}|jd}||z}|j|jks Jdt t jdddvrtd t|j|j|||d }| | | |||||||| | ||| dS) aApply attention with paged KV cache using XQA MLA (Multi-Head Latent Attention) kernel. Parameters ---------- q : torch.Tensor Query tensor with shape ``[batch_size, beam_width, num_q_heads, head_dim]``. Data type should be torch.float8_e4m3fn. Now only beam_width 1 is supported. k_cache: torch.Tensor Paged K cache tensor with shape ``[total_num_cache_heads, head_dim]``. Data type should be torch.float8_e4m3fn v_cache: torch.Tensor Paged V cache tensor with shape ``[total_num_cache_heads, head_dim]``. Data type should be torch.float8_e4m3fn page_table : torch.Tensor Page table tensor with shape ``batch_size, nb_pages_per_seq``. Data type should be torch.int32. K and V share the same table. seq_lens : torch.Tensor Sequence lengths tensor with shape ``[batch_size, beam_width]``. Data type should be torch.uint32. output : torch.Tensor Output tensor with shape ``[batch_size, beam_width, num_q_heads, head_dim]``. Data type should be torch.bfloat16. This tensor will be modified in-place. workspace_buffer : torch.Tensor Workspace buffer for temporary computations. Data type should be torch.uint8. semaphores : torch.Tensor Semaphore buffer for synchronization. Data type should be torch.uint32. page_size : int Size of each page in the paged KV cache. Must be one of [16, 32, 64, 128]. q_scale : Union[float, torch.Tensor], default=1.0 Scale factor for query tensor. kv_scale : Union[float, torch.Tensor], default=1.0 Scale factor for KV cache. sm_count : Optional[int], default=None Number of streaming multiprocessors to use. If None, will be inferred from the device. enable_pdl : Optional[bool], default=None Whether to enable PDL (Persistent Data Loader) optimization. If None, will be set to True if hardware supports it. Note ---- The function automatically infers several parameters from tensor shapes: - batch_size from q.shape[0] - head_dim from q.shape[-1] - head_group_ratio is fixed to 128 for MLA - max_seq_len from page_table.shape[-1] * page_size NrrRrSrVrW)r[z'XQA MLA is only supported on SM120 GPUsF) r rXrr\r]rr>r`rjrg)r,r.r/r0r2r"r#r5rr*r4r'r6r3rrrbr1rcs rBrgrgs2F&qx00)5;Mah;W;WJJwr{H#(,#i/K =GM ) ) )+S ) ) )el&999::1=TIIDEEE#   J rD) Nr;r;rrNNNr;rN)F)r;r;NN) __doc__ functoolstypesrtypingrrr> api_loggingrjit.xqarr jit.utilsr utilsr r r rrcacher]rKrJrMr?r@strrCrjrgrGrDrBrws= !!!!!!"""""""" ''''''77777777...... mmKmm m  m  m+mmmmm`%)*-+."!%#')xx |x \x\x x l x L xlx xxx EL !x5%,& 'xE5<'(xxx sm!x"#x$%x&'x( 5< )x* +xxxxv  % OOOKOO O  O  OOOOd+.+."!%nn |n \n\n n l n L nln nn5%,& 'nE5<'(nsmnn nnnnnnrD