)`iL dZddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZmZejd Ze dd ejd ejd ejdeejdeejdeedeejdedeejdedeejdeejdeejeejejfffdZdS)a3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)SimpleNamespace)OptionalUnionTuple)flashinfer_api)gen_gdn_prefill_sm90_module)register_custom_opregister_fake_opc(ttdddtjdtjdtjdtjdtjd tjd t tjd t tjd t tjd t ddffd }tddtjdtjdtjdtjdtjd tjd t tjd t tjd t tjd t ddfd}t|S)Nzflashinfer::gdn_prefill)output output_state) mutates_argsr rqkv cu_seqlens initial_stategbetascalereturnc D |||||||||| dSN gdn_prefill) r rrrrrrrrrmodules j/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/gdn_prefill.pyrz+get_gdn_prefill_module..gdn_prefill"sE        c dSr) r rrrrrrrrrs r_fake_gdn_prefillz1get_gdn_prefill_module.._fake_gdn_prefill>s  rr) r build_and_loadr torchTensorrfloatr r)rr"rs @rget_gdn_prefill_moduler's ( * * 9 9 ; ;F!0J   l  <  <  <  L    -  EL ! u|$         2/00     l   <   <  <  L    -   EL !  u|$         10   { 3 3 33rFrrrrrrroutput_final_stateruse_qk_l2norm_in_kernelr rrc | Jd|ddz } |d} |d}|d}|d}t||}|}| $tj| ||f|j|j} |r-| +tj| |||ftj|j} n.|s,| *tj| |||ftj|j} t| | |||| tj |||||nd |r| | fS| S)a2 Chunked Gated Delta Rule (GDN) attention for prefill. This implements the gated delta rule linear attention mechanism for efficient training and inference. Supports both GQA (grouped query attention) and GVA (grouped value attention) configurations. Args: q (torch.Tensor): Queries of shape ``[total_seq_len, num_q_heads, head_size]``. Must be contiguous and on CUDA. k (torch.Tensor): Keys of shape ``[total_seq_len, num_k_heads, head_size]``. Must be contiguous and on CUDA. v (torch.Tensor): Values of shape ``[total_seq_len, num_v_heads, head_size]``. Must be contiguous and on CUDA. g (Optional[torch.Tensor]): Forget gate (alpha) of shape ``[total_seq_len, num_sab_heads]`` where ``num_sab_heads = max(num_q_heads, num_v_heads)``. Must be float32. If None, defaults to all ones. Default: ``None``. beta (Optional[torch.Tensor]): Update gate (beta) of shape ``[total_seq_len, num_sab_heads]``. Must be float32. If None, defaults to all ones. Default: ``None``. scale (Optional[float]): Scale factor for the attention scores. If not provided, defaults to ``1 / sqrt(head_size)``. Default: ``None``. initial_state (Optional[torch.Tensor]): Initial KV state of shape ``[num_seqs, num_sab_heads, head_size, head_size]``. Must be float32. If None, starts from zero state. Default: ``None``. output_final_state (bool): Whether to output the final state. Default: ``False``. cu_seqlens (torch.Tensor): Cumulative sequence lengths of shape ``[num_seqs + 1]``, int64. Required for variable-length sequences (varlen mode). use_qk_l2norm_in_kernel (bool): Whether to use QK L2 normalization in kernel. Default: ``False``. output (Optional[torch.Tensor]): Pre-allocated output tensor of shape ``[total_seq_len, num_o_heads, head_size]`` where ``num_o_heads = max(num_q_heads, num_v_heads)``. If None, will be allocated automatically. Default: ``None``. output_state (Optional[torch.Tensor]): Pre-allocated output state tensor of shape ``[num_seqs, num_sab_heads, head_size, head_size]``, float32. Required if ``output_final_state=True``. Default: ``None``. Returns: Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]]: - If ``output_final_state=False``: Returns output tensor of shape ``[total_seq_len, num_o_heads, head_size]``. - If ``output_final_state=True``: Returns tuple of (output, final_state) where final_state has shape ``[num_seqs, num_sab_heads, head_size, head_size]``. Note: - Supports GQA: ``num_q_heads > num_k_heads = num_v_heads`` - Supports GVA: ``num_v_heads > num_q_heads = num_k_heads`` - The final state is in k-major layout ``[N, H, K, V]``. - Requires SM90 (Hopper) architecture. Nz&cu_seqlens is required for varlen moderr)dtypedeviceg) sizemaxr$emptyr,r-float32r'rtoint64)rrrrrrrr(rr)r rnum_seqs total_seq_len num_q_heads num_v_heads head_size num_o_heads num_sab_headss rchunk_gated_delta_ruler;PsR  ! !#K ! ! !q!!A%HFF1IIM&&))K&&))Kq Ik;//KM~ K 3'8     l2{ }i ;-8     L$8{ }i ;-8   ((    ek""  "   |## r) NNNNFNFNN)__doc__ functoolstypesrtypingrrrr$ api_loggingrjit.gdnr utilsr r cacher'r%r&boolr;r!rrrEs !!!!!!)))))))))) ''''''000000  .4.4.4b !%#'!,0$)-$)%)+/yy |y |y |y  y 5< y E? yEL)yy&y"y U\ "y5<(y 5<u|U\9: :;yyyyyyr