)`i1dZddlZddlmZmZddlZddlmZddlm Z ddl m Z m Z ej dZe d d d ejd ejdejdejdejdejdededededdfdZe d d ejd ejdejdejdejdejdededededdfdZe dd d ejd ejdejdejdejdejdedededededededdfdZe dd ejd ejdejdejdejdejdedededededededdfdZe dd d ejd ejdejdejd ejdededededdfd!Ze dd ejd ejdejdejd ejdededededdfd"Ze d#d$ d%ejd&ejd'ejd(ejd)ejd ejd*ejd+ejd,ejd-ejd.ed/eded0eddfd1Ze d#d%ejd&ejd'ejd(ejd)ejd ejd*ejd+ejd,ejd-ejd.ed/eded0eddfd2Ze d3d4 d%ejd&ejd'ejd(ejd5ejd*ejd,ejd)ejd ejd6ejd7ejd8ejd9ejd:ejd;ejded?ed.ed/eded0eddf0d@Ze d3d%ejd&ejd'ejd(ejd5ejd*ejd,ejd)ejd ejd6ejd7ejd8ejd9ejd:ejd;ejded?ed.ed/eded0eddf0dAZe dBd d ejd ejdejdejd)ejd ejdeddfdCZe dBd ejd ejdejdejdDejdEejd ejdeddfdFZe dGd d ejd ejdejdejd ejdedededededededdfdHZe dGd ejd ejdejdejd ejdedededededededdfdIZ e dkd ejd ejdejdejdeededededdfdLZ!e dkd ejd ejd ejdeededededdfdMZ"e dld ejd ejdejdejdeedededededededdfdRZ#e dld ejd ejd ejdeedededededededdfdSZ$e dkd ejd ejdejdejdeededededeejejffdTZ%e dkd ejd ejd ejdeededededeejejffdUZ&e dld ejd ejdejdejdeedededededededeejejffdVZ'e dld ejd ejd ejdeedededededededeejejffdWZ(e dmd=ejdYejdZejd[ed)ejd\edeejejffd]Z)e dmd=ejdYejdZejd[ed)ejd\eddfd^Z*e dndejdejd`ejdaejd)ejd ejd\edbeej+d.ed/ed*eejd+eejd,eejd-eejd0edeejejejejff dcZ,e dndejdejd`eejdaeejd)ejd ejd\edbeej+d.ed/ed*eejd+eejd,eejd-eejd0edeejejejejff ddZ-e dodejdejd`eejdaeejdgeejd)ejd ejdheejejfd:ejd;ejds r_rope_quantizerBsV2##rcdSr r rAs r_fake_rope_quantizerDs " Drz/flashinfer::rope_quantize_append_paged_kv_cache)r3r5k_cachev_cache ckv_cache kpe_cachev_inrErFrGrH kv_indices kv_indptr batch_indices positionskv_layout_code page_sizectt|||||||||| | | | | |||||||||dS)a%Custom operator that routes to the CUDA kernel implementation. Fuses RoPE application, FP8 quantization, and paged KV cache append into a single kernel. Converts is_neox parameter to interleave format and dispatches to the underlying CUDA kernel via the JIT-compiled module. N)r#rope_quantize_append_paged_kv_cacher7r8r9r:rIr3r5r;r,rErFrGrHrJrKrLrMrNrOr<r=rr>s r(_rope_quantize_fp8_append_paged_kv_cacherSsrV99 /rcdSr r rRs r-_fake_rope_quantize_fp8_append_paged_kv_cacherU,s 4 Drz,flashinfer::apply_rope_pos_ids_cos_sin_cachec Tt|||||||dSr )r apply_rope_pos_ids_cos_sin_cacherrrrr;r,rs r!_apply_rope_pos_ids_cos_sin_cacherYIsA66  r cos_cache sin_cachecdSr r )rrrrrZr[r,rs r&_fake_apply_rope_pos_ids_cos_sin_cacher]`s  Drz&flashinfer::apply_llama31_rope_pos_idsc ^t|||||||||| | | dSr )rapply_llama31_rope_pos_ids rrrrr,rrrrr#r$r%s r_apply_llama31_rope_pos_idsransP"00       rc dSr r r`s r _fake_apply_llama31_rope_pos_idsrcs  DrF@c d||d}t|||||||||| dS)a Apply rotary embedding to a batch of queries/keys (stored as RaggedTensor) inplace. cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)`, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. indptr : torch.Tensor Indptr tensor, shape: ``(batch_size + 1)``. offsets : torch.Tensor The relative position offsets of each query in the batch, shape: ``(batch_size)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``1``. rope_theta : float The theta value used in the rope embedding, default: ``1e4``. Examples -------- >>> import torch >>> import flashinfer >>> batch_size = 128 >>> qkv_len = 1024 >>> num_qo_heads = 32 >>> num_kv_heads = 32 >>> head_dim = 128 >>> nnz = batch_size * qkv_len >>> qkv_packed = torch.randn( >>> nnz, >>> (num_qo_heads + 2 * num_kv_heads) * head_dim, >>> dtype=torch.float16, >>> device="cuda:0", >>> ) >>> q = qkv_packed[:, : num_qo_heads * head_dim].reshape(nnz, num_qo_heads, head_dim) >>> k = qkv_packed[ ... :, num_qo_heads * head_dim : (num_qo_heads + num_kv_heads) * head_dim ... ].reshape(nnz, num_kv_heads, head_dim) >>> indptr = torch.tensor( ... [i * qkv_len for i in range(batch_size + 1)], dtype=torch.int32, device="cuda:0" >>> ) >>> offsets = torch.full((batch_size,), 10, dtype=torch.int32, device="cuda:0") >>> flashinfer.apply_rope_inplace(q, k, indptr, offsets) See Also -------- apply_rope N)sizer )rrrrrrrrs rapply_rope_inplacerhsKbVVBZZ  1aFGZZrc b||d}t||||||||| dS)aApply rotary embedding to a batch of queries/keys (stored as RaggedTensor) inplace. cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)`, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. pos_ids : torch.Tensor Position indices, shape: ``(nnz)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``1``. rope_theta : float The theta value used in the rope embedding, default: ``1e4``. See Also -------- apply_rope_pos_ids Nrf)rgr0)rrr,rrrrs rapply_rope_pos_ids_inplacerjsIhVVBZZ  1aGZZrA c ||d}t|||||||||||| t|  dS)a> Apply Llama 3.1 style rotary embedding to a batch of queries/keys (stored as RaggedTensor) inplace. cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. indptr : torch.Tensor Indptr tensor, shape: ``(batch_size + 1)``. offsets : torch.Tensor The relative position offsets of each query in the batch, shape: ``(batch_size)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``8``. rope_theta : float The theta value used in the rope embedding, default: ``5e5``. low_freq_factor : float The low frequency factor used in Llama 3.1 RoPE, default: ``1``. high_freq_factor : float The high frequency factor used in Llama 3.1 RoPE, default: ``4``. old_context_len : int The old context length used in Llama 3.1 RoPE, default: ``8192``. Examples -------- >>> import torch >>> import flashinfer >>> batch_size = 128 >>> qkv_len = 1024 >>> num_qo_heads = 32 >>> num_kv_heads = 32 >>> head_dim = 128 >>> nnz = batch_size * qkv_len >>> qkv_packed = torch.randn( >>> nnz, >>> (num_qo_heads + 2 * num_kv_heads) * head_dim, >>> dtype=torch.float16, >>> device="cuda:0", >>> ) >>> q = qkv_packed[:, : num_qo_heads * head_dim].reshape(nnz, num_qo_heads, head_dim) >>> k = qkv_packed[ ... :, num_qo_heads * head_dim : (num_qo_heads + num_kv_heads) * head_dim ... ].reshape(nnz, num_kv_heads, head_dim) >>> indptr = torch.tensor( ... [i * qkv_len for i in range(batch_size + 1)], dtype=torch.int32, device="cuda:0" >>> ) >>> offsets = torch.full((batch_size,), 10, dtype=torch.int32, device="cuda:0") >>> flashinfer.apply_llama31_rope_inplace(q, k, indptr, offsets) See Also -------- apply_llama31_rope Nrf)rgr)float) rrrrrrrrr#r$r%s rapply_llama31_rope_inplacerq4setVVBZZ      orc ||d}t|||||||||||t|  dS)a Apply Llama 3.1 style rotary embedding to a batch of queries/keys (stored as RaggedTensor) inplace. cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. pos_ids : torch.Tensor Position indices, shape: ``(nnz)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``8``. rope_theta : float The theta value used in the rope embedding, default: ``5e5``. low_freq_factor : float The low frequency factor used in Llama 3.1 RoPE, default: ``1``. high_freq_factor : float The high frequency factor used in Llama 3.1 RoPE, default: ``4``. old_context_len : int The old context length used in Llama 3.1 RoPE, default: ``8192``. See Also -------- apply_llama31_rope_pos_ids Nrf)rgrarp) rrr,rrrrr#r$r%s r"apply_llama31_rope_pos_ids_inplacerssbzVVBZZ      o     rc tj|}tj|} ||d}t|||| |||||| || fS)aE Apply rotary embedding to a batch of queries/keys (stored as RaggedTensor). cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)`, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. indptr : torch.Tensor Indptr tensor, shape: ``(batch_size + 1)``. offsets : torch.Tensor The relative position offsets of each query in the batch, shape: ``(batch_size)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``1``. rope_theta : float The theta value used in the rope embedding, default: ``1e4``. Returns ------- q_rope : torch.Tensor The rotated query tensor, shape: ``(nnz, num_q_heads, head_dim)``. k_rope : torch.Tensor The rotated key tensor, shape: ``(nnz, num_k_heads, head_dim)``. Examples -------- >>> import torch >>> import flashinfer >>> batch_size = 128 >>> qkv_len = 1024 >>> num_qo_heads = 32 >>> num_kv_heads = 32 >>> head_dim = 128 >>> nnz = batch_size * qkv_len >>> qkv_packed = torch.randn( >>> nnz, >>> (num_qo_heads + 2 * num_kv_heads) * head_dim, >>> dtype=torch.float16, >>> device="cuda:0", >>> ) >>> q = qkv_packed[:, : num_qo_heads * head_dim].reshape(nnz, num_qo_heads, head_dim) >>> k = qkv_packed[ ... :, num_qo_heads * head_dim : (num_qo_heads + num_kv_heads) * head_dim ... ].reshape(nnz, num_kv_heads, head_dim) >>> indptr = torch.tensor( ... [i * qkv_len for i in range(batch_size + 1)], dtype=torch.int32, device="cuda:0" >>> ) >>> offsets = torch.full((batch_size,), 10, dtype=torch.int32, device="cuda:0") >>> q_rope, k_rope = flashinfer.apply_rope(q, k, indptr, offsets) >>> q_rope.shape torch.Size([131072, 32, 128]) >>> k_rope.shape torch.Size([131072, 32, 128]) See Also -------- apply_rope_inplace Nrf)torch empty_likergr ) rrrrrrrrrrs rrrsvx a F  a FVVBZZ       6>rc tj|}tj|}||d}t||||||||| ||fS)aApply rotary embedding to a batch of queries/keys (stored as RaggedTensor). cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)`, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. pos_ids : torch.Tensor Position indices, shape: ``(batch_size + 1)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``1``. rope_theta : float The theta value used in the rope embedding, default: ``1e4``. Returns ------- q_rope : torch.Tensor The rotated query tensor, shape: ``(nnz, num_q_heads, head_dim)``. k_rope : torch.Tensor The rotated key tensor, shape: ``(nnz, num_k_heads, head_dim)``. See Also -------- apply_rope_inplace Nrf)rurvrgr0) rrr,rrrrrrs rr.r._skv a F  a FVVBZZ  1ffgz:z: 6>rc tj|} tj|} ||d}t||| | |||||||| t |  | | fS)aApply Llama 3.1 style rotary embedding to a batch of queries/keys (stored as RaggedTensor). cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. indptr : torch.Tensor Indptr tensor, shape: ``(batch_size + 1)``. offsets : torch.Tensor The relative position offsets of each query in the batch, shape: ``(batch_size)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. rope_scale : float The scaling factor used in the rope embedding, default: ``8``. rope_theta : float The theta value used in the rope embedding, default: ``5e5``. low_freq_factor : float The low frequency factor used in Llama 3.1 RoPE, default: ``1``. high_freq_factor : float The high frequency factor used in Llama 3.1 RoPE, default: ``4``. old_context_len : int The old context length used in Llama 3.1 RoPE, default: ``8192``. Returns ------- q_rope : torch.Tensor The rotated query tensor, shape: ``(nnz, num_q_heads, head_dim)``. k_rope : torch.Tensor The rotated key tensor, shape: ``(nnz, num_k_heads, head_dim)``. Examples -------- >>> import torch >>> import flashinfer >>> batch_size = 128 >>> qkv_len = 1024 >>> num_qo_heads = 32 >>> num_kv_heads = 32 >>> head_dim = 128 >>> nnz = batch_size * qkv_len >>> qkv_packed = torch.randn( >>> nnz, >>> (num_qo_heads + 2 * num_kv_heads) * head_dim, >>> dtype=torch.float16, >>> device="cuda:0", >>> ) >>> q = qkv_packed[:, : num_qo_heads * head_dim].reshape(nnz, num_qo_heads, head_dim) >>> k = qkv_packed[ ... :, num_qo_heads * head_dim : (num_qo_heads + num_kv_heads) * head_dim ... ].reshape(nnz, num_kv_heads, head_dim) >>> indptr = torch.tensor( ... [i * qkv_len for i in range(batch_size + 1)], dtype=torch.int32, device="cuda:0" >>> ) >>> offsets = torch.full((batch_size,), 10, dtype=torch.int32, device="cuda:0") >>> q_rope, k_rope = flashinfer.apply_llama31_rope(q, k, indptr, offsets) >>> q_rope.shape torch.Size([131072, 32, 128]) >>> k_rope.shape torch.Size([131072, 32, 128]) See Also -------- apply_llama31_rope_inplace Nrf)rurvrgr)rp) rrrrrrrrr#r$r%rrs rr'r'sJ a F  a FVVBZZ    o 6>rc tj|} tj|} ||d}t||| | |||||||t |  | | fS)a Apply Llama 3.1 style rotary embedding to a batch of queries/keys (stored as RaggedTensor). cos/sin values are computed on the fly inside the kernel. We use :attr:`indptr` to denote the start pointer of each segment in the batch, the i-th segment the query of the i-th segment is ``q[indptr[i]:indptr[i+1]]`` and the key of the i-th segment is ``k[indptr[i]:indptr[i+1]]``, the first element of :attr:`indptr` is always 0 and the last element of :attr:`indptr` is the total number of queries/keys in the batch. Please see :ref:`Ragged Tensor tutorial ` for more details about the ragged tensor. Parameters ---------- q : torch.Tensor Query ragged tensor, shape: ``(nnz, num_q_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. k : torch.Tensor Key ragged tensor, shape: ``(nnz, num_k_heads, head_dim)``, where ``nnz`` is the last element of ``indptr``. pos_ids : torch.Tensor Position indices, shape: ``(nnz)``. rotary_dim : Optional[int] The dimensions to apply RoPE, if ``None``, we apply RoPE to the entire head dimension, otherwise, we apply RoPE to the first ``rotary_dim`` dimensions, default: ``None``. interleave : bool Whether to use interleaved layout in the last dimension, default: ``False``. * If ``True``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. * If ``False``, the last dimension of the query/key tensor is not interleaved, i.e., we rotate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])`` rope_scale : float The scaling factor used in the rope embedding, default: ``8``. rope_theta : float The theta value used in the rope embedding, default: ``5e5``. low_freq_factor : float The low frequency factor used in Llama 3.1 RoPE, default: ``1``. high_freq_factor : float The high frequency factor used in Llama 3.1 RoPE, default: ``4``. old_context_len : int The old context length used in Llama 3.1 RoPE, default: ``8192``. Returns ------- q_rope : torch.Tensor The rotated query tensor, shape: ``(nnz, num_q_heads, head_dim)``. k_rope : torch.Tensor The rotated key tensor, shape: ``(nnz, num_k_heads, head_dim)``. See Also -------- apply_llama31_rope_pos_ids_inplace Nrf)rurvrgrarp) rrr,rrrrr#r$r%rrs rr_r_sF a F  a FVVBZZ    o    6>rTquerykey head_sizeis_neoxc |jtjkrtdtj|}tj|}t ||jdd|||jdd|||jdd|||jdd|||| ||fS)a Apply rotary embedding to keys and queries with precomputed cos/sin values. This is designed to be compatible with the SGL/vLLM implementation. Parameters ---------- positions : torch.Tensor Position indices, shape: ``(nnz)``. query : torch.Tensor Query tensor, shape: ``(nnz, num_q_heads * head_size)``. key : torch.Tensor Key tensor, shape: ``(nnz, num_k_heads * head_size)``. cos_sin_cache : torch.Tensor Cosine and Sine cache tensor, shape: ``(max_seq_len, rotary_dim)``. Cosine is the first half and Sine is the second half on rotary_dim. is_neox : bool Whether to use Neox style RoPE, default: ``True``. * If ``True``, the last dimension of the query/key tensor is not interleaved, i.e., we rorate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. * If ``False``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. Returns ------- query_out : torch.Tensor The rotated query tensor, shape: ``(nnz, num_q_heads * head_size)``. key_out : torch.Tensor The rotated key tensor, shape: ``(nnz, num_k_heads * head_size)``. Note ---- The rotary dimension is determined by the cosine cache and sine cache. cos_sin_cache should be float32rrfrX)dtyperufloat32 ValueErrorrvrYviewshape)rMrzr{r|r;r} query_outkey_outs rapply_rope_with_cos_sin_cacherwsZem++:;;; ''Is##G% **U[^R 3 3 ((39Q<Y / /~~ioa0"i@@||GM!,b)<<#K g rc ||jtjkrtdt ||jdd|||jdd|||jdd|||jdd|||| dS)a Apply rotary embedding to keys and queries with precomputed cos/sin values. This is designed to be compatible with the SGL/vLLM implementation. The result is inplace applied to the input tensors. Parameters ---------- positions : torch.Tensor Position indices, shape: ``(nnz)``. query : torch.Tensor Query tensor, shape: ``(nnz, num_q_heads * head_size)``. key : torch.Tensor Key tensor, shape: ``(nnz, num_k_heads * head_size)``. cos_sin_cache : torch.Tensor Cosine and Sine cache tensor, shape: ``(max_seq_len, rotary_dim)``. Cosine is the first half and Sine is the second half on rotary_dim. is_neox : bool Whether to use Neox style RoPE, default: ``True``. * If ``True``, the last dimension of the query/key tensor is not interleaved, i.e., we rorate the first half dimensions ``([..., :head_dim//2])`` and the second half dimensions ``([..., head_dim//2:])``. * If ``False``, the last dimension of the query/key tensor is interleaved, i.e., we rotate the even dimensions ``([..., ::2])`` and odd dimensions ``([..., 1::2])``. Note ---- The rotary dimension is determined by the cosine cache and sine cache. rrrfrXN)rrurrrYrr)rMrzr{r|r;r}s r%apply_rope_with_cos_sin_cache_inplacersLem++:;;;& **U[^R 3 3 ((39Q<Y / /zz%+a."i88xx ! b)44#Kr?q_nopek_nopequantize_dtypec<t|||||||||| | | | | |Sr )rope_quantize_fp8)rrrrr;r,r}rr<r=r3r4r5r6r>s rmla_rope_quantize_fp8rsC$   rc|jtjkrtd|jd}|jd}|jdk}|rdn |jd}|#tj||d|j|j}|H|r#tj|d|j|j}n#tj||d|j|j}| | | | | fD] }| |j}n tj}| | ntj ||} | | ntj ||} | | ntj ||} | | ntj ||} t||||||| | | | || | || | | | fS)aI Apply RoPE (Rotary Positional Embeddings) and quantize to FP8 format. This function takes pre-split query/key tensors (rotary and non-rotary dimensions separated), applies RoPE to the rotary dimension tensors, and quantizes both rotary and non-rotary tensors to FP8 format. Supports MLA, GQA, and MHA architectures. Parameters ---------- q_rope : torch.Tensor Query tensor (rotary dimensions), shape: ``(nnz, num_qo_heads, rope_dim)``. Must be float16 or bfloat16. k_rope : torch.Tensor Key tensor (rotary dimensions). For GQA/MHA: ``(nnz, num_kv_heads, rope_dim)``. For MLA: ``(nnz, rope_dim)``. Must be float16 or bfloat16. q_nope : Optional[torch.Tensor] Query tensor (non-rotary dimensions), shape: ``(nnz, num_qo_heads, no_rope_dim)``. If ``None``, treated as zero-dim: a size-0 tensor will be created internally. k_nope : Optional[torch.Tensor] Key tensor (non-rotary dimensions). For GQA/MHA: ``(nnz, num_kv_heads, no_rope_dim)``. For MLA: ``(nnz, no_rope_dim)``. If ``None``, treated as zero-dim and created internally. cos_sin_cache : torch.Tensor Precomputed cosine and sine values, shape: ``(max_seq_len, rope_dim)``. First half contains cosine values, second half contains sine values. Must be float32. pos_ids : torch.Tensor Position indices for each token, shape: ``(nnz,)``. is_neox : bool RoPE layout style. If ``True`` (default), use non-interleaved layout (first/second half). If ``False``, use interleaved layout (even/odd dimensions). quantize_dtype : Optional[torch.dtype] Target quantization dtype. If ``None``, inferred from output tensors or defaults to ``torch.float8_e4m3fn``. Must be ``torch.float8_e4m3fn`` or ``torch.float8_e5m2``. quant_scale_q : float Quantization scaling factor for query tensors, default: ``1.0``. quant_scale_kv : float Quantization scaling factor for key tensors, default: ``1.0``. q_rope_out : Optional[torch.Tensor] Pre-allocated output tensor for quantized query (rotary). If ``None``, allocated automatically. k_rope_out : Optional[torch.Tensor] Pre-allocated output tensor for quantized key (rotary). If ``None``, allocated automatically. q_nope_out : Optional[torch.Tensor] Pre-allocated output tensor for quantized query (non-rotary). If ``None``, allocated automatically. k_nope_out : Optional[torch.Tensor] Pre-allocated output tensor for quantized key (non-rotary). If ``None``, allocated automatically. enable_pdl : bool Whether to enable PDL (Programmatic Dependent Launch). Default: ``False``. Returns ------- Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor] Quantized tensors: (q_rope_out, k_rope_out, q_nope_out, k_nope_out). rrrNrdevicer) rrurrrndimemptyr float8_e4m3fnrvrB)rrrrr;r,r}rr<r=r3r4r5r6r>nnz num_qo_headsis_mla num_kv_headsouts rrrsJem++:;;; ,q/C<?L [A F311FLOL ~ q V]   ~  [av|FMRRRFF[\1FLF  J C 1 1C!$#0N  !   fN ; ; ;  !   fN ; ; ;  !   fN ; ; ;  !   fN ; ; ;   " z:z 99rNHDvpaged_kv_cache kv_layoutcR|jtjkrtd|jdk}|jd}|jd}|#tj||d|j|j}|U|r#tj|d|j|j}n0|jd}tj||d|j|j}| ||j} n||j} n tj} |tj || }|tj || }|rs|jdkr| d}|jdkr| d}|"tjd|j|j}ntdt|dkrtd |\}}|r|}|}|j| ks |j| kr"td | d |jd |j|jd ks |jd kr td|jd|jdtjd| |j}tjd| |j}n|}|}|td|j| ks |j| kr"td| d|jd|j|jdks |jdkr td|jd|jdtjd| |j}tjd| |j}ddl m }||j} | } | } |}| } t!||||||||||||||| | | | |||| |||fS)axApply RoPE (Rotary Positional Embeddings), quantize to FP8, and append K/V to paged cache. This fused function applies RoPE to query/key (Q/K) rotary dimension tensors, quantizes all Q/K tensors (and V for GQA/MHA) to FP8 format, and directly appends the quantized K/V to a paged KV cache. It returns quantized Q tensors for use in attention computation. Supports MLA, GQA, and MHA architectures with automatic detection based on input tensor shapes. Parameters ---------- q_rope : torch.Tensor Query tensor (rotary dimensions), shape: ``(nnz, num_qo_heads, rope_dim)``. Must be float16 or bfloat16. k_rope : torch.Tensor Key tensor (rotary dimensions). For GQA/MHA: ``(nnz, num_kv_heads, rope_dim)``. For MLA: ``(nnz, rope_dim)``. Must be float16 or bfloat16. q_nope : torch.Tensor Query tensor (non-rotary dimensions), shape: ``(nnz, num_qo_heads, no_rope_dim)``. Must be float16 or bfloat16. k_nope : torch.Tensor Key tensor (non-rotary dimensions). For GQA/MHA: ``(nnz, num_kv_heads, no_rope_dim)``. For MLA: ``(nnz, no_rope_dim)``. Must be float16 or bfloat16. v : Optional[torch.Tensor] Value tensor for GQA/MHA: ``(nnz, num_kv_heads, head_dim)``. Must be float16 or bfloat16. For MLA: pass ``None`` (MLA does not use separate V; K non-RoPE acts as compressed KV). cos_sin_cache : torch.Tensor Precomputed cosine and sine values, shape: ``(max_seq_len, rope_dim)``. First half contains cosine values, second half contains sine values. Must be float32. pos_ids : torch.Tensor Position indices for each token, shape: ``(nnz,)``. paged_kv_cache : Tuple[torch.Tensor, torch.Tensor] For MLA: ``(ckv_cache, kpe_cache)`` where: - ckv_cache: ``(max_pages, page_size, no_rope_dim)`` in FP8 - kpe_cache: ``(max_pages, page_size, rope_dim)`` in FP8 For GQA/MHA: ``(k_cache, v_cache)`` where: - k_cache: ``(max_pages, page_size, num_kv_heads, head_dim)`` or ``(max_pages, num_kv_heads, page_size, head_dim)`` depending on layout, in FP8 - v_cache: same shape as k_cache, in FP8 kv_indices : torch.Tensor Page indices mapping, shape: ``(total_pages,)``. Typically ``torch.arange(total_pages)``. kv_indptr : torch.Tensor Page indptr array for each request, shape: ``(batch_size + 1,)``. ``kv_indptr[i]`` is the starting page index for request ``i``. batch_indices : torch.Tensor Batch index for each token, shape: ``(nnz,)``. Maps each token to its request. positions : torch.Tensor Position within each request's sequence for each token, shape: ``(nnz,)``. is_neox : bool RoPE layout style. If ``True`` (default), use non-interleaved layout (first/second half). If ``False``, use interleaved layout (even/odd dimensions). quantize_dtype : Optional[torch.dtype] Target quantization dtype. If ``None``, inferred from output tensors or defaults to ``torch.float8_e4m3fn``. Must be ``torch.float8_e4m3fn`` or ``torch.float8_e5m2``. quant_scale_q : float Quantization scaling factor for query tensors, default: ``1.0``. quant_scale_kv : float Quantization scaling factor for key/value tensors, default: ``1.0``. page_size : int Number of entries per page in the paged cache, default: ``16``. kv_layout : str Cache memory layout for GQA/MHA. Options: ``"NHD"`` (page, seq, head, dim) or ``"HND"`` (page, head, seq, dim). Default: ``"NHD"``. Ignored for MLA. q_rope_out : Optional[torch.Tensor] Pre-allocated output tensor for quantized query (rotary). If ``None``, allocated automatically. q_nope_out : Optional[torch.Tensor] Pre-allocated output tensor for quantized query (non-rotary). If ``None``, allocated automatically. enable_pdl : bool Whether to enable PDL (Programmatic Dependent Launch). Default: ``False``. Returns ------- Tuple[torch.Tensor, torch.Tensor] Quantized query tensors: (q_rope_out, q_nope_out). K/V are written directly to the paged cache and not returned. Notes ----- - Architecture detection: Automatically distinguishes MLA (2D K tensors) from GQA/MHA (3D K tensors). - MLA writes K-RoPE to ``kpe_cache`` and K-noRoPE to ``ckv_cache``; V is not used. - GQA/MHA writes full K (RoPE+noRoPE) to ``k_cache`` and V to ``v_cache``. - The ``batch_indices`` and ``positions`` tensors are typically obtained from ``flashinfer.get_batch_indices_positions()``. - Cache tensors must already be allocated in the target FP8 dtype. rrrrNrrz'MLA should not have V input (pass None)z+paged_kv_cache must be a tuple of 2 tensorsz#MLA cache dtype mismatch: expected z , got ckv=z, kpe=z;MLA cache must be 3D: (max_pages, page_size, dim), got ckv=zD, kpe=DzbGQA/MHA expects a V tensor, but got None. Only MLA uses None for V (compressed KV representation).z'GQA/MHA cache dtype mismatch: expected z, got k=z, v=rmz GQA/MHA cache must be 4D, got k=zD, v=) TensorLayout)rrurrrrrrrrv unsqueezelenutilsrvalueintrS)!rrrrrr;r,rrJrKrLrMr}rr<r=rOrr3r5r>rrrrcache_0cache_1rGrHrErFrrNs! r'rope_quantize_fp8_append_paged_kv_cachersVem++:;;;[A F ,q/C<?L ~ q V]   ~  [av|FMRRRFF!<?L[\1FLF   !'-NN  #'-NN"0N%fNCCC %fNCCC  H ;!  %%a((F ;!  %%a((F 9 AV\&-HHHAAFGG G >aFGGG%GW &O  ?n , , >0Q0QDnDD$?DD2;/DD  >Q  ).A"5"5D$>DD2;.DDD  +a~fmLLL+a~fmLLL 9K  =N * *gm~.M.M<.<< <<,3M<<  <1    1 1U7<UUglUUU K NNN K NNN $#####!),2N!%%''M I!!J I-   /4 z !!r)NFrrd)NFrkrlrrmrn)T) TNrrNNNNF) TNrrrrNNF)0__doc__ functoolstypingrrru api_loggingrjit.roperrrr cacherTensorrboolrpr r"r)r+r0r2rBrDrSrUrYr]rarcrhrjrqrsrr.r'r_rrrrrstrrr rrrs2 """""""" ''''''%%%%%%77777777 ...,;OPPP | | L L  L  \  QP4*++   |   |   L   L  L  \               ,+  4CWXXX | | L L  L  \  YX@233  |  |  L  L  L  \             43 $4CWXXX | | L L  \    YX0233   |   |   L   L  \               43  I$|$|$|$| $ < $ \ $ $ $ $ $$$$$ $$$ $N-.. | | | |  <  \                  /. &5   8|8|8|8| 8 , 8  8 8<8\8\8\8|8|8 8|8