)`i8,dZddlZddlmZmZmZddlZddlmZddl m Z ddl m Z m Z mZmZmZmZejdZed d d ejd ejdejdejdeejdeejdejdejdejddfdZedd dejdejdejdejdeejdeejdejdejdejdeddfdZeddejdejdejdejdeejdeejdejdejdejdeddfdZe d+d ejd!ejd"edeejdeejdeejejff d#Zdejdejd$edejfd%Zed ejd ejdejdejdeejdeejdejdejdejddfd&Ze d,dejdejdejdejd(eejeejejffdejdejdejd)eddfd*ZdS)-a3 Copyright (c) 2023 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)OptionalTupleUnion)flashinfer_api)gen_page_module) TensorLayout_check_kv_layoutcheck_shape_dtype_device_unpack_paged_kv_cacheregister_custom_opregister_fake_opcBtSN)rbuild_and_loadc/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/page.pyget_page_moduler"s    + + - --rz%flashinfer::append_paged_mla_kv_cache) ckv_cache kpe_cache) mutates_args append_ckv append_kpe batch_indices positionsrr kv_indices kv_indptrkv_last_page_lenreturnc  |}|}|}|}|}t||||||||| dSr)intrappend_paged_mla_kv_cache rrrrrrrrrs r!_append_paged_mla_kv_cache_kernelr%'s"%%''M I!!J I'++--//     rz!flashinfer::append_paged_kv_cache) paged_k_cache paged_v_cache append_key append_valuer&r'layoutc "|}|}|}|}|}t|||||||||| dSr)r"rappend_paged_kv_cache r(r)rrr&r'rrrr*s r_append_paged_kv_cache_kernelr.Hs "%%''M I!!J I'++--++     rc dSrrr-s r"_fake_append_paged_kv_cache_kernelr0ks  Dr append_indptrseq_lensnnzcV|ddz }|j}tj}|tj|f||}nt ||f||d|tj|f||}nt ||f||dddlm}||f||||d ||fS) aConvert append indptr and sequence lengths to batch indices and positions. Parameters ---------- append_indptr : torch.Tensor The indptr of the ragged tensor, shape: ``[batch_size + 1]``. seq_lens: torch.Tensor The sequence lengths of each request in the KV-Cache, shape: ``[batch_size]``. nnz : int The number of entries in the ragged tensor. batch_indices : Optional[torch.Tensor] Pre-allocated output tensor for batch_indices. If ``None``, allocated automatically. positions : Optional[torch.Tensor] Pre-allocated output tensor for positions. If ``None``, allocated automatically. Returns ------- batch_indices: torch.Tensor The batch indices of each entry in the ragged tensor, shape: ``[nnz]``. positions: torch.Tensor The positions of each entry in the ragged tensor, shape: ``[nnz]``. Example ------- >>> import torch >>> import flashinfer >>> nnz_kv = 10 >>> append_indptr = torch.tensor([0, 1, 3, 6, 10], dtype=torch.int32, device="cuda:0") >>> seq_lens = torch.tensor([5, 5, 5, 5]) >>> batch_indices, positions = flashinfer.get_batch_indices_positions(append_indptr, seq_lens, nnz_kv) >>> batch_indices tensor([0, 1, 1, 2, 2, 2, 3, 3, 3, 3], device='cuda:0', dtype=torch.int32) >>> positions # the rightmost column index of each row tensor([4, 3, 4, 2, 3, 4, 1, 2, 3, 4], device='cuda:0', dtype=torch.int32) Note ---- This function is similar to `CSR2COO `_ conversion in cuSPARSE library, with the difference that we are converting from a ragged tensor (which doesn't require a column indices array) to a COO format. See Also -------- append_paged_kv_cache rrN)devicedtyperr)"get_batch_indices_positions_kernel) num_stages)sizer5torchint32emptyr triton.pager7) r1r2r3rr batch_sizer5r6r7s rget_batch_indices_positionsr@{sj##A&&*J  !F KE SF6GGG  vWWWKvUCCC  SFE6;OOO??????5& }5x a ) ##r page_sizecftj|dd|ddz dz d|z|zS)a5Convert KV indptr and last page length to sequence lengths. Parameters ---------- kv_indptr : torch.Tensor The indptr of the paged kv-cache, shape: ``[batch_size + 1]``. kv_last_page_len : torch.Tensor The number of entries in the last page of each request in the paged kv cache, shape: ``[batch_size]``. page_size : int The size of a page in the paged kv-cache. Returns ------- seq_lens: torch.Tensor The sequence lengths of each request in the paged kv-cache, shape: ``[batch_size]``. rNr)min)r;clamp)rrrAs r get_seq_lensrFsB*  IabbMIcrcN2Q6A>>>J  rc 4t||||||||| dS)aAppend a batch of key-value pairs to a paged key-value cache, Note: current only support ckv=512 and kpe=64 Parameters ---------- append_ckv : torch.Tensor The compressed kv tensor to append in ragged tensor format, shape: ``[append_indptr[-1], ckv_dim]``. append_kpe : torch.Tensor The value tensor to append in ragged tensor format, shape: ``[append_indptr[-1], kpe_dim]``. batch_indices : torch.Tensor The batch indices of the each entry in the appended key-value pairs, shape: ``[append_indptr[-1]]``. positions : torch.Tensor The positions of the each entry in the appended key-value pairs, shape: ``[append_indptr[-1]]``. ckv_cache : cache for compressed kv, torch.Tensor, shape: [page_num, page_size, ckv_dim] kpe_cache : cache for key position embedding, torch.Tensor, shape: [page_num, page_size, kpe_dim] kv_indices : torch.Tensor The page indices of the paged kv-cache, shape: ``[kv_indptr[-1]]``. kv_indptr : torch.Tensor The indptr of the paged kv-cache, shape: ``[batch_size + 1]``. kv_last_page_len : torch.Tensor The number of entries in the last page of each request in the paged kv cache, shape: ``[batch_size]``. N)r%r$s rr#r#s<J&     rNHDpaged_kv_cache kv_layoutc t|t||||gt|||||t|jRdS)a7Append a batch of key-value pairs to a paged key-value cache. Parameters ---------- append_key : torch.Tensor The key tensor to append in ragged tensor format, shape: ``[append_indptr[-1], num_kv_heads, head_dim]``. append_value : torch.Tensor The value tensor to append in ragged tensor format, shape: ``[append_indptr[-1], num_kv_heads, head_dim]``. batch_indices : torch.Tensor The batch indices of the each entry in the appended key-value pairs, shape: ``[append_indptr[-1]]``. positions : torch.Tensor The positions of the each entry in the appended key-value pairs, shape: ``[append_indptr[-1]]``. paged_kv_cache : Union[torch.Tensor, Tuple[torch.Tensor, torch.Tensor]] The paged KV-Cache stored as a tuple of tensors or a single tensor: * a tuple ``(k_cache, v_cache)`` of 4-D tensors, each with shape: ``[max_num_pages, page_size, num_kv_heads, head_dim]`` if :attr:`kv_layout` is ``NHD``, and ``[max_num_pages, num_kv_heads, page_size, head_dim]`` if :attr:`kv_layout` is ``HND``. * a single 5-D tensor with shape: ``[max_num_pages, 2, page_size, num_kv_heads, head_dim]`` if :attr:`kv_layout` is ``NHD``, and ``[max_num_pages, 2, num_kv_heads, page_size, head_dim]`` if :attr:`kv_layout` is ``HND``. Where ``paged_kv_cache[:, 0]`` is the key-cache and ``paged_kv_cache[:, 1]`` is the value-cache. kv_indices : torch.Tensor The page indices of the paged kv-cache, shape: ``[kv_indptr[-1]]``. kv_indptr : torch.Tensor The indptr of the paged kv-cache, shape: ``[batch_size + 1]``. kv_last_page_len : torch.Tensor The number of entries in the last page of each request in the paged kv cache, shape: ``[batch_size]``. kv_layout : str The layout of the paged kv-cache, either ``NHD`` or ``HND``. Example ------- >>> import torch >>> import flashinfer >>> nnz_kv = 100 >>> num_kv_heads = 32 >>> head_dim = 128 >>> k_append = torch.randn(nnz_kv, num_kv_heads, head_dim).half().to(0) >>> v_append = torch.randn(nnz_kv, num_kv_heads, head_dim).half().to(0) >>> # 45 + 8 + 25 + 22 = nnz_kv >>> kv_append_length = torch.tensor([45, 8, 25, 22], dtype=torch.int32, device="cuda:0") >>> kv_append_indptr = torch.cat( ... [torch.zeros(1).int().to(0), torch.cumsum(kv_append_length, dim=0)] ... ).int() # [0, 45, 53, 78, 100] >>> max_num_pages = 1000 >>> page_size = 16 >>> paged_kv_cache = torch.randn(max_num_pages, 2, page_size, num_kv_heads, head_dim).half().to(0) >>> num_pages_per_req = torch.tensor([3, 1, 2, 2], dtype=torch.int32, device="cuda:0") >>> kv_page_indptr = torch.cat( ... [torch.zeros(1).int().to(0), torch.cumsum(num_pages_per_req, dim=0)] ... ).int() >>> # use first 8 pages in the paged-kv >>> kv_page_indices = torch.arange(8, dtype=torch.int32, device="cuda:0") >>> # 45 = (3 - 1) * 16 + 13 >>> # 8 = (1 - 1) * 16 + 8 >>> # 25 = (2 - 1) * 16 + 9 >>> # 22 = (2 - 1) * 16 + 6 >>> kv_last_page_len = torch.tensor([13, 8, 9, 6], dtype=torch.int32, device="cuda:0") >>> batch_indices, positions = flashinfer.get_batch_indices_positions( ... kv_append_indptr, flashinfer.get_seq_lens(kv_page_indptr, kv_last_page_len, page_size), nnz_kv ... ) >>> batch_indices tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3], device='cuda:0', dtype=torch.int32) >>> positions tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 0, 1, 2, 3, 4, 5, 6, 7, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21], device='cuda:0', dtype=torch.int32) >>> flashinfer.append_paged_kv_cache( ... k_append, ... v_append, ... batch_indices, ... positions, ... paged_kv_cache, ... kv_page_indices, ... kv_page_indptr, ... kv_last_page_len ... ) Note ---- The function assumes that the space for appended k/v has already been allocated, which means :attr:`kv_indices`, :attr:`kv_indptr`, :attr:`kv_last_page_len` has incorporated appended k/v. See Also -------- get_batch_indices_positions N)r r.r r value) r(r)rrrIrrrrJs rr,r,shY!   : :        Y%      r)NN)rH)__doc__ functoolstypingrrrr; api_loggingrjit.pagerutilsr r r r r rcacherTensorr%r"r.r0r@rFr#strr,rrrrVs5 )))))))))) ''''''%%%%%% ...++  <|   %   %  |l  :'3 ,<|  EL)  EL)  |l   >566     ,  <  |  EL)  EL)     |  l          76   -1(, G$G$<G$lG$ G$EL) G$  % G$  5< %& G$G$G$G$T|/4|HK \4. . .<.| .  % .  % . .|.l. ....b~~ ~,~<~| ~ %,elEL.H(IIJ ~  ~|~l~~ ~~~~~~r