.`iv=dZddlmZmZddlmZddlZddlmZm Z erddl m Z Gdd eZ Gd d e Z dS) z Based on: Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., & Krishnamurthy, A. (2023). Punica: Multi-Tenant LoRA Serving. https://arxiv.org/abs/2310.18547 )ABCabstractmethod) TYPE_CHECKINGN) compute_metaconvert_mapping) LoRAMappingc eZdZdZedddeedzdededdf d Zed ee j d fe j zd e j d ee j d fde de j dzf dZ e dd e j d ee j d fe j zdee j d fdeed fdede j dzf dZ e dd e j d e j de j dede j dzf dZeddd e j d e j d ee j d fdee j d fde deed fdee j d fdzde j dzfdZeddd e j d e j d e j de j de j dzde j dzf dZdS)PunicaWrapperABCz PunicaWrapper ABC. mappingr lora_index_to_idN max_loras vocab_sizereturnc t)z2 Update the lora-related metadata NotImplementedErrorselfr r rrkwargss x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/lora/punica_wrapper/punica_base.pyupdate_metadataz PunicaWrapperABC.update_metadatas "!y.xlora_a_stackedscalec t)z? Performs GEMM for multiple slices of lora_a. rrrrrrrs r add_shrinkzPunicaWrapperABC.add_shrink) "!rrTlora_b_stacked output_slices offset_startc t)z> Performs GEMM for multiple slices of lora_b. rrrrr"r#r$ add_inputsrs r add_expandzPunicaWrapperABC.add_expand8s "!rr'c t)z Applies lora specifically for VocabParallelEmbeddingWithLoRA, and this layer only requires the expand operation. rrrrr"r'rs radd_lora_embeddingz#PunicaWrapperABC.add_lora_embeddingHr!rbufferr-c t)z4 Applicable to linear-related lora. r rrrrr"rr#r-rs radd_lora_linearz PunicaWrapperABC.add_lora_linearWs ""!rc t)zI Applies lora specifically for LogitsProcessorWithLoRA. rrrrrr"rr-rs radd_lora_logitsz PunicaWrapperABC.add_lora_logitsjs "!rrTT)__name__ __module__ __qualname____doc__rlistintrtupletorchTensorfloatr r(boolr+r0r3rrr r s% " "sTz* " "  "  " " "^ " " s" #el 2 " < "elC/0 "  "   " " "^ " " " < " s" #el 2 "elC/0 " S#X "  "   " " "^ "  " " < " < " "  "   " " "^ "37""" <" <"elC/0 " elC/0 "  "S#X"elC'(4/"  """^"$'+""" <" <" "  " t#"  """^"""rr ceZdZdZdededejezfdZddde ed zd ed efd Z d ej dd fdZ e deej ej ej eeeffdZe dej fdZe dej fdZe dej fdZe dej fdZddde ed zd ed efdZedeej dfej zdej deej dfdedej d zf dZe d:dej deej dfej zdeej dfdeedfd edej d zf d!Ze d;dej dej dej d"edej d zf d#Zed d$dej dej deej dfdeej dfdedeedfd%eej dfd zdej d zfd&Zed d$dej dej dej dej d%ej d zdej d zf d'Z dPunicaWrapperBasez PunicaWrapperBase is designed to manage and provide metadata for the punica kernel. The main function is to maintain the state information for Multi-LoRA, and to provide the interface for the punica. max_num_batched_tokens max_batchesdevicec tj|tj||_tj|tj||_tj|tj||_tjd|tj||_dgdz|_tj|tj||_tj|tj||_ tj|tj||_ ||_ d|_ d|_ d|_d|_d|_dS)N)dtyperFrF)r=emptylong_token_lora_indices_sampler_indices_sampler_indices_padded_embeddings_indices indices_len_seq_start_locs _seq_lengths_lora_indices_per_batchrF max_length token_nums batch_size is_prefillno_lora)rrDrErFrs r__init__zPunicaWrapperBase.__init__s.$); "%*V$ $ $  !& "%*V! ! ! (-{ "%*V( ( ( $$); %UZ$ $ $  /3VaZ${;ejQWXXX!K 5:fUUU',{ uz&( ( ( $%+   ! rr r r Nrrcd}t||||||j\}}}} } |jd|jd||jd|jd||jd|jd||jd| jdd| jdf| | |jdd<dS)Nrr) rrFrNshapecopy_rOrPrQrR) rr r rrextra_vocab_size base_indicessampler_indicessampler_indices_paddedembeddings_indicesrRs r_update_base_metadataz'PunicaWrapperBase._update_base_metadatas#       K      "    !8<#5a#8!89?? MMM 8 5a 889??PPP $%F'='CA'F%FGMM "     ) &q) )+H-?-Ea-H+H H %" # # #)rtoken_lora_tensorrczt|\}}}}}}}|jd|jd||jd|jd||jd|jd|||_||_||_||_ dS)Nr) rrSr]r^rTrUrXrVrWrZ) rreb_seq_start_tensorseq_length_tensorlora_indices_tensorrXrVrWrZs r_update_prefill_metadataz*PunicaWrapperBase._update_prefill_metadatas * + +        :17::;AABTUUU 6-3A667==>OPPP $%C':'@'C%CDJJ    %$$ rc|jd|j|jd|j|jd|j|j|j|jfS)aY This property provides a convenient way to access the necessary metadata for prefill-related kernel computations. 1. seq_start_locs: Tensor of sequence start positions. 2. seq_lengths: Tensor of sequence lengths. 3. lora_indices_per_batch: Tensor of lora indices, and an index of -1 means no lora should be applied. 4. batch_size: Batch size after clustering identical lora indices. 5. max_length: The maximum sequence length in the batch. 6. token_nums: The token numbers in the batch. N)rSrXrTrUrVrW)rs rprefill_metadataz"PunicaWrapperBase.prefill_metadatasV  !24?!2 3  // 0  ():4?): ; O O O   rc:|jd}|jd|S)z This property provides the lora indices corresponding to each token in the batch. An index of -1 means no lora should be applied. rN)rRrN)rtoken_lora_lens rtoken_lora_indicesz$PunicaWrapperBase.token_lora_indicess# )!,'88rc:|jd}|jd|S)zt This property is used to access the lora indices specifically for LogitsProcessorWithLoRA. rN)rRrO)rsampler_indices_lens rraz!PunicaWrapperBase.sampler_indicess' #.q1$%9&9%9::rc:|jd}|jd|S)zJ This property provides access to padded sampler indices. rIN)rRrP)rindices_padded_lens rrbz(PunicaWrapperBase.sampler_indices_padded s' "-a0+,?-?,?@@rcB|jd}|jddd|fS)z This property provides access to the indices used for lora embeddings, specifically for VocabParallelEmbeddingWithLoRA. N)rRrQ)rembeddings_indices_lens rrcz$PunicaWrapperBase.embeddings_indicess0 "&!1!!4'+B,B+B(BCCrc ||||||jr#||jd|_dSd|_dS)NTF)rdrYrjrors rrz!PunicaWrapperBase.update_metadatasW ""7,"!rr'c t)a Applies lora specifically for VocabParallelEmbeddingWithLoRA. and this layer only requires the expand operation. Semantics: y += x @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_b_stacked (torch.Tensor): lora_b's weights. add_inputs (bool): Default to True. rr*s rr+z$PunicaWrapperBase.add_lora_embeddinghs ,"!rr,r-c t)aB Applicable to linear-related lora. Semantics: for i in range(len(lora_a_stacked)): y[i] += ( x[i].unsqueeze(0) @ lora_a_stacked[indices[i], layer_idx, :, :] @ lora_b_stacked[indices[i], layer_idx, :, :] * scale ).squeeze(0) Args: y (torch.Tensor): Output tensor. Will be changed in-place. x (torch.Tensor): Input tensor lora_a_stacked (tuple[torch.Tensor, ...]): lora_a's weight. lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight. scale (float): Scaling factor. output_slices (tuple[int, ...]): Every slice's size. buffer (Optional[tuple[torch.Tensor, ...]]): Defaults to None. rr/s rr0z!PunicaWrapperBase.add_lora_linears F"!rc t)a Applies lora specifically for LogitsProcessorWithLoRA. Semantics: buffer = (x @ lora_a_stacked) * scale y += buffer @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_a_stacked (torch.Tensor): lora_a's weights. lora_b_stacked (torch.Tensor):lora_b's weights. scale (float): Scaling factor. buffer (Optional[torch.Tensor]):Default to None. rr2s rr3z!PunicaWrapperBase.add_lora_logitss 8"!rFtopk_ids num_tokens block_size num_expertsadapter_enabled expert_mappad_sorted_idsc t)z~ Aligns tokens and experts into block-sized chunks for LoRA-based mixture-of-experts (MoE) execution. r) rr~rrrrrrrs rmoe_lora_align_block_sizez+PunicaWrapperBase.moe_lora_align_block_sizes "!r topk_weightssorted_token_ids expert_idsnum_tokens_post_padded max_lora_rank top_k_num fully_shardedoffsetct)zj Performs a fused forward computation for LoRA of Mixture-of-Experts (MoE) layer. r)rrrrr"rrrrrr shrink_config expand_configrmul_routed_weightrrs radd_lora_fused_moez$PunicaWrapperBase.add_lora_fused_moeryrr4r5)NF)FFr)r6r7r8r9r;r=rFstrr[r:rdr>rjpropertyr<rlrorarbrcrrr?r r(r@r+r0r3rrrArrrCrC|s # ### s" ####J"*"*sTz*"* "*  "*"*"*"*H%,4* u|U\5<c3F G   X .9EL999X9;;;;X;A AAAXADELDDDXD$$sTz*$ $  $$$$"" s" #el 2" <"elC/0 "  "  """^"2"" <" s" #el 2"elC/0 " S#X "  "  """^"@  "" <" <" "  "  """^".37"""""" <"" <""elC/0 "" elC/0 ""  ""S#X""elC'(4/""  """"""^""H'+""" <" <" "  " t#"  """^"J+/$"","" "  "  ""L4'"" u|U\5<7 8""""B ##"" <" <"elC/0 " elC/0 " l " ,"L"!& """" !""#""""""rrC)r9abcrrtypingrr=utilsrrvllm.lora.layersr r rCrArrrs$####### 00000000-,,,,,,c"c"c"c"c"sc"c"c"Lq"q"q"q"q"(q"q"q"q"q"r