.`iB"vdZddlmZddlZddlmZddlmZmZm Z ddl m Z eGdd e Z dS) z Based on: Chen, L., Ye, Z., Wu, Y., Zhuo, D., Ceze, L., & Krishnamurthy, A. (2023). Punica: Multi-Tenant LoRA Serving. https://arxiv.org/abs/2310.18547 )finalN) LoRAMapping) bgmv_expandbgmv_expand_slice bgmv_shrink)PunicaWrapperBaseceZdZdZdededejezfdZde de edzd ed efd Z d ej d ej fdZdej d ej dej defdZdej d ej dej dededef dZdej d ej deej dfdefdZ d&dej d ej deej dfdeedfded df dZ d'dej d ej dej ded df d Zdd!dej d ej deej dfdeej dfdedeedfd"ej dzd dfd#Zed ej fd$Zdd!dej d ej dej dej d"ej dzd df d%ZdS)(PunicaWrapperXPUz PunicaWrapperXPU is designed to manage and provide metadata for the punica kernel. The main function is to maintain the state information for Multi-LoRA, and to provide the interface for the punica ipex kernel. max_num_batched_tokens max_batchesdevicec tj||||tj|jdtj|jdtj|jddS)Nrr)r __init__torch_dynamo mark_dynamic_token_lora_indices_embeddings_indices_sampler_indices_padded)selfr r rkwargss w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/lora/punica_wrapper/punica_xpu.pyrzPunicaWrapperXPU.__init__ss "4)?fUUU ""4#;Q??? ""4#;Q??? ""4#?CCCCCmappinglora_index_to_idN max_loras vocab_sizec N|j|_|||||dSN) is_prefill_update_base_metadata)rrrrrrs rupdate_metadataz PunicaWrapperXPU.update_metadata(s/", ""7,2233 R RI   q|Qy0I5 Q Q Q Q R RrrTlora_b_stacked output_slices offset_startc |}|d|jd}|jdksJ|dt |ksJt t |D]:} |||| || ||| |||| z };||dS)a2 Performs GEMM for multiple slices of lora_b. Semantics: for i in range(len(lora_b_stacked)): slice = output_slices[i] y[:, offset:offset+slice] += x[i] @ lora_b_stacked[i] offset += slice Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensors lora_b_stacked (tuple[torch.Tensor, ...]): lora_b's weight output_slices (tuple[int, ...]): Every slice's size add_inputs (bool): Defaults to True. r7rr1N)r8r9ndimr(r;r:r4view_as) rr*r$r>r?r@r1ry_orgr<s r add_expandzPunicaWrapperXPU.add_expandgs4 FF2qwr{ # #v{{{{vvayyC ......s>2233 5 5I   ) y)i(%     M)4 4LL %rc V||}t|||||dS)a] Applies lora specifically for VocabParallelEmbeddingWithLoRA. Semantics: y += x @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_b_stacked (torch.Tensor): lora_b's weights. add_inputs (bool): Default to True. N)r)r)rr*r$r>r1rr3s radd_lora_embeddingz#PunicaWrapperXPU.add_lora_embeddings5("99!<<A~q*r,r?rJrrs radd_lora_linearz PunicaWrapperXPU.add_lora_linearsD>""c.&9&9OOOOS=O=OOOOOOO >q!&&r**A[]##QVVAYY2mxF                          rc |jddS)zJ This property provides access to padded sampler indices. N)r)rs rsampler_indices_paddedz'PunicaWrapperXPU.sampler_indices_paddeds +AAA..rc |}|d|jd}|d|jd}|d} |;tj|d| ftj|j}tj|jdd|d} t|||| |t|||| d| |S)a Applies lora specifically for LogitsProcessorWithLoRA. Semantics: buffer = (x @ lora_a_stacked) * scale y += buffer @ lora_b_stacked Args: y (torch.Tensor): Output tensor. x (torch.Tensor): Input tensor. lora_a_stacked (torch.Tensor): lora_a's weights. lora_b_stacked (torch.Tensor): lora_b's weights. scale (float): Scaling factor. buffer (Optional[torch.Tensor]): Default to None. r7NrrLTrC) r8r9r(rrNrOrr'_sampler_indicesrrrE) rr*r$r5r>r,rJrrFrPsampler_indicess radd_lora_logitsz PunicaWrapperXPU.add_lora_logitss4 FF2qwr{ # # FF2qwr{ # #    # # >[!&&))Qu}QXVVVF,t'rjs ((((((NNNNNNNNNN******     (     r