.`iYddlZddlmZddlmZddlmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZmZdd ZGdd eZGddeZGddeZGddeZGddeZGddeZGddeZGddeZGddeZdS) N)PretrainedConfig) LoRAConfig) tensor_model_parallel_all_gather)divide)ColumnParallelLinearMergedColumnParallelLinearQKVParallelLinear)current_platform)BaseLinearLayerWithLoRA)_fully_sharded_can_replace_not_fully_sharded_can_replacelayerColumnParallelLinearWithLoRAcF|jt|jcxkr1t|jcxkrt|jksnJ|jj|j||}|d|j d}|d|j d|j }}tj |j|j d|jdj dftj |j }|j|||jd}t!js|}t%|}|j|||j|jdd}t!js|}|j|}|S)z For `ColumnParallelLinearWithLoRA` or classes that inherit from `ColumnParallelLinearWithLoRA`, they share the same `apply` logic. rdtypedeviceg?T) offset_start add_input)n_sliceslenlora_a_stackedlora_b_stacked output_slices base_layer quant_methodapplyviewshapetorchzerosfloat32rpunica_wrapper add_shrinkr can_update_inplacer add_expand)xbiasroutputout_orig_shapebuffersshrunk_buffers lora_outputs {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/lora/layers/column_parallel_linear.py _mcp_applyr2s  u# $ $ $ $ $ $ u# $ $ $ $ $ $ u" # # $ $ $ $ $ $   * 0 01A1d K KF r172;A#[[V\"-=>> NFk U%9!%<%B1%EFmxG +0*>*I*IE(#++N  . 0 0! .w77G',';'F'F   (G((K  . 0 0 V[. )F Mc eZdZdZdeddffd ZdejdejfdZdejdejfd Z d ejdeje ejejdzfzfd Z e e dd ejd edededzdef dZxZS)ra$ LoRA on top of ColumnParallelLinear layer. LoRA B is sliced for tensor parallelism. There are two types for the `base_layer`: 1. ColumnParallelLinear, e.g.`dense_h_to_4h` in `FalconForCausalLM`. 2. MergedColumnParallelLinear, e.g.`gate_up_proj` in `Phi3ForCausalLM`. rreturnNct|t|tu|_|jj|_d|_dSNr ) super__init__typeris_merged_col_linearroutput_size_per_partition output_sizerselfr __class__s r1r9z%ColumnParallelLinearWithLoRA.__init__SsK $$$%)$4$48R$R!?D r3lora_ac|SNr?rAs r1 slice_lora_az)ColumnParallelLinearWithLoRA.slice_lora_a]s r3lora_bcb|jr{|jdz}|jddz}||j|z|jdz|zddf}|||j|zz||jdz|zzddf}t j||gd}n,|j}|j|z}|jdz|z}|||ddf}|S)Nrrr dim)r;r=r"tp_rankr#cat)r?rG shard_sizeoffset left_weight right_weight start_idxend_idxs r1 slice_lora_bz)ColumnParallelLinearWithLoRA.slice_lora_b`s  $ 2)Q.J\!_)F  z)T\A-=,KKQQQNK" 22Vt|dkrdSt |dot|jdk SdS)NTr F output_sizes)r:rrrhasattrrbclsr]r^r_r`s r1can_replace_layerz.ColumnParallelLinearWithLoRA.can_replace_layers   !5 5 54   !; ; ;&''1,,u  n558 122a7 ur3rC)__name__ __module__ __qualname____doc__rr9r#TensorrFrStupler\ classmethodrnnModulerlistrboolrg __classcell__r@s@r1rrJsN#7D5<EL5<EL2#l# elEL4,??@ @####8# 15 i "  '-   $#[r3c eZdZdZdeezddffd Z ddedede dzddfd Z d e e j dzde e j dzfd Zd e e j dzde e j dzfd Zded e j e e j zd e j e e j zfdZee ddejdede de dzdef dZxZS)"MergedColumnParallelLinearWithLoRAzColumnParallelLinear layer that is composed of 2 sublayers (slices) packed together (e.g. gate_proj + up_proj -> gate_up_proj). This means we have 2 LoRAs, each applied to one half of the layer. Both slices must have the same size. rr5Nct|jj}t fd|D_t j_jfjz_ dS)Nc3BK|]}t|jVdSrC)rrX).0r=r?s r1 z>MergedColumnParallelLinearWithLoRA.__init__..s@# # 2=F; - -# # # # # # r3) r8r9rrbrmrrrrK output_ids)r?rrbr@s` r1r9z+MergedColumnParallelLinearWithLoRA.__init__s $$$3 "# # # # AM# # #   D.// .sc $ $  K0!,{     $ $ $ $ $ $ r3c 3jK|]-}tjd|jjjV.dSr)r#r$ max_lora_rankrr)ryr=r^r|r?s r1rzzIMergedColumnParallelLinearWithLoRA.create_lora_weights..sd $ $  K)!,{     $ $ $ $ $ $ r3N) r^fully_sharded_lorasrrrXrmrangerrrr)r?r|r^r`rs``` @r1create_lora_weightsz6MergedColumnParallelLinearWithLoRA.create_lora_weightss'2 AK % % 14<@@ ) $ $ $ $ $ $ $ $ 4=)) $ $ $   $ $ $ $ $ $ $  $1 $ $ $   r3rAc|SrCrDrEs r1rFz/MergedColumnParallelLinearWithLoRA.slice_lora_as  r3rGcdg|jz}tt|j|jD],\}\}}||x}|||z||dzzddf||<-|Sr7)r enumeratezipr{r)r?rG sliced_lora_bishard_idrMlora_b_is r1rSz/MergedColumnParallelLinearWithLoRA.slice_lora_bs. )2 !3 4 4* *    %A%*#1I%2#+)J(Q,,GGJ$ a r3indexc|||jdkr*||}||}t |jD]}||x}F|j||dd|jdd|jdf|d||x}F|j ||dd|jdd|jdf|ddS)Nr rT) non_blocking) reset_lorarXrFrSrrrr"copy_r)r?rrArGrlora_a_irs r1set_loraz+MergedColumnParallelLinearWithLoRA.set_loras  s r1r9z"QKVParallelLinearWithLoRA.__init__1s $$$ O +do.G G "&!:T_=V!V O (4?+D D  O .1J J  r3rGc|j|_|j|jjz|_||j|jz|j|jdzzddf}|j}|||j|jzz||j|jdzzzddf}||jz}|||j|jzz||j|jdzzzddf}tj |||gd}|S)Nr rrI) rK q_shard_idrnum_kv_head_replicas kv_shard_idrrrrr#rL)r?rGlora_b_qk_offsetlora_b_kv_offsetlora_b_vs r1rSz&QKVParallelLinearWithLoRA.slice_lora_b@s(,<4?+OO  "T_ 4t7M"8$ $ AA   ) t.1AA AH%)9A)=>E? ? AA   d55 t.1AA AH%)9A)=>E? ? AA   Hh9qAAA r3r]r^r_r`cTt|tuot|dkSr7r:r rres r1rgz+QKVParallelLinearWithLoRA.can_replace_layerW,L!!%66X3?R;S;SWX;XXr3rC)rhrirjrkr r9r#rlrSrnrrorprrqrrrrgrsrts@r1rr$s   #4       5<EL.# 15 YYiY Y" Y '- Y  YYY$#[YYYYYr3rc eZdZdZdeddffd Z d dedededzddffd Z e e d d e j ded ededzdef d ZxZS)MergedQKVParallelLinearWithLoRAaKMergedColumnParallelLinear layer that is composed of 3 sublayers (slices) packed together in qkv proj fashion (q_proj + k_proj + v_proj -> qkv_proj). This means we have 3 LoRAs, each applied to one slice of the layer. Q slice may have different shape than K and V slices (which both have the same shape). rr5Nct|t|jj|_|jj|jjz|_|jj |jjz|_ |j |_ |j |jj z|_|j|j |j f|_|j |j|jf|_dSrC)r8r9rrrbrrrrrrrKrrrrr{r>s r1r9z(MergedQKVParallelLinearWithLoRA.__init__ns $$$DO899 !%!:T_=V!V O (4?+D D ,<4?+OO  "  #  #  O     r3r|r^r`cNt|||dS)z The main reason for overloading this function is to handle inconsistent weight dimensions in qkv lora. N)r8r)r?r|r^r`r@s r1rz3MergedQKVParallelLinearWithLoRA.create_lora_weightss' ##I{LIIIIIr3r]r_cTt|tuot|dkS)Nrcrres r1rgz1MergedQKVParallelLinearWithLoRA.can_replace_layerrr3rC)rhrirjrkr r9rrrrrnrrorprqrrrgrsrts@r1rrcs. #4       615 J J J  J'- J  J J J J J J# 15 YYiY Y" Y '- Y  YYY$#[YYYYYr3rc eZdZdZdejdejfdZddejdejdzdejfdZee dd e j d e d e d edzdef fd ZxZS)#ColumnParallelLinearWithShardedLoRAz Differs from ColumnParallelLinearWithLoRA by slicing LoRA A also. Based on S-LoRA, slicing happens along the rank dim. rAr5cl|jdjd}|j|z}||||zddf}|SNrrrr"rKr?rArMrQs r1rFz0ColumnParallelLinearWithShardedLoRA.slice_lora_aE(+1!4 L:-  I $::AAA=> r3Nr*r+c$t|||SrCr2r?r*r+s r1r z)ColumnParallelLinearWithShardedLoRA.apply!T4(((r3r]r^r_r`cPt||||dSNF)r]r^r_r`decorater8rgrfr]r^r_r`r@s r1rgz5ColumnParallelLinearWithShardedLoRA.can_replace_layer5ww((%# 3% )   r3rCrhrirjrkr#rlrFr rnr rorprrqrrrrgrsrts@r1rrs5<EL ))u|)5<$+>)%,)))) 15   i   "  '-         [     r3rc eZdZdZdeejdzdeejdzfdZddejdejdzdejfdZe e dd e j d e d ed edzdef fd ZxZS))MergedColumnParallelLinearWithShardedLoRAz Differs from MergedColumnParallelLinearWithLoRA by slicing the LoRA A's also. Based on S-LoRA, slicing happens along the rank dim. rANr5c|jdjd}|j|z}|d|d|||zddfnd|d|d|||zddfndg}|S)Nrrr r)r?rAoutput_shard_sizeoutput_start_idxs r1rFz6MergedColumnParallelLinearWithShardedLoRA.slice_lora_as!/28;<*;;ay$ 1I&)9)%,)))) 15   i   "  '-         [     r3rc eZdZdZdejdejfdZddejdejdzdejfdZee dd e j d e d e d edzdef fd ZxZS) QKVParallelLinearWithShardedLoRAz Differs from QKVParallelLinearWithLoRA by slicing the LoRA A's also. Based on S-LoRA, slicing happens along the rank dim. rAr5cl|jdjd}|j|z}||||zddf}|Srrrs r1rFz-QKVParallelLinearWithShardedLoRA.slice_lora_arr3Nr*r+c$t|||SrCrrs r1r z&QKVParallelLinearWithShardedLoRA.applyrr3r]r^r_r`cPt||||dSrrrs r1rgz2QKVParallelLinearWithShardedLoRA.can_replace_layer rr3rCrrts@r1rrs5<EL ))u|)5<$+>)%,)))) 15   i   "  '-         [     r3rc eZdZdZdeejdzdeejdzfdZddejdejdzdejfdZe e dd e j d e d ed edzdef fd ZxZS)&MergedQKVParallelLinearWithShardedLoRAz Differs from MergedQKVParallelLinearWithLoRA by slicing the LoRA A's also. Based on S-LoRA, slicing happens along the rank dim. rANr5cfdtdDfdtdD}|d)|d|d|ddzddfnd|d)|d|d|ddzddfnd|d)|d|d|ddzddfndg}|S)Nc@g|]}j|jdS)r)rr")ryrr?s r1 zGMergedQKVParallelLinearWithShardedLoRA.slice_lora_a..(s)HHH!d)!,215HHHr3rcc0g|]}j|zSrD)rK)ryrr?rMs r1rzGMergedQKVParallelLinearWithShardedLoRA.slice_lora_a..)s$DDDaT\JqM1DDDr3rr r)r)r?rArQrMs` @r1rFz3MergedQKVParallelLinearWithShardedLoRA.slice_lora_a$sIHHHuQxxHHH DDDDD588DDD ay$ 1IilYq\JqM%AA111D E Eay$ 1IilYq\JqM%AA111D E Eay$ 1IilYq\JqM%AA111D E E   r3r*r+c$t|||SrCrrs r1r z,MergedQKVParallelLinearWithShardedLoRA.apply7rr3r]r^r_r`cPt||||dSrrrs r1rgz8MergedQKVParallelLinearWithShardedLoRA.can_replace_layer:rr3rCrrts@r1rrs5<$./ elT! "&))u|)5<$+>)%,)))) 15   i   "  '-         [     r3rc eZdZdZee d dejdede de dzde f dZ d e d eje ejzd eje ejzffd ZxZS)/MergedColumnParallelLinearVariableSliceWithLoRAzMergedColumnParallelLinear with variable number of slices (3+). This handles cases where the checkpoint has a single weight for the whole module (not split into slices), but the layer itself has multiple slices. Nr]r^r_r`r5ct|turdSt|dkrdSt|dkrdSt|dot|jdkS)NFrcTrrb)r:rrrdrbres r1rgzAMergedColumnParallelLinearVariableSliceWithLoRA.can_replace_layerVs   %? ? ?5 " # #q ( (4 " # #q ( (5 L. 1 1 4L-..!3 r3rrArGct||t|tjr |g|jz}t|tjr?|jj}g}d}|D]*}||z}||||ddf|}+|}t |||dS)zSOverride to handle single tensor weights that need to be split into slices.rN) r isinstancer#rlrrrbappendr8r) r?rrArGrb lora_b_listrQr=rRr@s r1rz8MergedColumnParallelLinearVariableSliceWithLoRA.set_loravs  fel + + .X -F fel + + !?7LKI+ $ $ #k1""6)G*;QQQ*>#?@@@#  F /////r3rC)rhrirjrkrnrrorprrqrrrrgrr#rlrrsrts@r1rrMs # 15   i   "  '-      $#[ <00 tEL110 tEL11 0000000000r3r)rr) r#torch.nnro transformersrvllm.config.lorarvllm.distributedrvllm.distributed.utilsr!vllm.model_executor.layers.linearrrr vllm.platformsr base_linearr utilsr rr2rrvrrrrrrrrDr3r1rs  ))))))''''''======)))))) ,+++++000000MMMMMMMM0000f`````#:```Ft t t t t )Et t t n<Y<Y<Y<Y<Y <<Y<Y<Y~7Y7Y7Y7Y7Y&H7Y7Y7Y~% % % % % *F% % % P+ + + + + 0R+ + + \! ! ! ! ! '@! ! ! H. . . . . -L. . . bE0E0E0E0E0&E0E0E0E0E0r3