.`iA)UddlZddlmZmZddlZddlmZmZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#dd l$m%Z%dd l&m'Z'dd l(m)Z)m*Z*erdd l+m,Z,ddl-m.Z.ddl/m0Z0ee1Z2da3dZ4e#eeeee!e eeeeeee"eehZ5e6e7ee8d<de j9de:fdZ; d+de j9de d+dddddezis_moe_model..Ns, F FF:fh ' ' F F F F F Fr(z8MoE model detected. Using fused MoE LoRA implementation.TF)anymoduleslogger info_once)r,s r) is_moe_modelr9LsH F Femmoo F F FFFSTTTt 5r(layer max_loras lora_configpacked_modules_list model_configctD]A}|||||r&||}|||||cSB|S)N) source_layerr<r=r>)r+can_replace_layercreate_lora_weights)r:r;r<r=r>lora_clsinstance_layers r) from_layerrETs{& " "  % %# 3% &   " &Xe__N  . .y+| T T T! ! ! ! " Lr(r!lm_headr"ct||j|jj|jj|}|||||Sr0)r embedding_dimweightdtypedeviceget_sharded_to_full_mappingrB)r:rFr;r<r>rets r)from_layer_logits_processorrNisZ " ++--   CI{LAAA Jr( module_name new_modulec|d|ddd}|dd}t||||S)z1Replace a submodule in a model with a new module..N) get_submodulejoinsplitsetattr)r,rOrPparent target_names r)replace_submodulerZ{sj +*;*;C*@*@"*E!F!F G GF##C((,K FK,,, r(nameweights_mapperr#cl|dr5|dd}|r||n|}d|z}n|r||n|}|drdnd}|d}|ddkrC|dd ks |dd kr+d||d}||dd kfS|dd ks |dd kr+d||d}||dd kfSt |d )aParse the name of lora weights. args: name: the name of the fine-tuned LoRA, e.g. base_model.model.dense1.weight weights_mapper: maps the name of weight, e.g. `model.` -> `language_model.model.`, return: tuple(module_name, is_lora_a): module_name: the name of the module, e.g. model.dense1, is_lora_a whether the tensor is lora_a or lora_b. zbase_model.model.rrRrSrIlora_Alora_Blora_embedding_Alora_embedding_Bz is unsupported LoRA weight) startswithreplace _map_namerVrU ValueError)r[r\ start_indexpartsnew_names r)parse_fine_tuned_lora_namerls^& *++J||/441?I~''---T"T)1?I~''---T ':;;B!!K JJsOOE RyH%)x"7"759;P;P88E+b.122rh... Ry&&&%)7I*I*I88E+b.122r&8888 999 : ::r(c0d}||S)N)z.embed_tokens.base_layer.weightz.lm_head.base_layer.weight)endswith)r[embedding_suffixess r)is_base_embeddding_weightsrps ==+ , ,,r(ct}|D]\}}t|dd}||D]}||t |t fr.||ddt |tfr.||ddt|S)z2 In vLLM, all linear layers support LoRA. embedding_modulesNrRrS) set named_modulesgetattraddr1rrVrlist)r,supported_lora_modulesr[r3rrs r)get_supported_lora_modulesrys (+uu++-- < < f$F,?FF  () 1 1&**40000 fzm , , < " & &tzz#r': ; ; ; fxk * * < " & &tzz#r': ; ; ; & ' ''r( lora_pathctjrSdrtjStjrtjStjrddl m }m ddl m }fd}||f}d}nfd}ttf}d} |}n'#|$rt |cYSwxYw|S) a' Resolves the given lora_path to an absolute local path. If the lora_path is identified as a Hugging Face model identifier, it will download the model and return the local snapshot path. Otherwise, it treats the lora_path as a local file path and converts it to an absolute path. Parameters: lora_path (str): The path to the lora model, which can be an absolute path, a relative path, or a Hugging Face model identifier. Returns: str: The resolved absolute local path to the lora model. ~r)InvalidParametersnapshot_download) HTTPErrorcS)N)model_idr')rzr~sr)z+get_adapter_absolute_path..s//CCCr(z&Error downloading the ModelScope modelc.tjS)N)repo_id)huggingface_hubr~)rzsr)rz+get_adapter_absolute_path..so? RRRr(z'Error downloading the HuggingFace model)ospathisabsre expanduserexistsabspathr VLLM_USE_MODELSCOPE modelscope.hub.snapshot_downloadr}r~requestsrrrr7 exception)rzr}r download_fndownload_exceptions error_loglocal_snapshot_pathr~s` @r)get_adapter_absolute_pathrsR$ w}}YC  -w!!),,, w~~i  *wy)))  >XXXXXXXX&&&&&&CCCCC (*:;< SRRR -/@A= )kmm  ###  s C!DDct|rGt|x}r't|}|jsd|D|d<|St dt|S)NcJg|] \}}}}d|v |d!S)z..rR)rstrip)r2_ weight_names r) z2process_packed_modules_mapping..sC555,;1;.. &&s++...r(expertszGTo support LoRA for MoE model, 'get_expert_mapping' must be implemented)r9rr is_3d_moe_weightAttributeError)r,moe_packed_mappingpacked_modules_mappings r)process_packed_modules_mappingr sE1!7!>!> >   &@%F%F ")  550B555&y1 * ) ;  *%000r(r0)Irtypingrrrhuggingface_hub.utilsrrtorchr transformersrvllmr vllm.config.lorar vllm.loggerr vllm.lora.layersr r rrrrrrrrrrrrrrr$vllm.model_executor.layers.fused_moer!vllm.model_executor.layers.linearrvllm.model_executor.utilsrr +vllm.model_executor.layers.logits_processorr!3vllm.model_executor.layers.vocab_parallel_embeddingr" vllm.model_executor.models.utilsr#__name__r7r&r*r+rstype__annotations__Moduleboolr9intrwrErNstrrZtuplerlrpryrdictrr'r(r)rsa ********CCCCCCCC))))))''''''######&:99999888888XXXXXXXX?KKKKKKRRRRRR>>>>>> X  # &#'$-3*$!33t-./( d-1  9  #T)  Y 4-1     #T)   $ 9#&46IY<@);); );'8); 39););););X-S-T----(bi(DI((((0444444n1")1S$s)^8L111111r(