.`it. ddlmZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z ddlmZmZddlmZmZddlmZdd lmZdd lmZdd lmZmZdd lmZeeZ Gd dZ!dej"fde#de#dede fdZ$dS))Lock) defaultdict) HTTPStatus) EngineClient) ErrorInfo ErrorResponse ModelCard ModelListModelPermission) BaseModelPathLoRAModulePath)LoadLoRAAdapterRequestUnloadLoRAAdapterRequest)sanitize_message) init_logger) LoRARequest) LoRAResolverLoRAResolverRegistry) AtomicCounterceZdZdZdddedeedeedzffdZdZ d e fd Z dd e dzd e fd Zd efd Z ddede dzd ee zfdZded ee zfdZded edzfdZded edzfdZde d e ezfdZxZS)OpenAIServingModelszShared instance to hold data about the loaded base model(s) and adapters. Handles the routes: - /v1/models - /v1/load_lora_adapter - /v1/unload_lora_adapter N) lora_modules engine_clientbase_model_pathsrct||_||_||_i|_t d|_g|_tj D].}|j tj |/tt|_|jj|_|jj|_|jj|_|jj|_|jj|_dS)Nr)super__init__rrstatic_lora_modules lora_requestsrlora_id_counterlora_resolversrget_supported_resolversappend get_resolverrrlora_resolver_lockinput_processor io_processorrenderer model_config max_model_len)selfrrrlora_resolver_name __class__s z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/entrypoints/openai/models/serving.pyrzOpenAIServingModels.__init__'s * 0#/ 57,Q//24"6"N"P"P      & &$12DEE    4?t3D3D#1A .;*3  .;!.<cK|jdS|jD]m}t|j|j}|||jd{V}t |trt|j j ndS)zALoads all static LoRA modules. Raises if any fail to loadN) lora_path lora_name)requestbase_model_name) rrpathnameload_lora_adapterr4 isinstancer ValueErrorerrormessage)r+lora load_request load_results r.init_static_lorasz%OpenAIServingModels.init_static_lorasDs  # + F, < z4OpenAIServingModels.is_base_model..Ts*OO5:+OOOOOOr/)anyr)r+rFs `r. is_base_modelz!OpenAIServingModels.is_base_modelSs)OOOO9NOOOOOOr/ lora_requestc8||jS|jdjS)a&Returns the appropriate model name depending on the availability and support of the LoRA or base model. Parameters: - lora: LoRARequest that contain a base_model_name. Returns: - str: The name of the base model or the first available model path. Nr)r2rr6)r+rJs r.rFzOpenAIServingModels.model_nameVs$  #) )$Q',,r/cKfdjD}fdjD}||t |S)zLShow available models. This includes the base model and all adaptersc ng|]1}t|jj|jt g2S))idr*root permission)r r6r* model_pathr )rD base_modelr+s r. z=OpenAIServingModels.show_available_models..esW    ?"0*+--.       r/c g|]J}t|j|j|jr|jnjdjt gKS)r)rNrOparentrP)r r2r5r4rr6r )rDr<r+s r.rSz=OpenAIServingModels.show_available_models..nsq     >Y'3t++*1-2+--.        r/)data)rrvaluesextendr )r+ model_cards lora_cardss` r.show_available_modelsz)OpenAIServingModels.show_available_modelsbs    #3        *1133     :&&&k****r/r3r4cRK|j}|j|4d{V||d{V}||cdddd{VS|j}||jvr|j|jn|jd}t||||j }|| |r||_ |j |d{Vnu#t$rh}d} tj} dt#|vrd} tj} t't#|| | cYd}~cdddd{VSd}~wwxYw||j|<t(d||d|d cdddd{VS#1d{VswxYwYdS) N)r2 lora_int_idr1 load_inplaceBadRequestErrorzNo adapter found NotFoundErrorr;err_type status_codez-Loaded new LoRA adapter: name '%s', path '%s'Success: LoRA adapter 'z' added successfully.)r2r% _check_load_lora_adapter_requestr1rr^r incrr_rIr4radd_lora Exceptionr BAD_REQUESTstr NOT_FOUNDcreate_error_responseloggerinfo) r+r3r4r2error_check_retr1r^rJe error_typerds r.r7z%OpenAIServingModels.load_lora_adapter|sY% *95' N' N' N' N' N' N' N' N$($I$I'$R$RRRRRRRO*&' N' N' N' N' N' N' N' N' N' N' N' N' N' N  )I 222"9-99)--a00  '#'#$1 L *t/A/A//R/R*/> , (11,??????????   . (4 %Q//!0J","6K,FFZ[?' N' N' N' N' N' N' N' N' N' N' N' N' N' N0 -9D y ) KK?I   NYMMMO' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' N' NsIFA2F C#"F# E-A E8E9FE.F F #F c:K|j}|j|4d{V||d{V}||cdddd{VS|j|=td|d|dcdddd{VS#1d{VswxYwYdS)NzRemoved LoRA adapter: name '%s'rez' removed successfully.)r2r%"_check_unload_lora_adapter_requestrrnro)r+r3r2rps r.unload_lora_adapterz'OpenAIServingModels.unload_lora_adapters% *95 P P P P P P P P$($K$KG$T$TTTTTTTO*& P P P P P P P P P P P P P P "9- KK99 E E EOYOOO P P P P P P P P P P P P P P P P P P P P P P P P P P P P P PsB (B  BBcK|jr|jstddtjS|js3|j|jvr%td|jddtjSdS)Nz2Both 'lora_name' and 'lora_path' must be provided.InvalidUserInputrbThe lora adapter 'z`' has already been loaded. If you want to load the adapter in place, set 'load_inplace' to True.)r2r1rmrrjr_rr+r3s r.rfz4OpenAIServingModels._check_load_lora_adapter_requests  (9 (L+&2 # (9T=O(O(O(W->,&2  tr/cK|jstddtjS|j|jvr%td|jddtjSdS)Nz:'lora_name' needs to be provided to unload a LoRA adapter.rwrbrxz' cannot be found.ra)r2rmrrjrrlrys r.rtz6OpenAIServingModels._check_unload_lora_adapter_requests  (T+&2   D$6 6 6(RW->RRR(&0  tr/r2c lK|j|4d{V||jvr|j|cdddd{VS|jj}|jd}d}|jD]}|||d{V}|d}||_ |j |d{V||j|<t d||j j|ccdddd{VS#t$r1}td||j j|Yd}~d}~wwxYw|r2t#d|dd t$j cdddd{VSt#d |d d t$j cdddd{VS#1d{VswxYwYdS)adAttempt to resolve a LoRA adapter using available resolvers. Args: lora_name: Name/identifier of the LoRA adapter Returns: LoRARequest if found and loaded successfully. ErrorResponse (404) if no resolver finds the adapter. ErrorResponse (400) if adapter(s) are found but none load. Nr]FTz.Resolved and loaded LoRA adapter '%s' using %szBFailed to load LoRA '%s' resolved by %s: %s. Trying next resolver.zLoRA adapter 'z$' was found but could not be loaded.r`rbz LoRA adapter z does not existra)r%rr)rEr rgr! resolve_lorar^rrhrnror-__name__ BaseExceptionwarningrmrrjrl)r+r2r4 unique_id found_adapterresolverrJrqs r.r|z OpenAIServingModels.resolve_loras^*953 3 3 3 3 3 3 3 D...))43 3 3 3 3 3 3 3 3 3 3 3 3 3  #/5O,0033I!M!/ ! !%-%:%:?I%V%VVVVVVV +$(M/8L,!"099,GGGGGGGGG8D*95 L%$.7 ,++33 3 3 3 3 3 3 3 3 3 3 3 3 3 4)!!!4%$.7 !!,. ,XXXX. * 6 M3 3 3 3 3 3 3 3 3 3 3 3 3 3 ^-FIFFF, * 4_3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 sHF#AF#AC>)F#> D9'D4/F#4D99%F#1F## F-0F-rC)r} __module__ __qualname____doc__rlistr r rr?boolrIrrkrFr r[rrr7rrurfrtr| __classcell__)r-s@r.rrs59 ===#=}-= >*T1 ======: < < <P4PPPP - -{T'9 -S - - - -+Y++++6NR-N-N--N@Cd -N  -N-N-N-N^P/P  PPPP -  0/  *>C>K-4O>>>>>>>>r/rr`r;rcrdr@cfttt|||jS)N)r;typecode)r:)rrrvaluerbs r.rmrm)sA $W--"      r/N)%asyncior collectionsrhttprvllm.engine.protocolr'vllm.entrypoints.openai.engine.protocolrrr r r 'vllm.entrypoints.openai.models.protocolr r $vllm.entrypoints.serve.lora.protocolrrvllm.entrypoints.utilsr vllm.loggerrvllm.lora.requestrvllm.lora.resolverrrvllm.utils.counterrr}rnrrjrkrmr/r.rs######------RQQQQQQQ433333######))))))AAAAAAAA,,,,,, X  HHHHHHHHZ&(4            r/