-`itddlZddlmZddlmZddlmZmZmZddl Z ddl m Z m Z m Z ddlmZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lm Z ddl!m"Z"erddl#m$Z$ddl%m&Z&ddl'm(Z(neZ$eZ&eZ(ee)Z*edZ+edZ,edZ-edZ.edZ/eeGddZ0eeGddZ1dS)N)Callablereplace) TYPE_CHECKINGAnyLiteral)Fieldfield_validatormodel_validator) dataclass) ProcessGroupReduceOp)Self)config) init_logger)vllm_is_batch_invariant)current_platform)get_open_ports_list)cuda_device_count_stateless) RuntimeEnv)PlacementGroupExecutor)linear round_robin)raympuniexternal_launcher)rrdefault)naivepplxdeepep_high_throughputdeepep_low_latencymoriallgather_reducescatterflashinfer_all2allvceZdZUdZdZeed< dZeed< eddZ eed< d Z e ed < d Z eed < d Z e ed < dZeed< eddefdZdS) EPLBConfigz6Configuration for Expert Parallel Load Balancing (EP).i window_sizei step_intervalrr genum_redundant_expertsFlog_balancednesslog_balancedness_interval use_asyncr policyaftermodereturnc|jr|jdkrtd|jr|jdkrtd|S)Nr z5Async EPLB is only supported with the default policy.rz1log_balancedness_interval must be greater than 0.)r2r3 ValueErrorr/r1selfs h/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/config/parallel.py_validate_eplb_configz EPLBConfig._validate_eplb_configWsU > VdkY66TUU U   RT%Cq%H%HPQQ Q N)__name__ __module__ __qualname____doc__r*int__annotations__r+r r.r/boolr1r2r3EPLBPolicyOptionr rr=r>r<r)r)4sA@K0M3"'qQ!7!7!73777D"d"""&'s&&&It )F (((D_'"""t#"r>r)ceZdZUdZdZeed< dZeed< dZeed< dZ eed< dZ eed< dZ eed < d Z ed zed < d Z eed < dZeed< dZeed< dZeed< dZeed< dZeed< d Zed zed< dZeed< dZeed< eeZeed< dZeed< dZeed< d Z ed zed < dZ!eed!< dZ"eed"< dZ#eed#< d$Z$eed%< d&Z%eed'< ed (Z&eed)< dZ'eed*< d Z(e)d zed+< d Z*e+d zed,< d Z,ee-ze.e/zd zed-< d.Z0eed/< d.Z1eed0< d1Z2eed2< d Z3eed3< d4Z4eed5< dZ5eed6< dZ6eed7< ed8Z7eed9< dZ8eed:< ee9Z:e9eed;< dZ;eed<< dZ< ed8Z>eed?< edd@Z?eedA< eddBCZ@eedD< eAd)dEFeBdGeCdHeDdIeCfdJZEeFdKFdIeGfdLZHeIdIefdMZJeIdIefdNZKeIdIefdOZLeIdIefdPZMdIefdQZNdIeOfdRZPeIdIefdSZQeIdIefdTZReIdIefdUZSeIdIefdVZTeUdWeOdXedIefdYZVeUdWeOdZedIefd[ZWd\ZXdad]ZYeIdIefd^ZZeFdKFdIeGfd_Z[dIeGfd`Z\d S)bParallelConfigz,Configuration for the distributed execution.r0pipeline_parallel_sizetensor_parallel_sizeprefill_context_parallel_sizedata_parallel_sizedata_parallel_size_localrdata_parallel_rankNdata_parallel_rank_localz 127.0.0.1data_parallel_master_ipinsdata_parallel_rpc_portir4cF|j|jkrtd|jd|j|j|jkr td|jd|jd|jdkr|jrtd|jrltjstd|j std |j |jzdkr td |j d |jd n-|j j d krtd|j j d|S)NzBInvalid value of `_api_process_rank`. Expected to be `-1` or `[0, z)`, but found: zdata_parallel_size_local (z!) must be <= data_parallel_size ()r0zEdata_parallel_external_lb can only be set when data_parallel_size > 1zXExpert parallelism load balancing is only supported on CUDA devices or ROCm devices now.z0enable_expert_parallel must be True to use EPLB.zZEPLB requires tensor_parallel_size or data_parallel_size to be greater than 1, but got TP=z,DP=.rz num_redundant_experts is set to zL but EPLB is not enabled. Either enable EPLB or unset num_redundant_experts.) r~r|r9rNrMrUrYr is_cuda_alikerXrKr[r.r:s r<_validate_parallel_configz(ParallelConfig._validate_parallel_config4s  !T%< < <7/3/F77"477   (4+B B BMT-JMM262IMMM   "a ' 'D,J 'W    #133  8. U !STTT(4+BBaGG T3TT9=9PTTTH5:: -'=--- r>c |j|jzS)zaworld_size_across_dp is TPxPPxDP, it is the size of the world including data parallelism.)rtrMr:s r<world_size_across_dpz#ParallelConfig.world_size_across_dpas!888r>c&|jp |jdkSNr0r`rar:s r< use_ubatchingzParallelConfig.use_ubatchinggs6$"2Q"66r>c"|jrdn|jS)Nrr:s r< num_ubatcheszParallelConfig.num_ubatchesksO9qq)99r>c|jp|jS)z Client manages local+remote EngineCores in pure internal LB case. Client manages local EngineCores in hybrid and external LB case. )rUrVr:s r<local_engines_onlyz!ParallelConfig.local_engines_onlyos -M1MMr>cv|jr|j}n|j}|xjdz c_|S)a We might need to initialize process groups in multiple processes that is related to data parallelism, e.g. both in the worker and in the engine, which can live in different processes. To avoid port conflicts, we pop a new port from the prepared port list each time we need to initialize a new process group related to data parallelism. r0)rvpoprS)r;answers r<get_next_dp_init_portz$ParallelConfig.get_next_dp_init_portwsF  / 09==??FF3F  * *a / * * r>cNddlm}ddlm}d}d}t |D]} ||j||j|jtj cS#|$r9}dt|vr!t d|}Yd}~w|d}~wwxYw|J|)Nr)DistNetworkError).stateless_init_torch_distributed_process_group)backend EADDRINUSEz1Address already in use. Retrying with a new port.)torch.distributedrvllm.distributed.utilsrrangerQrrOrMr dist_backendstrloggerwarning)r;rr max_retrieslast_exc_es r<stateless_init_dp_groupz&ParallelConfig.stateless_init_dp_groups 766666       %){##  A EE0..00++,9 $   3q66))NN#VWWW HHHHH  ###s:A  B%-BBBcN|jdvo|jo|jdko |jdkS)N)r&r!r#r$r%r0)r]rXrKrMr:s r<use_sequence_parallel_moez(ParallelConfig.use_sequence_parallel_moesJ    ,+ ,)A- ,'!+ r>c |j|jzSN)rqnnodes_within_dpr:s r<node_rank_within_dpz"ParallelConfig.node_rank_within_dps~ 555r>cN|jdkrdS|j|jz}|j|zSr)rrrMrN)r;data_parallel_node_sizes r<rzParallelConfig.nnodes_within_dps7 ;!  1  #t'D D {555r>c |j|jzSr)rtrr:s r<local_world_sizezParallelConfig.local_world_sizes$"777r>dp_grouphas_unfinishedctj|gtjd}tj|t j|t|}|S)Ncpudtypedeviceopgroup) torchtensorint32 distributed all_reducerMAXrEitem)rrraggregated_has_unfinisheds r<has_unfinished_dpz ParallelConfig.has_unfinished_dps[~.ek%PPP $$V H$MMM$($7$7!((r>kv_cache_memoryc|dkr#tjtjj}tj|gtjd}tj|tj|| S)Nr}rrr) riinfoint64maxrrrrMINr)rrrs r<sync_kv_cache_memory_sizez(ParallelConfig.sync_kv_cache_memory_sizesl b #k%+66:O/u{5QQQ $$V H$MMM{{}}r>cHhd}ddlm}m}|||}||S)a Provide a hash that uniquely identifies all the configs that affect the structure of the computation graph from input ids/embeddings to the final hidden states, excluding anything before input ids/embeddings and after the final hidden states. This hash is also used for DP worker configuration validation to prevent hangs from mismatched collective communication patterns. >rurrrqrkrorprlrhrgr~r|rOrzrnrTrRrfrVrQrPrNrUrSr_rir^rvr)get_hash_factors hash_factors)vllm.config.utilsrr)r;ignored_factorsrrfactorss r< compute_hashzParallelConfig.compute_hashsP   > EDDDDDDD""499|G$$$r>ctjdr+tdtj|_|j|jz|jz|_ |j dkr/t d|xj |j zc_ |j dks |j dkr|j dkrTttjd|j |j zz|_t d|j|jst'd |_|j|_d|jcxkr |j ks"nt-d |jd |j d nxtj|_ tj|_tj|_tj|_tj|_|j dkr|jd urt-d|j|_|j dkr)dtjd<t d|j 6|j dkr*ddl m!}d}|"}tGj$rtj%rd}ntGj&r|j'dkrd}ntGj&r@tQ|j kr)tQ}t-d|j d|d|j)dkrt dd}n.|r,|j*rd}n"ddl+m,}|rddl-m.}|rd}||_ t/d||j |j dkrd|_ |j0t1dd}|j |vr|j'dkrt-ddSdS) NVLLM_ALL2ALL_BACKENDzVLLM_ALL2ALL_BACKEND environment variable is deprecated and will be removed in v0.15.0. Please use the --all2all-backend command-line argument instead.rz2Using external launcher for distributed inference.r0rRANKz+Set data_parallel_rank to %d automatically.rzdata_parallel_rank (z) must be in the range [0, rFzDOffline data parallel mode is not supported/useful for dense models.0VLLM_ENABLE_V1_MULTIPROCESSINGz3Disabling V1 multiprocessing for external launcher. ray_utilsrrz World size (z/) is larger than the number of available GPUs (z) in this node. If this is intentional and you are using: - ray, set '--distributed-executor-backend ray'. - multiprocessing, set '--nnodes' appropriately.rzDUsing ray distributed inference because data_parallel_backend is ray)is_initialized)get_current_placement_groupz.Defaulting to use %s for distributed inferencezLmax_parallel_loading_workers is currently not supported and will be ignored.)rrrz]nnodes > 1 can only be set when distributed executor backend is mp, uni or external_launcher.)2envsis_setr warning_oncerr]rJrKrLrtriinforMrNrCosenvironrOrvrrrSr9 VLLM_DP_SIZE VLLM_DP_RANKVLLM_DP_RANK_LOCALrPVLLM_DP_MASTER_IPrQVLLM_DP_MASTER_PORTrWrzvllm.v1.executorrray_is_availableris_tpuVLLM_XLA_USE_SPMDis_cudarrrrTrhrrray.utilrdebugr^r)r;rr ray_found gpu_countray_is_initializedrallowed_backendss r< __post_init__zParallelConfig.__post_init__s ;- . . =   C    $(#>+' A+7 N7J17M7M4-1-Q-U-U-W-WD *0JJJJ43JJJJJ L4+BLL151HLLLK'+&7D #&*&7D #,0,CD )+/+AD (-1-ED *&**t/@E/I/I ) $(#:  ,0C C C;>BJ7 8 KKM N N N  , 419L9L 3 2 2 2 2 226G!2244I&(( ,T-C ,!)++ , a (** ,/11DOCC799  G4?GG'0GGG+u44 3  ,' ,#GGHHHHHH))++,HHHHHH6688,&+G07D - LLI7 S S S  , 4A9M9M05D -  , 8 NN5   >  -5E E E a;  F Er>cx|jdkp/t|jtot|jddS)Nruses_rayF)ri isinstancetypegetattrr:s r<use_rayzParallelConfig.use_rays?0E9 t8$ ? ? N9:uMM r>cHddlm}trd|_|jat |jt sGt |jtrt|j|std|jd|j rddlm }| tjs!d|_td|jdkr!d|_td |jr|j std |S) NrrTz*Unrecognized distributed executor backend zl. Supported values are 'ray', 'mp' 'uni', 'external_launcher', custom Executor subclass or its import path.rzVDisabled the custom all-reduce kernel because it is not supported on current platform.r0zBDisabled the custom all-reduce since we are running on multi-node.z;Unable to use nsight profiling unless workers run with Ray.)rrrr_rirrr issubclassr9rrassert_ray_availableruse_custom_allreducerrrrrf)r;rrs r< _verify_argszParallelConfig._verify_argssr .----- # $ $ 2-1D *  - 9t@#FF :4c t|fi|Srr)r;kwargss r<rzParallelConfig.replacest&&v&&&r>)r7N)]r?r@rArBrJrCrDrKrLrMrNrOrPrQrrRrSrTDataParallelBackendrUrErVrWrXrYr r)r[r\ExpertPlacementStrategyr]All2AllBackendr^r_r`rarcrdrerfrgrrhrriDistributedExecutorBackendrrrkrlrnrorprqrrrtrulistrvrwrxryrzr|r~r classmethodrrrr rrpropertyrrrrrr rrrrr staticmethodrrrrrrrrGr>r<rIrI`s76"#C###- !#!!!+)*!3***4G$%c%%%/*+/cDj///#.S...)"'C'''+%*s***+15.555A&+t+++-%*T)))# !%L$+$$$7#(D(((NKB#eJ???K???+9A6AAA>'@O^???K04 #*333"',t+++EJ;K &(((((+***$ 16d0C0C0C'CCC $)D(((`)-OZ$&---A-1O^d*1118 ! ((4>9D@   AJ1M3M "#"""#K""":K:Is:FCOOO:e'''J'''JD#MMM+16t1L1L1L#T#YLLL)* #)))*)* #))) ()(((  %u%000000,$eA!444444#U1444s444_:HHH9#99S999[IH9_'"""*4***#"*X9c999X9 7t777X7:c:::X:NDNNNXNs"$$$$$^  4    X  6S666X66#666X68#888X8)L)$)4)))\)L3SV\-%-%-%^zzzzx    X _'"""+d+++#"+Z'4''''''r>rI)2rcollections.abcr dataclassesrtypingrrrrpydanticr r r pydantic.dataclassesr rr rtyping_extensionsr vllm.envsrrr vllm.loggerr*vllm.model_executor.layers.batch_invariantrvllm.platformsrvllm.utils.network_utilsrvllm.utils.torch_utilsrray.runtime_envrray.util.placement_grouprrrr?rrrrrFrr)rIrGr>r<rsn $$$$$$.......... <<<<<<<<<<******44444444""""""$$$$$$######,+++++888888>>>>>>******777777)))))))JNH X  !"9:$%LMk*9% ''''''' 'T l 'l 'l 'l 'l 'l 'l ' l 'l 'l 'r>