.`iK6ddlZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl mZddlmZddlmZdd l mZdd lmZdd lmZdd lmZdd lmZerddlmZmZddlmZee Z!dZ" ddl#Z#ddl$m%Z%ddl&m'Z' ddl(m)Z)n#e*$rddl(m+Z,e,j-Z)YnwxYwGddeZ.dZ/n!#e*$rZ0dZ#e1e0Z/dZ.YdZ0[0ndZ0[0wwxYwGddeZ2de3fdZ4dZ5ddde de1fd Z6d(d"Z7d(d#Z8 d)de d$e1dzfd%Z9de:fd&Z;de:fd'Zr:len model_runner execute_model_is_intermediate_tensors isinstancer get_outputr is_last_rankreq_ids sample_tokens)r r?scheduler_outputgrammar_outputintermediate_tensorsoutputs r$execute_model_rayz"RayWorkerWrapper.execute_model_raySs^  * * , , ,;**,G***&''1,,'G .2F2F4G0 .'+$;+777[-;; "6F,,V44 @'??&"899 -**,,>>. 1"7777)>4?1??OOf&<==1#..00FMr%varscDtj|dSr')osenvironupdate)r rPs r$override_env_varsz"RayWorkerWrapper.override_env_vars}s J  d # # # # #r%c,t|tSr')rFr )r rNs r$rEz)RayWorkerWrapper._is_intermediate_tensorssf&9:: :r%)rN)__name__ __module__ __qualname____doc__rstrr)tuplelistintr7r>rrOdictrUboolrE __classcell__r#s@r$rr)s, C C 6 6 6 6 6 6      $%T#Y*? $ $ $ $ 9 9 9 ( !&'I!JMN"O(   K L N ( ( ( ( T $$sCx. $ $ $ $ ;d ; ; ; ; ; ; ; ;r%rc8eZdZdZddedzffd ZddZxZS) FutureWrapperaRA wrapper around Ray output reference to meet the interface of .execute_model(): The top level (core busy loop) expects .result() api to block and return a single output. If aggregator is provided, the outputs from all workers are aggregated upon the result() call. If not only the first worker's output is returned. N aggregatorcdt||_||_dSr')rr ref_or_refsre)r rgrer#s r$rzFutureWrapper.__init__s, &$r%ctj|j|}|j|S|j|dS)Ntimeoutr) output_rank)r+getrgre aggregate)r rjoutputss r$resultzFutureWrapper.resultsA'$*G<<< ? "N((a(@@@r%r')rWrXrYrZr rrorarbs@r$rdrdsu%%0BT0I%%%%%% AAAAAAAAr%rdrctduS)z!Returns True if Ray is available.N)r+r%r$ray_is_availablerrs d?r%cDttdtddS)z+Raise an exception if Ray is not available.NzFailed to import Ray: z+.Please install Ray with `pip install ray`.)r+ ValueErrorray_import_errrqr%r$assert_ray_availablervs6 { 9^ 9 9 9   {r%placement_grouprparallel_config device_strc |tjs Jdt|}|d}|d}tt}|D]&\}}||||'tj} | |vrtd| d|j d|d|D]T\}}t||j kr7td|j |t||||j Ud S) zVerify a given placement group has bundles located in the right place. There are 2 rules. - Warn if all tensor parallel workers cannot fit in a single node. - Fail if driver node is not included in a placement group. zDRay is not initialized although distributed-executor-backend is ray.bundles_to_node_idbundleszdriver node id z& is not included in a placement group z. Node id -> bundles z. You don't have enough GPUs available in a current node. Check `ray status` and `ray list nodes` to see if you have available GPUs in a node `{driver_node_id}` before starting an vLLM engine.aCtensor_parallel_size=%d is bigger than a reserved number of %ss (%d %ss) in a node %s. Tensor parallel workers can be spread out to 2+ nodes which can degrade the performance unless you have fast interconnect across nodes, like Infiniband. To resolve this issue, make sure you have more than %d GPUs available at each node.N)r+is_initializedrrr]itemsappendr,r-r1idrBtensor_parallel_sizeloggerwarning) rwrxrypg_databundle_to_node_idsr|node_id_to_bundle bundle_idxr4driver_node_ids r$_verify_bundlesrs    N $O44G !56i G;Ft;L;L17799?? G'"))'**=>>>>,..::< > > NN7 4G 4   r%current_placement_groupc |j}tj}|}d}tj|z tkrt j|g|\}}t |dkrna|dz}tdttj|z |tj|z tk t j |dd S#tj j $r[td|D}|dkrtd|d |d td d td |dtdd wxYw)zWait until a placement group is ready. It prints the informative log messages if the placement group is not created within time. rira6Waiting for creating a placement group of specs for %d seconds. specs=%s. Check `ray status` and `ray list nodes` to see if you have enough resources, and make sure the IP addresses used by ray cluster are the same as VLLM_HOST_IP environment variable specified in each node if you are running on a multi-node.c3BK|]}|ddVdS)GPUrN)rl).0specs r$ z'_wait_until_pg_ready.. s0 V V%!3!3 V V V V V Vr%z+Cannot provide a placement group requiring z GPUs (placement_group_specs=z ) within z seconds. Tensor parallel size may exceed available GPUs in your cluster. Check resources with `ray status` and `ray list nodes`. If running on K8s with limited GPUs, consider reducing --tensor-parallel-size to match available GPU resources.Nz:Cannot provide a placement group of placement_group_specs=z within z^ seconds. See `ray status` and `ray list nodes` to make sure the cluster has enough resources.) bundle_specstimereadyPG_WAIT_TIMEOUTr+waitrBrinfor^rl exceptionsGetTimeoutErrorsumrt)rplacement_group_specss pg_ready_ref wait_intervalr_total_gpu_requireds r$_wait_until_pg_readyrs4@ A*0022LM )++/O + +8\NMBBBq u::>>     J  a !  )++/O + +$  a(((((( > ) V V@U V V VVV  ! !L%LL*?LL#LLL   (((("(((   +s C88A/E'ctj|tj}d}tj|z tkrtj}|dS|dz}t dttj|z tj |tj|z tkdSdS)Nrrz?Waiting for removing a placement group of specs for %d seconds.) r+utilremove_placement_grouprrget_current_placement_grouprrr^sleep)rrrpgs r$_wait_until_pg_removedr'sH##$;<<< AM )++/O + + X 1 1 3 3 : E   M  a     =!!! )++/O + + + + + +r% ray_addressctddlm}|jrN|jdkrCddlm}|}|j|kr(td|j|||jtj rt dn|j s|j ra tjdnf#t$r>tdtj||j|j YnwxYwtj||j |jst%d |jd |jr|j}ntj}|rt d |j}d}|D]8}|d} | dkrt%dd| r|dz }9|j|kr!t%ddd|jd|d nIt dtjd} |j| krtdfdt5|jD} t7} tj} t=| }|ddkr"t%dd|ddd| d| d d| dd| <tj| d }t?||JtA||||_d!S)"aInitialize the distributed cluster with Ray. it will connect to the Ray cluster and create a placement group for the workers, which includes the specification of the resources for each distributed worker. Args: parallel_config: The configurations for parallel execution. ray_address: The address of the Ray cluster. If None, uses the default Ray cluster address. rr r)cuda_device_count_statelesszTensor parallel size (%d) exceeds available GPUs (%d). This may result in Ray placement group allocation failures. Consider reducing tensor_parallel_size to %d or less, or ensure your Ray cluster has %d GPUs available.z8Ray is already initialized. Skipping Ray initialization.autoz_No existing RAY instance detected. A new instance will be launched with current node resources.)addressnum_gpus runtime_env)rrzcurrent platform z does not support ray.z"Using the existing placement groupz/Placement group bundle cannot have more than 1 .zThe number of required z(s exceeds the total number of available z6s in the placement group. Required number of devices: z. Total number of devices: zANo current placement group found. Creating a new placement group.z\The number of required %ss exceeds the total number of available %ss in the placement group.cg|]}diS)g?rq)rrrys r$ z*initialize_ray_cluster..s+9 9 9 "#Z 9 9 9 r%zCurrent node has no z" available. current_node_resource=z#. vLLM engine cannot start without z . Make sure you have at least 1 z% available in a node current_node_id=z current_ip=gMbP?znode:PACK)strategyN)!rvvllm.platformsr is_cuda world_sizevllm.utils.torch_utilsrrrr+r}ris_rocmis_xpuinitConnectionErrorray_runtime_envr0rtr2rwrrrrlcluster_resourcesrangerr,r-rrr)rxrr ravailable_gpusrr|device_bundlesbundlebundle_devicesnum_devices_in_clusterr current_ipcurrent_node_idcurrent_node_resourcerys @r$initialize_ray_clusterr9s////// !!o&@1&D&DFFFFFF4466  % 6 6 NND **    S NOOOO !  ! # #S'>'7'>'@'@S  HV        NNO    H#(3+;        /2QRRRR!0J   T 0 < T T T   &I"1"A"%("F"F"H"H=6 8999*6 $ $F#ZZ A66N!! SjSSS $!#  % 6 6>*>>'1>>/>/I>>-;>>>  7  WXXX!$!6!8!8! > > NNB    9 9 9 9 ',_-G'H'H9 9 9 XX 133??AA < > > O $ $Z 3 3a 7 7IzII(IIII?III(7II;EIII :?a !5!5!56#&(":": !F#;# #  4555 " . . .+_jIII&=O###sCADDcddlm}tj}t |d}|}||zdksJ||zS)Nr)TPUAcceleratorManagerTPU)ray._private.acceleratorsrr+rr^!get_current_node_num_accelerators)rr total_tpus tpus_per_nodes r$get_num_tpu_nodesrsh??????-//&u-..J)KKMMM  % * * * *  &&r%ctj}tj}d}|rt }|D]W\}}||jkr5|dD]\}}||Xt|}|S)Nrr{) r+rrrsetr~rhexaddrB)pg_table current_pg num_nodes nodes_in_pgpg_keyrrnodes r$ get_num_nodes_in_placement_grouprsx--//H5577JI%ee "..** * *JFB**,,,,!"67==??**GAtOOD)))) $$ r%)rrr')=rRr collectionsrconcurrent.futuresrtypingrrrr. vllm.configrvllm.distributedr/vllm.distributed.kv_transfer.kv_connector.utilsr vllm.loggerr r vllm.sequencer vllm.utils.network_utilsrvllm.v1.outputsrvllm.v1.worker.worker_baservllm.v1.core.sched.outputrrrrWrrr+ray.utilrray.util.placement_grouprray._private.stater ImportErrorr_state_available_resources_per_noderruer[rdr`rrrvrrrrr^rrrqr%r$rs ######%%%%%%''''''''&&&&&&))))))NNNNNN######++++++------++++++2222228888882HHHHHHHH111111 X  nJJJ......777777LCCCCCCC LLL666666'-'K$$$ L X;X;X;X;X;,X;X;X;tNN CSVVN AAAAAFAAA,$    2%28F2TW2222j>>>>B""""(#D>D>#D>tD>D>D>D>N'3'''' #      s<5B6B B6 B"B6!B""B66C;CC