&`iR 4ddlZddlZddlZddlZddlmZmZddlmZddl m Z m Z m Z m Z mZddlZddlmZddlmZddlmZmZmZmZmZddlmZmZmZdd lmZm Z m!Z!m"Z"m#Z#d Z$d Z%d Z&d Z'ej(ddZ)e*ej(ddZ+ej(dZ,ej(ddZ-ej(ddZ.ee-e.Z/dZ0ej1dZ2ej3e4Z5de e6e fdefdZ7de e6e fdeeeffdZ8de e6e fde e6fdZ9de e6e fdefdZ:de e6e fdefd Z;d!e6d"e efd#Zdee e6e6fe6ffd)Z?e)e/fd*e6d&e6d+e6d,e6de6f d-Z@d.e e6e fd/e6de*fd0ZAd.e e6e fd!e*de e*fd1ZBd.e e6e fd!e*fd2ZCd.e e6e fd!e*fd3ZDGd4d5eZEGd6d7eEZFGd8d9eZGdS):N)ABCabstractmethod) defaultdict)AnyDictListOptionalTuple) build_address)WORKER_LIVENESS_CHECK_KEY)NodeIDNodeIPNodeKind NodeStatusNodeType)BatchingNodeProviderNodeData ScaleRequest)NODE_KIND_HEADNODE_KIND_WORKERSTATUS_UP_TO_DATESTATUS_UPDATE_FAILEDTAG_RAY_USER_NODE_TYPEzray.io/node-typez ray.io/groupheadworkerKUBERAY_CRD_VERv1alpha1KUBERAY_REQUEST_TIMEOUT_S<RAY_HEAD_POD_NAMEKUBERNETES_SERVICE_HOSTzhttps://kubernetes.defaultKUBERNETES_SERVICE_PORT_HTTPS443 replicaIndex)minutespodreturnct|\}}t|}t|}t|}t |||||S)zpConverts a Ray pod extracted from K8s into Ray NodeData. NodeData is processed by BatchingNodeProvider. )kindtype replica_indexstatusip) kind_and_type status_tagpod_ip_replica_index_labelr)r'r*r+r-r.r,s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/autoscaler/_private/kuberay/node_provider.pynode_data_from_podr4VsZs##JD$ __F B(--M  M&R   c|dd}|ttkrtnt}|t}||fS)zmDetermine Ray node kind (head or workers) and node type (worker group name) from a Ray pod's labels. metadatalabels)KUBERAY_LABEL_KEY_KINDKUBERAY_KIND_HEADrrKUBERAY_LABEL_KEY_TYPE)r'r8r*r+s r3r/r/csL_X &F ( )-> > >   ( )D :r5cT|dd}|tdS)aReturns the replicaIndex label for a Pod in a multi-host TPU worker group. The replicaIndex label is set by the GKE TPU Ray webhook and is of the form {$WORKER_GROUP_NAME-$REPLICA_INDEX} where $REPLICA_INDEX is an integer from 0 to Replicas-1. r7r8N)getREPLICA_INDEX_KEY)r'r8s r3r2r2qs' _X &F ::' . ..r5c:|dddS)Nr-podIPzIP not yet assignedr=)r's r3r1r1{s x=  W&; < <z'_worker_group_index..s*"[r5rvworkerGroupSpecs)r=index)rprq group_namess r3_worker_group_indexr{sM&0&8&<&<=OQS&T&TK   Z ( ((r5cP|dd|dS)zExtract the maxReplicas of a worker group. If maxReplicas is unset, return None, to be interpreted as "no constraint". At time of writing, it should be impossible for maxReplicas to be unset, but it's better to handle this anyway. rvrx maxReplicasrArprJs r3_worker_group_max_replicasrs( f 0 1+ > B B= Q QQr5cR|dd|ddS)Nrvrxreplicasr%rAr~s r3_worker_group_replicasrs( f 0 1+ > B B:q Q QQr5cR|dd|ddS)Nrvrx numOfHostsr%rAr~s r3_worker_group_num_of_hostsrs) f 0 1+ > B B= .zray.io/cluster=zpods?labelSelector=z&resourceVersion=z"&resourceVersionMatch=NotOlderThanr7resourceVersionz%Fetched pod data at resource version itemsdeletionTimestampname) _getr _raycluster_get_pods_resource_versionrrrdrutilsquoter4) rresource_versionlabel_selector resource_pathpod_listfetched_resource_versionnode_data_dictr'pod_names r3 get_node_dataz!KubeRayNodeProvider.get_node_dataos  99%GD4E%G%GHH  ::<<   KKDt/@DD!%DD0@DDD   "--.S@Q.S.STT>n>>   6$46667 M 99]++#+J#78I#J  S8P S S S   G$ ? ?C #c*o55:v.H'9#'>'>N8 $ $r5 scale_requestc|||j}td|jd|jdt|||dS)zConverts the scale request generated by BatchingNodeProvider into a patch that modifies the RayCluster CR's replicas and/or workersToDelete fields. Then submits the patch to the K8s API server. z;Autoscaler is submitting the following patch to RayCluster rrN)_scale_request_to_patch_payloadrrrrrd_submit_raycluster_patch)rr patch_payloads r3submit_scale_requestz(KubeRayNodeProvider.submit_scale_requests << 4+    B  B B04 B B B     M""" %%m44444r5ct|j}|jddg}g}t |D]q\}}|didg}|r|||D](}||vr"td|ddS)rg}|D](}t|g} || )|rMt d t d |d| |d S) aReturns False iff non_terminated_nodes contains any pods in the RayCluster's workersToDelete lists. Explanation: If there are any workersToDelete which are non-terminated, we should wait for the operator to do its job and delete those pods. Therefore, we back off the autoscaler update. If, on the other hand, all of the workersToDelete have already been cleaned up, then we patch away the workersToDelete lists and return True. In the future, we may consider having the operator clean up workersToDelete on it own: https://github.com/ray-project/kuberay/issues/733 Note (Dmitri): It is stylistically bad that this function has a side effect. rvrx scaleStrategyrNz&Waiting for operator to remove worker rF)rKzCleaning up workers to delete.zSubmitting patch T) setrkeysrr= enumerateappendrwarningrSrr) rnode_set worker_groupsnon_empty_worker_group_indicesrJ worker_grouprNrrrs r3 safe_to_scalez!KubeRayNodeProvider.safe_to_scales&t*//1122(0445GLL *,&)2=)A)A ! ! %K*..CCGG!2O C.55kBBB) ! !X%%NN#UF#U#U#UVVV 555 & ! 9 ( (K' rJJJE   ' ' ' '  9 KK8 9 9 9 KK| D]:\}} t||}t|| }| |;|S)zEConverts autoscaler scale request into a RayCluster CR patch payload.NzAutoscaler attempted to create z&more than maxReplicas pods of type {}.)desired_num_workersrr{rrrrmrrVrrlistrK node_tagsrrS) rrrpr node_typerTrJgroup_max_replicasrdeletion_groupsrrKs r3rz3KubeRayNodeProvider._scale_request_to_patch_payloads *7*K*Q*Q*S*S ( ( &I-j)DDK!;J !T!T !-2D2V2V5>EEiPPQ#5"8["Q"QQQ(oFFE   ' ' ' '&d++#5 6 6Fv../EFI I & - -f 5 5 5 5,;,A,A,C,C ( ( (I(-j)DDK' 5FGGE   ' ' ' 'r5rcfd|j}|||dS)z*Submits a patch to modify a RayCluster CR.zrayclusters/{}N)rmr_patch)rrrQs r3rz,KubeRayNodeProvider._submit_raycluster_patchs2&&t'899 D-(((((r5rQc6|j|S)r)rr=rs r3rzKubeRayNodeProvider._gets"&&t,,,r5rc8|j||S)r)rrrs r3rzKubeRayNodeProvider._patchs"((w777r5N)rrrrrrrr rrrrboolrrrrrrrrZr5r3rr^sKc3hKKKKK .tFH$45....`5,5555$2t2222h7C7777#)#7;CH~# d38n ####J)d4S>6J)))) --c3h----838d38n)=8$sCx.888888r5r)Hrrloggingosabcrr collectionsrtypingrrrr r rray._common.network_utilsr !ray.autoscaler._private.constantsr ray.autoscaler._private.utilr rrrr%ray.autoscaler.batching_node_providerrrrray.autoscaler.tagsrrrrrr9r;r:KUBERAY_KIND_WORKERgetenvrintrr r!KUBERNETES_SERVICE_PORTKUBERNETES_HOSTr> timedeltar getLoggerrrrr4r/r2r1r0rSrVrPrcror{rrrrrrrZr5r3rsu  ##############33333333333333333333GGGGGGWWWWWWWWWWWWWW ,( ")-z::C  *Er J JKKBI122 $");$")$CUKK- 79PQQ")x)!4440  8 $ $ DcN x     tCH~ %(0B*C    /d38n/#////=S#X=6====4DcN4z44442&S&T&\&&&& &c&C&&&& ;;C;DcN;;;;%S#X 34, /* "Q"Q"Q "Q"Q "Q  "Q"Q"Q"QJ)DcN)))))) RS#X R-0 R c] R R R RRtCH~RCRRRR T4S>TTTTT      s   $KKKKK6KKK\A8A8A8A8A8.A8A8A8A8A8r5