&`ioQ ddlZddlZddlZddlZddlmZddlmZmZm Z ddl Z ddl m Z ddl mZmZmZmZddlmZmZddlmZejeZdZd Zd Zd Zd Zd ZdZ dZ!dZ"dZ#dZ$GddZ%dee&efdee&effdZ'de&dee&effdZ(dee&effdZ)dee&efdee&effdZ*dee&efde+dee&effdZ,dee&efde+dee&e-ffdZ.dee&efdee&e&ffd Z/d!ee&e&fd"ee&e&fd#ee&ee&e&ffd$e&de-f d%Z0d!ee&e&fd"ee&e&fd#ee&ee&e&ffde e-fd&Z1d!ee&e&fd"ee&e&fd#ee&ee&e&ffd$e&de e-f d'Z2d!ee&e&fd(ee&e-fd#ee&ee&e&ffde e-fd)Z3d*e&de-fd+Z4d!ee&e&fd"ee&efd$e&dee&e-ffd,Z5dS)-N)chain)AnyDictOptional)validate_node_label_syntax)DISABLE_LAUNCH_CONFIG_CHECK_KEYDISABLE_NODE_UPDATERS_KEYFOREGROUND_NODE_LAUNCH_KEYWORKER_LIVENESS_CHECK_KEY) node_providerutils)validate_configautoscalerOptionsidleTimeoutSeconds upscalingMode AggressiveDefault Conservativez!cloud.google.com/gke-tpu-topologyz$cloud.google.com/gke-tpu-accelerator headgroupc:eZdZdZdZdZdeeeffdZ dS)AutoscalingConfigProduceraProduces an autoscaling config by reading data from the RayCluster CR. Used to fetch the autoscaling config at the beginning of each autoscaler iteration. In the context of Ray deployment on Kubernetes, the autoscaling config is an internal interface. The autoscaling config carries the strict subset of RayCluster CR data required by the autoscaler to make scaling decisions; in particular, the autoscaling config does not carry pod configuration data. This class is the only public object in this file. cNtj||_d||_dS)N) namespacez rayclusters/)r KubernetesHttpApiClientkubernetes_api_client _ray_cr_path)selfray_cluster_nameray_cluster_namespaces /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/autoscaler/_private/kuberay/autoscaling_config.py__init__z"AutoscalingConfigProducer.__init__9s8%2%J+& & & ">+;==cL|}t|}|S)N)#_fetch_ray_cr_from_k8s_with_retries&_derive_autoscaling_config_from_ray_cr)rray_crautoscaling_configs r!__call__z"AutoscalingConfigProducer.__call__?s'99;;CFKK!!r#returncDtdtdzD]} |j|jcS#t j$rL}|tkr4tdtj tn|dYd}~zd}~wwxYwt)zFetch the RayCluster CR by querying the K8s API server. Retry on HTTPError for robustness, in particular to protect autoscaler initialization. z1Failed to fetch RayCluster CR from K8s. Retrying.N) rangeMAX_RAYCLUSTER_FETCH_TRIESrgetrrequests HTTPErrorlogger exceptiontimesleepRAYCLUSTER_FETCH_RETRY_SAssertionError)ries r!r%z=AutoscalingConfigProducer._fetch_ray_cr_from_k8s_with_retriesDs q4q899 & &A &155d6GHHHHH% & & &111$$KJ78888%98888  &s=B ABBN) __name__ __module__ __qualname____doc__r"r)rstrrr%r#r!rr*s_  >>> """ T#s(^r#rr'r*ct|dd}t|d}td|D}t }|dt i}t|vr|tdz }nd}|ttkrd}nN|ttkrd}n(|ttkrd}nd}||dd t||||d |}t||S) Nmetadatarspecc3&K|] }|dV dS) max_workersNr?).0 node_types r! z9_derive_autoscaling_config_from_ray_cr..cs8%. - r#gN@g?r,iname)provider cluster_namehead_node_typeavailable_node_typesrDidle_timeout_minutesupscaling_speed)_generate_provider_config/_generate_available_node_types_from_ray_cr_specsumvalues*_generate_legacy_autoscaling_config_fieldsr/AUTOSCALER_OPTIONS_KEYIDLE_SECONDS_KEY UPSCALING_KEYUPSCALING_VALUE_CONSERVATIVEUPSCALING_VALUE_DEFAULTUPSCALING_VALUE_AGGRESSIVE_HEAD_GROUP_NAMEr) r'provider_configrLglobal_max_workerslegacy_autoscaling_fieldsautoscaler_optionsrMrNr(s r!r&r&Zsw/z0B;0OPPOJv 2F2M2M2O2O !K L L ++,BBGG---12BCdJ"m,,0LLL    . .2I I I    . .2L L L$z*62* 4)!5+   $  &''' r#r c Fddd|tdtdtdtdiS)zGenerates the `provider` field of the autoscaling config, which carries data required to instantiate the KubeRay node provider. typekuberayrTF)r rr r )r s r!rOrOs-  *!4'"D!5  r#c igdggggggid S)zEGenerates legacy autoscaling config fields required for compatibiliy.F) file_mountscluster_synced_filesfile_mounts_sync_continuouslyinitialization_commandssetup_commandshead_setup_commandsworker_setup_commandshead_start_ray_commandsworker_start_ray_commandsauthr?r?r#r!rSrSs1 ").#%!!##%%'   r# ray_cr_speccf|d}tt|did|dDS)z[Formats autoscaler "available_node_types" field based on the Ray CR's group specs. headGroupSpecTis_headc>i|]}|dt|dS) groupNameFrp)_node_type_from_group_spec)rEworker_group_specs r! zC_generate_available_node_types_from_ray_cr_spec..sF   " k *,F!5---   r#workerGroupSpecs)rZrt)rmros r!rPrPsT  0M4]DQQQ   &11C%D     r# group_specrqc:|rdx}}n>|d|ddz}|d|ddz}t||}t|}||i||d}|t}|t ||d<|S) z/Converts CR group spec to autoscaler node type.r minReplicas numOfHostsr, maxReplicas) min_workersrD node_config resourceslabelsNidle_timeout_s)r/"_get_ray_resources_from_group_spec_get_labels_from_group_specrUfloat)rxrqr}rDrrrFrs r!rtrtsR$%% kk!/*..q2Q2QQ  /*..q2Q2QQ 2:wGGI ( 4 4F#"I ^^$455N!&+N&;&; "# r#c|di}|di}|dddddi}|rtn|d}t||||}t||||}t |||}t |||} t |||} i} t|tsJ|| d<||| d <| d |vr| | d < d |ddvrt|ddd vrzt|ddd vr_|ddd t} |ddd t} tj | | }|r d | d|d<n+tdtdtd| | | d<| || S)z Infers Ray resources from group `Resources` field, rayStartCommands, or K8s limits. The resources extracted are used in autoscaling calculations. rrayStartParamstemplaterB containersrrsCPUNGPUTPU nodeSelectorr,zTPU-z-headzPods using TPUs require both `z` and `z` node selectors. See https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/tpu.html#configuring-ray-pods-for-tpu-usage and https://cloud.google.com/kubernetes-engine/docs/how-to/tpus.memory)r/rZ _get_num_cpus _get_num_gpus_get_custom_resources _get_num_tpus _get_memory isinstanceintGKE_TPU_TOPOLOGY_LABELGKE_TPU_ACCELERATOR_LABELr tpu_node_selectors_to_typer2errorupdate)rxrqgroup_resourcesray_start_params k8s_resources group_namenum_cpusnum_gpuscustom_resource_dictnum_tpusrrtopology acceleratoraccelerator_types r!rrsg!nn["55O!~~&6;;z*62<@CGG UWXXM%,I!!*[2IJ)=*H)=*H1):_.BMRRH *:M J JF I h $ $$$ $Ie# % , , ,'Ie   j4V< < <&*Z*@*H*XXX)*%f-n=>>"*-f5nE&H%Z08H)K %?+VV  ><= 8!18889 LLS1GSSPiSSS    $ ( )*** r#c@i}|di}|d}|rtd|dd|vrLt|dtr$|d}t ||S)z Parses Ray node labels for the autoscaling config based on the following priority: 1. Top-level `labels` field in the group spec. 2. `labels` field in `rayStartParams`. rrzIgnoring labels: zi set in rayStartParams. Group labels are supported in the top-level Labels field starting in KubeRay v1.5)r/r2warningrdictr)rx labels_dictr labels_strs r!rr7sK!~~&6;;!%%h//J  V V V V   :*Z^^H-E-Et"L"L nnX.. ";/// r#rrrrcd|vr=d|vr$td|ddt|dSd|vrt|dSd|divr|dd}t|Sd|divr|dd}t|St d|d ) z{Get CPU annotation from `resources` field, ray_start_params or k8s_resources, with priority for `resources` field. rznum-cpuszs'CPU' specified in both the top-level 'resources' field and in 'rayStartParams'. Using the value from 'resources': .cpulimitsr0z6Autoscaler failed to detect `CPU` resources for group zY. Set the `--num-cpus` rayStartParam and/or the CPU resource limit for the Ray container.)r2r_round_up_k8s_quantityrr/ ValueError)rrrr cpu_quantitys r!rrPs   ) ) ) NNO5DU5KOOO   &oe&<===%%%#J/000 -##Hb11 1 1)(3E: %l333 -##J33 3 3)*5e< %l333 $>$B$BCS$T$T !$022GHHa<<#OOO 4r#quantityctj|}|tj}t |S)zRounds a Kubernetes resource quantity up to the nearest integer. Args: quantity: Resource quantity as a string in the canonical K8s form. Returns: The quantity, rounded up, as an integer. )rounding)r parse_quantityto_integral_valuedecimalROUND_UPr)rresource_decimalroundeds r!rrs:).(rrrrc>i|]\}}|v |t|Sr?)r)rEkv standard_keyss r!rvz)_get_custom_resources..s=   AqM)))!,,)))r#z#Error reading `resource` for group z. For the correct format, refer to example configuration at https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/kuberay/ray-cluster.complete.yaml.Nr,\z1Error reading `resource` rayStartParam for group ) r2rr Exceptionrreplacejsonloadsrrr>r) rrrcustom_resourcesr9resources_stringresources_jsonrrvaluers @r!rrs  * * * NNI6EIII   877      +1133          LLDjDDD    G  *** ' 4*!B$/77bAAJ~.. )T*****#//++ * *JCc3'' ' ''eS)) ) )) ) *   @  @ @ @    s0 A A; A66A;BD!! E+ E  E)6rrloggingr4 itertoolsrtypingrrrr0ray._private.label_utilsr!ray.autoscaler._private.constantsrr r r ray.autoscaler._private.kuberayr r ray.autoscaler._private.utilr getLoggerr:r2rTrUrVrYrXrWr.r6rrrZrr>r&rOrSrPboolrtrrrrrrrrrr?r#r!rs  &&&&&&&&&& A@@@@@@@888888  8 $ $,' )#-<B--------`44S>4d3PS8n4444n S T#s(^     DcN     c3h #s(^$S#X)- #s(^BRS#XR)-R #s(^RRRRjDcNtCH~2 #s(^ 38n T#s(^+,        B#s(^38nT#s(^+,c] 4!#s(^!38n!T#s(^+,! ! c] !!!!H#s(^sCx.T#s(^+,c] 2 S S    :#s(^:7;CH~:SV: #s(^::::::r#