&`i^|ddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z ejeZedefdZde deeeffd Zde defd Zedefd Zd Zd ZdZdZdZdZe ddedee defdZGddZ dS)N) lru_cache)DictOptionalTuple)_is_ray_cluster)Trialreturnc(tjSN)raycluster_resources/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/tune/execution/insufficient_resources_manager.py$_get_cluster_resources_no_autoscalerrs  " ""rtrialc|jjdd}|jjdd}||fS)NCPUrGPU)placement_group_factoryrequired_resourcesget)rcpugpus r_get_trial_cpu_and_gpursA  ' : > >ua H HC  ' : > >ua H HC 8Orc|jtjksJt|\}}|t ddko%|t ddkS)z^Calculates if there is enough resources for a PENDING trial. For no autoscaler case. rrr)statusrPENDINGrrr)r asked_cpus asked_gpuss r_can_fulfill_no_autoscalerr!s| <5= ( ( ( (3E::J =??CC q  M <>>BB5!LL LMrctr-ttjddSttjddS)N5TUNE_WARN_INSUFFICENT_RESOURCE_THRESHOLD_S_AUTOSCALER60*TUNE_WARN_INSUFFICENT_RESOURCE_THRESHOLD_S)rfloatosenvironrrrr-_get_insufficient_resources_warning_thresholdr)(s] Y JNNG     RZ^^$PRVWWXXXrzTraining has not started in the last {wait_time:.0f} seconds. This could be due to the cluster not having enough resources available. zYou asked for {asked_cpus} CPUs and {asked_gpus} GPUs, but the cluster only has {cluster_cpus} CPUs and {cluster_gpus} GPUs available. zStop the training and adjust the required resources (e.g. via the `ScalingConfig` or `resources_per_trial`, or `num_workers` for rllib), or add more resources to your cluster.zNo trial is running and no new trial has been started within the last {wait_time:.0f} seconds. This could be due to the cluster not having enough resources available. zYou asked for {asked_cpus} CPUs and {asked_gpus} GPUs per trial, but the cluster only has {cluster_cpus} CPUs and {cluster_gpus} GPUs available. zStop the tuning and adjust the required resources (e.g. via the `ScalingConfig` or `resources_per_trial`, or `num_workers` for rllib), or add more resources to your cluster.F for_trainc d}|rt}t}t}nt}t}t }||tz }|rdt|\}}t}||||| dd| ddz }||z }|S)Nz3Ignore this message if the cluster is autoscaling. ) wait_timerrr)rr cluster_cpus cluster_gpus) MSG_TRAIN_STARTMSG_TRAIN_INSUFFICIENT MSG_TRAIN_ENDMSG_TUNE_STARTMSG_TUNE_INSUFFICIENT MSG_TUNE_ENDformatr)rrr) r*rmsgstart insufficientendrr r s r'_get_insufficient_resources_warning_msgr:Vs @C- , 5<<"O"Q"Q< R RRC  !7!>!> J@BB |""!!*..ua88*..ua88 #   3JC Jrc&eZdZdZddefdZdZdS)_InsufficientResourcesManagerzInsufficient resources manager. Makes best effort, conservative guesses about if Tune loop is stuck due to infeasible resources. If so, outputs usability messages for users to act upon. Fr*c0d|_d|_||_dS)N)_no_running_trials_since_last_trial_num _for_train)selfr*s r__init__z&_InsufficientResourcesManager.__init__s(*%!#rct||jkr|jdkrtj|_ntj|jz t krrt d|D}|r d|_dSt|j|d}t |tj|_nd|_t||_dS)a^Tracks information across the life of Tune loop and makes guesses about if Tune loop is stuck due to infeasible resources. If so, outputs certain warning messages. The logic should be conservative, non-intrusive and informative. For example, rate limiting is applied so that the message is not spammy. r>c3^K|](}|jtjkot|V)dSr )rrrr!).0rs r zG_InsufficientResourcesManager.on_no_available_trials..sN&&LEM1W6PQV6W6W&&&&&&rNr)r*r) lenr@r?time monotonicr)anyr:rAloggerwarning)rB all_trialscan_fulfill_anyr6s ron_no_available_trialsz4_InsufficientResourcesManager.on_no_available_trialss z??d2 2 2,22040@0@--  4#@@?AABB#&&&!+&&&## #46D1F>"oZ]s###040@0@-,.D )":rN)F)__name__ __module__ __qualname____doc__boolrCrPrrrr<r<wsM$$$$$$$ "/"/"/"/"/rr<)FN)!loggingr'rI functoolsrtypingrrrr ray.tune.execution.cluster_inforray.tune.experimentr getLoggerrQrLrintrrUr!r&r)r/r0r1r2r3r4strr:r<rrrr^s (((((((((( ;;;;;;%%%%%%  8 $ $  #d### #%E#s(O Me M M M M M  Yu Y Y Y  YO B -O W -  6:$,UO @0/0/0/0/0/0/0/0/0/0/r