&`i9ddlZddlmZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZmZddlmZddlmZdd lmZdd lmZmZdd lmZdd l m!Z!erdd l"m#Z#ej$e%Z&eGddeZee!dGddZ'eGddeZee!dGddZ(dS)N) dataclass)cached_property)Path) TYPE_CHECKINGDictListLiteralOptionalUnion) FailureConfig ScalingConfig) RuntimeEnv) _DEPRECATED)StorageContext)FAIL_FAST_DEPRECATION_MESSAGE%TRAINER_RESOURCES_DEPRECATION_MESSAGE)date_str) PublicAPI) UserCallbackceZdZUdZdZeeed<dZe e ed<dZ ee ed<dZ ee ee e feee e ffed<fdZefd Zed Zed ZxZS) r ac Configuration for scaling training. Args: num_workers: The number of workers (Ray actors) to launch. Each worker will reserve 1 CPU by default. The number of CPUs reserved by each worker can be overridden with the ``resources_per_worker`` argument. If the number of workers is 0, the training function will run in local mode, meaning the training function runs in the same process. use_gpu: If True, training will be done on GPUs (1 per worker). Defaults to False. The number of GPUs reserved by each worker can be overridden with the ``resources_per_worker`` argument. resources_per_worker: If specified, the resources defined in this Dict is reserved for each worker. Define the ``"CPU"`` and ``"GPU"`` keys (case-sensitive) to override the number of CPU or GPUs used by each worker. placement_strategy: The placement strategy to use for the placement group of the Ray actors. See :ref:`Placement Group Strategies ` for the possible options. label_selector: A list of label selectors for Ray Train worker placement. If a single label selector is provided, it will be applied to all Ray Train workers. If a list is provided, it must be the same length as the max number of Ray Train workers. accelerator_type: [Experimental] If specified, Ray Train will launch the training coordinator and workers on the nodes with the specified type of accelerators. See :ref:`the available accelerator types `. Ensure that your cluster has instances with the specified accelerator type or is able to autoscale to fulfill the request. This field is required when `use_tpu` is True and `num_workers` is greater than 1. use_tpu: [Experimental] If True, training will be done on TPUs (1 TPU VM per worker). Defaults to False. The number of TPUs reserved by each worker can be overridden with the ``resources_per_worker`` argument. This arg enables SPMD execution of the training workload. topology: [Experimental] If specified, Ray Train will launch the training coordinator and workers on nodes with the specified topology. Topology is auto-detected for TPUs and added as Ray node labels. This arg enables SPMD execution of the training workload. This field is required when `use_tpu` is True and `num_workers` is greater than 1. Ntrainer_resourcesFuse_tputopologylabel_selectorc|jtt|jr|jrt d|js|jdkrt d|jr|jdkrt d|jrM|jdkrB|jst d|j st d|j rt dt|j trFt|jtr,t|j |jkrt d |jdkrtd t#dS) Nz6Cannot specify both `use_gpu=True` and `use_tpu=True`.rz`use_tpu` is False but `TPU` was found in `resources_per_worker`. Either set `use_tpu` to True or remove `TPU` from `resources_per_worker.z`use_tpu` is True but `TPU` is set to 0 in `resources_per_worker`. Either set `use_tpu` to False or request a positive number of `TPU` in `resources_per_worker.zY`topology` must be specified in ScalingConfig when `use_tpu=True` and `num_workers` > 1.z``accelerator_type` must be specified in ScalingConfig when `use_tpu=True` and `num_workers` > 1.z}Cannot set `label_selector` when `use_tpu=True` because Ray Train automatically reserves a TPU slice with a predefined label.zKIf `label_selector` is a list, it must be the same length as `num_workers`.zRunning in local mode. The training function will run in the same process. If you are using it and running into issues please file a report at https://github.com/ray-project/ray/issues.)rDeprecationWarningruse_gpur ValueErrornum_tpus_per_worker num_workersraccelerator_typer isinstancelistintlenloggerinfosuper __post_init__self __class__s k/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/train/v2/api/config.pyr*zScalingConfig.__post_init__Ms  ! -$%JKK K < WDL WUVV V|  81 < <;  < D499)  < D,q00=  .(  <"  \ t*D 1 1 4+S11 D'((D,<<<]   q KK=    cN|j |jrddiStjS)NTPUr)resources_per_workerrr)_resources_per_worker_not_noner+s r.r3z,ScalingConfig._resources_per_worker_not_nones-  $ ,| "qz!ww55r/ciSNr,s r._trainer_resources_not_nonez)ScalingConfig._trainer_resources_not_nones r/c8|jddS)z%The number of TPUs to set per worker.r1r)r3getr7s r.r z!ScalingConfig.num_tpus_per_workers266ua@@@r/)__name__ __module__ __qualname____doc__rr dict__annotations__rr boolrstrrrrr*propertyr3r8r __classcell__)r-s@r.r r s''R)-x~,,, GU4[   "Hhsm"""LPNHU4S>4S#X3G#GHIPPP7 7 7 7 7 r6666X6XAAXAAAAAr/r stable) stabilityceZdZUdZdZeeed<dZee ed<dZ e ded<e Z eeee e fed<e Zeeee e fed <d ZdS) CheckpointConfigaConfiguration for checkpointing. Default behavior is to persist all checkpoints reported with :meth:`ray.train.report` to disk. If ``num_to_keep`` is set, the default retention policy is to keep the most recent checkpoints. Args: num_to_keep: The maximum number of checkpoints to keep. If you report more checkpoints than this, the oldest (or lowest-scoring, if ``checkpoint_score_attribute`` is set) checkpoint will be deleted. If this is ``None`` then all checkpoints will be kept. Must be >= 1. checkpoint_score_attribute: The attribute that will be used to score checkpoints to determine which checkpoints should be kept. This attribute must be a key from the metrics dictionary attached to the checkpoint. This attribute must have a numerical value. checkpoint_score_order: Either "max" or "min". If "max"/"min", then checkpoints with highest/lowest values of the ``checkpoint_score_attribute`` will be kept. Defaults to "max". checkpoint_frequency: [Deprecated] checkpoint_at_end: [Deprecated] N num_to_keepcheckpoint_score_attributemaxrKmincheckpoint_score_ordercheckpoint_frequencycheckpoint_at_endc|jtkrtd|jtkrtd|j#|jdkrt d|jd|jdvrt d|jddS) Nz`checkpoint_frequency` is deprecated since it does not apply to user-defined training functions. Please remove this argument from your CheckpointConfig.z`checkpoint_at_end` is deprecated since it does not apply to user-defined training functions. Please remove this argument from your CheckpointConfig.rzReceived invalid num_to_keep: z". Must be None or an integer >= 1.rLz)Received invalid checkpoint_score_order: z. Must be 'max' or 'min'.)rOrrrPrIrrNr7s r.r*zCheckpointConfig.__post_init__s  $ 3 3$J   ![ 0 0$J    'D,<,A,A31A333   &n < <*DrIr r%r@rJrBrNr rrOr rPrAr*r6r/r.rHrHs."&K#%%%04 44449GL1999GR% w{/C CDRRREPuXd^W[-AABPPPr/rHcFeZdZUdZeZeeefe d<dZ e e d<dZ dS)r ahConfiguration related to failure handling of each training run. Args: max_failures: Tries to recover a run from training worker errors at least this many times. Will recover from the latest checkpoint if present. Setting to -1 will lead to infinite recovery retries. Setting to 0 will disable retries. Defaults to 0. controller_failure_limit: [DeveloperAPI] The maximum number of controller failures to tolerate. Setting to -1 will lead to infinite controller retries. Setting to 0 will disable controller retries. Defaults to -1. fail_fastcontroller_failure_limitcN|jtkrttdSr5)rSrrrr7s r.r*zFailureConfig.__post_init__s& >[ ( ($%BCC C ) (r/N) r;r<r=r>rrSr rArBr@rUr%r*r6r/r.r r s`  #.IuT3Y---$&c&&&DDDDDr/r cbeZdZUdZdZeeed<dZeeed<dZ ee j j ed<dZ eeed<dZeeed<dZeeded <dZeeeefed <eZeed <eZeed <eZeed <eZeed<eZeed<dZedefdZ dS) RunConfiga6Runtime configuration for training runs. Args: name: Name of the trial or experiment. If not provided, will be deduced from the Trainable. storage_path: Path where all results and checkpoints are persisted. Can be a local directory or a destination on cloud storage. For multi-node training/tuning runs, this must be set to a shared storage location (e.g., S3, NFS). This defaults to the local ``~/ray_results`` directory. storage_filesystem: A custom filesystem to use for storage. If this is provided, `storage_path` should be a path with its prefix stripped (e.g., `s3://bucket/path` -> `bucket/path`). failure_config: Failure mode configuration. checkpoint_config: Checkpointing configuration. callbacks: [DeveloperAPI] A list of callbacks that the Ray Train controller will invoke during training. worker_runtime_env: [DeveloperAPI] Runtime environment configuration for all Ray Train worker actors. Nname storage_pathstorage_filesystemfailure_configcheckpoint_configr callbacksworker_runtime_env sync_configverbosestopprogress_reporter log_to_filecLddlm}|j||_|jst |_|jst |_t|jtr|j |_d}gd}|D]=}t||tkr"t| |>|jsdt|_|jpg|_|jpi|_ddlmt)fd|jDst+dt|jt st+d |jjd t|jtst+d |jjd dS) Nr)DEFAULT_STORAGE_PATHaI`RunConfig({})` is deprecated. This configuration was a Ray Tune API that did not support Ray Train usage well, so we are dropping support going forward. If you heavily rely on these configurations, you can run Ray Train as a single Ray Tune trial. See this issue for more context: https://github.com/ray-project/ray/issues/49454)r`rarbrcrdzray_train_run-)RayTrainCallbackc38K|]}t|VdSr5)r#).0cbrgs r. z*RunConfig.__post_init__..<s.MM:b"233MMMMMMr/zAll callbacks must be instances of `ray.train.UserCallback`. Passing in a Ray Tune callback is no longer supported. See this issue for more context: https://github.com/ray-project/ray/issues/49454z!Invalid `CheckpointConfig` type: z|. Use `ray.train.CheckpointConfig` instead. See this issue for more context: https://github.com/ray-project/ray/issues/49454zInvalid `FailureConfig` type: zy. Use `ray.train.FailureConfig` instead. See this issue for more context: https://github.com/ray-project/ray/issues/49454)ray.train.constantsrfrZr\r r]rHr#ras_posixgetattrrrformatrYrr^r_ray.train.v2.api.callbackrgallrr-)r,rfrun_config_deprecation_messageunsupported_paramsparamrgs @r.r*zRunConfig.__post_init__s<<<<<<   $ 4D " 2"///D % 8%5%7%7D " d' . . = $ 1 : : <  '   ( W WEtU##{22()G)N)Nu)U)UVVV3y 6555DI-2"&"9"?R>>>>>>MMMMdnMMMMM B $02BCC BD4J4TBBB $-}== B1D1NBBB   r/returncDt|j|j|jS)N)rZexperiment_dir_namer[)rrZrYr[r7s r.storage_contextzRunConfig.storage_contextTs** $ #6    r/)!r;r<r=r>rYr rBr@rZr[pyarrowfs FileSystemr\r r]rHr^rr_r r?rrr`rarbrcrdr*rrrxr6r/r.rXrXsV*D(3-"&L(3-&&&:>!67>>>.2NH]+22248x 0188804Ix^,-444<@tZ'7!89@@@"K"""GSD#(s((("K"""BBBH    _   r/rX))logging dataclassesr functoolsrpathlibrtypingrrrr r r pyarrow.fsryray.air.configr FailureConfigV1r ScalingConfigV1ray.runtime_envr ray.train.v2._internal.constantsr(ray.train.v2._internal.execution.storager&ray.train.v2._internal.migration_utilsrrray.train.v2._internal.utilrray.util.annotationsr ray.trainr getLoggerr;r'rHrXr6r/r.rs!!!!!!%%%%%%FFFFFFFFFFFFFFFF'&&&&&888888CCCCCC100000******'&&&&&&  8 $ $ wAwAwAwAwAOwAwA wAt  X7777777 7t DDDDDODD D*  Xn n n n n n n  n n n r/