&`i/RddlZddlmZmZddlmZmZmZmZm Z Gdde Z Gdde Z Gdd e Z Gd d e ZGd d e ZGdde ZGdde ZGdde ZGddeZGdde ZdS)N)ListOptional)COLLECTIVE_TIMEOUT_S_ENV_VAR$DEFAULT_WORKER_GROUP_START_TIMEOUT_S%DEFAULT_WORKER_HEALTH_CHECK_TIMEOUT_S$WORKER_GROUP_START_TIMEOUT_S_ENV_VAR%WORKER_HEALTH_CHECK_TIMEOUT_S_ENV_VARceZdZdZdS) RayTrainErrorz(Base class for all Ray Train exceptions.N__name__ __module__ __qualname____doc__u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/train/v2/_internal/exceptions.pyr r s2222rr c"eZdZdZfdZxZS)WorkerHealthCheckTimeoutErrorzBException raised when a worker health check hangs for long enough.ctjtt}|dtd|dz }t |dS)Nz Set the z> environment variable to increase the timeout (current value: z seconds).)osgetenvr rsuper__init__)selfmessagetimeout __class__s rrz&WorkerHealthCheckTimeoutError.__init__sj) 13X    3> 3 3& 3 3 3  !!!!!r)r rrrr __classcell__rs@rrrs>LL " " " " " " " " "rrc4eZdZdZdeffd ZdZdZxZS)WorkerHealthCheckFailedErrorz2Exception raised when a worker health check fails.failurecft|||_||_dSN)rr_messagehealth_check_failure)rrr#rs rrz%WorkerHealthCheckFailedError.__init__$s0 !!! $+!!!rc,|j|j|jffSr%)rr&r'rs r __reduce__z'WorkerHealthCheckFailedError.__reduce__)s0I JKKrc@|jdzt|jzS)N )r&strr'r)s r__str__z$WorkerHealthCheckFailedError.__str__,s}t#c$*C&D&DDDr) r rrr Exceptionrr*r.rr s@rr"r"!sr<<,,,,,,, LLLEEEEEEErr"c.eZdZdZdeffd ZdZxZS)WorkerGroupStartupTimeoutErrora1Exception raised when the worker group startup times out. Example scenario: 4 GPUs are detected in the cluster, but when the worker are actually scheduled, one of the nodes goes down and only 3 GPUs are available. One of the worker tasks may be stuck pending, until a timeout is reached. num_workersc ttjtt }||_td|d|dtddS)Nz)The worker group startup timed out after z seconds waiting for a workers. Potential causes include: (1) temporary insufficient cluster resources while waiting for autoscaling (ignore this warning in this case), (2) infeasible resource request where the provided `ScalingConfig` cannot be satisfied), and (3) transient network issues. Set the z. environment variable to increase the timeout.) floatrenvirongetrrr2rr)rr2rrs rrz'WorkerGroupStartupTimeoutError.__init__8s JNN44     '  < < < < << < < < rc |j|jffSr%)rr2r)s rr*z)WorkerGroupStartupTimeoutError.__reduce__Ms!1 344r)r rrrintrr*rr s@rr1r10s] C      *5555555rr1ceZdZdZdS)WorkerGroupStartupFailedErrorzException raised when the worker group fails to start. Example scenario: A worker is scheduled onto a node that dies while the worker actor is initializing. Nr rrrr:r:Qrr:ceZdZdZdS)!InsufficientClusterResourcesErrorzException raised when the cluster has insufficient resources. Example scenario: A worker that requires 1 GPU is scheduled onto a cluster that only has CPU worker node types. Nr rrrr=r=Yr;rr=ceZdZdZdS)$CheckpointManagerInitializationErroraException raised when the checkpoint manager fails to initialize from a snapshot. Example scenarios: 1. The checkpoint manager snapshot version is old and incompatible with the current version of Ray Train. 2. The checkpoint manager snapshot JSON file is corrupted. 3. The checkpoint manager snapshot references checkpoints that cannot be found in the run storage path. Nr rrrr?r?asrr?ceZdZdZdS)CollectiveTimeoutErrorzhException raised when an internal Ray Train collective operation of the worker group times out. Nr rrrrArAmsrrAcNeZdZdZdeedeedeffd ZdZ xZ S)BroadcastCollectiveTimeoutErroraException raised when the broadcast operation times out. There are two main timeout examples: 1. If not all workers call `ray.train.report`, the entire worker group will hang until the timeout before raising. This prevents indefinite worker group hangs. 2. If a worker is slow in the training loop and fails to reach the broadcast time, the collective will time out. time_elapsed missing_ranks timeout_sc ||_||_||_d|dd|dtd|dd }t |dS)Nz)The collective operation timed out after z.2fzH seconds. The following ranks have not joined the collective operation: z" You can set the timeout with the z& environment variable (current value: zI seconds). Disable the timeout by setting the environment variable to -1.) _time_elapsed_missing_ranks _timeout_srrr)rrDrErFrrs rrz(BroadcastCollectiveTimeoutError.__init__~s*+# M T M MMZ M M0L M M5> N M M M  !!!!!rc8|j|j|j|jffSr%)rrHrIrJr)s rr*z*BroadcastCollectiveTimeoutError.__reduce__s# N  !4do F  r) r rrrrr4rr8rr*rr s@rrCrCssv"$UO"<@I"RW""""""        rrCc.eZdZdZdedefdZdZdZdS)UserExceptionWithTracebacka"This class wraps a user code exception raised on the worker with its original traceback string, for logging and debugging purposes. This is needed because the original exception traceback is not serialized with the exception when it is *returned* back to the main process. exc traceback_strc"||_||_dSr%) _base_exc_traceback_str)rrNrOs rrz#UserExceptionWithTraceback.__init__s+rc,|j|j|jffSr%)rrQrRr)s rr*z%UserExceptionWithTraceback.__reduce__s1D EFFrc|jSr%)rRr)s rr.z"UserExceptionWithTraceback.__str__s ""rN) r rrr BaseExceptionr-rr*r.rrrrMrMsa,M,#,,,,GGG#####rrM)rtypingrr ray.train.v2._internal.constantsrrrrr r/r rr"r1r:r=r?rArCrMrrrrXs% !!!!!!!!33333I333 " " " " "M " " " E E E E E= E E E55555]555BM      =   ]      &<   D##########r