&`i.ddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z ddl ZddlmZmZmZmZddlmZddlmZejeZdZ dd ed e ejjd e efd ZGd dZ dS)N)Counter)Path)CallableDictOptionalUnion)StorageContext_download_from_fs_path_list_at_fs_pathget_fs_and_path)Trial)out_of_band_serialize_datasetaThis could be due to a large number of trials, large logfiles from lots of reported metrics, or throttling from the remote storage if uploading too frequently. You may want to consider switching the `RunConfig(storage_filesystem)` to a more performant storage backend such as s3fs for a S3 storage path. You can suppress this error by setting the environment variable TUNE_WARN_SLOW_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S to a higher value than the current threshold ({threshold}).experiment_pathfsreturnc ddlm}t||\}}t||}|jd}t j||}|sdSt|}t|| S)aReturns file name of most recently created experiment checkpoint. Args: experiment_path: Local or remote path to the experiment directory containing at least one experiment checkpoint file. Returns: str: The local or remote path to the latest experiment checkpoint file based on timestamp. None if no experiment checkpoints were found. r)TuneController)storage_filesystemrfs_path*N) "ray.tune.execution.tune_controllerrr r CKPT_FILE_TMPLformatfnmatchfiltermaxras_posix)rrrexperiment_fs_path filenamespatternmatchingfilenames w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/tune/execution/experiment_state.py"_find_newest_experiment_checkpointr%$sBAAAAA,_QSTTTB B0BCCCI+22377G~i11H t8}}H "H - - 6 6 8 88ceZdZdZdddeedeeee fdeefdZ e dZ d efd Z dd egdfd edefdZddZdefdZdS)_ExperimentCheckpointManageraOHelper class for managing experiment-level checkpoints. This class implements the ``checkpoint()`` method used to checkpoint experiment state. When called, this will serialize and write to disk the state of the trial runner, trial executor, and search algorithm, to a specified checkpoint file. The checkpoint period is automatically adjusted to ``max(10, time_per_checkpoint * 19)``. This means that at most 5% of the time (1/20) will be used for writing checkpoints, while 95% of the time (19/20) will be used to handle the rest of the training loop. N)sync_every_n_trial_checkpointsstoragecheckpoint_periodr)c||_td|_d|_|dk|_|jrd|_nt||_||_t|_d|_ ttj dd|_ ttj dd|_dS) Nz-infauto$@F:TUNE_WARN_EXCESSIVE_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S55TUNE_WARN_SLOW_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S30)_storagefloat_last_save_time_last_sync_time_auto_checkpoint_enabled_checkpoint_period_sync_every_n_trial_checkpointsr&_trial_num_checkpoints_since_last_sync_should_force_sync_uposenvironget_excessive_sync_threshold_slow_sync_threshold)selfr*r+r)s r$__init__z%_ExperimentCheckpointManager.__init__Ks  $V}}#):V(C%  ( ?&*D # #&+,=&>&>D # 0N,HO 3+0"). JNNLc  * * & %* JNNG  % % !!!r&c|jSN)r7rAs r$auto_checkpoint_enabledz4_ExperimentCheckpointManager.auto_checkpoint_enabledqs ,,r& time_takenc|jrBtd|dz|_td|dd|jdddSdS)Nr.z#Experiment state snapshotting took .2fz+ seconds. Adjusting snapshotting period to z seconds.)r7rr8loggerdebug)rArGs r$_update_auto_checkpoint_timez9_ExperimentCheckpointManager._update_auto_checkpoint_timeus~  ( '*$ R&@&@D # LL:-::*9:::       r&Fsave_fnforcewaitcjj}|pj}tj}|jz jkr|sdStj}t5|dddn #1swxYwYfd}|r}tj}|tj|z } | jkr?t d| ddt jj tjj z nd} jj|jj} | rx| =| jkr2jr+t djdjd tj_ jd _nL| J| jkr?t d | dd t j|r |tj|z } | tj_dS)aSaves execution state to the experiment directory on the storage path. This includes an experiment checkpoint file that contains trial statuses and the searcher state. Overwrites the current session checkpoint, which starts when self is instantiated. Throttle depends on self._checkpoint_period. Args: save_fn: Function to call to actually save data to the driver staging path. The files in the driver staging path will be uploaded to the storage path. force: Forces an experiment checkpoint and launches a sync to storage. This happens regardless of checkpoint_period wait: Waits for the sync up to complete before returning. Nc jjdS#t$r.tdjjddYdSwxYw)Nz'Saving experiment state to storage at 'z' failed with exception: T)exc_info)r3syncerrP ExceptionrKerrorrrEsr$ wait_for_synczL_ExperimentCheckpointManager.sync_up_experiment_state..wait_for_syncs  $))+++++    T 8TTT! s#4AAzvSaving the experiment state (which holds a global view of trial statuses and is used to restore the experiment) took ~rJz1 seconds, which may be a performance bottleneck. ) thresholdzLExperiment state snapshotting has been triggered multiple times in the last a seconds and may become a bottleneck. A snapshot is forced if `CheckpointConfig(num_to_keep)` is set, and a trial has checkpointed >= `num_to_keep` times since the last snapshot. You may want to consider increasing the `CheckpointConfig(num_to_keep)` or decreasing the frequency of saving checkpoints. You can suppress this warning by setting the environment variable TUNE_WARN_EXCESSIVE_EXPERIMENT_CHECKPOINT_SYNC_THRESHOLD_S to a smaller value than the current threshold (z3). Set it to 0 to completely suppress this warning.FzSaving the experiment state (which holds a global view of trial statuses and is used to restore the experiment) has already taken zh seconds, which may cause consistency issues upon restoration if your driver script ungracefully exits. )rG)r3experiment_driver_staging_pathr;time monotonicr5r8rr@rKwarning_SLOW_SYNC_WARNINGrr6rTsync_uprr?r:clearrM) rArNrOrPdriver_staging_pathnowcheckpoint_time_startrW start_time wait_timetime_since_last_sync launched_synccheckpoint_time_takens ` r$sync_up_experiment_statez5_ExperimentCheckpointManager.sync_up_experiment_states3*#mJ33n % %(? ? ? ? F!% 0 0+ , ,   GIII                      ))J MOOO((:5I4444X&QXX*004;T0UU XX#/ N  t3 3 3   ,44 !A   ( $0(4+III.J G)-)G G G6 G G G"$(>#3#3D   7 = = ? ? ?).D & &%0(4+DDDX)=MXX *004;T0UU XX   MOOO $ 0 03H H ))5J)KKK $~//s" A99A=A=rc |jj}t||jj}d|D}|D]l}t |jj|}t |jj|}t|||mt d|d|jjj d|jjd|jjdS)Nrcfg|].}|ds|d,|/S)z.jsonz.pkl)endswith).0paths r$ zK_ExperimentCheckpointManager.sync_down_experiment_state..sP   }}W%% *.v)>)>    r&)rr local_pathzCopied z from: (fs, path) = (z, z) -> ) r3rr rrrrYr rKrL type_name)rAr filepathsmatchesrelpathrros r$sync_down_experiment_statez7_ExperimentCheckpointManager.sync_down_experiment_states( ] -$DM4TUUU   !     R RG4=;WEENNPPG  A A     r&trialc|jsdS|j|xxdz cc<|j||jkr d|_dSdS)NT)r9r:r;)rArus r$on_trial_checkpointz0_ExperimentCheckpointManager.on_trial_checkpointsh3  F 3E:::a?:::  7 >3 4 4*.D & & & 4 4r&)FF)rN)__name__ __module__ __qualname____doc__rr rintr4strrBpropertyrFrMrboolrhrtr rxr&r$r(r(=s(  $9= $ $ $ .)$ !eS1 $ )1 $ $ $ $ L--X- u     z0z0"d(#z0z0 z0z0z0z0x    0 . . . . . . .r&r(rD)!rloggingr<rZ collectionsrpathlibrtypingrrrr pyarrow.fspyarrowray.train._internal.storager r r r ray.tune.experiment.trialr +ray.tune.impl.out_of_band_serialize_datasetr getLoggerryrKr]r~r FileSystemr%r(rr&r$rs} 222222222222 ,+++++UUUUUU  8 $ $6AE999&wz'<=9 c]99992b.b.b.b.b.b.b.b.b.b.r&