&`iPddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z mZmZddlmZddlmZmZddlmZmZmZmZddlmZmZmZdd lmZdd l m!Z!dd l"m#Z#m$Z$m%Z%dd l&m'Z'e d Z(e dZ)ej*e+Z,e'GddZ-dS)N)Path)AnyCallableDictListOptionalTypeVarUnion)StartTracebackStartTracebackWithWorkerRankskip_exceptions)Dataset) Checkpoint DataConfig)BackendExecutorInactiveWorkerGroupErrorTrainBackendErrorTrainingWorkerError)_TrainingResult _TrainSession get_session) ActorWrapper) BackendConfig) BaseTrainer GenDatasetTrainingFailedError) DeveloperAPITSc2eZdZdZdeeefdedeege fee e e fge ffde e e fde e e fdedeee e eeffd Zd Z ddeefd Zd efdZdZdeee fdZdZdefdZd S)TrainingIteratorzEAn iterator over Train results. Returned by ``trainer.run_iterator``.backend_executorbackend_config train_funcdatasetsmetadata data_config checkpointc||_||_||_||_||_||_|||j|j|j|d|_dS)N)r$r%r&r'r(F) _backend_executor backend_cls_backend _train_func _datasets _metadata _data_config_start_training_finished_training)selfr"r#r$r%r&r'r(s e/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/train/trainer.py__init__zTrainingIterator.__init__'s"2&2244 %!!' !^^)!    #(c|SNr3s r4__iter__zTrainingIterator.__iter__Bs r6Nc t}|s Jd|jfddS)Nz8`_start_training` should only be called from within TunecDjS)N)r$r%r&r'storager()r*start_training)r(r'r%r&r3r>r$sr4z2TrainingIterator._start_training..Rs1D*99%!!'% :r6)rr>_run_with_error_handling)r3r$r%r&r'r( tune_sessionr>s`````` @r4r1z TrainingIterator._start_trainingEs'2mm WWWWW|& %%           r6funccR |S#t$r^td||j|j|j|j||cYSt$rtddt$rtddwxYw)NzSWorkers have been successfully restarted. Resuming training from latest checkpoint.zThis Trainer is not active. It is either shutdown already or never started in the first place. Either create a new Trainer or start this one.zTraining failed. You should not be seeing this error and this is a bug. Please create a new issue at https://github.com/ray-project/ray.) rloggerinfor1r-r.r/r0rAr RuntimeErrorr)r3rCs r4rAz)TrainingIterator._run_with_error_handling\s 466M" 7 7 7 KK3      !     0066 6 6 6'   A   !   6    s A%B&33B&c|r"|jdt ||j}|C|jd||jd|_t|S#t$r}t|tr|j }nd}t|}d tjt!|||j}|jd|| |jdd|_n#t&$rYnwxYwd}~wwxYw)NF)erroredT)rI stack_trace failed_rank)graceful_termination) is_finishedr*report_final_run_status StopIterationrA_fetch_next_result_finish_trainingr2r isinstancer worker_rankr join tracebackformat_exceptiontype __traceback__shutdown Exception)r3 next_resultserLrKs r4__next__zTrainingIterator.__next__|s       " : :5 : I I I ! 889PQQL#&>>u>MMM--d.CDDD*.'####   !9:: #m "  ""A''*477AqGGK  " : :+; ;    &//U/KKK*.''     1 s=A B E##BE*"E  E EEEEE#returncv|j}|dStd|DsJ|S)aFetch next results produced by ``session.report()`` from each worker. Assumes ``start_training`` has already been called. Returns: A list of dictionaries of values passed to ``session.report()`` from each worker. Each item corresponds to an intermediate result a single worker. If there are no more items to fetch, returns None. Nc3@K|]}t|tVdSr8)rSr).0results r4 z6TrainingIterator._fetch_next_result..s,MM6:fo66MMMMMMr6)r*get_next_resultsall)r3resultss r4rQz#TrainingIterator._fetch_next_resultsJ(99;; ?4MMWMMMMMMMMr6c4|jS)atFinish training and return final results. Propagate any exceptions. Blocks until training is finished on all workers. Assumes `start_training` has already been called. Returns: A list of return values from calling ``train_func`` on each worker. Each item corresponds to the return value from a single worker. )r*finish_trainingr:s r4rRz!TrainingIterator._finish_trainings%55777r6c|jSr8)r2r:s r4rNzTrainingIterator.is_finisheds &&r6r8)__name__ __module__ __qualname____doc__r rrrrrrstrrrrrrrr5r;r1rAr^rrQrRboolrNr9r6r4r!r!#s}OO( =>(&((2q5/8T#s(^4Da4G+HHI ( sG|$ ( sCx. ( (U4dJ#>?@((((6,0   Z(     .X@%%%NHT$Z$8" 8 8 8'T''''''r6r!).loggingrVpathlibrtypingrrrrrr r ray.air._internal.utilr r r ray.datar ray.trainrr$ray.train._internal.backend_executorrrrrray.train._internal.sessionrrrray.train._internal.utilsrray.train.backendrray.train.base_trainerrrrray.util.annotationsrrr getLoggerrkrEr!r9r6r4r~sFFFFFFFFFFFFFFFFFF ,,,,,,,, TSSSSSSSSS222222++++++ .----- GCLL GCLL  8 $ $^'^'^'^'^'^'^'^'^'^'r6