&`i ddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z m Z mZmZmZmZddlZddlmZddlZddlmZddlmZmZmZddlmZddl m!Z!ddl"m#Z#m$Z$dd l%m&Z&m'Z'dd l(m)Z)dd l*m+Z,dd l-m+Z.dd l/m0Z0dd l1m2Z2ddl3m4Z4ej5e6Z7ej8dZ9ej8dZ:e4dGddej;Zde?de@fdZAdej=jBde@fdZCe# d&dee?e,e.fdedde e?e ffdZDe#dee?e,e.fde?de?fd ZEe#d!ee?e,e.fde?de?fd"ZFe4d'd$e@fd%ZGdS)(N)MappingProxyType)Any CollectionDictListOptionalTupleUnion)version)COMPONENT_LEARNERCOMPONENT_LEARNER_GROUPCOMPONENT_RL_MODULE) force_list)FaultTolerantActorManager) OldAPIStack5OverrideToImplementCustomLogic_CallToSuperRecommended)NOT_SERIALIZABLEserialize_type) StateDict) Checkpoint)sync_dir_between_nodes)log_once) PublicAPIz1.1z2.1alpha) stabilityc ~eZdZdZdZdZdZ ddddddeee e j fdee d ed d e d e f d Zddddee e j fdee d ed d dfdZe ddee e j fd ed d dfdZej ddddeee ee fdeee ee fd e fdZejde d dfdZejd eeee efffdZed efdZd eee dffdZd e fdZdZ ddZ!dS)CheckpointableaMAbstract base class for a component of RLlib that can be checkpointed to disk. Subclasses must implement the following APIs: - save_to_path() - restore_from_path() - from_checkpoint() - get_state() - set_state() - get_ctor_args_and_kwargs() - get_metadata() - get_checkpointable_components() statezclass_and_ctor_args.pklz metadata.jsonNF)r filesystem use_msgpackpathrpyarrow.fs.FileSystemr returnc  |Qddl}tjtj}t |}|p||z }t|t r|n|}|r)|s'tj j |\}}| |dtj|}|}d|vrt t|d<|||jz 5} | t'j|ddddn #1swxYwY|||jz 5} t/jt3||d| dddn #1swxYwY|du} |p1|d|D }|D]\} } | r| |vr || z } t| t:r~| d}dd }t?tA| !||g }|j"s|#|#}|}d}| r'tIj%|&| }||kr| !| ||fd |g ||fd }t?tA| !||g }|j"s|#|#}tO|||t | |fd}| !||g | r|&| }n|| | }| (| |||||j)|rdndzz }||5} |r'tUd}||| nt/j|| dddn #1swxYwYt |S)aSaves the state of the implementing class (or `state`) to `path`. The state of the implementing class is always saved in the following format: .. testcode:: :skipif: True path/ [component1]/ [component1 subcomponentA]/ ... [component1 subcomponentB]/ ... [component2]/ ... [cls.METADATA_FILE_NAME] (json) [cls.STATE_FILE_NAME] (pkl|msgpack) The main logic is to loop through all subcomponents of this Checkpointable and call their respective `save_to_path` methods. Then save the remaining (non subcomponent) state to this Checkpointable's STATE_FILE_NAME. In the exception that a component is a FaultTolerantActorManager instance, instead of calling `save_to_path` directly on that manager, the first healthy actor is interpreted as the component and its `save_to_path` method is called. Even if that actor is located on another node, the created file is automatically synced to the local node. Args: path: The path to the directory to save the state of the implementing class to. If `path` doesn't exist or is None, then a new directory will be created (and returned). state: An optional state dict to be used instead of getting a new state of the implementing class through `self.get_state()`. filesystem: PyArrow FileSystem to use to access data at the `path`. If not specified, this is inferred from the URI scheme of `path`. use_msgpack: Whether the state file should be written using msgpack and msgpack_numpy (file extension is `.msgpack`), rather than pickle (file extension is `.pkl`). Returns: The path (str) where the state has been saved. NrT recursivecheckpoint_versionzutf-8)classctor_args_and_kwargscg|] }|d Sr.0cs o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/utils/checkpoints.py z/Checkpointable.save_to_path..sOOOQAaDOOOnot_componentsc<ddl}|jSNr)rayutilget_node_ip_address)_r7s r0_get_ipz,Checkpointable.save_to_path.._get_ipsJJJ877999r2remote_actor_idsc|||tj|n||S)Nrr ) save_to_pathr7get get_state)w_path_state _use_msgpacks r0z-Checkpointable.save_to_path..sENN %(.'9%(GFOOO)*,8+r2cddl}|j}|||tj|n|||S)Nrr?)tempfilemkdtempr@r7rArB)rCrErFrItmpdirs r0_savez*Checkpointable.save_to_path.._savesa'"2!1!3!3"393E1;;==(4 ' & r2c8ddl}||dSr6)shutilrmtree)r:_dirrNs r0_rmdirz+Checkpointable.save_to_path.._rmdir"s#%  d+++++r2) components)rrr .msgpack.pklerrorN)+uuidpathlibPathrI gettempdirstruuid4 isinstanceas_posixpyarrowfs FileSystemfrom_uri create_dir get_metadata)CHECKPOINT_VERSION_LEARNER_AND_ENV_RUNNERopen_output_streamMETADATA_FILE_NAMEwritejsondumpsencodeCLASS_AND_CTOR_ARGS_FILE_NAMEpickledumptypeget_ctor_args_and_kwargsrBget_checkpointable_componentsrhealthy_actor_idsnextiter foreach_actorokrAr7putpoprr@STATE_FILE_NAMEtry_import_msgpack)selfr!rrr rXtmp_dirrandom_dir_namemetadataf_state_provided comp_namecomp comp_path actor_to_user;_resultworker_ip_addr self_ip_addrcomp_state_refrLworker_temp_dirrQ comp_statefilenamemsgpacks r0r@zCheckpointable.save_to_path^sAh < KKKl8#6#8#899G!$**,,//O47_4D"$,,Att$--//  D  D 'z4==dCC J dd333|D!!$$&& x / /-09..H) * * * D+ + 5 5 7 7   :  GGDJx((//88 9 9 9 : : : : : : : : : : : : : : : * * D6 6 @ @ B B    K!$ZZ,0,I,I,K,K                    t+ OO$*L*L*N*NOOO(   $AACCi i OIt 9E#9#9y(I $ 9::_ #5577: :::: **#.:^+z(!++--'!(&wyy "&"C%(WUYYy-A-A%B%BN!\11&&(1.Wb   +7'     )7[ & & & &#T///WWXXG#:,%kkmm+&-kkmmO+&'$I (7,,,, &&v&OOOO#Q!&9!5!5JJ!%9!E!Ei!PJ!!)$ + "  +$IJJ6 J  * *8+<+<+>+> ? ? &1 &,4888 UA&&&& E1%%%  & & & & & & & & & & & & & & &4yys63;E::E>E>48G88G<?G<z3Checkpointable.restore_from_path..sNNNA1Q4NNNr2rNrSTrUF)strict_map_keyrTch|] }|d Sr+r,r-s r0rz3Checkpointable.restore_from_path..sQQQqadQQQr2only_comp_names)r^r\r_r`rarbrcrYrZ_exists_at_fs_pathFileNotFoundErrorrr$_restore_all_subcomponents_from_pathrz with_suffixis_filer{open_input_streamloadrn set_state) r|r!rrkwargsorig_comp_namesrrrrnew_comp_namesdiff_comp_namess r0restore_from_pathz Checkpointable.restore_from_pathFs`"$,,Att$--//  D  D 'z4==dCC J|D!!!*dmmoo>> C#$At$A$A$ABB BON)K)K)M)MNNN11 *  (1 5;    d22H##J//7799 +,488811((44==??B#LL5LAAEBBBBBBBBBBBBBBB  11((0099;;+"KNNE+++++++++++++++ NN5 ! ! !QQD,N,N,P,PQQQN,>O 99*6EIO%  "  s$ E11E58E57GGGc t|tr|n|}|r)|s'tjj|\}}tj|} | ||j z 5}tj |}dddn #1swxYwY|d}t|dd}|dd}ttj|jD]Y\} \} } | |vrM|| } | jtjjkrt/|| kr| || <T| || <Zn#t0$r |}g}|}YnwxYwt3||st5d|d|d|d t3|t6st5d|d|d ||i|} | j|fd |i|| S) aCreates a new Checkpointable instance from the given location and returns it. Args: path: The checkpoint path to load (a) the information on how to construct a new instance of the implementing class and (b) the state to restore the created instance to. filesystem: PyArrow FileSystem to use to access data at the `path`. If not specified, this is inferred from the URI scheme of `path`. kwargs: Forward compatibility kwargs. Note that these kwargs are sent to each subcomponent's `from_checkpoint()` call. Returns: A new instance of the implementing class, already set to the state stored under `path`. Nr(r)rz The class (z) stored in checkpoint (z+) does not seem to be a subclass of `cls` (z)!zA) does not seem to be an implementer of the `Checkpointable` API!r)r^r\r_r`rarbrcrYrZrrmrnrr enumerateinspect signature parametersitemsrykind_ParameterKindPOSITIONAL_OR_KEYWORDlen Exception issubclass ValueErrorrr)clsr!rrr ctor_infoctor ctor_args ctor_kwargsi param_nameparamvalobjs r0from_checkpointzCheckpointable.from_checkpoints."$,,Att$--//  D  D 'z4==dCC J|D!! !--99CCEE +"KNN  + + + + + + + + + + + + + + +W%D"9-C#DQ#GHHI#$:;A>K+4!$''288::++ 6 6&&J'' **Z00C g&<&RRR NNQ..'* ! 25 J/ 6 ! ! !DI KKK  !$$$ 0d00D00(+000 D.11 >d>>D>>>  dI---dDDzDVDDD s7,/F B<0 F <CF CCF FFr3rRr4c dS)aReturns the implementing class's current state as a dict. The returned dict must only contain msgpack-serializable data if you want to use the `AlgorithmConfig._msgpack_checkpoints` option. Consider returning your non msgpack-serializable data from the `Checkpointable.get_ctor_args_and_kwargs` method, instead. Args: components: An optional collection of string keys to be included in the returned state. This might be useful, if getting certain components of the state is expensive (e.g. reading/compiling the weights of a large NN) and at the same time, these components are not required by the caller. not_components: An optional list of string keys to be excluded in the returned state, even if the same string is part of `components`. This is useful to get the complete state of the class, except one or a few components. kwargs: Forward-compatibility kwargs. Returns: The current state of the implementing class (or only the `components` specified, w/o those in `not_components`). Nr,)r|rRr4rs r0rBzCheckpointable.get_stater2cdS)abSets the implementing class' state to the given state dict. If component keys are missing in `state`, these components of the implementing class will not be updated/set. Args: state: The state dict to restore the state from. Maps component keys to the corresponding subcomponent's own state. Nr,)r|rs r0rzCheckpointable.set_staterr2cdS)zReturns the args/kwargs used to create `self` from its constructor. Returns: A tuple of the args (as a tuple) and kwargs (as a Dict[str, Any]) used to construct `self` from its class constructor. Nr,r|s r0rqz'Checkpointable.get_ctor_args_and_kwargs$rr2cL|j|jtjtjdS)aReturns JSON writable metadata further describing the implementing class. Note that this metadata is NOT part of any state and is thus NOT needed to restore the state of a Checkpointable instance from a directory. Rather, the metadata will be written into `self.METADATA_FILE_NAME` when calling `self.save_to_path()` for the user's convenience. Returns: A JSON-encodable dict of metadata information. )class_and_ctor_args_file state_file ray_version ray_commit)rmrzr7 __version__ __commit__rs r0rezCheckpointable.get_metadata-s))-(J.?.    r2cgS)aReturns the implementing class's own Checkpointable subcomponents. Returns: A list of 2-tuples (name, subcomponent) describing the implementing class' subcomponents, all of which have to be `Checkpointable` themselves and whose state is therefore written into subdirectories (rather than the main state file (self.STATE_FILE_NAME) when calling `self.save_to_path()`). r,rs r0rrz,Checkpointable.get_checkpointable_components@s  r2ct|}t|}|tfd|Ds|vr||vrdSdS)aoReturns True if a component should be checkpointed. Args: name: The checkpoint name. components: A list of components that should be checkpointed. non_components: A list of components that should not be checkpointed. Returns: True, if the component should be checkpointed and otherwise False. Nc3HK|]}|dzVdS)/N) startswith)r.r/names r0 z2Checkpointable._check_component..Zs3??1<<s ++??????r2TF)rany)r|rrRr4 comp_list not_comp_lists ` r0_check_componentzCheckpointable._check_componentKsoz** ">22  ????Y????? y  %])B)B4ur2c|dSt|}g}|D]G}||dzr-||t|dzdH|sdn|S)Nrr)rrappendr)r|rrR subcomponentsrs r0_get_subcomponentsz!Checkpointable._get_subcomponents`s}  4 ++   < ._restores$OOOJJJ%(X%A%A%C%CN&11++ERRYR'RRRRR8X8::h2 (%0A/ (4=AH s #A99A=A=r<)rr)rrrr_rrr^rr7r8r9rsrrvr) r|r!rrrrrrcomp_argcomp_dir head_node_ipall_healthy_actorsrs r0rz3Checkpointable._restore_all_subcomponents_from_pathls $AACCA A OIt*y/O/OH )+**h6G6G6I6IJJ '' C88(Y!);)=)=>HH)++ $ 9::% "x;;== %)%;%;%=%="-V44")& 4""8>P"QQQQ '&)3xKQA A r2rW)NN)"__name__ __module__ __qualname____doc__rzrmrhrr r\rYrZrboolr@r classmethodrabcabstractmethodrrBrr rrrqrrerrrrrrr,r2r0rr?s  $O %>!)48f&*8<! fffuS',./0f " f 45 f  f ffffX$(8< XXXC%&XC= X 45 X XXXXt9=UUC%&U45U  UUU[Un =A AE    U3 3#789 !sJsO';!<=       >   y  T        %tCH~0E*F    ; d   ;: $ tE#?O:O4P/Q    D* < < <AEDDDDDDr2rrar!r#cj||}|jtjjjkS)z:Returns `True` if the path can be found in the filesystem.) get_file_inforpr`raFileTypeNotFound)rar!valids r0rrs*   T " "E :,5 55r2 file_infoc@|jtjjjkS)z5Returns `True`, if the file info is from a directory.)rpr`rar Directory)rs r0_is_dirrs >WZ0: ::r2 checkpointrr"c  ddtddddd}t|ttfr|}|r)|s't jj|\}}tj |}t|| r t|| r|dt!|i|t j| d}|D]c}|jrZt'jd|jr@|t-jd t!|jd |cSdt||d z r||d z 5}t3j| }dddn #1swxYwYd |vrt-j|d |d <||n-t7drt8d|ddD]h}t||d|zz r>|d|dkrdndtt!|d|z d|cSid}dD]d}|d|z } t|| r8|| jr |dkrdnd}ne|t=d||t!| d|dz } t|| rt|| rt?} |t j| d}|D]}| |j|d| i|tBz tDz tFz } t|| rt|| rt?} |t j| d}|D]Z}| |jz }t|| r| |j[|d| int|| rw|| jrK|t-jd t!|j$t!|dn t=dt!|d |S)!aReturns a dict with information about an Algorithm/Policy checkpoint. If the given checkpoint is a >=v1.0 checkpoint directory, try reading all information from the contained `rllib_checkpoint.json` file. Args: checkpoint: The checkpoint directory (str) or a Checkpoint object. filesystem: PyArrow FileSystem to use to access data at the `checkpoint`. If not specified, this is inferred from the URI scheme provided by `checkpoint`. Returns: A dict containing the keys: "type": One of "Policy" or "Algorithm". "checkpoint_version": A version tuple, e.g. v1.0, indicating the checkpoint version. This will help RLlib to remain backward compatible wrt. future Ray and checkpoint versions. "checkpoint_dir": The directory with all the checkpoint files in it. This might be the same as the incoming `checkpoint` arg. "state_file": The main file with the Algorithm/Policy's state information in it. This is usually a pickle-encoded file. "policy_ids": An optional set of PolicyIDs in case we are dealing with an Algorithm checkpoint. None if `checkpoint` is a Policy checkpoint. Algorithm cloudpickleN)rpformatr'checkpoint_dirr policy_ids module_idsrFr%zcheckpoint-\d+z0.1)r'rrllib_checkpoint.json)fpr'no_rllib_checkpoint_json_filez>No `rllib_checkpoint.json` file found in checkpoint directory zG! Trying to extract checkpoint info from other files found in that dir.)pklmsgpckz policy_state.Policyrr)rprr'r)rrrzalgorithm_state.zGiven checkpoint does not seem to be valid! No file with the name `algorithm_state.[pkl|msgpack|msgpck]` (or `checkpoint-[0-9]+`) found.)rrpoliciesrr)r'rrzGiven checkpoint (zV) not found! Must be a checkpoint directory (or a file for older checkpoint versions).)%CHECKPOINT_VERSIONr^Checkpoint_trainCheckpoint_tune to_directoryr`rarbrcrYrZrr_rrupdater\ FileSelectorrrematch base_namer Versionrrjrrloggerwarningrsetaddr r rparent)rrinfofile_info_listrrrllib_checkpoint_info extensionrr policies_dirr modules_dirr module_dirs r0get_checkpoint_inforsF<0  D*/ABB/,,.. L*L")!6!?!? !K!K Jj))J*j&9&9&;&;<<N   !4!4!6!677BBN  %s:7888$11 J # #J$7$7$9$9U # K K  ( I  8-y/BCC KK29/%2H2H*-i.A*B*B  KKK  &==GGII   --55??AA 8(, Q%  8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 $'<<<>Eo)*>???%&:; KK- . . . .788 *!***+  I!Z?Y+FGQQSS  (3<3E3E--9.@&)*7Ry7R7R*R&S&S   5  I#&D&D&DDJ":z/B/B/D/DEE ,,Z-@-@-B-BCCK +4u*<*<) >Y   !*oo     "J. j,*?*?*A*A B B 4w  $ $\%:%:%<%< = =H H 4J'55 '' (=(=(?(?5'QQN, 4 4 y23333 KKz2 3 3 3 % & " "  j**=*=*?*? @ @ 4W  $ $[%9%9%;%; < <F F 4J'55 '' (<(<(>(>%'PPN, 8 8 )9+>> :33J4G4G4I4IJJKK8NN9#6777 KKz2 3 3 3 :z':':'<'<==  $ $Z%8%8%:%: ; ; C  &-oe&<&<"%j&7"8"8!*oo       NZ N N N   Ks6HHHmsgpack_checkpoint_dirc rddlm}ddlm}ddlm}t d}||}|}t|d|d<t|dts|d |d<n| |d|d<i}d |vr&d |d vr|d d i}t|d d <t|d d <t!t"|d <t$j|d} t+| d5} ||| dddn #1swxYwYt+t$j|dd5} t/jd|d d| t1|t4jt4jd| dddn #1swxYwY|D]v\} } || dt$j|d| } t%j| d|| }| | | dw|!|S)a Converts an Algorithm checkpoint (pickle based) to a msgpack based one. Msgpack has the advantage of being python version independent. Args: checkpoint: The directory, in which to find the Algorithm checkpoint (pickle based). msgpack_checkpoint_dir: The directory, in which to create the new msgpack based checkpoint. Returns: The directory in which the msgpack checkpoint has been created. Note that this is the same as `msgpack_checkpoint_dir`. r)r)AlgorithmConfig)validate_module_idTrUalgorithm_classconfigworker policy_statespolicy_mapping_fnis_policy_to_trainr'zalgorithm_state.msgpckwbNrrCrr)rpr'rrrrrrexist_ok policy_statecheckpoint_format)"ray.rllib.algorithmsr%ray.rllib.algorithms.algorithm_configrray.rllib.core.rl_modulerr{r __getstate__rr^dict serialize_serialize_dictryrr\rosr!joinopenrorjlistkeysr7rrrmakedirs get_policyexport_checkpointstop)rrrrrralgorr!rrpidr( policy_dirpolicys r0convert_to_msgpack_checkpointr>s[&/.....EEEEEE;;;;;;!t,,,G  $ $Z 0 0D     E .e4E.FGGE  eHot , ,K/3355h)99%/JJhM5_h??h++OR@@ ,rBr{r,r2r0r[s    """"""FFFFFFFFFFFFFFFFFF  '&&&&&CCCCCCKJJJJJJJ,,,,,,444444222222??????******  8 $ $.%W_U++,;GOE,B,B) Wp p p p p SWp p p f67:0666666 ;wz*;t;;;;  59CCc+_<=C01C #s(^CCC CL X"c+_<=X"X" X"X"X" X"v  "S"2OCD " "  " " "  "F d r2