&`i0ddlZddlZddlmZmZmZmZmZddlZ ddl Z ddl Z ddl m Z ddlmZddlmZddlmZddlmZddlmZmZdd lmZdd lmZdd lmZmZdd l m!Z!m"Z"dd l#m$Z$erddl%m&Z&ej'dZ(e\Z)Z*Z+dZ,dZ-dZ.e"dGddeej/Z0dS)N) TYPE_CHECKINGAnyDictOptionalTuple)COMPONENT_RL_MODULE)StepFailedRecreateEnvError)FaultAwareApply)update_global_seed_if_necessary) try_import_tf)ENV_RESET_TIMERENV_STEP_TIMER) MetricsLogger)convert_to_torch_tensor) StateDict TensorType) DeveloperAPI PublicAPI)Counter)AlgorithmConfigz ray.rllibenv_reset_failureenv_step_failurenum_env_step_failuresalpha) stabilityc eZdZdZdfd ZejdZdZdZ ejde fd Z de fd Z e deejeeffd Zejdeeeejejfffd ZddZddZd d ddeedeedee e ffdZdZdefdZ de!dedd fdZ"xZ#S) EnvRunneraBase class for distributed RL-style data collection from an environment. The EnvRunner API's core functionalities can be summarized as: - Gets configured via passing a AlgorithmConfig object to the constructor. Normally, subclasses of EnvRunner then construct their own environment (possibly vectorized) copies and RLModules/Policies and use the latter to step through the environment in order to collect training data. - Clients of EnvRunner can use the `sample()` method to collect data for training from the environment(s). - EnvRunner offers parallelism via creating n remote Ray Actors based on this class. Use `ray.remote([resources])(EnvRunner)` method to create the corresponding Ray remote class. Then instantiate n Actors using the Ray `[ctor].remote(...)` syntax. - EnvRunner clients can get information about the server/node on which the individual Actors are running. configrc |d|_d|_|d|_|d|jj|_d|_t|j d|_ t trI|jjdks|jr2tstd|_|jj8t)|jj|jpdzd |jjzz|_t-|jj|j t/d d d |_|jd|jjit/ddd |_|jd|jjidS)zInitializes an EnvRunner instance. Args: config: The AlgorithmConfig to use to setup this EnvRunner. **kwargs: Forward compatibility kwargs. F) copy_frozenr worker_index num_workersN)stats_cls_lookuproottf2g.A) frameworkseed)rllib_env_runner_num_try_env_step_counterz8Number of env.step() calls attempted in this Env Runner.)rllib)name descriptiontag_keysr).rllib_env_runner_num_env_steps_sampled_counterz/Number of env steps sampled in this Env Runner.)copyrnum_env_steps_sampled_lifetimegetr!num_env_runnersr"envrr#metricssuper__init__tf1 framework_strenable_tf1_exec_eagerlyexecuting_eagerlyenable_eager_execution_seedr'int in_evaluationr r_metrics_num_try_env_stepset_default_tags __class____name___metrics_num_env_steps_sampled)selfrkwargsr@s l/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/env/env_runner.pyr5zEnvRunner.__init__5s(.{{u{'E'E ./+ "(N!;!; & =$+:U V V&3#4' ' '    )*e33v7U3))++4  & & ( ( (%) ;  '  $)+224DJ (k/    *1<R* * * & &77 dn- .   /6AI/ / / + +<< dn- .     cdS)aHChecks that self.__init__() has been completed properly. Useful in case an `EnvRunner` is run as @ray.remote (Actor) and the owner would like to make sure the Ray Actor has been properly initialized. Raises: AssertionError: If the EnvRunner Actor has NOT been properly initialized. NrCs rEassert_healthyzEnvRunner.assert_healthy{rFcdS)aCreates the RL environment for this EnvRunner and assigns it to `self.env`. Note that users should be able to change the EnvRunner's config (e.g. change `self.config.env_config`) and then call this method to create new environments with the updated configuration. It should also be called after a failure of an earlier env in order to clean up the existing env (for example `close()` it), re-create a new one, and then continue sampling with that new env. NrHrIs rEmake_envzEnvRunner.make_envs  rFcdS)a(Creates the RLModule for this EnvRunner and assigns it to `self.module`. Note that users should be able to change the EnvRunner's config (e.g. change `self.config.rl_module_spec`) and then call this method to create a new RLModule with the updated configuration. NrHrIs rE make_modulezEnvRunner.make_modules  rFreturnc dS)a`Returns experiences (of any form) sampled from this EnvRunner. The exact nature and size of collected data are defined via the EnvRunner's config and may be overridden by the given arguments. Args: **kwargs: Forward compatibility kwargs. Returns: The collected experience in any form. NrH)rCrDs rEsamplezEnvRunner.samplerKrFcdS)zReturns metrics (in any form) of the thus far collected, completed episodes. Returns: Metrics of any form. NrHrIs rE get_metricszEnvRunner.get_metrics rFc|}|t}|}t j|||fS)a Convenience method for fast, async algorithms. Use this in Algorithms that need to sample Episode lists as ray.ObjectRef, but also require (in the same remote call) the metrics and the EnvRunner states, except for the module weights. )not_components)rR get_staterrTrayput)rC _episodes_connector_states_metricss rEsample_get_state_and_metricsz&EnvRunner.sample_get_state_and_metricssQKKMM  NN:MNNN##%%wy!!#4h>>rFcdS)zFReturns a dict mapping ModuleIDs to 2-tuples of obs- and action space.NrHrIs rE get_spaceszEnvRunner.get_spacesrKrFNcdS)zReleases all resources used by this EnvRunner. For example, when using a gym.Env in this EnvRunner, you should make sure that its `close()` method is called. NrHrIs rEstopzEnvRunner.stoprUrFcdS)z:If this Actor is deleted, clears all resources used by it.NrHrIs rE__del__zEnvRunner.__del__s rFr'optionsr'rfc |jt5|j||\}}dddn #1swxYwY||fS#t $rk}|jjrXt d|j d| | ||cYd}~S|d}~wwxYw)aTries resetting the env and - if an error occurs - handles it gracefully. Args: seed: An optional seed (int) to be passed to the Env.reset() call. options: An optional options-dict to be passed to the Env.reset() call. Returns: The results of calling `Env.reset()`, which is a tuple of observations and info dicts. Raises: Exception: In case `config.restart_failed_sub_environments` is False and `Env.reset()` resulted in an error. reNz?Resetting the env resulted in an error! The original error is: r) r3log_timer r2reset Exceptionrrestart_failed_sub_environmentslogger exceptionargsrM_try_env_reset)rCr'rfobsinfoses rErozEnvRunner._try_env_resetsD* &&77 H H!X^^w^GG U H H H H H H H H H H H H H H H:    {:   '6!9''  **g*FFFFFFFF sGA A  A AAAA C&AC C C  Cc  |jt5|j|}dddn #1swxYwY||jd|S#t$r}|jtdd|j j r_t|ts*td|jjd||t&cYd}~Std|jjdt)|d |t*|d}~wwxYw) zHTries stepping the env and - if an error occurs - handles it gracefully.N)metric num_steps lifetime_sum)reduceRLlib z]: Environment step failed. Will force reset env(s) in this EnvRunner. The original error is: a&: Environment step failed and 'config.restart_failed_sub_environments' is False. This env will not be recreated. Consider setting 'fault_tolerance(restart_failed_sub_environments=True)' in your AlgorithmConfig in order to automatically re-create and force-reset an env.The original error type: z. )r3rhrr2step_log_env_stepsr>rj log_valueNUM_ENV_STEP_FAILURES_LIFETIMErrk isinstancer rlrmr@rArMENV_STEP_FAILUREtype RuntimeError)rCactionsresultsrrs rE _try_env_stepzEnvRunner._try_env_steps *&&~66 1 1(--00 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1   t'EQR  S S SN * * * L " ".. #   {: *!!%?@@$$[!8[[XY[[ (''''''  T^4 15Q   #)5 *sGA1A A1A  A1A  A11 E;BE EAE  Ec~|jjdkrt|Stjt j|S)z0Converts structs to a framework-specific tensor.torch)rr7rtree map_structuretfconvert_to_tensor)rCstructs rE_convert_to_tensorzEnvRunner._convert_to_tensor#s8 ; $ / /*622 2%b&:FCC CrFrurvc |dkr||dStd|jjd|jdd|ddS)Nr)valueryz*: Skipping Prometheus logging for metric 'r*z'. Received num_steps=z1, but the number of steps must be greater than 0.)incrlwarningr@rAinfo)rCrurvs rEr{zEnvRunner._log_env_steps+s q== JJYJ ' ' ' ' ' NNc0cc\b\ghn\occ&/ccc     rF)rr)rPN)$rA __module__ __qualname____doc__r5abcabstractmethodrJrMrOrrRrTrrrY ObjectRefrr^rstrgymSpacer`rbrdrr<dictrorrrrr{ __classcell__)r@s@rErr#s% D D D D D D L              #       S    ? s}i2 3???\?" UDeCIsy,@&A!ABUUUU        #"& '''sm'$ ' sCx ''''R"*"*"*HDJDDDDWrFr) metaclass)1rloggingtypingrrrrr gymnasiumrrrYray.rllib.corerray.rllib.env.env_errorsr ray.rllib.utils.actor_managerr ray.rllib.utils.debugr ray.rllib.utils.frameworkr ray.rllib.utils.metricsr r&ray.rllib.utils.metrics.metrics_loggerrray.rllib.utils.torch_utilsrray.rllib.utils.typingrrray.util.annotationsrrray.util.metricsr%ray.rllib.algorithms.algorithm_configr getLoggerrlr6r_ENV_RESET_FAILURErr}ABCMetarrHrFrErs <<<<<<<<<<<<<< ......??????999999AAAAAA333333CCCCCCCC@@@@@@??????8888888888888888$$$$$$FEEEEEE  ; ' ' ]__ R'%!8  WNNNNN3;NNNNNNrF