&`i>lddlZddlmZmZmZmZmZmZmZm Z m Z ddl Z ddl Z ddlmZddlmZmZmZmZerddlmZdZejeZeGddZd Zeefd eeefd d d efdZe ddedeegefdededededded dfdZ dS)N) TYPE_CHECKINGAnyCallableDictListOptionalSetTupleUnion) OldAPIStack)AgentIDEnvIDEnvType MultiEnvDict) RolloutWorkerasync_reset_returnc eZdZdZ ddeeegefdeded ed ed df d Z d e e e e e e e ffd Z de d dfdZ dddddeedeedeed e ee ee ffdZddeed dfdZd ded eeeeffdZd eefdZddeed dfdZd!dZed ejfdZed ejfdZd e e e e e e ffdZdS)"BaseEnva3The lowest-level env interface used by RLlib for sampling. BaseEnv models multiple agents executing asynchronously in multiple vectorized sub-environments. A call to `poll()` returns observations from ready agents keyed by their sub-environment ID and agent IDs, and actions for those agents can be sent back via `send_actions()`. All other RLlib supported env types can be converted to BaseEnv. RLlib handles these conversions internally in RolloutWorker, for example: gym.Env => rllib.VectorEnv => rllib.BaseEnv rllib.MultiAgentEnv (is-a gym.Env) => rllib.VectorEnv => rllib.BaseEnv rllib.ExternalEnv => rllib.BaseEnv .. testcode:: :skipif: True MyBaseEnv = ... env = MyBaseEnv() obs, rewards, terminateds, truncateds, infos, off_policy_actions = ( env.poll() ) print(obs) env.send_actions({ "env_0": { "car_0": 0, "car_1": 1, }, ... }) obs, rewards, terminateds, truncateds, infos, off_policy_actions = ( env.poll() ) print(obs) print(terminateds) .. testoutput:: { "env_0": { "car_0": [2.4, 1.6], "car_1": [3.4, -3.2], }, "env_1": { "car_0": [8.0, 4.1], }, "env_2": { "car_0": [2.3, 3.3], "car_1": [1.4, -0.2], "car_3": [1.2, 0.1], }, } { "env_0": { "car_0": [4.1, 1.7], "car_1": [3.2, -4.2], }, ... } { "env_0": { "__all__": False, "car_0": False, "car_1": True, }, ... } NFrmake_envnum_envs remote_envsremote_env_batch_wait_msrestart_failed_sub_environmentsreturnc|S)aConverts an RLlib-supported env into a BaseEnv object. Supported types for the `env` arg are gym.Env, BaseEnv, VectorEnv, MultiAgentEnv, ExternalEnv, or ExternalMultiAgentEnv. The resulting BaseEnv is always vectorized (contains n sub-environments) to support batched forward passes, where n may also be 1. BaseEnv also supports async execution via the `poll` and `send_actions` methods and thus supports external simulators. TODO: Support gym3 environments, which are already vectorized. Args: env: An already existing environment of any supported env type to convert/wrap into a BaseEnv. Supported types are gym.Env, BaseEnv, VectorEnv, MultiAgentEnv, ExternalEnv, and ExternalMultiAgentEnv. make_env: A callable taking an int as input (which indicates the number of individual sub-environments within the final vectorized BaseEnv) and returning one individual sub-environment. num_envs: The number of sub-environments to create in the resulting (vectorized) BaseEnv. The already existing `env` will be one of the `num_envs`. remote_envs: Whether each sub-env should be a @ray.remote actor. You can set this behavior in your config via the `remote_worker_envs=True` option. remote_env_batch_wait_ms: The wait time (in ms) to poll remote sub-environments for, if applicable. Only used if `remote_envs` is True. policy_config: Optional policy config dict. Returns: The resulting BaseEnv object. )selfrrrrrs j/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/env/base_env.py to_base_envzBaseEnv.to_base_envYs V ct)a Returns observations from ready agents. All return values are two-level dicts mapping from EnvID to dicts mapping from AgentIDs to (observation/reward/etc..) values. The number of agents and sub-environments may vary over time. Returns: Tuple consisting of: New observations for each ready agent. Reward values for each ready agent. If the episode is just started, the value will be None. Terminated values for each ready agent. The special key "__all__" is used to indicate episode termination. Truncated values for each ready agent. The special key "__all__" is used to indicate episode truncation. Info values for each ready agent. Agents may take off-policy actions, in which case, there will be an entry in this dict that contains the taken action. There is no need to `send_actions()` for agents that have already chosen off-policy actions. NotImplementedErrorrs rpollz BaseEnv.polls <"!r! action_dictct)a Called to send actions back to running agents in this env. Actions should be sent for each ready agent that returned observations in the previous poll() call. Args: action_dict: Actions values keyed by env_id and agent_id. r#)rr's r send_actionszBaseEnv.send_actionss "!r!)seedoptionsenv_idr*r+cdS)a+Attempt to reset the sub-env with the given id or all sub-envs. If the environment does not support synchronous reset, a tuple of (ASYNC_RESET_REQUEST, ASYNC_RESET_REQUEST) can be returned here. Note: A MultiAgentDict is returned when using the deprecated wrapper classes such as `ray.rllib.env.base_env._MultiAgentEnvToBaseEnv`, however for consistency with the poll() method, a `MultiEnvDict` is returned from the new wrapper classes, such as `ray.rllib.env.multi_agent_env.MultiAgentEnvWrapper`. Args: env_id: The sub-environment's ID if applicable. If None, reset the entire Env (i.e. all sub-environments). seed: The seed to be passed to the sub-environment(s) when resetting it. If None, will not reset any existing PRNG. If you pass an integer, the PRNG will be reset even if it already exists. options: An options dict to be passed to the sub-environment(s) when resetting it. Returns: A tuple consisting of a) the reset (multi-env/multi-agent) observation dict and b) the reset (multi-env/multi-agent) infos dict. Returns the (ASYNC_RESET_REQUEST, ASYNC_RESET_REQUEST) tuple, if not supported. )NNr)rr,r*r+s r try_resetzBaseEnv.try_resets @zr!cdS)a<Attempt to restart the sub-env with the given id or all sub-envs. This could result in the sub-env being completely removed (gc'd) and recreated. Args: env_id: The sub-environment's ID, if applicable. If None, restart the entire Env (i.e. all sub-environments). Nrrr,s r try_restartzBaseEnv.try_restarts tr!as_dictc|riSgS)zReturn a reference to the underlying sub environments, if any. Args: as_dict: If True, return a dict mapping from env_id to env. Returns: List or dictionary of the underlying sub environments or [] / {}. r)rr2s rget_sub_environmentszBaseEnv.get_sub_environmentss  I r!ciS)z|Return the agent ids for the sub_environment. Returns: All agent ids for each the environment. rr%s r get_agent_idszBaseEnv.get_agent_idss  r!cdS)zTries to render the sub-environment with the given id or all. Args: env_id: The sub-environment's ID, if applicable. If None, renders the entire Env (i.e. all sub-environments). Nrr0s r try_renderzBaseEnv.try_renders  r!c||D]&}t|dr|'dS)zReleases all resources used.closeN)r4hasattrr:)renvs rstopz BaseEnv.stopsI,,..  CsG$$    r!ct)a Returns the observation space for each agent. Note: samples from the observation space need to be preprocessed into a `MultiEnvDict` before being used by a policy. Returns: The observation space for each environment. r#r%s robservation_spacezBaseEnv.observation_space "!r!ct)a Returns the action space for each agent. Note: samples from the action space need to be preprocessed into a `MultiEnvDict` before being passed to `send_actions`. Returns: The observation space for each environment. r#r%s r action_spacezBaseEnv.action_spacer@r!cDtdiiiiifS)a Returns the last observations, rewards, done- truncated flags and infos ... that were returned by the environment. Returns: The last observations, rewards, done- and truncated flags, and infos for each sub-environment. z3last has not been implemented for this environment.)loggerwarningr%s rlastz BaseEnv.lasts( LMMM2r2r!!r!)NrFrFN)F)rN) __name__ __module__ __qualname____doc__rrintrboolr r rr&r)rdictr.r1r rr4r r r6r8r=propertygymSpacer?rBrFrr!rrrsCCN8<!()05 ++8SE7N34++ + #& + *. + ++++Z"    """"@ " " " " " "#' #"&    sm  $  x %x '== >    D  (5/ T      D U4=RVCV=W    s7|   %  D      "39 " " "X " "ci " " "X " " |\<|S T " " " " " "r!ragent0env_id_to_valuesdummy_idr rczi}|D]#\}}t|tr|n||i||<$|SrG)items isinstance Exception)rSrTretr,values rwith_dummy_agent_idr[1sT C+1133SS *%;;Ree(EARF Jr!rFr<rrrrworkerrrc ddlm}ddlm}ddlm} ddlm} m} |r|dkrtdt|t|| |fr| |||||S|rQt|d r+tj|jnd } | ||| ||g|| }n1| ||g||j|j| }| |}t|ts J||S) aConverts an RLlib-supported env into a BaseEnv object. Supported types for the `env` arg are gym.Env, BaseEnv, VectorEnv, MultiAgentEnv, ExternalEnv, or ExternalMultiAgentEnv. The resulting BaseEnv is always vectorized (contains n sub-environments) to support batched forward passes, where n may also be 1. BaseEnv also supports async execution via the `poll` and `send_actions` methods and thus supports external simulators. TODO: Support gym3 environments, which are already vectorized. Args: env: An already existing environment of any supported env type to convert/wrap into a BaseEnv. Supported types are gym.Env, BaseEnv, VectorEnv, MultiAgentEnv, ExternalEnv, and ExternalMultiAgentEnv. make_env: A callable taking an int as input (which indicates the number of individual sub-environments within the final vectorized BaseEnv) and returning one individual sub-environment. num_envs: The number of sub-environments to create in the resulting (vectorized) BaseEnv. The already existing `env` will be one of the `num_envs`. remote_envs: Whether each sub-env should be a @ray.remote actor. You can set this behavior in your config via the `remote_worker_envs=True` option. remote_env_batch_wait_ms: The wait time (in ms) to poll remote sub-environments for, if applicable. Only used if `remote_envs` is True. worker: An optional RolloutWorker that owns the env. This is only used if `remote_worker_envs` is True in your config and the `on_sub_environment_created` custom callback needs to be called on each created actor. restart_failed_sub_environments: If True and any sub-environment (within a vectorized env) throws any error during env stepping, the Sampler will try to restart the faulty sub-environment. This is done without disturbing the other (still intact) sub-environment and without the RolloutWorker crashing. Returns: The resulting BaseEnv object. r) ExternalEnv) MultiAgentEnv) RemoteBaseEnv) VectorEnvVectorEnvWrapperrz_Remote envs only make sense to use if num_envs > 1 (i.e. environment vectorization is enabled).)rrrrr_is_multi_agentF) multiagentr existing_envsr\r)rrerrBr?r)ray.rllib.env.external_envr^ray.rllib.env.multi_agent_envr_ray.rllib.env.remote_base_envr`ray.rllib.env.vector_envrarb ValueErrorrWrr r;raygetrcremotevectorize_gym_envsrBr?) r<rrrrr\rr^r_r`rarbrds rconvert_to_base_envro>sl766666;;;;;;;;;;;;DDDDDDDD x1}} ;   # ;GHH((#%=,K      ( 3 122+2244555   -%)A"e0OCC..!"e! -"%"70O /C#"3''C c7 # #((S(( # Jr!)NrFrNF)!loggingtypingrrrrrrr r r gymnasiumrPrkray.rllib.utils.annotationsr ray.rllib.utils.typingr rrr#ray.rllib.evaluation.rollout_workerrASYNC_RESET_RETURN getLoggerrHrDr_DUMMY_AGENT_IDr[rLrMrorr!rrys XXXXXXXXXXXXXXXXXXXXXX 333333HHHHHHHHHHHHBAAAAAA)  8 $ $ W"W"W"W"W"W"W" W"v >M  5#:& 2;       *.$%(,,1nn nug~&nn n " n _ % n&*nnnn nnnr!