&`iddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lm Z m!Z!m"Z"dd l#m$Z$e$d GddZ%dS)N) defaultdict) AnyCallable Collection DefaultDictDictListOptionalSetUnion) Deprecated)SingleAgentEpisode)InfiniteLookbackBuffer)MultiAgentBatch) force_list)MultiAgentEnvError)batch)AgentIDModuleIDMultiAgentDict) PublicAPIalpha) stabilityc% eZdZdZdZdZ drdddddddddddddddddeed eee d ee j d eee d eee d ee j deee de e e fde e e fdeee deedeeeefde eefdeeeefdeeeefdeeedgeff dZddd e d ee ddfdZ drddddd e d e de d ee dee dee dee ddfdZdsdZedZede fd Zede fd!Zed"Zdtd#Zdud%Zdvdeddfd'Zedeefd(Z ede fd)Z!d*edeefd+Z" dwd,ddddd-d.ee eeee#fd/ee e$eefd0e d1e d2ee%d3e d4e de e ee ffd5Z& dwd,dddd6d.ee eeee#fd/ee e$eefd0e d1e d2ee%d4e de e ee ffd7Z' dwd,ddddd-d.ee eeee#fd/ee e$eefd0e d1e d2ee%d3e d4e de e ee ffd8Z( dwd,dddd6d.ee eeee#fd/ee e$eefd0e d1e d2ee)d4e de e ee ffd9Z* dxd,dddd6d:eed.ee eeee#fd/ee e$eefd0e d1e d2ee%d4e de e ee ffd;Z+de fd<Z,de fd=Z-dd>d?e#deeddfd@Z#dAZ.dBZ/dsdCZ0deee%ffdDZ1e2dEeee%fddfdFZ3de4fdGZ5 dydHe de)fdIZ6deefdJZ7deefdKZ8de)fdLZ9dddMdNe dOee eeee#fd1e ddfdPZ:dddMdNe dOee eeee#fd1e ddfdQZ;dddMdNe dOee eeee#fd1e ddfdRZdefdUZ?dVe#ddfdWZ@ddddddddddX deeeefdeeeefd eee d eee deee d eee de e e fde e e fdeee fdYZAdd,ddddddZd[ZBd\ZCd]ed.e ee#eefd/e$ed1e d2e%d^edee fd_ZDd]ed.e ee#eefd/e$ed1e d2e%d3e d^ede fd`ZEd]edaeFd*edbe eefd2e%d3eGd^edce%de%fddZHddddded]ed*edfe eefd2ee%d3e d^eedcee%de%fdgZId]ed*ede%fdhZJd*ed$dddfdiZKd*eddfdjZLd*eddfdkZM dxd*ed]ed^eedcee%fdlZNeOdmd,ndoZPeOdpd,ndqZQdS)zMultiAgentEpisodeaStores multi-agent episode data. The central attribute of the class is the timestep mapping `self.env_t_to_agent_t` that maps AgentIDs to their specific environment steps to the agent's own scale/timesteps. Each AgentID in the `MultiAgentEpisode` has its own `SingleAgentEpisode` object in which this agent's data is stored. Together with the env_t_to_agent_t mapping, we can extract information either on any individual agent's time scale or from the (global) multi-agent environment time scale. Extraction of data from a MultiAgentEpisode happens via the getter APIs, e.g. `get_observations()`, which work analogous to the ones implemented in the `SingleAgentEpisode` class. Note that recorded `terminateds`/`truncateds` come as simple `MultiAgentDict`s mapping AgentID to bools and thus have no assignment to a certain timestep (analogous to a SingleAgentEpisode's single `terminated/truncated` boolean flag). Instead we assign it to the last observation recorded. Theoretically, there could occur edge cases in some environments where an agent receives partial rewards and then terminates without a last observation. In these cases, we duplicate the last observation. Also, if no initial observation has been received yet for an agent, but some rewards for this same agent already occurred, we delete the agent's data up to here, b/c there is nothing to learn from these "premature" rewards. )id_agent_to_module_mapping_fn_agent_to_module_mappingobservation_space action_space env_t_startedenv_tagent_t_startedenv_t_to_agent_t_hanging_actions_end _hanging_extra_model_outputs_end_hanging_rewards_end_hanging_rewards_begin is_terminated is_truncatedagent_episodes_last_step_time_len_lookback_buffers _start_time _custom_dataSNFauto) observationsrinfosactionsr rewards terminateds truncatedsextra_model_outputsr!r#len_lookback_bufferagent_episode_idsagent_module_idsrrr2rr3r4r r5r6r7r8r!r#r9r:r;rc |ptjj|_| ddlm}|j}||_|pi|_| dkrt|pg} | |_ |pi|_ |pi|_ |pi}| pi} | pd|_ |t|nd|j z |j z|_tt | pi|_tt$|_i|_tt*|_tt.|_tt.|_t5|t6r|n|dd|_t5| t6r| n| dd|_i|_| |||||||| |  i|_!d|_"d|_#|$dS)aInitializes a `MultiAgentEpisode`. Args: id_: Optional. Either a string to identify an episode or None. If None, a hexadecimal id is created. In case of providing a string, make sure that it is unique, as episodes get concatenated via this string. observations: A list of dictionaries mapping agent IDs to observations. Can be None. If provided, should match all other episode data (actions, rewards, etc.) in terms of list lengths and agent IDs. observation_space: An optional gym.spaces.Dict mapping agent IDs to individual agents' spaces, which all (individual agents') observations should abide to. If not None and this MultiAgentEpisode is numpy'ized (via the `self.to_numpy()` method), and data is appended or set, the new data will be checked for correctness. infos: A list of dictionaries mapping agent IDs to info dicts. Can be None. If provided, should match all other episode data (observations, rewards, etc.) in terms of list lengths and agent IDs. actions: A list of dictionaries mapping agent IDs to actions. Can be None. If provided, should match all other episode data (observations, rewards, etc.) in terms of list lengths and agent IDs. action_space: An optional gym.spaces.Dict mapping agent IDs to individual agents' spaces, which all (individual agents') actions should abide to. If not None and this MultiAgentEpisode is numpy'ized (via the `self.to_numpy()` method), and data is appended or set, the new data will be checked for correctness. rewards: A list of dictionaries mapping agent IDs to rewards. Can be None. If provided, should match all other episode data (actions, rewards, etc.) in terms of list lengths and agent IDs. terminateds: A boolean defining if an environment has terminated OR a MultiAgentDict mapping individual agent ids to boolean flags indicating whether individual agents have terminated. A special __all__ key in these dicts indicates, whether the episode is terminated for all agents. The default is `False`, i.e. the episode has not been terminated. truncateds: A boolean defining if the environment has been truncated OR a MultiAgentDict mapping individual agent ids to boolean flags indicating whether individual agents have been truncated. A special __all__ key in these dicts indicates, whether the episode is truncated for all agents. The default is `False`, i.e. the episode has not been truncated. extra_model_outputs: A list of dictionaries mapping agent IDs to their corresponding extra model outputs. Each of these "outputs" is a dict mapping keys (str) to model output values, for example for `key=STATE_OUT`, the values would be the internal state outputs for that agent. env_t_started: The env timestep (int) that defines the starting point of the episode. This is only larger zero, if an already ongoing episode chunk is being created, for example by slicing an ongoing episode or by calling the `cut()` method on an ongoing episode. agent_t_started: A dict mapping AgentIDs to the respective agent's (local) timestep at which its SingleAgentEpisode chunk started. len_lookback_buffer: The size of the lookback buffers to keep in front of this Episode for each type of data (observations, actions, etc..). If larger 0, will interpret the first `len_lookback_buffer` items in each type of data as NOT part of this actual episode chunk, but instead serve as "historical" record that may be viewed and used to derive new data from. For example, it might be necessary to have a lookback buffer of four if you would like to do observation frame stacking and your episode has been cut and you are now operating on a new chunk (continuing from the cut one). Then, for the first 3 items, you would have to be able to look back into the old chunk's data. If `len_lookback_buffer` is "auto" (default), will interpret all provided data in the constructor as part of the lookback buffers. agent_episode_ids: An optional dict mapping AgentIDs to their corresponding `SingleAgentEpisode`. If None, each `SingleAgentEpisode` in `MultiAgentEpisode.agent_episodes` will generate a hexadecimal code. If a dictionary is provided, make sure that IDs are unique, because the agents' `SingleAgentEpisode` instances are concatenated or recreated by it. agent_module_ids: An optional dict mapping AgentIDs to their respective ModuleIDs (these mapping are always valid for an entire episode and thus won't change during the course of this episode). If a mapping from agent to module has already been provided via this dict, the (optional) `agent_to_module_mapping_fn` will NOT be used again to map the same agent (agents do not change their assigned module in the course of one episode). agent_to_module_mapping_fn: A callable taking an AgentID and a MultiAgentEpisode as args and returning a ModuleID. Used to map agents that have not been mapped yet (because they just entered this episode) to a ModuleID. The resulting ModuleID is only stored inside the agent's SingleAgentEpisode object. Nr)AlgorithmConfigr1__all__F) r;r:r2r3r4r5r6r7r8)%uuiduuid4hexr%ray.rllib.algorithms.algorithm_configr="DEFAULT_AGENT_TO_MODULE_MAPPING_FNrrlenr-rr r!r"rintr#rr$r%dictr&floatr'r( isinstanceboolgetr)r*r+_init_single_agent_episodesr/r.r,validate)selfrr2rr3r4r r5r6r7r8r!r#r9r:r;rr=s u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/env/multi_agent_episode.py__init__zMultiAgentEpisode.__init__Ws/T/tz||/ % - M M M M M M B '+E' BRAWUW% & ( ("%gm"4"4 %8"!2!8b(.B!'R %2 +/a$0S\\\a( )  ! +30E2FF" . / / %'!0;D0A0A-$/$6$6! '2%&8&8# +t,, 3KKE22 *d++ 2JJ 511 BD ((-/%#! 3 )  # )r3returnc f|jrJ|j|jcxkrdksnJ|pi}|D]\}}|j|d||jvrdt||||j |j ||j ||j|<|j| || ||tj|_dS)aEStores initial observation. Args: observations: A dictionary mapping agent IDs to initial observations. Note that some agents may not have an initial observation. infos: A dictionary mapping agent IDs to initial info dicts. Note that some agents may not have an initial info dict. If not None, the agent IDs in `infos` must be a subset of those in `observations` meaning it would not be allowed to have an agent with an info dict, but not with an observation. ragent_id module_idmulti_agent_episode_idrr  observationr3N)is_doner"r!itemsr$appendr+r module_forrrrJr add_env_resetrLtime perf_counterr.)rMr2r3rT agent_obss rNr]zMultiAgentEpisode.add_env_reset+sV"<zT/44441444444 $0#5#5#7#7   Hi  !( + 2 21 5 5 5t2220B%"ooh77+/8&*&<&@&@&J&J!%!2!6!6x!@!@ 111#H-   ) 7 7%ii)) 8      ,..rP)r6r7r8cX |jrtd|pi}pipi|pi}|xjdz c_dd|_dd|_t d|jDfdDzfdDztfd|j Drd |_t | }|j D]#\} } | |vr| |j$t | t | zt | zt  zt  zt dsdr|j n t zdhz } | D]} | |jvr]t| || |j|j| |j|  } n |j| } || } || }|| }|| }| dp|j}| dp|j}|| }| jr@t)d | ||||fDr"td | d | d|d|d|d|d |pd}| || |vrtd | dn$| ||j| d}|;|j| d}|j| d|z}n|s|r|| | |j vsJ|j | |jg|jzdgz|j|j | _| | ||j| xx|z cc<| |j| <| |vrT| |vrP|rtd | d|r| j} | j }n| |jvsJ||j| <||j| <||j| <n|j| }|s|rv||| |rtd | d| j} | j }|j| d}|j| d|z}n4| |jvr|j| xx|z cc<n|j| xx|z cc<| V| !| |||||||j |  tE| | j#jz|s|r|| |$tKj&|_'|j(|j'|_(dSdS)aAdds a timestep to the episode. Args: observations: A dictionary mapping agent IDs to their corresponding next observations. Note that some agents may not have stepped at this timestep. actions: Mandatory. A dictionary mapping agent IDs to their corresponding actions. Note that some agents may not have stepped at this timestep. rewards: Mandatory. A dictionary mapping agent IDs to their corresponding observations. Note that some agents may not have stepped at this timestep. infos: A dictionary mapping agent IDs to their corresponding info. Note that some agents may not have stepped at this timestep. terminateds: A dictionary mapping agent IDs to their `terminated` flags, indicating, whether the environment has been terminated for them. A special `__all__` key indicates that the episode is terminated for all agent IDs. terminateds: A dictionary mapping agent IDs to their `truncated` flags, indicating, whether the environment has been truncated for them. A special `__all__` key indicates that the episode is `truncated` for all agent IDs. extra_model_outputs: A dictionary mapping agent IDs to their corresponding specific model outputs (also in a dictionary; e.g. `vf_preds` for PPO). zGCannot call `add_env_step` on a MultiAgentEpisode that is already done!r>Fc&g|]\}}|j |S)rY.0aidsa_epss rN z2MultiAgentEpisode.add_env_step..s# R R R[S&6> RS R R RrPc$g|] }| | Srdrdrfrgr6s rNriz2MultiAgentEpisode.add_env_step..s#>>>s[-=>s>>>rPc$g|] }| | Srdrdrfrgr7s rNriz2MultiAgentEpisode.add_env_step..s"<<z1MultiAgentEpisode.add_env_step..s0AA3sc+&&&AAAAAArPTrSc3K|]}|duV dSrord)rfvs rNrrz1MultiAgentEpisode.add_env_step..s:** ******rPzAgent zl already had its `SingleAgentEpisode.is_done` set to True, but still received data in a following step! obs=z act=z rew=z info=z extra_model_outputs=.NzL acted (and received next obs), but did NOT receive any reward from the env!rrWz acted and then got truncated, but did NOT receive a last (truncation) observation, required for e.g. value function bootstrapping!)rXactionrewardr3 terminated truncatedr8))rYrr"rJr)r*rpr+rZall agent_idskeysr$r[SKIP_ENV_TS_TAGrr\rrr anyr%popr&r' _del_hangingextendr-lookbackr]r(_last_added_observation_last_added_infos add_env_steprDr2rLr^r_r,r.)rMr2r4r5r3r6r7r8stepped_agent_idsrTr$agent_ids_with_data sa_episode _observation_action_reward_infos _terminated _truncated_extra_model_outputsrqs `` @rNrzMultiAgentEpisode.add_env_step]sN < $   !'R %2 17R a )__Y>>&NN9e<< R RD$7$=$=$?$? R R R>>>>k>>> ?<<< > &H&000 ''(<===  !!## $ $',,..!! "',,..!! "+""$$%% &*//##$$  % ??9--1; 1J1J#((***UU K ,F ,F ,Ht222/%"ooh77+/8&*&<&@&@&J&J!%!2!6!6x!@!@  "0: (++H55Lkk(++Gkk(++GYYx((F%//(E::Pd>PK#%88MD>!12TZ?1#E &*%?-$""00\QW0XXX3H===H===8B+H5 --(g2E2E-,8888!- $.#EL'9FF $4+DDDDD:AD-h7:AD-h7-9 377AA'G*'G))(333 "0EXEEE$.#EL'9F,0+P+T+T $,,(#7;;HcJJWTGG t':::3H===H====1(;;;wF;;;''' ,"" *((<(%h/66 OOj&=&FF  ,j ,!!(+++  $022   ##3D    $ #rPcf|jD]}|dS)zValidates the episode's data. This function ensures that the data stored to a `MultiAgentEpisode` is in order (e.g. that the correct number of observations, actions, rewards are there). N)r+valuesrL)rMepss rNrLzMultiAgentEpisode.validates;&--//  C LLNNNN  rPc|jSro)r/rMs rN custom_datazMultiAgentEpisode.custom_datas   rPcbtd|jDS)z?Returns True if `self.add_env_reset()` has already been called.c3FK|]}t|jdkVdS)rN)rDr2)rfrs rNrrz-MultiAgentEpisode.is_reset..sF    ' ( (1 ,      rP)rr+rrs rNis_resetzMultiAgentEpisode.is_resetsA  "188::      rPctt|jjt fd|jDst d|dS)zDTrue, if the data in this episode is already stored as numpy arrays.c3*K|] }|juVdSro)is_numpy)rfrrs rNrrz-MultiAgentEpisode.is_numpy..s*TT3<8+TTTTTTrPz(Only some SingleAgentEpisode objects in z( are converted to numpy, others are not!)nextiterr+rrr{ RuntimeError)rMrs @rNrzMultiAgentEpisode.is_numpysT07799::;;DTTTTt7J7Q7Q7S7STTTTT *4*** rPc|jp|jS)aWhether the episode is actually done (terminated or truncated). A done episode cannot be continued via `self.add_env_step()` or being concatenated on its right-side with another episode chunk or being succeeded via `self.cut()`. Note that in a multi-agent environment this does not necessarily correspond to single agents having terminated or being truncated. `self.is_terminated` should be `True`, if all agents are terminated and `self.is_truncated` should be `True`, if all agents are truncated. If only one or more (but not all!) agents are `terminated/truncated the `MultiAgentEpisode.is_terminated/is_truncated` should be `False`. This information about single agent's terminated/truncated states can always be retrieved from the `SingleAgentEpisode`s inside the 'MultiAgentEpisode` one. If all agents are either terminated or truncated, but in a mixed fashion, i.e. some are terminated and others are truncated: This is currently undefined and could potentially be a problem (if a user really implemented such a multi-agent env that behaves this way). Returns: Boolean defining if an episode has either terminated or truncated. )r)r*rs rNrYzMultiAgentEpisode.is_dones6!6T%66rPc|jD]\}}||S)ah Converts this Episode's list attributes to numpy arrays. This means in particular that this episodes' lists (per single agent) of (possibly complex) data (e.g. an agent having a dict obs space) will be converted to (possibly complex) structs, whose leafs are now numpy arrays. Each of these leaf numpy arrays will have the same length (batch dimension) as the length of the original lists. Note that Columns.INFOS are NEVER numpy'ized and will remain a list (normally, a list of the original, env-returned dicts). This is due to the heterogeneous nature of INFOS returned by envs, which would make it unwieldy to convert this information to numpy arrays. After calling this method, no further data may be added to this episode via the `self.add_env_step()` method. Examples: .. testcode:: import numpy as np from ray.rllib.env.multi_agent_episode import MultiAgentEpisode from ray.rllib.env.tests.test_multi_agent_episode import ( TestMultiAgentEpisode ) # Create some multi-agent episode data. ( observations, actions, rewards, terminateds, truncateds, infos, ) = TestMultiAgentEpisode._mock_multi_agent_records() # Define the agent ids. agent_ids = ["agent_1", "agent_2", "agent_3", "agent_4", "agent_5"] episode = MultiAgentEpisode( observations=observations, infos=infos, actions=actions, rewards=rewards, # Note: terminated/truncated have nothing to do with an episode # being converted `to_numpy` or not (via the `self.to_numpy()` method)! terminateds=terminateds, truncateds=truncateds, len_lookback_buffer=0, # no lookback; all data is actually "in" episode ) # Episode has not been numpy'ized yet. assert not episode.is_numpy # We are still operating on lists. assert ( episode.get_observations( indices=[1], agent_ids="agent_1", ) == {"agent_1": [1]} ) # Numpy'ized the episode. episode.to_numpy() assert episode.is_numpy # Everything is now numpy arrays (with 0-axis of size # B=[len of requested slice]). assert ( isinstance(episode.get_observations( indices=[1], agent_ids="agent_1", )["agent_1"], np.ndarray) ) Returns: This `MultiAgentEpisode` object with the converted numpy data. )r+copyrZto_numpy)rMrT agent_epss rNrzMultiAgentEpisode.to_numpysN^$(#6#;#;#=#=#C#C#E#E ! ! Hi     rPotherc |j|jksJ|jrJ|j|jksJ|t |jt |jz}|D]2}|j|}|V|j||j|<|j ||j |<|j ||j |<| ||u|jr}||jvr||j vr||j vsJ||j|d|j|d|j ||j||j|z|j |||j|||j vrct)j|j ||j |<|j||j|<t)j|j ||j |<|j}t-|j |ddD]\\}}||jkr&|j ||j6|j ||dz|z]4|j|_|jrd|_n|jrd|_|j|j|dS)aAdds the given `other` MultiAgentEpisode to the right side of `self`. In order for this to work, both chunks (`self` and `other`) must fit together. This is checked by the IDs (must be identical), the time step counters (`self.env_t` must be the same as `episode_chunk.env_t_started`), as well as the observations/infos of the individual agents at the concatenation boundaries. Also, `self.is_done` must not be True, meaning `self.is_terminated` and `self.is_truncated` are both False. Args: other: The other `MultiAgentEpisode` to be concatenated to this one. Returns: A `MultiAgentEpisode` instance containing the concatenated data from both episodes (`self` and `other`). Nr)rXr3rwrxr8rbT)rrYr"r!rLrpr|r+rJr$r# _copy_hangingr%r&rget_observations get_infosr'r(concat_episoderdeepcopy enumerater~r[r)r*rupdate)rMr all_agent_idsrTrjivals rNrz MultiAgentEpisode.concat_episodesF$yDH$$$$<zU00000 DN++c%/.B.BB %7 J7 JH,00::J!050DX0N#H-272H2R%h/161Fx1P$X.""8U3333#* JU111t888#t'LLLLL++$)$8$B$S$STU$V$V#28<FFqII#8B 5h?#:8DE!A(K,   ))%*>x*HIIIu999:>-28<;;D-h7;@:T ;D-h7GKm>xHGGD9(C J'(>x(H(LMMJJFAsd222-h7>>t?STTTT-h7>>q1uqyIIII[   %!%D     % $D   1222 rPrc|dksJjrtd|}jD]\}}j|djksJt|tj|j dzD]A}j|| jkrt||dz }nBt| dz d}|dkrt| dntdd} |d} |d} |d} |d} d|d} t#did jd |d jd | d | djd| d| dddjdfdjDdjdfdjDdjdd} j| _t9jj| _ | S)aReturns a successor episode chunk (of len=0) continuing from this Episode. The successor will have the same ID as `self`. If no lookback buffer is requested (len_lookback_buffer=0), the successor's observations will be the last observation(s) of `self` and its length will therefore be 0 (no further steps taken yet). If `len_lookback_buffer` > 0, the returned successor will have `len_lookback_buffer` observations (and actions, rewards, etc..) taken from the right side (end) of `self`. For example if `len_lookback_buffer=2`, the returned successor's lookback buffer actions will be identical to the results of `self.get_actions([-2, -1])`. This method is useful if you would like to discontinue building an episode chunk (b/c you have to return it from somewhere), but would like to have a new episode instance to continue building the actual gym.Env episode at a later time. Vie the `len_lookback_buffer` argument, the continuing chunk (successor) will still be able to "look back" into this predecessor episode's data (at least to some extend, depending on the value of `len_lookback_buffer`). Args: len_lookback_buffer: The number of environment timesteps to take along into the new chunk as "lookback buffer". A lookback buffer is additional data on the left side of the actual episode data for visibility purposes (but without actually being part of the new chunk). For example, if `self` ends in actions: agent_1=5,6,7 and agent_2=6,7, and we call `self.cut(len_lookback_buffer=2)`, the returned chunk will have actions 6 and 7 for both agents already in it, but still `t_started`==t==8 (not 7!) and a length of 0. If there is not enough data in `self` yet to fulfil the `len_lookback_buffer` request, the value of `len_lookback_buffer` is automatically adjusted (lowered). Returns: The successor Episode chunk of this one with the same ID and state and the only observation being the last observation in self. rzFCan't call `MultiAgentEpisode.cut()` when the episode is already done!rbNT)indices return_list)keyrrrr2rr3r4r r5r8r6r7r!r#cZi|]'}j|j|j|j(Srd)r+rYtrfrgrMs rN z)MultiAgentEpisode.cut..sG*3/7T(-/rPr:r;c6i|]}|j|jSrd)r+rUrs rNrz)MultiAgentEpisode.cut..s3RRRRR;D,A(,K,P(Q(QTU(UVV  (266r::d>RRR*-.A1q5*I*I'ES !&':&:Q&> E E#Q&& && - - -tQ  ,,)t-  '<$OO""! ,#44- .CG>/ 4(,'F'F5 8!'9 B,0+D+I+I+K+K ("&t/?!@!@ rPcNt|jS)zReturns the agent ids.)rpr+r}rs rNr|zMultiAgentEpisode.agent_idss!4&++--...rPcHd|jDS)z3Returns ids from each agent's `SingleAgentEpisode`.c$i|] \}}||jSrdrrfrTrs rNrz7MultiAgentEpisode.agent_episode_ids..s0   #) im   rP)r+rZrs rNr:z#MultiAgentEpisode.agent_episode_idss2  '+':'@'@'B'B    rPrTcr||jvr"|||x}|j|<|S|j|S)aReturns the ModuleID for a given AgentID. Forces the agent-to-module mapping to be performed (via `self.agent_to_module_mapping_fn`), if this has not been done yet. Note that all such mappings are stored in the `self._agent_to_module_mapping` property. Args: agent_id: The AgentID to get a mapped ModuleID for. Returns: The ModuleID mapped to from the given `agent_id`. )rr)rMrTrUs rNr\zMultiAgentEpisode.module_forsR 48 8 8//$?? @I5 0: :rPT) env_stepsneg_index_as_lookbackfillone_hot_discreterrr|rrrrrc <|d|||||||S)a Returns agents' observations or batched ranges thereof from this episode. Args: indices: A single int is interpreted as an index, from which to return the individual observation stored at this index. A list of ints is interpreted as a list of indices from which to gather individual observations in a batch of size len(indices). A slice object is interpreted as a range of observations to be returned. Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. If None, will return all observations (from ts=0 to the end). agent_ids: An optional collection of AgentIDs or a single AgentID to get observations for. If None, will return observations for all agents in this episode. env_steps: Whether `indices` should be interpreted as environment time steps (True) or per-agent timesteps (False). neg_index_as_lookback: If True, negative values in `indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with agent A's observations [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will respond to `get_observations(-1, agent_ids=[A], neg_index_as_lookback=True)` with {A: `6`} and to `get_observations(slice(-2, 1), agent_ids=[A], neg_index_as_lookback=True)` with {A: `[5, 6, 7]`}. fill: An optional value to use for filling up the returned results at the boundaries. This filling only happens if the requested index range's start/stop boundaries exceed the episode's boundaries (including the lookback buffer on the left side). This comes in very handy, if users don't want to worry about reaching such boundaries and want to zero-pad. For example, an episode with agent A' observations [10, 11, 12, 13, 14] and lookback buffer size of 2 (meaning observations `10` and `11` are part of the lookback buffer) will respond to `get_observations(slice(-7, -2), agent_ids=[A], fill=0.0)` with `{A: [0.0, 0.0, 10, 11, 12]}`. one_hot_discrete: If True, will return one-hot vectors (instead of int-values) for those sub-components of a (possibly complex) observation space that are Discrete or MultiDiscrete. Note that if `fill=0` and the requested `indices` are out of the range of our data, the returned one-hot vectors will actually be zero-hot (all slots zero). return_list: Whether to return a list of multi-agent dicts (instead of a single multi-agent dict of lists/structs). False by default. This option can only be used when `env_steps` is True due to the fact the such a list can only be interpreted as one env step per list item (would not work with agent steps). Returns: A dictionary mapping agent IDs to observations (at the given `indices`). If `env_steps` is True, only agents that have stepped (were ready) at the given env step `indices` are returned (i.e. not all agent IDs are necessarily in the keys). If `return_list` is True, returns a list of MultiAgentDicts (mapping agent IDs to observations) instead. r2whatrr|rrrrr_getrMrr|rrrrrs rNrz"MultiAgentEpisode.get_observationss8Fyy"7-#   rP)rrrrc :|d||||||S)a Returns agents' info dicts or list (ranges) thereof from this episode. Args: indices: A single int is interpreted as an index, from which to return the individual info dict stored at this index. A list of ints is interpreted as a list of indices from which to gather individual info dicts in a list of size len(indices). A slice object is interpreted as a range of info dicts to be returned. Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. If None, will return all infos (from ts=0 to the end). agent_ids: An optional collection of AgentIDs or a single AgentID to get info dicts for. If None, will return info dicts for all agents in this episode. env_steps: Whether `indices` should be interpreted as environment time steps (True) or per-agent timesteps (False). neg_index_as_lookback: If True, negative values in `indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with agent A's info dicts [{"l":4}, {"l":5}, {"l":6}, {"a":7}, {"b":8}, {"c":9}], where the first 3 items are the lookback buffer (ts=0 item is {"a": 7}), will respond to `get_infos(-1, agent_ids=A, neg_index_as_lookback=True)` with `{A: {"l":6}}` and to `get_infos(slice(-2, 1), agent_ids=A, neg_index_as_lookback=True)` with `{A: [{"l":5}, {"l":6}, {"a":7}]}`. fill: An optional value to use for filling up the returned results at the boundaries. This filling only happens if the requested index range's start/stop boundaries exceed the episode's boundaries (including the lookback buffer on the left side). This comes in very handy, if users don't want to worry about reaching such boundaries and want to auto-fill. For example, an episode with agent A's infos being [{"l":10}, {"l":11}, {"a":12}, {"b":13}, {"c":14}] and lookback buffer size of 2 (meaning infos {"l":10}, {"l":11} are part of the lookback buffer) will respond to `get_infos(slice(-7, -2), agent_ids=A, fill={"o": 0.0})` with `{A: [{"o":0.0}, {"o":0.0}, {"l":10}, {"l":11}, {"a":12}]}`. return_list: Whether to return a list of multi-agent dicts (instead of a single multi-agent dict of lists/structs). False by default. This option can only be used when `env_steps` is True due to the fact the such a list can only be interpreted as one env step per list item (would not work with agent steps). Returns: A dictionary mapping agent IDs to observations (at the given `indices`). If `env_steps` is True, only agents that have stepped (were ready) at the given env step `indices` are returned (i.e. not all agent IDs are necessarily in the keys). If `return_list` is True, returns a list of MultiAgentDicts (mapping agent IDs to infos) instead. r3rrr|rrrrrrMrr|rrrrs rNrzMultiAgentEpisode.get_infosks5|yy"7#   rPc <|d|||||||S)aReturns agents' actions or batched ranges thereof from this episode. Args: indices: A single int is interpreted as an index, from which to return the individual actions stored at this index. A list of ints is interpreted as a list of indices from which to gather individual actions in a batch of size len(indices). A slice object is interpreted as a range of actions to be returned. Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. If None, will return all actions (from ts=0 to the end). agent_ids: An optional collection of AgentIDs or a single AgentID to get actions for. If None, will return actions for all agents in this episode. env_steps: Whether `indices` should be interpreted as environment time steps (True) or per-agent timesteps (False). neg_index_as_lookback: If True, negative values in `indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with agent A's actions [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will respond to `get_actions(-1, agent_ids=[A], neg_index_as_lookback=True)` with {A: `6`} and to `get_actions(slice(-2, 1), agent_ids=[A], neg_index_as_lookback=True)` with {A: `[5, 6, 7]`}. fill: An optional value to use for filling up the returned results at the boundaries. This filling only happens if the requested index range's start/stop boundaries exceed the episode's boundaries (including the lookback buffer on the left side). This comes in very handy, if users don't want to worry about reaching such boundaries and want to zero-pad. For example, an episode with agent A' actions [10, 11, 12, 13, 14] and lookback buffer size of 2 (meaning actions `10` and `11` are part of the lookback buffer) will respond to `get_actions(slice(-7, -2), agent_ids=[A], fill=0.0)` with `{A: [0.0, 0.0, 10, 11, 12]}`. one_hot_discrete: If True, will return one-hot vectors (instead of int-values) for those sub-components of a (possibly complex) observation space that are Discrete or MultiDiscrete. Note that if `fill=0` and the requested `indices` are out of the range of our data, the returned one-hot vectors will actually be zero-hot (all slots zero). return_list: Whether to return a list of multi-agent dicts (instead of a single multi-agent dict of lists/structs). False by default. This option can only be used when `env_steps` is True due to the fact the such a list can only be interpreted as one env step per list item (would not work with agent steps). Returns: A dictionary mapping agent IDs to actions (at the given `indices`). If `env_steps` is True, only agents that have stepped (were ready) at the given env step `indices` are returned (i.e. not all agent IDs are necessarily in the keys). If `return_list` is True, returns a list of MultiAgentDicts (mapping agent IDs to actions) instead. r4rrrs rNrzMultiAgentEpisode.get_actionss8Dyy"7-#   rPc :|d||||||S)a) Returns agents' rewards or batched ranges thereof from this episode. Args: indices: A single int is interpreted as an index, from which to return the individual rewards stored at this index. A list of ints is interpreted as a list of indices from which to gather individual rewards in a batch of size len(indices). A slice object is interpreted as a range of rewards to be returned. Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. If None, will return all rewards (from ts=0 to the end). agent_ids: An optional collection of AgentIDs or a single AgentID to get rewards for. If None, will return rewards for all agents in this episode. env_steps: Whether `indices` should be interpreted as environment time steps (True) or per-agent timesteps (False). neg_index_as_lookback: If True, negative values in `indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with agent A's rewards [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will respond to `get_rewards(-1, agent_ids=[A], neg_index_as_lookback=True)` with {A: `6`} and to `get_rewards(slice(-2, 1), agent_ids=[A], neg_index_as_lookback=True)` with {A: `[5, 6, 7]`}. fill: An optional float value to use for filling up the returned results at the boundaries. This filling only happens if the requested index range's start/stop boundaries exceed the episode's boundaries (including the lookback buffer on the left side). This comes in very handy, if users don't want to worry about reaching such boundaries and want to zero-pad. For example, an episode with agent A' rewards [10, 11, 12, 13, 14] and lookback buffer size of 2 (meaning rewards `10` and `11` are part of the lookback buffer) will respond to `get_rewards(slice(-7, -2), agent_ids=[A], fill=0.0)` with `{A: [0.0, 0.0, 10, 11, 12]}`. return_list: Whether to return a list of multi-agent dicts (instead of a single multi-agent dict of lists/structs). False by default. This option can only be used when `env_steps` is True due to the fact the such a list can only be interpreted as one env step per list item (would not work with agent steps). Returns: A dictionary mapping agent IDs to rewards (at the given `indices`). If `env_steps` is True, only agents that have stepped (were ready) at the given env step `indices` are returned (i.e. not all agent IDs are necessarily in the keys). If `return_list` is True, returns a list of MultiAgentDicts (mapping agent IDs to rewards) instead. r5rrrs rNrzMultiAgentEpisode.get_rewardss5xyy"7#   rPrc <|d|||||||S)aReturns agents' actions or batched ranges thereof from this episode. Args: key: The `key` within each agents' extra_model_outputs dict to extract data for. If None, return data of all extra model output keys. indices: A single int is interpreted as an index, from which to return the individual extra model outputs stored at this index. A list of ints is interpreted as a list of indices from which to gather individual extra model outputs in a batch of size len(indices). A slice object is interpreted as a range of extra model outputs to be returned. Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. If None, will return all extra model outputs (from ts=0 to the end). agent_ids: An optional collection of AgentIDs or a single AgentID to get extra model outputs for. If None, will return extra model outputs for all agents in this episode. env_steps: Whether `indices` should be interpreted as environment time steps (True) or per-agent timesteps (False). neg_index_as_lookback: If True, negative values in `indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with agent A's actions [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will respond to `get_actions(-1, agent_ids=[A], neg_index_as_lookback=True)` with {A: `6`} and to `get_actions(slice(-2, 1), agent_ids=[A], neg_index_as_lookback=True)` with {A: `[5, 6, 7]`}. fill: An optional value to use for filling up the returned results at the boundaries. This filling only happens if the requested index range's start/stop boundaries exceed the episode's boundaries (including the lookback buffer on the left side). This comes in very handy, if users don't want to worry about reaching such boundaries and want to zero-pad. For example, an episode with agent A' actions [10, 11, 12, 13, 14] and lookback buffer size of 2 (meaning actions `10` and `11` are part of the lookback buffer) will respond to `get_actions(slice(-7, -2), agent_ids=[A], fill=0.0)` with `{A: [0.0, 0.0, 10, 11, 12]}`. one_hot_discrete: If True, will return one-hot vectors (instead of int-values) for those sub-components of a (possibly complex) observation space that are Discrete or MultiDiscrete. Note that if `fill=0` and the requested `indices` are out of the range of our data, the returned one-hot vectors will actually be zero-hot (all slots zero). return_list: Whether to return a list of multi-agent dicts (instead of a single multi-agent dict of lists/structs). False by default. This option can only be used when `env_steps` is True due to the fact the such a list can only be interpreted as one env step per list item (would not work with agent steps). Returns: A dictionary mapping agent IDs to actions (at the given `indices`). If `env_steps` is True, only agents that have stepped (were ready) at the given env step `indices` are returned (i.e. not all agent IDs are necessarily in the keys). If `return_list` is True, returns a list of MultiAgentDicts (mapping agent IDs to extra_model_outputs) instead. r8)rextra_model_outputs_keyrr|rrrrr)rMrrr|rrrrs rNrz)MultiAgentEpisode.get_extra_model_outputsFs8Jyy&$'"7#   rPcffdjD}|dji|S)z&Gets the terminateds at given indices.c6i|]}|j|jSrd)r+r)rfrTrMs rNrz5MultiAgentEpisode.get_terminateds..s6    d)(3A   rPr>r|rr))rMr6s` rNrz!MultiAgentEpisode.get_terminatedssQ     N    It'9:;;;rPcffdjD}|dji|S)Nc6i|]}|j|jSrd)r+r*rs rNrz4MultiAgentEpisode.get_truncateds..s6    d)(3@   rPr>r)rMr7s` rNrz MultiAgentEpisode.get_truncatedssQ     N    9d&89:::rP)r9slice_c|jdvrtd|jd|j}|j}|d}nI|dkr!t t ||zd}n"|t |krt |}|t |}nI|dkr!t t ||zd}n"|t |krt |}d |jD]}\}} |jj |jj ksJ|j j ksJ|j j ksJtfd|j DsJ~n#t$rt!dwxYwiii}|jD]\}}|j|}|t |kr>|||jkr-t |||kr|j|<|j|<|t |krFt+|t |D](} || |jkr|j|| z||<n)tfd|jDd <tfd |jDd <||n} || z dkr| |z kr|z} |t1|| z |d zd d } |t1|| z |d d } |t1|| z |d d } |t1|| z |d d }t9|j| |j| |j| || |j |zd|jD||j!|j"}|j#r|$|S)a7 Returns a slice of this episode with the given slice object. Works analogous to :py:meth:`~ray.rllib.env.single_agent_episode.SingleAgentEpisode.slice` However, the important differences are: - `slice_` is provided in (global) env steps, not agent steps. - In case `slice_` ends - for a certain agent - in an env step, where that particular agent does not have an observation, the previous observation will be included, but the next action and sum of rewards until this point will be stored in the agent's hanging values caches for the returned MultiAgentEpisode slice. .. testcode:: from ray.rllib.env.multi_agent_episode import MultiAgentEpisode from ray.rllib.utils.test_utils import check # Generate a simple multi-agent episode. observations = [ {"a0": 0, "a1": 0}, # 0 { "a1": 1}, # 1 { "a1": 2}, # 2 {"a0": 3, "a1": 3}, # 3 {"a0": 4}, # 4 ] # Actions are the same as observations (except for last obs, which doesn't # have an action). actions = observations[:-1] # Make up a reward for each action. rewards = [ {aid: r / 10 + 0.1 for aid, r in o.items()} for o in observations ] episode = MultiAgentEpisode( observations=observations, actions=actions, rewards=rewards, len_lookback_buffer=0, ) # Slice the episode and check results. slice = episode[1:3] a0 = slice.agent_episodes["a0"] a1 = slice.agent_episodes["a1"] check((a0.observations, a1.observations), ([3], [1, 2, 3])) check((a0.actions, a1.actions), ([], [1, 2])) check((a0.rewards, a1.rewards), ([], [0.2, 0.3])) check((a0.is_done, a1.is_done), (False, False)) # If a slice ends in a "gap" for an agent, expect actions and rewards to be # cached for this agent. slice = episode[:2] a0 = slice.agent_episodes["a0"] check(a0.observations, [0]) check(a0.actions, []) check(a0.rewards, []) check(slice._hanging_actions_end["a0"], 0) check(slice._hanging_rewards_end["a0"], 0.1) Args: slice_: The slice object to use for slicing. This should exclude the lookback buffer, which will be prepended automatically to the returned slice. len_lookback_buffer: If not None, forces the returned slice to try to have this number of timesteps in its lookback buffer (if available). If None (default), tries to make the returned slice's lookback as large as the current lookback buffer of this episode (`self`). Returns: The new MultiAgentEpisode representing the requested slice. )rbNz9Slicing MultiAgentEnv with a step other than 1 (you used z) is not supported!Nrc3.K|]}|jkVdSro)r)rfilb ref_lookbacks rNrrz*MultiAgentEpisode.slice..s>LL0rPzdCan only slice a MultiAgentEpisode if all lookback buffers in this episode have the exact same size!c3BK|]}|VdSrorJrks rNrrz*MultiAgentEpisode.slice..As@% % %(KOOC % % % % % % rPr>c3BK|]}|VdSrorrms rNrrz*MultiAgentEpisode.slice..Ds/#W#WCJNN3$7$7#W#W#W#W#W#WrPrbT)rr)rrrc$i|] \}}||jSrdr)rfrgeids rNrz+MultiAgentEpisode.slice..ms-!)cSWrP)rr2rr4r r5r8r6r7r9r!r:r#r;r)%stepNotImplementedErrorstartstoprrDr+rZr2rr4r5r{r8rAssertionError ValueErrorr$r~r)r*r t_startedrrrrrrrrr r!rrrr)rMrr9rrrgrr#mappingr_lbr2r4r5r8 ma_episoderr6r7s @@@rNrzMultiAgentEpisode.slices\ ;i ' '%5K555  { =EE QYYD E)1--EE SYY  IIE <t99DD AXXs4yy4'++DD CII  t99D  #'#6#<#<#>#>  Z'#-#:#CL!.7<GGGG!)2lBBBB!)2lBBBB)=DDFF    4    #288::  OC+C0Gs7||##DMT%999 OOwt}44#-#; C ","9 3s7||##uc'll33AqzT%999/9/Cgaj/P,:"%% % % % ,0,?% % % " " I!$#W#W#W#W4CV#W#W#W W W 9&9%D!!, 3;??|sU{;;&C,, %#+tax ( ("&-  "" %#+t $ $"&#   "" %#+t $ $"&#   #::%#+t,,"&;  '%"4* 3#! #,u4-1-@-F-F-H-H,!:'+'F%   , = "    ! ! !s BE77Fc |j|jz S)a7Returns the length of an `MultiAgentEpisode`. Note that the length of an episode is defined by the difference between its actual timestep and the starting point. Returns: An integer defining the length of the episode or an error if the episode has not yet started. )r"r!rs rN__len__zMultiAgentEpisode.__len__{szD...rPc d|jD}dt|d|jd|d|jd S)Nc>i|]\}}||Srd get_returnres rNrz.MultiAgentEpisode.__repr__..s7   )4fC""$$   rPz MAEps(len=z done=z Rs=z id_=))r+rZrDrYr)rMsa_eps_returnss rN__repr__zMultiAgentEpisode.__repr__sz  8<8K8Q8Q8S8S    3T 3 3$, 3 3  3 3'+x 3 3 3 rPc td|jD}tt |jj}td|jD}dd|zzddt| ||z Dzdz}g}|j D]b\}}|dd|t|z zz}|j D]} | d kr|d z }|d z }| | ct|d|zd S) zHPrints this MultiAgentEpisode as a table of observations for the agents.c3>K|]}|VdSro)len_incl_lookback)rftss rNrrz*MultiAgentEpisode.print..s.UUR))++UUUUUUrPc34K|]}t|VdSrorD)rfrgs rNrrz*MultiAgentEpisode.print..s(??CHH??????rPr z c34K|]}t|VdSro)str)rfrs rNrrz*MultiAgentEpisode.print..s(MMAQMMMMMMrP z r0z z x N)rr$rrrrr|joinrrZrDrr[rstripprint) rMmax_tsr longest_agentheaderrowsagent inf_bufferrowrs rNrzMultiAgentEpisode.printsUUd6K6R6R6T6TUUUUUT299;;<<==F???????  ]" $jjMMy&8:K)L)LMMMMM N  !%!6!!> & & E:,,,#U)C"DEC_ " "886MCC6MCC KK % % % % ftyy&'''''rPcid|jd|jd|jd|jd|jd|jd|jd|jd |jd |j d |j d |j d |j d|j d|jdtd|jDd|j|j|jdS)zReturns the state of a multi-agent episode. Note that from an episode's state the episode itself can be recreated. Returns: A dicitonary containing pickable data for a `MultiAgentEpisode`. rrrrr r!r"r#r$r%r&r'r(r)r*r+c>i|]\}}||Srd) get_staters rNrz/MultiAgentEpisode.get_state..s:+)i1133rPr.)r,r)rrrrr r!r"r#r$r%r&r'r(r)r*listr+rZr.r,rrs rNrzMultiAgentEpisode.get_states_ 48 ($*I  '(E  !7  D-  T/   TZ  t3   5  #D$=  /0U  #D$=  %d&A  T/ D-! " d/3/B/H/H/J/J%'' # . 4+/ 0 $3+3    rPstatec^t|d}|d|_|d|_|d|_|d|_|d|_|d|_|d |_|d |_|d |_ |d |_ |d |_ |d|_ |d|_ |d|_d|dD|_|d|_|d|_|di|_||S)a>Creates a multi-agent episode from a state dictionary. See `MultiAgentEpisode.get_state()` for creating a state for a `MultiAgentEpisode` pickable state. For recreating a `MultiAgentEpisode` from a state, this state has to be complete, i.e. all data must have been stored in the state. Args: state: A dict containing all data required to recreate a MultiAgentEpisode`. See `MultiAgentEpisode.get_state()`. Returns: A `MultiAgentEpisode` instance created from the state data. rrrrrr r!r"r#r$r%r&r'r(r)r*c>i|]\}}|tj|Srd)r from_state)rfrT agent_states rNrz0MultiAgentEpisode.from_state..s:" " " %+ (3K@@" " " rPr+r.r,r)rrrrr r!r"r#r$r%r&r'r(r)r*r+r.r,rJr/rL)repisodes rNr zMultiAgentEpisode.from_statesL"$e 555-23O-P*+01K+L($)*=$>!$^4 %o 6g "'(9":#();#< ',-C'D$38 .4 0(--C'D$)./G)H& %o 6$^4" " )./?)@" " " $M2"'(9":$yy;; rPctd|jD|j|jz S)aConverts this `MultiAgentEpisode` into a `MultiAgentBatch`. Each `SingleAgentEpisode` instances in `MultiAgentEpisode.agent_epiosdes` will be converted into a `SampleBatch` and the environment timestep will be passed as the returned MultiAgentBatch's `env_steps`. Returns: A MultiAgentBatch containing all of this episode's data. cdi|]-\}}|j|jz dk||.Sr)rrget_sample_batchrs rNrz6MultiAgentEpisode.get_sample_batch..sI'Hi;!44q88)4466888rP)policy_batchesr)rr+rZr"r!rs rNr&z"MultiAgentEpisode.get_sample_batchsU+/+>+D+D+F+F j4#55     rPinclude_hanging_rewardsctd|jD}|rB|jD]}||z }|jD]}||z }|S)aXReturns all-agent return. Args: include_hanging_rewards: Whether we should also consider hanging rewards wehn calculating the overall return. Agents might have received partial rewards, i.e. rewards without an observation. These are stored in the "hanging" caches (begin and end) for each agent and added up until the next observation is received by that agent. Returns: The sum of all single-agents' returns (maybe including the hanging rewards per agent). c3>K|]}|VdSror)rfrs rNrrz/MultiAgentEpisode.get_return..+s?  '0I " "      rP)sumr+rr(r')rMr( env_return hanging_rs rNrzMultiAgentEpisode.get_returns$  484G4N4N4P4P      # (!8??AA ( ( i' !6==?? ( ( i' rPcjfddDS)aDReturns a set of agent IDs required to send an action to `env.step()` next. Those are generally the agents that received an observation in the most recent `env.step()` call. Returns: A set of AgentIDs that are supposed to send actions to the next `env.step()` call. c8h|]}j|j|Srd)r+rYrs rN z6MultiAgentEpisode.get_agents_to_act..@s:   &s+3    rPr)rr}rs`rNget_agents_to_actz#MultiAgentEpisode.get_agents_to_act6sH    ,,R005577    rPcjt|dS)aReturns a set of agent IDs of those agents that just finished stepping. These are all the agents that have an observation logged at the last env timestep, which may include agents, whose single agent episode just terminated or truncated. Returns: A set of AgentIDs of those agents that just finished stepping (that have a most recent observation on the env timestep scale), regardless of whether their single agent episodes are done or not. r)rprr}rs rNget_agents_that_steppedz)MultiAgentEpisode.get_agents_that_steppedFs,4((,,1133444rPc2|jdS|j|jz S)z8Returns the duration of this Episode (chunk) in seconds.Nrv)r,r.rs rNget_duration_sz MultiAgentEpisode.get_duration_sTs!   '3#d&666rP) at_indicesrnew_datar6c|D]D\}}||jvrtd|d|j||||EdS)a Overwrites all or some single-agent Episode's observations with the provided data. This is a helper method to batch `SingleAgentEpisode.set_observations`. For more detail, see `SingleAgentEpisode.set_observations`. Args: new_data: A dict mapping agent IDs to new observation data. Each value in the dict is the new observation data to overwrite existing data with. This may be a list of individual observation(s) in case this episode is still not numpy'ized yet. In case this episode has already been numpy'ized, this should be (possibly complex) struct matching the observation space and with a batch size of its leafs exactly the size of the to-be-overwritten slice or segment (provided by `at_indices`). at_indices: A single int is interpreted as one index, which to overwrite with `new_data` (which is expected to be a single observation). A list of ints is interpreted as a list of indices, all of which to overwrite with `new_data` (which is expected to be of the same size as `len(at_indices)`). A slice object is interpreted as a range of indices to be overwritten with `new_data` (which is expected to be of the same size as the provided slice). Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. neg_index_as_lookback: If True, negative values in `at_indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with observations = [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will handle a call to `set_observations(individual_observation, -1, neg_index_as_lookback=True)` by overwriting the value of 6 in our observations buffer with the provided "individual_observation". Raises: IndexError: If the provided `at_indices` do not match the size of `new_data`. AgentID '' not found in this episode.r7r6rN)rZr+KeyErrorset_observationsrMr7r6rrTnew_agent_datas rNr=z"MultiAgentEpisode.set_observationsZsZ)1(8(8   $Hnt222Q8QQQRRR   ) : :'%&; ;      rPc|D]D\}}||jvrtd|d|j||||EdS)aG Overwrites all or some of this Episode's actions with the provided data. This is a helper method to batch `SingleAgentEpisode.set_actions`. For more detail, see `SingleAgentEpisode.set_actions`. Args: new_data: A dict mapping agent IDs to new action data. Each value in the dict is the new action data to overwrite existing data with. This may be a list of individual action(s) in case this episode is still not numpy'ized yet. In case this episode has already been numpy'ized, this should be (possibly complex) struct matching the action space and with a batch size of its leafs exactly the size of the to-be-overwritten slice or segment (provided by `at_indices`). at_indices: A single int is interpreted as one index, which to overwrite with `new_data` (which is expected to be a single observation). A list of ints is interpreted as a list of indices, all of which to overwrite with `new_data` (which is expected to be of the same size as `len(at_indices)`). A slice object is interpreted as a range of indices to be overwritten with `new_data` (which is expected to be of the same size as the provided slice). Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. neg_index_as_lookback: If True, negative values in `at_indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with actions = [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will handle a call to `set_actions(individual_action, -1, neg_index_as_lookback=True)` by overwriting the value of 6 in our actions buffer with the provided "individual_action". Raises: IndexError: If the provided `at_indices` do not match the size of `new_data`. r9r:r;N)rZr+r< set_actionsr>s rNrAzMultiAgentEpisode.set_actionsZ)1(8(8   $Hnt222Q8QQQRRR   ) 5 5'%&; 6      rPc|D]D\}}||jvrtd|d|j||||EdS)a Overwrites all or some of this Episode's rewards with the provided data. This is a helper method to batch `SingleAgentEpisode.set_rewards`. For more detail, see `SingleAgentEpisode.set_rewards`. Args: new_data: A dict mapping agent IDs to new reward data. Each value in the dict is the new reward data to overwrite existing data with. This may be a list of individual reward(s) in case this episode is still not numpy'ized yet. In case this episode has already been numpy'ized, this should be a np.ndarray with a length exactly the size of the to-be-overwritten slice or segment (provided by `at_indices`). at_indices: A single int is interpreted as one index, which to overwrite with `new_data` (which is expected to be a single reward). A list of ints is interpreted as a list of indices, all of which to overwrite with `new_data` (which is expected to be of the same size as `len(at_indices)`). A slice object is interpreted as a range of indices to be overwritten with `new_data` (which is expected to be of the same size as the provided slice). Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. neg_index_as_lookback: If True, negative values in `at_indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with rewards = [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will handle a call to `set_rewards(individual_reward, -1, neg_index_as_lookback=True)` by overwriting the value of 6 in our rewards buffer with the provided "individual_reward". Raises: IndexError: If the provided `at_indices` do not match the size of `new_data`. r9r:r;N)rZr+r< set_rewardsr>s rNrDzMultiAgentEpisode.set_rewardsrBrPc|D]E\}}||jvrtd|d|j|||||FdS)a+ Overwrites all or some of this Episode's extra model outputs with `new_data`. This is a helper method to batch `SingleAgentEpisode.set_extra_model_outputs`. For more detail, see `SingleAgentEpisode.set_extra_model_outputs`. Args: key: The `key` within `self.extra_model_outputs` to override data on or to insert as a new key into `self.extra_model_outputs`. new_data: A dict mapping agent IDs to new extra model outputs data. Each value in the dict is the new extra model outputs data to overwrite existing data with. This may be a list of individual reward(s) in case this episode is still not numpy'ized yet. In case this episode has already been numpy'ized, this should be a np.ndarray with a length exactly the size of the to-be-overwritten slice or segment (provided by `at_indices`). at_indices: A single int is interpreted as one index, which to overwrite with `new_data` (which is expected to be a single extra model output). A list of ints is interpreted as a list of indices, all of which to overwrite with `new_data` (which is expected to be of the same size as `len(at_indices)`). A slice object is interpreted as a range of indices to be overwritten with `new_data` (which is expected to be of the same size as the provided slice). Thereby, negative indices by default are interpreted as "before the end" unless the `neg_index_as_lookback=True` option is used, in which case negative indices are interpreted as "before ts=0", meaning going back into the lookback buffer. neg_index_as_lookback: If True, negative values in `at_indices` are interpreted as "before ts=0", meaning going back into the lookback buffer. For example, an episode with extra_model_outputs[key][agent_id] = [4, 5, 6, 7, 8, 9], where [4, 5, 6] is the lookback buffer range (ts=0 item is 7), will handle a call to `set_extra_model_outputs(key, individual_output, -1, neg_index_as_lookback=True)` by overwriting the value of 6 in our extra_model_outputs[key][agent_id] buffer with the provided "individual_output". Raises: IndexError: If the provided `at_indices` do not match the size of `new_data`. r9r:)rr7r6rN)rZr+r<set_extra_model_outputs)rMrr7r6rrTr?s rNrFz)MultiAgentEpisode.set_extra_model_outputss`)1(8(8   $Hnt222Q8QQQRRR   ) A A'%&; B      rPc t|S)zReturns the number of environment steps. Note, this episode instance could be a chunk of an actual episode. Returns: An integer that counts the number of environment steps this episode instance has seen. r rs rNrzMultiAgentEpisode.env_steps6s4yyrPcbtd|jDS)zNumber of agent steps. Note, there are >= 1 agent steps per environment step. Returns: An integer counting the number of agent steps executed during the time this episode instance records. c34K|]}t|VdSror )rfrs rNrrz0MultiAgentEpisode.agent_steps..Js(DD3s88DDDDDDrP)r+r+rrs rN agent_stepszMultiAgentEpisode.agent_stepsAs0DDt':'A'A'C'CDDDDDDrPitemc~t|tr||Std|d)zGEnable squared bracket indexing- and slicing syntax, e.g. episode[-4:].)rz1MultiAgentEpisode does not support getting item 'z>'! Only slice objects allowed with the syntax: `episode[a:b]`.)rHrr)rMrKs rN __getitem__zMultiAgentEpisode.__getitem__LsT dE " " ::T:** *%NDNNN rP) r;r:r2r4r5r3r6r7r8c  |dS||rJ| rJg}g}g} |&dtt|D}| &dtt|D} tt} tt} tt} tt} tttt}tfd}t ||ng}|pi}tt||D]C\}\}}t||kr||ni}t| |kr| |ni}t||kr||ni}| D]<\}| |  ||  | it| dkr|  j  j|  jnBjvr9jjg|z|xx|zcc<|vrX|j <| ij<jxx| dz cc<nP| s| rd|<n |t|dz kr dx|<|<j t| dz >|D]|vr|vrj jtjjz dkr|xxdzcc<jxx| dz cc<EtjD]| vr j=t| t| cxkrVt| dzcxkr9tdzcxkrt| dzksnJjj_| D]p\}t|dkr+| rD| }t1|| nd|j|j | | j | r-fddDnd| d | d jt;|d }|j<rdS) Ncg|]}iSrdrdrf_s rNrizAMultiAgentEpisode._init_single_agent_episodes..os:::AR:::rPcg|]}iSrdrdrPs rNrizAMultiAgentEpisode._init_single_agent_episodes..qs"C"C"C!2"C"C"CrPcjSro)r-rsrNz?MultiAgentEpisode._init_single_agent_episodes..ys D.. sRRRQAaDRRRrPrd)rfrWrTextra_model_outputs_per_agents @rNrzAMultiAgentEpisode._init_single_agent_episodes..sGRRRR*G*QRRRrPF)rrTrUrVr2rr3r4r r5r8ryrzrr9)rrDrrrIrpr}rziprZaddr[rJr%rr&r'r$rr~r-r _del_agentrrrrr r#rr+)rMr;r:r2r4r5r3r6r7r8observations_per_agentinfos_per_agentactions_per_agentrewards_per_agentdone_per_agentlen_lookback_buffer_per_agentrdata_idxobsinfact extra_outsrewr`rUrrTrXs` @@rNrKz-MultiAgentEpisode._init_single_agent_episodesVs    F ?  ;* * **GG"$  =::s<'8'8!9!9:::E  &"C"CuS\\/B/B"C"C"C !,T!2!2%d++'--'--(3D(9(9%$T**(34V4V4V4V(W(W%(9(E  " " $ $ $2  ,1r%.c,.F.F$G$GM RM R HjsC (+7||h'>'>'(##BC*++h66$H--  (+7||h'>'>'(##BC'*yy{{/ / #)!!(+++&x077 BBB)0021F1FGGG-h7881<<%h/66155h??2(;BB=AA(KK&h/66155h??  t'<<<-h7>>!12X=6h???8K???s??;>h-D-h7FPnn "GGD9(C-h7773778S;Q;QQ7777 __X..L*..2J2JL/3N8,,L 1 1A 555GKKN8,{8/D%h/66.x899A=* R R3&&8>+I+I)(3::4;OPPP D1(;<<456h???1D???-h7773778S;Q;QQ777 R T27799:: R RH555)(3*8455x0118888(233a788884X>??!C8888(233a7 888888 8<7QD !( + 4 4$:#?#?#A#A+ 7+ 7 Hi9~~""~'9'9('C'C"))) ),,$99(DIII,)4&))(333!#'+x&"&"8"<" !>x!H!K!P!P!R!R &??8U;;$..599.x8$'(Eh(OQR$S$S5J:-7D  ) )W+ 7+ 7rP)r|rrrrrrc tt|p|j}t|||||t|sind|i| } |dur"|rt d|d|jdi| S|r |jdi| S|jdi| S)Nr)rrr|rrrrFz`MultiAgentEpisode.get_zG()` can't be called with both `env_steps=False` and `return_list=True`!rd)rprr|rFr_get_data_by_agent_steps_get_data_by_env_steps_as_list_get_data_by_env_steps) rMrrr|rrrrrrkwargss rNrzMultiAgentEpisode._get s 9--..@$. "7!*V1CEU0V%<         @d@@@140::6:: : 9646@@@@ @/4.8888 8rPc i}|jD]b\} } | |vr t| |} ||| } || |} | |} | jd|||| d|} | | gkr]| || <c|S)Nrrr_add_last_ts_valuerd)r+rZgetattr_get_hanging_valuerJ)rMrrr|rrrrretrTrinf_lookback_buffer hanging_val agent_values rNriz*MultiAgentEpisode._get_data_by_agent_steps< s%)$7$=$=$?$? ( ( Hjy((")*d";"; 11$AAK&2&9:Q&R#)*AB 1-1&;#.  # K"kR&7&7'CMM rPrrc |i}|jD]5} | |vr|j| |||j|dv|| <6|sgSg} t t tt| D]} i} | D]m\} } | || }| | |||| | \}}|dkr|s|sI| ||| |||||}||| | <n| | | S)Nr2r3rr_ignore_last_tsfilter_for_skip_indicesr8rrsrTindex_incl_lookbackrrrrt)r+r}r$rJr~rrDrrrrZrq _get_inf_lookback_buffer_or_dict_get_single_agent_data_by_indexr[)rMrrr|rrrr agent_indicesrTrrrret2idxesrtrsindices_to_userus rNrjz0MultiAgentEpisode._get_data_by_env_steps_as_list` s +0022  Hy((&*&;H&E&I&I&;)!%,E E'J''M( # # Is4]%9%9%;%; < <==>>??  AD#0#6#6#8#8 1 1%"55dHEE 99+,1!H :'"111/2'2"BB(;%(6%5,C +C   *%0DN JJt     rPc |dv}i} |jD]\} } | |vr ||| } |j| ||||jnd|} || ||| | \}} t| tr5| || | ||| |}t|dkr|| | <| ||| | |||| }||| | <| S)Nrwrxrz)rrTindices_incl_lookbackrrrtrrr|) r+rZrqr$rJr~r~rHr*_get_single_agent_data_by_env_step_indicesrDr)rMrrr|rrrrignore_last_tsrrrTrrtrrs agent_valuess rNrkz(MultiAgentEpisode._get_data_by_env_steps sw%>>$($7$=$=$?$?, 1, 1 Hjy((11$AAK 1(;??&;-1-=T))4!/@M261V1V'(5 2W22 . -.. 1#NN%*7%5 +,C O   |$$q(($0CM#CC(;%(5%5,C + D    +$0CM rPrsr}rtc  |j|} ||jkr6|dSt| d|d dd|dt|ind|i|S|dkr||t |tsJi} |rB|D],\} } | jd || jz d||dn|| d|| | <-n>|D])} tjd |d||| d|| | <*| S|jd ||jz d||d|S) Nget_J)Frrrrr8Trnrd) r+r~rprFrHrZrJrr}r) rMrrsrTr}rrrrtrrrr sub_buffers rNrz1MultiAgentEpisode._get_single_agent_data_by_index s(2 $"6 6 6|57:}d}}55 %&+  .6B!89 #   $* * */F/N&*[$*G*G&&GC" ':'@'@'B'B  OC-z~  3j6I I.2!$/$7DD[=M   +  CHH '++--C;577;  3.2!+6s+;    +   CHHJ+&*+.A.JJ&*#.  #  rP)rrrrtrc  j|}t|| | | j|vr| jddd|d|} g} |D]^} | jkr| | #|  jd| t||jz d||d|_jrt| } n! fd|D}  jd| d||d|} | S) a} Returns single data item from the episode based on given (env step) indices. The returned data item will have a batch size that matches the env timesteps defined via `indices_incl_lookback`. Args: what: A (str) descriptor of what data to collect. Must be one of "observations", "infos", "actions", "rewards", or "extra_model_outputs". indices_incl_lookback: A list of ints specifying, which indices to pull from the InfiniteLookbackBuffer defined by `agent_id` and `what` (and maybe `extra_model_outputs_key`). Note that these indices disregard the special logic of the lookback buffer. Meaning if one index in `indices_incl_lookback` is 0, then the first value in the lookback buffer should be returned, not the first value after the lookback buffer (which would be normal behavior for pulling items from an `InfiniteLookbackBuffer` object). agent_id: The individual agent ID to pull data for. Used to lookup the `SingleAgentEpisode` object for this agent in `self`. fill: An optional float value to use for filling up the returned results at the boundaries. This filling only happens if the requested index range's start/stop boundaries exceed the buffer's boundaries (including the lookback buffer on the left side). This comes in very handy, if users don't want to worry about reaching such boundaries and want to zero-pad. For example, a buffer with data [10, 11, 12, 13, 14] and lookback buffer size of 2 (meaning `10` and `11` are part of the lookback buffer) will respond to `indices_incl_lookback=[-1, -2, 0]` and `fill=0.0` with `[0.0, 0.0, 10]`. one_hot_discrete: If True, will return one-hot vectors (instead of int-values) for those sub-components of a (possibly complex) space that are Discrete or MultiDiscrete. Note that if `fill=0` and the requested `indices_incl_lookback` are out of the range of our data, the returned one-hot vectors will actually be zero-hot (all slots zero). extra_model_outputs_key: Only if what is "extra_model_outputs", this specifies the sub-key (str) inside the extra_model_outputs dict, e.g. STATE_OUT or ACTION_DIST_INPUTS. hanging_val: In case we are pulling actions, rewards, or extra_model_outputs data, there might be information "hanging" (cached). For example, if an agent receives an observation o0 and then immediately sends an action a0 back, but then does NOT immediately reveive a next observation, a0 is now cached (not fully logged yet with this episode). The currently cached value must be provided here to be able to return it in case the index is -1 (most recent timestep). Returns: A data item corresponding to the provided args. NrFrTrnc:g|]}|jk |jz Srd)r~r)rfrrsrMs rNrizPMultiAgentEpisode._get_single_agent_data_by_env_step_indices.. s:,,,'00,,,rPrd)r+rpr~rJr[rrr)rMrrTrrrrrtrsingle_fill_valuerrrrrss` @rNrzBm6x@??D 1( ; ; ; 2 1rPc|j|d|j|d|j|d|j|ddS)zGDeletes all hanging action, reward, extra_model_outputs of given agent.N)r(rr%r&r'rMrTs rNrzMultiAgentEpisode._del_hanging sl #''$777 !%%h555 -11(DAAA !%%h55555rPc<|||j|d|j||j|d|j|d|j|ddS)z2Deletes all data of given agent from this episode.N)rr+rr|discardr$rr#rs rNr[zMultiAgentEpisode._del_agent s (### $/// x((( !!(D111 %))(D999   400000rPc t|j||x}}|dkrD| ||}n9|r/tt|}n|||fS|S|St ||jz|duz |dvt|tr fd|D}n r| krd}||fS|S)aReturns a single InfiniteLookbackBuffer or a dict of such. In case `what` is "extra_model_outputs" AND `extra_model_outputs_key` is None, a dict is returned. In all other cases, a single InfiniteLookbackBuffer is returned. r8Nrwc(g|]}r|krdn|S)r0rd)rfrrinf_lookback_buffer_lens rNrizFMultiAgentEpisode._get_inf_lookback_buffer_or_dict.. s>+++*Qa3J.J.JCCPQ+++rPr0) rpr+rrrrDrrHr) rMrTrrrtr{inf_lookback_buffer_or_dictrsrrs @@rNr~z2MultiAgentEpisode._get_inf_lookback_buffer_or_dict sG=D   )4= =  #&9 ( ( (&2&A+'##- 3&*40K0R0R0T0T+U+U&V&V##(424KKK22 " .'((%./d*, $ ")BBN1488 .+++++4+++''  .$;?V$V$V*-'.0GG G. .rPz-MultiAgentEpisode.custom_data[some-key] = ...)newerrorcdSrordrs rNadd_temporary_timestep_dataz-MultiAgentEpisode.add_temporary_timestep_data  rPz'MultiAgentEpisode.custom_data[some-key]cdSrordrs rNget_temporary_timestep_dataz-MultiAgentEpisode.get_temporary_timestep_data rrPro)rQN)rQr)rrrQNr%)NN)NNN)F)R__name__ __module__ __qualname____doc__ __slots__r~r r r rgymSpacer rIrErrrrrOr]rrLpropertyrrrrYrrrr r|r:r\rrrrrrrGrrrrrrrr staticmethodr rr&rr1r3r5r=rArDrFrrJrMrKrrirjrkrrFrrrqrrr[r~r rrrdrPrNrr s8I.O"R8<150426,0263827>B'+8>B )RRR c]RtN34 R $CI. R ^,- R$~./Rsy)R$~./R>4/0R.$./R&d>&:;R }R"$w|"45R#38_R $D#$67!R"#4(9#:;#R$%- g23X= >% %RRRRp+/ 0/0/0/%0/' 0/  0/0/0/0/n+/ h415/38<h4h4h4$h4 h4 h4 ' h4n-h4^,h4&n5h4 h4h4h4h4T !!X! $   X  $   X 77X78RRRRhhhhhTppsp3Fppppd/3w<///X/ >   X ;7;x/A;;;;0;?CGL &+"!&!L L L %T#Y 567L E*W"5w">?@L  L  $L smL L L  ~tN33 4L L L L `;?CGF &+"!F F F %T#Y 567F E*W"5w">?@F  F $ F smF F  ~tN33 4F F F F T;?CGK &+"!&!K K K %T#Y 567K E*W"5w">?@K  K $ K smK K K  ~tN33 4K K K K ^;?CGD &+ $!D D D %T#Y 567D E*W"5w">?@D  D $ D uoD D  ~tN33 4D D D D P":>CG N &+"!N N N c]N %T#Y 567N E*W"5w">?@ N  N  $N smN N  ~tN33 4N N N N `.2 RRRR&c] R  RRRRh / / /   ((((># 4S># # # # J-$sCx.--@---\-^ /    0).!% : 3w<    5W 5 5 5 577777>B&+ 444!4U3S 5#89: 4 $ 4  4444t>B&+ 444!4U3S 5#89: 4 $ 4  4444t>B&+ 444!4U3S 5#89: 4 $ 4  4444v>B&+ 888! 8 U3S 5#89: 8 $ 8 8888t 3     ES E E E E*=?C:>7;2626043827>Bv7v7v7#4(9#:;v7$D#$67 v7 tN34 v7 $~./ v7$~./v7^,-v7>4/0v7.$./v7&d>&:;v7v7v7v7z#  $,9,9,9,9,9\"""H>>sE49,- > g& > $ >>"%> n >>>>@::sE49,- : g& : $ :::"%: ::::xFF4 F  F #38_ FFF"%FF FFFF\#!&15%)gggg g %S#X g sm gg"*#gc]g ggggR;s;g;#;;;; g 6I d    6W66666171t111126%) $ -/-/-/-/"*# -/ c] -/-/-/-/^ZC4PPP  QP Z=TJJJ  KJ   rPr)&rr^r? collectionsrtypingrrrrrr r r r gymnasiumrray._common.deprecationr "ray.rllib.env.single_agent_episoder,ray.rllib.env.utils.infinite_lookback_bufferrray.rllib.policy.sample_batchrray.rllib.utilsrray.rllib.utils.errorr"ray.rllib.utils.spaces.space_utilsrray.rllib.utils.typingrrrray.util.annotationsrrrdrPrNrs ######                      ......AAAAAAOOOOOO999999&&&&&&444444444444DDDDDDDDDD******  WJ+ J+ J+ J+ J+ J+ J+ J+ J+ J+ rP