&`ie:ddlZddlmZddlmZmZmZmZmZm Z m Z ddl Z ddl mZddlmZddlmZmZddlmZddlmZdd lmZdd lmZmZmZmZmZer dd l m!Z!dd l"m#Z#eGd dZ$dS)N) defaultdict) TYPE_CHECKINGAnyCallableDictListOptionalTuple)_DUMMY_AGENT_ID)AgentCollector)_PolicyCollector_PolicyCollectorGroup) PolicyMap) SampleBatch) OldAPIStack)AgentIDEnvID EnvInfoDictPolicyID TensorType) RLlibCallback) RolloutWorkerc eZdZdZddddededeeddgefde dd e d f d Z e d fd ede defdZ deefdZd edefdZd(dZddd ededeeefdeddf dZd edeeefddfdZ d)dede de ddfdZd*d ede fd Zd ede fd!Zd ede fd"Zd ede fd#Zd ed$efd%Ze fd ede efd&Z e!d'Z"dS)+ EpisodeV2z=Tracks the current state of a (possibly multi-agent) episode.N)worker callbacksenv_idpoliciespolicy_mapping_fnrrrrctjtd|_||_d|_d|_d|_d|_d|_ i|_ i|_ i|_ i|_ ||_||_||_||_i|_i|_d|_i|_t-t.|_t-t2|_i|_i|_i|_i|_dS)aInitializes an Episode instance. Args: env_id: The environment's ID in which this episode runs. policies: The PolicyMap object (mapping PolicyIDs to Policy objects) to use for determining, which policy is used for which agent. policy_mapping_fn: The mapping function mapping AgentIDs to PolicyIDs. worker: The RolloutWorker instance, in which this episode runs. gNgmCgrN)random randrangeint episode_idr total_rewardactive_env_stepstotal_env_stepsactive_agent_stepstotal_agent_stepscustom_metrics user_data hist_datamediarr policy_mapr_agent_to_policy_agent_collectors_next_agent_index_agent_to_indexrfloat agent_rewardslist_agent_reward_history _has_init_obs_last_terminateds_last_truncateds _last_infos)selfrrrrrs s/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/evaluation/episode_v2.py__init__zEpisodeV2.__init__s* &/D :: #&&(%''(&' 13*,24%'  "%-  :<@B&'35EPPUDVDV?J4?P?P"24685713Fagent_idrefreshreturnc||jvs|r(||||jx}|j|<n |j|}||jvrt d|d|S)aReturns and stores the policy ID for the specified agent. If the agent is new, the policy mapping fn will be called to bind the agent to a policy for the duration of the entire episode (even if the policy_mapping_fn is changed in the meantime!). Args: agent_id: The agent ID to lookup the policy ID for. Returns: The policy ID for the specified agent. )rz.policy_mapping_fn returned invalid policy id 'z'!)r0rrr/KeyError)r<r@rA policy_ids r= policy_forzEpisodeV2.policy_fores$ 40 0 0G 0:>:P:P{;Q;; I-h77-h7I DO + +QIQQQ r?cNt|jS)zReturns list of agent IDs that have appeared in this episode. Returns: The list of all agent IDs that have appeared so far in this episode. )r6r3keysr<s r= get_agentszEpisodeV2.get_agentss!D(--//000r?cl||jvr|j|j|<|xjdz c_|j|S)zGet the index of an agent among its environment. A new index will be created if an agent is seen for the first time. Args: agent_id: ID of an agent. Returns: The index of this agent. )r3r2r<r@s r= agent_indexzEpisodeV2.agent_indexsD 4/ / /-1-CD  *  " "a ' " "#H--r?cF|xjdz c_|xjdz c_dS)z(Advance the episode forward by one step.rLN)r'r(rIs r=stepzEpisodeV2.steps0 " !r?r!)tinit_obs init_infosrQc|j||}||jvsJt|j|jdd|jdd||d|j|<|j| |j | ||j |||d|j |<dS) zAdd initial env obs at the start of a new episode Args: agent_id: Agent ID. init_obs: Initial observations. init_infos: Initial infos dicts. t: timestamp. model max_seq_len_disable_action_flatteningF)rVdisable_action_flatteningis_policy_recurrent intial_states_enable_new_api_stack)r%rNrrRrSrQTN)r/rFr1r view_requirementsconfigget is_recurrentget_initial_state add_init_obsr%rNrr8)r<r@rRrSrQpolicys r=razEpisodeV2.add_init_obss !:!:;t55555+9  $ g.}=&,m&7&7,e''!' 3 3 5 5 2244"' , , , x( x(55((22;! 6   (,8$$$r?valuesc||jvsJ|xjdz c_|xjdz c_|tkr||d<|j|||t j}|xj|z c_|j|| |fxx|z cc<|j | |t j |vr|t j |j |<t j|vr|t j|j|<t j|vr(|||t jdSdS)zAdd action, reward, info, and next_obs as a new step. Args: agent_id: Agent ID. values: Dict of action, reward, info, and next_obs. rLr@N)r1r)r*r add_action_reward_next_obsrREWARDSr&r5rFr7append TERMINATEDSr9 TRUNCATEDSr:INFOS set_last_info)r<r@rcrewards r=add_action_reward_done_next_obsz)EpisodeV2.add_action_reward_done_next_obssh411111 1$ !#  & &!)F:  x(CCFKKK +, V# Hdooh&?&?@AAAVKAAA "8,33F;;;  "f , ,/5k6M/ND "8 ,  !V + +.4[5K.LD !( +   & &   x 0A)B C C C C C ' &r? batch_builderis_done check_donesc i}|jD]U\}}|jdkr||}|j|}||j} ||| f||<V|D]\}\}}} |rU|rS| s?td |j |||dz|j |i dds| r5ttj| t"jdkrtd| t|dkr|} | |=ni} | } t)|dd .|j|| || d || | |} dd lm} |j| ||||j| | ||jvrt=||j|<|j|| |j|xj|j z c_|xj!|j"z c_!d|_ d|_"d S) aBuild and return currently collected training samples by policies. Clear agent collector states if this episode is done. Args: batch_builder: _PolicyCollectorGroup for saving the collected per-agent sample batches. is_done: If this episode is done (terminated or truncated). check_dones: Whether to make sure per-agent trajectories are actually done. rzkEpisode {} terminated for all agents, but we still don't have a last observation for agent {} (policy {}). zkPlease ensure that you include the last observations of all live agents when setting done[__all__] to True.training_enabledTrLzPBatches sent to postprocessing must only contain steps from a single trajectory. explorationN)get_global_worker)repisoder@rErpostprocessed_batchoriginal_batches)#r1items agent_stepsrFr/build_for_trainingr\is_terminated_or_truncated ValueErrorformatr%r;r^is_single_trajectorylennpuniquerEPS_IDcopygetattrrspostprocess_trajectory get_sessionset_get_interceptor#ray.rllib.evaluation.rollout_workerrtron_postprocess_trajectorypolicy_collectorsr $add_postprocessed_batch_for_trainingr) env_stepsr') r<rnrorp pre_batchesr@ collectorpidrb pre_batch other_batches post_batchrts r=postprocess_episodezEpisodeV2.postprocess_episodes+( #'#9#?#?#A#A = = Hi$))//(++C_S)F!44V5MNNI%(&)$W>WXX#''"5599:LdSS 2244 ry;+=!>??@@1DD 0 ;!## + 0 0 2 2 !(++ " #Jv}d33?"99J(:(:(<(<  * *4 0 0 066z=RVWWJ M M M M M M N 4 4((**!$.!, 5   -9997G7O7O /4  +C 0 U UF4     !!T%<<!!4#88#$ !r?c|||jvo |j|Stt|jS)zReturns whether this episode has initial obs for an agent. If agent_id is None, return whether we have received any initial obs, in other words, whether this episode is completely fresh. )r8anyr6rcrMs r= has_init_obszEpisodeV2.has_init_obs]sJ  t11Rd6H6R RtD.55778899 9r?cV||p||SN) is_terminated is_truncatedrMs r=rozEpisodeV2.is_donehs)!!(++Jt/@/@/J/JJr?c8|j|dSNF)r9r^rMs r=rzEpisodeV2.is_terminatedks%))(E:::r?c8|j|dSr)r:r^rMs r=rzEpisodeV2.is_truncatedns$((5999r?infoc||j|<dSr)r;)r<r@rs r=rkzEpisodeV2.set_last_infoqs%)"""r?c6|j|Sr)r;r^rMs r= last_info_forzEpisodeV2.last_info_forts##H---r?c|jSr)r(rIs r=lengthzEpisodeV2.lengthys ##r?)rBN)FFr)#__name__ __module__ __qualname____doc__rrrrrr r>r boolrFrrJr$rNrPrrstrrarmrrrrorrrkrrpropertyrr?r=rrsGG-1/3H3H3H3H3H3$Wk?$KX$UV H3 ) H3O,H3H3H3H3V#25!!!<@! !!!!F1DM1111.G..... """" (,(,(,(, (, j) (,  (, (,(,(,(,T'D'DS*_%'D  'D'D'D'DX! b"b",b"b" b"  b"b"b"b"H : :W : : : : :KKDKKKK;g;$;;;;:W:::::*g*T****#2... + .... $$X$$$r?r)%r" collectionsrtypingrrrrrr r numpyrray.rllib.env.base_envr /ray.rllib.evaluation.collectors.agent_collectorr 5ray.rllib.evaluation.collectors.simple_list_collectorr rray.rllib.policy.policy_maprray.rllib.policy.sample_batchrray.rllib.utils.annotationsrray.rllib.utils.typingrrrrrray.rllib.callbacks.callbacksrrrrrr?r=rs ######LLLLLLLLLLLLLLLLLL222222JJJJJJ211111555555333333TTTTTTTTTTTTTTB;;;;;;AAAAAA c$c$c$c$c$c$c$ c$c$c$r?