&`i% ddlZddlZddlmZmZddlmZmZddlm Z m Z m Z m Z m Z mZddlmZmZddlmZmZddlmZddlmZdd lmZmZdd lmZdd lmZdd l m!Z!dd l"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*e rddl+m,Z,ddl-m.Z.ddl/m0Z0e&\Z1Z2Z3ej4e5Z6edgdZ7e e e Z8GddeZ9e#GddeeZ:e#Gdde:Z;dS)N)ABCMetaabstractmethod) defaultdict namedtuple) TYPE_CHECKINGAnyListOptionalTypeUnion)DEPRECATED_VALUEdeprecation_warning)BaseEnvconvert_to_base_env)SampleCollector)SimpleListCollector) EnvRunnerV2 _PerfStats)RolloutMetrics) InputReader)concat_samples) OldAPIStackoverride) try_import_tf)SampleBatchType)log_once) RLlibCallback)ObservationFunction) RolloutWorker_PolicyEvalData)env_idagent_idobsinfo rnn_state prev_action prev_rewardceZdZdZdS)_NewEpisodeDefaultDictcf|jt|||x}||<|SN)default_factoryKeyError)selfr!rets p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/evaluation/sampler.py __missing__z"_NewEpisodeDefaultDict.__missing__-s:   '6"" "!%!5!5f!=!= =C$v,JN)__name__ __module__ __qualname__r1r2r0r)r),s#r2r)ceZdZdZeedefdZedefdZ ede e fdZ ede efdZ dS) SamplerInputz1Reads input experiences from an existing sampler.returnc|g}||t|dkrt dt |S)NrzNo data available from sampler.)get_dataextendget_extra_batcheslen RuntimeErrorr)r.batchess r0nextzSamplerInput.next9s^==??#t--//000 w<<1  @AA Ag&&&r2ct)zCalled by `self.next()` to return the next batch of data. Override this in child classes. Returns: The next batch of data. NotImplementedErrorr.s r0r;zSamplerInput.get_dataAs "!r2ct)a$Returns list of episode metrics since the last call to this method. The list will contain one RolloutMetrics object per completed episode. Returns: List of RolloutMetrics objects, one per completed episode since the last call to this method. rCrEs r0 get_metricszSamplerInput.get_metricsLs "!r2ct)aReturns list of extra batches since the last call to this method. The list will contain all SampleBatches or MultiAgentBatches that the user has provided thus-far. Users can add these "extra batches" to an episode by calling the episode's `add_extra_batch([SampleBatchType])` method. This can be done from inside an overridden `Policy.compute_actions_from_input_dict(..., episodes)` or from a custom callback's `on_episode_[start|step|end]()` methods. Returns: List of SamplesBatches or MultiAgentBatches provided thus-far by the user since the last call to this method. rCrEs r0r=zSamplerInput.get_extra_batchesXs "!r2N)r3r4r5__doc__rrrrArr;r rrGr=r6r2r0r8r85s;; Xk'o''''"/"""^" "T.1 " " "^ ""4#8"""^"""r2r8) metaclasscBeZdZdZddddddddddddeeedddd ed eeefd e d e d ddededede dde e e defdZeedefdZeedeefdZeedeefdZdS) SyncSamplerzHSync SamplerInput that collects experiences when `get_data()` is called. env_stepsFTN)count_steps_bymultiple_episodes_in_batchnormalize_actions clip_actionsobservation_fnsample_collector_classrenderpoliciespolicy_mapping_fn preprocessors obs_filterstf_sesshorizon soft_horizonno_done_at_endworkerrenv clip_rewardsrollout_fragment_lengthrN callbacksrrOrPrQrRrrSrTc <tdr| td|td|td|td|td|tkrtd d |tkrtd d |tkrtd d t||_||_t j|_t|j j |_ | st} | |j||||||_| |_t#||j|||j |||j|_|j|_t j|_dS)aNInitializes a SyncSampler instance. Args: worker: The RolloutWorker that will use this Sampler for sampling. env: Any Env object. Will be converted into an RLlib BaseEnv. clip_rewards: True for +/-1.0 clipping, actual float value for +/- value clipping. False for no clipping. rollout_fragment_length: The length of a fragment to collect before building a SampleBatch from the data and resetting the SampleBatchBuilder object. count_steps_by: One of "env_steps" (default) or "agent_steps". Use "agent_steps", if you want rollout lengths to be counted by individual agent steps. In a multi-agent env, a single env_step contains one or more agent_steps, depending on how many agents are present at any given time in the ongoing episode. callbacks: The RLlibCallback object to use when episode events happen during rollout. multiple_episodes_in_batch: Whether to pack multiple episodes into each batch. This guarantees batches will be exactly `rollout_fragment_length` in size. normalize_actions: Whether to normalize actions to the action space's bounds. clip_actions: Whether to clip actions according to the given action_space's bounds. observation_fn: Optional multi-agent observation func to use for preprocessing observations. sample_collector_class: An optional SampleCollector sub-class to use to collect, store, and retrieve environment-, model-, and sampler data. render: Whether to try to render the environment after each step. deprecated_sync_sampler_argsNrU)oldrVrWrXrYrZT)rderrorr[r\)ema_coef)rN)r]base_envrOra perf_statsr`rNrT)rrr rrgr`queueQueue extra_batchesrconfigsampler_perf_stats_ema_coefrhr policy_mapsample_collectorrTr_env_runner_objrun _env_runner metrics_queue)r.r]r^r_r`rNrarOrPrQrRrSrTrUrVrWrXrYrZr[r\s r0__init__zSyncSampler.__init__osx 2 3 3 F## 3333 ,#(;<<<<(#8888&# 6666"# 2222***# >>>>///#dCCCC!111#(8EEEE+C00 '>$"[]]$]>   & 9%8 " 6 6     & #) ! ! !   +]'A$;);  /3355"[]]r2r9c t|j}t|tr|j|n|SGr+)rArr isinstancerrsput)r.items r0r;zSyncSampler.get_datasM ())D$// "&&t,,,,  r2cg} ||j|jn#t j$rYnwxYwo|S)NT)rh)appendrs get_nowait_replacerhgetriEmpty)r. completeds r0rGzSyncSampler.get_metricss     &1133<<#'?#6#6#8#8= ;     sAAA/.A/cg} ||jn#tj$rYnwxYwD|Sr+)rzrkr{rir~)r.extras r0r=zSyncSampler.get_extra_batchessc   T/::<<====;       s,2AA)r3r4r5rIr rr boolfloatintstrr r rrtrr8rr;r rrGr=r6r2r0rLrLksRR*+0"&":>BF %'/m+m+m+ m+ m+ D%K( m+ "% m+m+#m+%)m+ m+m+!!67m+!)o)> ?m+m+m+m+m+^Xl/Xl T.1    Xl4#8r2rL)<loggingriabcrr collectionsrrtypingrrr r r r ray._common.deprecationr rray.rllib.env.base_envrr0ray.rllib.evaluation.collectors.sample_collectorr5ray.rllib.evaluation.collectors.simple_list_collectorr"ray.rllib.evaluation.env_runner_v2rrray.rllib.evaluation.metricsrray.rllib.offlinerray.rllib.policy.sample_batchrray.rllib.utils.annotationsrrray.rllib.utils.frameworkrray.rllib.utils.typingrray.util.debugrray.rllib.callbacks.callbacksr)ray.rllib.evaluation.observation_functionr#ray.rllib.evaluation.rollout_workerrtf1tf_ getLoggerr3loggerr StateBatchr)r8rLr6r2r0rs ''''''''////////JIIIIIII????????LLLLLLUUUUUUFFFFFFFF777777))))))888888========333333222222######B;;;;;;MMMMMMAAAAAA ]__ R  8 $ $*TTT $s)_ [ 2"2"2"2"2";'2"2"2" 2"j QQQQQ,QQ QQQr2