&`i7'HddlZddlZddlmZmZmZmZddlZddl m Z ddl m Z ddl mZddlmZmZddlmZddlmZdd lmZmZdd lmZerdd lmZejeZd eed ej fdZ!eGddZ"eGddZ#dS)N) TYPE_CHECKINGAnyDictList)deprecation_warning)_DUMMY_AGENT_ID)Policy)MultiAgentBatch SampleBatch) OldAPIStack) summarize)AgentIDPolicyID)log_once) RLlibCallbackvreturnctj|}|jtjkr|tjS|SN)nparraydtypefloat64astypefloat32)rarrs }/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/evaluation/sample_batch_builder.py_to_float_arrayrs7 (1++C yBJzz"*%%% JcHeZdZdZdZdZdeddfdZdeddfd Z defd Z dS) SampleBatchBuilderzUtil to build a SampleBatch incrementally. For efficiency, SampleBatches hold values in column form (as arrays). However, it is useful to add data one row (dict) at a time. rcPtjt|_d|_dS)Nr) collections defaultdictlistbufferscountselfs r__init__zSampleBatchBuilder.__init__'s(3(?(E(E  rvaluesrNc |D]%\}}|j||&|xjdz c_dS)z7Add the given dictionary (row) of values to this batch.N)itemsr&appendr')r)r+krs r add_valueszSampleBatchBuilder.add_values+sOLLNN & &DAq LO " "1 % % % % a rbatchc|D]%\}}|j||&|xj|jz c_dS)z,Add the given batch of values to this batch.N)r.r&extendr')r)r2r0columns r add_batchzSampleBatchBuilder.add_batch2sR + +IAv LO " "6 * * * * ek! rcNtd|jD}tj|vrFt jt j|j|tj<t xjdz c_|j d|_|S)z=Returns a sample batch including all previously added values.c4i|]\}}|t|S)r).0r0rs r z6SampleBatchBuilder.build_and_reset..<s&TTTtq!Q 2 2TTTrr-r) r r&r. UNROLL_IDrrepeatr!_next_unroll_idr'clear)r)r2s rbuild_and_resetz"SampleBatchBuilder.build_and_reset9sTTt|?Q?Q?S?STTTUU   - -+-9"2EK,,E+' (  . .! 3 . .   r) __name__ __module__ __qualname____doc__r>r*rr1r r6r@r9rrr!r!s O34"{"t""""       rr!ceZdZdZdeeefdeddfdZde fdZ defd Z d e d e d e dd fdZdddZddZddefdZd S)MultiAgentSampleBatchBuilderagUtil to build SampleBatches for each policy in a multi-agent env. Input data is per-agent, while output data is per-policy. There is an M:N mapping between agents and policies. We retain one local batch builder per agent. When an agent is done, then its local batch is appended into the corresponding policy batch for the agent's policy. policy_map clip_rewards callbacksrctdrtdd||_||_d|D|_i|_i|_||_d|_ dS)aSInitialize a MultiAgentSampleBatchBuilder. Args: policy_map (Dict[str,Policy]): Maps policy ids to policy instances. clip_rewards (Union[bool,float]): Whether to clip rewards before postprocessing (at +/-1.0) or the actual value to +/- clip. callbacks: RLlib callbacks. rFF)olderrorc,i|]}|tSr9)r!)r:r0s rr;z9MultiAgentSampleBatchBuilder.__init__..ds!SSSA#5#7#7SSSrrN) rrrGrHkeyspolicy_buildersagent_buildersagent_to_policyrIr')r)rGrHrIs rr*z%MultiAgentSampleBatchBuilder.__init__Qs 2 3 3 Q $B% P P P P$(SSARARSSS!!" rrcbtd|jDS)zReturns the total number of steps taken in the env (all agents). Returns: int: The number of steps taken in total in the environment over all agents. c3$K|] }|jV dSr)r')r:as r z5MultiAgentSampleBatchBuilder.total..ws$AAq17AAAAAAr)sumrPr+r(s rtotalz"MultiAgentSampleBatchBuilder.totalos0AAD$7$>$>$@$@AAAAAArc2t|jdkS)zReturns whether there is pending unprocessed data. Returns: bool: True if there is at least one per-agent builder (with data in it). r)lenrPr(s rhas_pending_agent_dataz3MultiAgentSampleBatchBuilder.has_pending_agent_datays4&''!++ragent_id policy_idr+Nc ||jvr t|j|<||j|<|tkr||d<|j|jdi|dS)aAdd the given dictionary (row) of values to this batch. Args: agent_id: Unique id for the agent we are adding values for. policy_id: Unique id for policy controlling the agent. values: Row of values to add for this agent. r[Nr9)rPr!rQrr1)r)r[r\r+s rr1z'MultiAgentSampleBatchBuilder.add_valuessp 4. . .,>,@,@D  )-6D  *  & &!)F: 0H%0::6:::::rc i}|jD]4\}}|j|j||f||<5i}|jdur;|D]%\}\}}t j|d|d<&nO|jrH|D]3\}\}}t j|d|j |j|d<4|D]\}\}}| }||=|j|j|} | r0tt|tjdkrtd||||<t!| dd4| j| ||| | ||||||<t)dr:t*d t1|d d lm} t7|D]i\}} |j| |||j||j| | |j|j|| j|j |j dS) a@Apply policy postprocessors to any unprocessed rows. This pushes the postprocessed per-agent batches onto the per-policy builders, clearing per-agent state. Args: episode (Optional[Episode]): The Episode object that holds this MultiAgentBatchBuilder object. Trewards)a_mina_maxr-zPBatches sent to postprocessing must only contain steps from a single trajectory. explorationN after_postz8Trajectory fragment after postprocess_trajectory(): {} r)get_global_worker)workerepisoder[r\policiespostprocessed_batchoriginal_batches)!rPr.rGrQr@rHrsignclipcopyis_single_trajectoryrYsetr EPS_ID ValueErrorgetattrrbpostprocess_trajectory get_sessionrloggerinfoformatr #ray.rllib.evaluation.rollout_workerrdsortedrIon_postprocess_trajectoryrOr6r?) r)rf pre_batchesr[builder post_batches_ pre_batch other_batchespolicyrd post_batchs rpostprocess_batch_so_farz5MultiAgentSampleBatchBuilder.postprocess_batch_so_farsb !%!4!:!:!?''))%K ! !    $ $%0%6%6%8%8 E E!>Ay')wy/C'D'D )$$ E   %0%6%6%8%8  !>Ay')wi(,,+((( )$$ )4(9(9(;(;   $Hnq)',,..Mh'_T%9(%CDF2244 s9[%7899::Q>> 0 &/L "v}d33?"99L2F4F4F4H4H&,%B%BX& w&&L " " L ! !  KKMTTl++    JIIIII$*<+=+=+?+?$@$@ W W Hj N 4 4((**!.x8$.!, 5     !5h!? @ J J: V V V V !!### ""$$$$$rc|jD]O\}}|js1t d||j|dzPdS)NzpThe environment terminated for all agents, but we still don't have a last observation for agent {} (policy {}). z}Please ensure that you include the last observations of all live agents when setting '__all__' terminated|truncated to True. )rPr.r&is_terminated_or_truncatedrprvrQ)r)r[r{s rcheck_missing_donesz0MultiAgentSampleBatchBuilder.check_missing_doness!%!4!:!:!"> F F Iw}q  ,3,C,C,E,Ey)J  -niHHHrr)rN)rArBrCrDrrr boolr*intrWrZrrr1rrr r@r9rrrFrFGs6)*# <BsBBBB,,,,,;7;w;#;RV;;;;&M%M%M%M%M%^    IIIIIIIIrrF)$r#loggingtypingrrrrnumpyrray._common.deprecationrray.rllib.env.base_envrray.rllib.policy.policyr ray.rllib.policy.sample_batchr r ray.rllib.utils.annotationsr ray.rllib.utils.debugr ray.rllib.utils.typingrrray.util.debugrray.rllib.callbacks.callbacksr getLoggerrArtndarrayrr!rFr9rrrs111111111111777777222222******FFFFFFFF333333++++++44444444######<;;;;;;  8 $ $tCyRZ &&&&&&& &R AIAIAIAIAIAIAI AIAIAIr