&`ic/ddlmZmZddlZddlZddlmZddl m Z ddl m Z m Z ddlmZddlmZmZe Gdd Ze d ed ed e d dfdZe d!de ded edededededefdZe d"dede deeee fd e fdZe de ded e fdZe dejd ed ejfd ZdS)#)DictOptionalN)Policy) SampleBatch) DeveloperAPI OldAPIStack)convert_to_numpy)AgentID TensorTypeceZdZdZdZdZdS)Postprocessingz(Constant definitions for postprocessing. advantages value_targetsN)__name__ __module__ __qualname____doc__ ADVANTAGES VALUE_TARGETSw/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/evaluation/postprocessing.pyr r s22J#MMMrr n_stepgammabatchreturnc ,|s Jdt|}tj|tj|dtj|tjdgt||zgd|tj<tj|tj |dz dtj |tj dt|dz |gd|tj <tj |vrztj|tj |dz dtj |tj dt|dz |gd|tj <t|D]]}td|D]J}||z|kr?|tj |xx||z|tj ||zzz cc<K^dS)aRewrites `batch` to encode n-step rewards, terminateds, truncateds, and next-obs. Observations and actions remain unaffected. At the end of the trajectory, n is truncated to fit in the traj length. Args: n_step: The number of steps to look ahead and adjust. gamma: The discount factor. batch: The SampleBatch to adjust (in place). Examples: n-step=3 Trajectory=o0 r0 d0, o1 r1 d1, o2 r2 d2, o3 r3 d3, o4 r4 d4=True o5 gamma=0.9 Returned trajectory: 0: o0 [r0 + 0.9*r1 + 0.9^2*r2 + 0.9^3*r3] d3 o0'=o3 1: o1 [r1 + 0.9*r2 + 0.9^2*r3 + 0.9^3*r4] d4 o1'=o4 2: o2 [r2 + 0.9*r3 + 0.9^2*r4] d4 o1'=o5 3: o3 [r3 + 0.9*r4] d4 o3'=o5 4: o4 r4 d4 o4'=o5 z8Unexpected terminated|truncated in middle of trajectory!Nraxis)is_single_trajectorylennp concatenaterOBSstackNEXT_OBSmin TERMINATEDStile TRUNCATEDSrangeREWARDS)rrrlen_ijs r adjust_nstepr2s2 ""$$BBABB$ u::D#%. +/ "677 + HeK01"56VT9J9JJ K K  ###E+ &(^ +) *6A:<< 8 GE+1226FQJ8M8M N N  &&&E+ !"&&(*k,-fqjll;k45b93vz4;P;PQQ  ) ) ) k$%4[[q&!!  A1ut||k)*1---1Hu[%89!a%@@--- r??Trolloutlast_rlambda_use_gae use_criticrewardsvf_predsctj|vs |r Jd|s |r Jdt|}||tj}||r|tj}|rt j|t j|gg}|||ddzz|ddz } t| ||z|tj <|tj |z tj |tj <nt j|t j|gg} t| |dd tj } |r"| |z |tj <| |tj <n@| |tj <t j |tj |tj <|tj  tj |tj <|S)aGiven a rollout, compute its value targets and the advantages. Args: rollout: SampleBatch of a single trajectory. last_r: Value estimation for last observation. gamma: Discount factor. lambda_: Parameter for GAE. use_gae: Using Generalized Advantage Estimation. use_critic: Whether to use critic (value estimates). Setting this to False will use 0 as baseline. rewards: Override the reward values in rollout. vf_preds: Override the value function predictions in rollout. Returns: SampleBatch with experience from rollout and processed rewards. z$use_critic=True but values not foundz,Can't use gae without using a value functionNr!r)rVF_PREDSr r.r$r%arraydiscount_cumsumr rastypefloat32r zeros_like) r5r6rr7r8r9r:r;vpred_tdelta_trewards_plus_vdiscounted_returnss rcompute_advantagesrGUs8 ''z''- ('9 TWTT&TTT $ f % %F+-.J;/0.(BHfX,>,>!?@@EGABBK//'#2#,>.=Wego-V-V)* N- . 9 &   ,--"(F82D2D(EFF,^UCCCRCHOO J    1Ch1NGN- .4FGN0 1 11CGN- .46M1255GN0 1*11J)K)R)R **GN %& Nrpolicy sample_batchother_agent_batchesc t||}tj|tj}tj|tj}t |jdkrA|j|jksJtj|d}tj|d}d}nd}t||tj d|j d|j d|j d |j d d|| }|r3tj |tjd|tj<|S) aAdds GAE (generalized advantage estimations) to a trajectory. The trajectory contains only data from one episode and from one agent. - If `config.batch_mode=truncate_episodes` (default), sample_batch may contain a truncated (at-the-end) episode, in case the `config.rollout_fragment_length` was reached by the sampler. - If `config.batch_mode=complete_episodes`, sample_batch will contain exactly one episode (no matter how long). New columns can be added to sample_batch and existing ones may be altered. Args: policy: The Policy used to generate the trajectory (`sample_batch`) sample_batch: The SampleBatch to postprocess. other_agent_batches: Optional dict of AgentIDs mapping to other agents' trajectory data (from the same episode). NOTE: The other agents use the same policy. episode: Optional multi-agent episode object in which the agents operated. Returns: The postprocessed, modified SampleBatch (or a new one). r!rTFrrlambdar8r9)r5r6rr7r8r9r;r:)compute_bootstrap_valuer$r>rr=r.r#shapesqueezerGVALUES_BOOTSTRAPPEDconfigget expand_dimsr r)rHrIrJepisoder;r:squeezedrs rcompute_gae_for_sample_batchrWsA>+<@@Lx [%9:;;Hh|K$7899G 8>a~....:hQ///*W1--- K; .+ ,1, , , n'( Lrc|tjdrd}n)||jd}|jd i|}t j|tj}t|j dkrt j |d}d}nd }t j t|dd t jt|gt j gd |tj<|rVt j|d|tj<t j|tjd|tj<|S)aPerforms a value function computation at the end of a trajectory. If the trajectory is terminated (not truncated), will not use the value function, but assume that the value of the last timestep is 0.0. In all other cases, will use the given policy's value function to compute the "bootstrapped" value estimate at the end of the given trajectory. To do so, the very last observation (sample_batch[NEXT_OBS][-1]) and - if applicable - the very last state output (sample_batch[STATE_OUT][-1]) wil be used as inputs to the value function. The thus computed value estimate will be stored in a new column of the `sample_batch`: SampleBatch.VALUES_BOOTSTRAPPED. Thereby, values at all timesteps in this column are set to 0.0, except or the last timestep, which receives the computed bootstrapped value. This is done, such that in any loss function (which processes raw, intact trajectories, such as those of IMPALA and APPO) can use this new column as follows: Example: numbers=ts in episode, '|'=episode boundary (terminal), X=bootstrapped value (!= 0.0 b/c ts=12 is not a terminal). ts=5 is NOT a terminal. T: 8 9 10 11 12 <- no terminal VF_PREDS: . . . . . VALUES_BOOTSTRAPPED: 0 0 0 0 X Args: sample_batch: The SampleBatch (single trajectory) for which to compute the bootstrap value at the end. This SampleBatch will be altered in place (by adding a new column: SampleBatch.VALUES_BOOTSTRAPPED). policy: The Policy object, whose value function to use. Returns: The altered SampleBatch (with the extra SampleBatch.VALUES_BOOTSTRAPPED column). rglast)indexrLr!rTFN)dtyperr)rr*get_single_step_input_dictview_requirements_valuer$r>r=r#rOrPr%r rArQrT)rIrHr6 input_dictr;rVs rrNrNs`JK+,R0 -"<<  $F=  ,,,,x [%9:;;H 8>a:hQ/// 57N Xabb\ * * H&v../rz B B B  555L01 -/^H1-M-M-M [)*8: 8 99 9 9 [45 rxctjdgdt| g|dddddddS)aCalculates the discounted cumulative sum over a reward sequence `x`. y[t] - discount*y[t+1] = x[t] reversed(y)[t] - discount*reversed(y)[t-1] = reversed(x)[t] Args: gamma: The discount factor gamma. Returns: The sequence containing the discounted cumulative sums for each individual reward in `x` till the end of the trajectory. .. testcode:: :skipif: True x = np.array([0.0, 1.0, 2.0, 3.0]) gamma = 0.9 discount_cumsum(x, gamma) .. testoutput:: array([0.0 + 0.9*1.0 + 0.9^2*2.0 + 0.9^3*3.0, 1.0 + 0.9*2.0 + 0.9^2*3.0, 2.0 + 0.9*3.0, 3.0]) r!Nrrr)scipysignallfilterfloat)r`rs rr?r?,sG8 <  av%744R4q  I I$$B$ OOr)r3r4TTNN)NN)typingrrnumpyr$ scipy.signalrbray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.utils.annotationsrrray.rllib.utils.numpyr ray.rllib.utils.typingr r r intrer2boolrGrWrNndarrayr?rrrrqs!!!!!!!!******555555AAAAAAAA22222266666666$$$$$$$$ <<U<;<4<<< <~ AA A A A A  A  AAAAA AH AE  @@ @@"$w ';"<=@  @@@ @F J+JvJ+JJJ JZ PrzP%PBJPPP PPPr