&`i6?dZddlZddlmZddlmZe\ZZZej dgdZ ej ddZ edfd Z d Z edd d d fd Z ddZ ddZdZdS)a Functions to compute V-trace off-policy actor critic targets. For details and theory see: "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures" by Espeholt, Soyer, Munos et al. See https://arxiv.org/abs/1802.01561 for the full paper. In addition to the original paper's code, changes have been made to support MultiDiscrete action spaces. behaviour_policy_logits, target_policy_logits and actions parameters in the entry point multi_from_logits method accepts lists of tensors instead of just tensors. N) Categorical) try_import_tfVTraceFromLogitsReturnsvs pg_advantageslog_rhosbehaviour_action_log_probstarget_action_log_probs VTraceReturnszvs pg_advantagesc6t|g|g||dS)Nr)'multi_log_probs_from_logits_and_actions) policy_logitsactions dist_classmodels y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/algorithms/impala/vtrace_tf.py!log_probs_from_logits_and_actionsr4s+ 3 'J    c g}tt|D]"}t||}t||}t||tdg|ddgd}t||tdg|ddgd} |t|||| |dd$|S)aComputes action log-probs from policy logits and actions. In the notation used throughout documentation and comments, T refers to the time dimension ranging from 0 to T-1. B refers to the batch size and ACTION_SPACE refers to the list of numbers each representing a number of actions. Args: policy_logits: A list with length of ACTION_SPACE of float32 tensors of shapes [T, B, ACTION_SPACE[0]], ..., [T, B, ACTION_SPACE[-1]] with un-normalized log-probabilities parameterizing a softmax policy. actions: A list with length of ACTION_SPACE of tensors of shapes [T, B, ...], ..., [T, B, ...] with actions. dist_class: Python class of the action distribution. Returns: A list with length of ACTION_SPACE of float32 tensors of shapes [T, B], ..., [T, B] corresponding to the sampling log probability of the chosen action w.r.t. the policy. Nraxis)rangelentfshapereshapeconcatappendlogp) rrrr log_probsip_shapea_shapepolicy_logits_flat actions_flats rrr<s*.I 3}%% & &   ((=+,,((71:&&ZZ ! bii"wqrr{(;!iDD  zz'!*bii"wqrr{8KRSi.T.TUU  JJ -u55::<HH'RTSTRT+      r?vtrace_from_logitsc t|g|g|g||||||| | |  } t| j| j| jt | jdt | jdS)z-multi_from_logits wrapper used only for tests)clip_rho_thresholdclip_pg_rho_thresholdnamerrr) multi_from_logitsrrrr rsqueezer r ) behaviour_policy_logitstarget_policy_logitsr discountsrewardsvaluesbootstrap_valuerrr,r-r.ress r from_logitsr8ds  !    -3    C # 6'#%::c.LST:#U#U " 3+FQ O O    rc tt|D]} t|| tj|| <t|| tj|| <|| jd|| jdt| |||||||g5t||||}t|dks| t||||} t|| }t|||||| | }td|| |d| cdddS#1swxYwYdS) a V-trace for softmax policies. Calculates V-trace actor critic targets for softmax polices as described in "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures" by Espeholt, Soyer, Munos et al. Target policy refers to the policy we are interested in improving and behaviour policy refers to the policy that generated the given rewards and actions. In the notation used throughout documentation and comments, T refers to the time dimension ranging from 0 to T-1. B refers to the batch size and ACTION_SPACE refers to the list of numbers each representing a number of actions. Args: behaviour_policy_logits: A list with length of ACTION_SPACE of float32 tensors of shapes [T, B, ACTION_SPACE[0]], ..., [T, B, ACTION_SPACE[-1]] with un-normalized log-probabilities parameterizing the softmax behaviour policy. target_policy_logits: A list with length of ACTION_SPACE of float32 tensors of shapes [T, B, ACTION_SPACE[0]], ..., [T, B, ACTION_SPACE[-1]] with un-normalized log-probabilities parameterizing the softmax target policy. actions: A list with length of ACTION_SPACE of tensors of shapes [T, B, ...], ..., [T, B, ...] with actions sampled from the behaviour policy. discounts: A float32 tensor of shape [T, B] with the discount encountered when following the behaviour policy. rewards: A float32 tensor of shape [T, B] with the rewards generated by following the behaviour policy. values: A float32 tensor of shape [T, B] with the value function estimates wrt. the target policy. bootstrap_value: A float32 of shape [B] with the value function estimate at time T. dist_class: action distribution class for the logits. model: backing ModelV2 instance behaviour_action_log_probs: precalculated values of the behaviour actions clip_rho_threshold: A scalar float32 tensor with the clipping threshold for importance weights (rho) when calculating the baseline targets (vs). rho^bar in the paper. clip_pg_rho_threshold: A scalar float32 tensor with the clipping threshold on rho_s in \rho_s \delta log \pi(a|x) (r + \gamma v_{s+1} - V(x_s)). name: The name scope that all V-trace operations will be created in. Returns: A `VTraceFromLogitsReturns` namedtuple with the following fields: vs: A float32 tensor of shape [T, B]. Can be used as target to train a baseline (V(x_t) - vs_t)^2. pg_advantages: A float 32 tensor of shape [T, B]. Can be used as an estimate of the advantage in the calculation of policy gradients. log_rhos: A float32 tensor of shape [T, B] containing the log importance sampling weights (log rhos). behaviour_action_log_probs: A float32 tensor of shape [T, B] containing behaviour policy action log probabilities (log \mu(a_t)). target_action_log_probs: A float32 tensor of shape [T, B] containing target policy action probabilities (log \pi(a_t)). dtyper5N)r r3r4r5r6r,r-)r r r )rrrconvert_to_tensorfloat32rassert_has_ranktf1 name_scoper get_log_rhosfrom_importance_weightsr_asdict)r1r2rr3r4r5r6rrr r,r-r.r$r r vtrace_returnss rr/r/sj3.// 0 0 9 9%'%9%9 #A &bj&:& & "#%"6"6  #2:#7# # Q "(88;;;Q%55a8888  #         ) ) #J ':u# #  & ' '! + +/I/Q*Q'*e** & 79STT0+1"7   ' 'A$;  $$&&   I) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) s!BE//E36E3vtrace_from_importance_weightsc t|tj}t|tj}t|tj}t|tj}t|tj}|&t|tj}|&t|tj}|jj}|j||j|dz |j||j|||jd||jdt ||||||g5tj |} |t || d} n| } t d| d } t |ddt |dgd } | ||| zz|z z} t |dg t | dg t | dg f}d }t|}tjtjt|||dd }t |dgd}t||d}t |ddt |dgd }|t || d}n| }||||zz|z z}t)t|t|cdddS#1swxYwYdS)aFV-trace from log importance weights. Calculates V-trace actor critic targets as described in "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures" by Espeholt, Soyer, Munos et al. In the notation used throughout documentation and comments, T refers to the time dimension ranging from 0 to T-1. B refers to the batch size. This code also supports the case where all tensors have the same number of additional dimensions, e.g., `rewards` is [T, B, C], `values` is [T, B, C], `bootstrap_value` is [B, C]. Args: log_rhos: A float32 tensor of shape [T, B] representing the log importance sampling weights, i.e. log(target_policy(a) / behaviour_policy(a)). V-trace performs operations on rhos in log-space for numerical stability. discounts: A float32 tensor of shape [T, B] with discounts encountered when following the behaviour policy. rewards: A float32 tensor of shape [T, B] containing rewards generated by following the behaviour policy. values: A float32 tensor of shape [T, B] with the value function estimates wrt. the target policy. bootstrap_value: A float32 of shape [B] with the value function estimate at time T. clip_rho_threshold: A scalar float32 tensor with the clipping threshold for importance weights (rho) when calculating the baseline targets (vs). rho^bar in the paper. If None, no clipping is applied. clip_pg_rho_threshold: A scalar float32 tensor with the clipping threshold on rho_s in \rho_s \delta log \pi(a|x) (r + \gamma v_{s+1} - V(x_s)). If None, no clipping is applied. name: The name scope that all V-trace operations will be created in. Returns: A VTraceReturns namedtuple (vs, pg_advantages) where: vs: A float32 tensor of shape [T, B]. Can be used as target to train a baseline (V(x_t) - vs_t)^2. pg_advantages: A float32 tensor of shape [T, B]. Can be used as the advantage in the calculation of policy gradients. r:Nr>rr= clipped_rhos)r.r)csrc$|\}}}|||z|zzS)Nr?)acc sequence_item discount_tc_tdelta_ts rscanfuncz)from_importance_weights..scanfunc~s#'4 $JWZ#-33 3rscan)fnelems initializerparallel_iterationsr. vs_minus_v_xsrclipped_pg_rhos)rr)rr@rArndimsrBrCrDmathexpminimumr expand_dimsreverse zeros_likenest map_structure stop_gradientrTaddr )r r3r4r5r6r,r-r.rho_rankrhosrKrLvalues_t_plus_1deltas sequencesrSinitial_valuesrYr vs_t_plus_1rZrs rrFrFs6h##HBJ#??H$$Ybj$AAI""7"*"==G  ! !& ! ; ;F**?"**MMO%112DBJ1WW( " 4 4 !!5! !  ~#H L  ***))(Q,777 O##H--- M!!(+++% 00333(#33A666  h 7FOL   ; ; w{{8$$  )::&8$^:TTLLL ZZTZ - -)) ABBZ;; <1$  9+F!F!OP JJysJ + + JJrJ $ $ JJvQCJ ( (  4 4 477--   GG*$%       =1#O LL VVM6V 5 5iiABB)K)K LSTiUU , jj%t2C)OO#O'7Y5L+Lv+UV ##23C3CM3R3R   s; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; ; sIP<<QQct|}t|}t||z d}|S)zWith the selected log_probs for multi-discrete actions of behaviour and target policies we compute the log_rhos for calculating the vtrace.rr)rstack reduce_sum)r r tbr s rrErEsF ())A +,,A}}QU}++H Or)Nr)r)r*)r)r)rI)__doc__ collections"ray.rllib.models.tf.tf_action_distrray.rllib.utils.frameworkrrCrtfv namedtuplerr rrr8r/rFrEr?rrrxsE"::::::333333} R0+0  ' &8JKK (3$    %%%`  %%%%d $ K K K K h )F F F F Rr