&`i% ddlZddlZddlmZmZmZmZddlZddl Z ddl m Z ddl mZddlmZddlmZmZddlmZddlmZdd lmZmZdd lmZmZdd lmZdd l m!Z!ej"Z#eGd deZ$dS)N)AnyDictListOptional)Dataset) FQETorchModel)OffPolicyEstimator)compute_is_weightscompute_q_and_v_values)OfflineEvaluator)Policy) SampleBatch convert_ma_batch_to_sample_batch) DeveloperAPIoverride)convert_to_numpy)SampleBatchTypec eZdZdZee ddedededed e e f fd Z eed e d e e effd Zeede d e e eeffdZeeded e e effdZeedddeded e e effdZxZS) DoublyRobusta.The Doubly Robust estimator. Let s_t, a_t, and r_t be the state, action, and reward at timestep t. This method trains a Q-model for the evaluation policy \pi_e on behavior data generated by \pi_b. Currently, RLlib implements this using Fitted-Q Evaluation (FQE). You can also implement your own model and pass it in as `q_model_config = {"type": your_model_class, **your_kwargs}`. For behavior policy \pi_b and evaluation policy \pi_e, define the cumulative importance ratio at timestep t as: p_t = \sum_{t'=0}^t (\pi_e(a_{t'} | s_{t'}) / \pi_b(a_{t'} | s_{t'})). Consider an episode with length T. Let V_T = 0. For all t in {0, T - 1}, use the following recursive update: V_t^DR = (\sum_{a \in A} \pi_e(a | s_t) Q(s_t, a)) + p_t * (r_t + \gamma * V_{t+1}^DR - Q(s_t, a_t)) This estimator computes the expected return for \pi_e for an episode as: V^{\pi_e}(s_0) = V_0^DR and returns the mean and standard deviation over episodes. For more information refer to https://arxiv.org/pdf/1911.06854.pdfTNpolicygammaepsilon_greedynormalize_weightsq_model_configc`t||||pi}||d<|dt|_||_||_|jd d|i||_t|jds Jdt|jds JddS) a=Initializes a Doubly Robust OPE Estimator. Args: policy: Policy to evaluate. gamma: Discount factor of the environment. epsilon_greedy: The probability by which we act acording to a fully random policy during deployment. With 1-epsilon_greedy we act according the target policy. normalize_weights: If True, the inverse propensity scores are normalized to their sum across the entire dataset. The effect of this is similar to weighted importance sampling compared to standard importance sampling. q_model_config: Arguments to specify the Q-model. Must specify a `type` key pointing to the Q-model class. This Q-model is trained in the train() method and is used to compute the state-value and Q-value estimates for the DoublyRobust estimator. It must implement `train`, `estimate_q`, and `estimate_v`. TODO (Rohan138): Unify this with RLModule API. rtyper estimate_vz'self.model must implement `estimate_v`! estimate_qz'self.model must implement `estimate_q`!N) super__init__popr _model_cls_model_configs_normalize_weightsmodelhasattr)selfrrrrr __class__s ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/offline/estimators/doubly_robust.pyr"zDoublyRobust.__init__3s: 777'-2"'w(,,V]CC,"3$T_       J    5 5 4 5 5  J    5 5 4 5 5  5 5episodereturncRi}|d|d}}||}||z }d}d}|j|} t| } |j|} t| } | j| jcxkr |jfksnJtt|jD]C} || |j |zz}| | || || |j |zz| | z zz}D| }||d<||d<|S)Nrewards action_probr v_behaviorv_target) compute_action_probsr'rrrshapecountreversedrangeritem) r)r-estimates_per_epsioder0old_probnew_probweightr2r3q_valuesv_valuests r+estimate_on_single_episodez'DoublyRobust.estimate_on_single_episodecsL "#I. 0F,,W55H$ :((11#H--:((11#H--~CCCCGM3CCCCCCC% ..//  A dj:&==J{VAY TZ(22Xa[@&HH==??.8l+,4j)$$r,batchc0i}|d|d}}||}|j|}t|}|j|}t|}|}||z } || ||z zz} ||d<| |d<|S)Nr0r1r2r3)r4r'rrr) r)rBr:r0r;r<r>r?r2r=r3s r+estimate_on_single_step_samplesz,DoublyRobust.estimate_on_single_step_sampless!#!),eM.B,,U33:((//#H--:((//#H-- H$f((:;;.8l+,4j)$$r,ct|}|j|}dtj|iS)zTrains self.model on the given batch. Args: batch: A SampleBatch or MultiAgentbatch to train on Returns: A dict with key "loss" and value as the mean training loss. loss)rr'trainnpmean)r)rBlossess r+rGzDoublyRobust.trains9177!!%((((r,.) n_parallelismdatasetrKc&t||zd}|t|d|j|jd}t||zd}|t|d|jj|jd}ddtj dtfd }|j r| d nd}|||dd|i}| d }| d }||z } |d |z |z tj|z } ||| | d S)aEstimates the policy value using the Doubly Robust estimator. The doubly robust estimator uses normalization of importance sampling weights (aka. propensity ratios) to the average of the importance weights across the entire dataset. This is done to reduce the variance of the estimate (similar to weighted importance sampling). You can disable this by setting `normalize_weights=False` in the constructor. Note: This estimate works for only discrete action spaces for now. Args: dataset: Dataset to compute the estimate on. Each record in dataset should include the following columns: `obs`, `actions`, `action_prob` and `rewards`. The `obs` on each row shoud be a vector of D dimensions. n_parallelism: Number of parallelism to use for the computation. Returns: A dict with the following keys: v_target: The estimated value of the target policy. v_behavior: The estimated value of the behavior policy. v_gain: The estimated gain of the target policy over the behavior policy. v_std: The standard deviation of the estimated value of the target. pandas) policy_stateestimator_class) batch_size batch_format fn_kwargs) model_class model_state?rB normalizercr|d|z }|d||d|dz zz|d<|d|d<|S)Nweightsr?r0r>r3r2r )rBrXrZs r+compute_v_targetz:DoublyRobust.estimate_on_dataset..compute_v_targetsTI&3G %j 1Gi 5#445!E* #( "2E, Lr,rZr2r3)r2r3 v_gain_mean v_gain_ste)rW)maxr6 map_batchesr r get_stater*r r'pd DataFramefloatr&rIstdmathsqrt) r)rLrKrR updated_dsr[rXr2r3r\r]s r+estimate_on_datasetz DoublyRobust.estimate_on_datasets<M91== (( !! $ 5 5 7 7#'> )  ))++}rs ,,,,,,,,,,,,FFFFFFPPPPPPA@@@@@######WWWWWWWW>>>>>>>>222222222222    a a a a a %a a a a a r,