&`i) ddlZddlmZmZmZddlZddlZddl Z ddl m Z ddl m Z ddlmZddlmZmZddlmZmZmZddlmZed ejd efd Zed ejd efd Z dd ejdeeefdedefdZe j de j!j d edeejegdfdedeeeff dZ"eGdde Z#dS)N)AnyCallableDict)Dataset)OfflineEvaluator)Policy) SampleBatch convert_ma_batch_to_sample_batch) DeveloperAPIExperimentalAPIoverride)SampleBatchTypebatchindexc|tj|jd}|||f|dd|f<dS)Nr)nprandom permutationshape)rr random_indss x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/offline/feature_importance.py _perturb_fnrs>)'' A77KK./E!!!U(OOOctj|dj}t||t ||d<|S)Nobs perturbed_obs)rvstackvaluesrlist)rr obs_batchs r _perturb_dfr!s= %,-..I 5!!!!)__E/ Lr policy_state input_key output_keyc|s tj}tj|}ttjt j||ji}||d\}}}|sd}|||<|S)aA custom local function to do batch prediction of a policy. Given the policy state the action predictions are computed as a function of `input_key` and stored in the `output_key` column. Args: batch: A sub-batch from the dataset. policy_state: The state of the policy to use for the prediction. input_key: The key to use for the input to the policy. If not given, the default is SampleBatch.OBS. output_key: The key to use for the output of the policy. If not given, the default is "predicted_actions". Returns: The modified batch with the predicted actions added as a column. Fexplorepredicted_actions)r OBSr from_staterrrcompute_actions_from_input_dict)rr#r$r%policy sample_batchactions_s r_compute_actionsr1s, $O  | , ,F ORYuY'7'>?? L ::.delta_fnos,m 4u=P7Q QRRg r)r4r7) map_batchesr1) r2rr3r4r# perturbed_dsr9r>r<s rget_feature_importance_on_indexrAGs.&&E" 'L %00-((  1    ) )Zh *  E Lrc eZdZeeddefdededede e j ege j fffd Z de d eeeffd Zeed d d eded eeeffdZxZS)FeatureImportanceg?r-repeatlimit_fractionr3ctt|||_||_||_dS)aDFeature importance in a model inspection technique that can be used for any fitted predictor when the data is tablular. This implementation is also known as permutation importance that is defined to be the variation of the model's prediction when a single feature value is randomly shuffled. In RLlib it is implemented as a custom OffPolicyEstimator which is used to evaluate RLlib policies without performing environment interactions. Example usage: In the example below the feature importance module is used to evaluate the policy and the each feature's importance is computed after each training iteration. The permutation are repeated `self.repeat` times and the results are averages across repeats. ```python config = ( AlgorithmConfig() .offline_data( off_policy_estimation_methods= { "feature_importance": { "type": FeatureImportance, "repeat": 10, "limit_fraction": 0.1, } } ) ) algorithm = DQN(config=config) results = algorithm.train() ``` Args: policy: the policy to use for feature importance. repeat: number of times to repeat the perturbation. perturb_fn: function to perturb the features. By default reshuffle the features within the batch. limit_fraction: fraction of the dataset to use for feature importance This is only used in estimate_on_dataset when the dataset is too large to compute feature importance on. N)super__init__rEr3rF)selfr-rErFr3 __class__s rrIzFeatureImportance.__init__~s:d     $,rrreturncx t|}|d}|jd}tj|j|f |j|d\}}}t|jD]}t|D]u}tj |}t|||j|d\} }}tj tj | |z  ||f<v d fdtt D} | S)a)Estimate the feature importance of the policy. Given a batch of tabular observations, the importance of each feature is computed by perturbing each feature and computing the difference between the perturbed policy and the reference policy. The importance is computed for each feature and each perturbation is repeated `self.repeat` times. Args: batch: the batch of data to use for feature importance. Returns: A dict mapping each feature index string to its importance. rFr')rrc(i|]}d||Sfeature_.0i importances r z.FeatureImportance.estimate..%QQQQ>a>>:a=QQQr)r rrzerosrEr-compute_actionsrangecopydeepcopyrmeanr=len) rJrr n_featuresr;r0rrUcopy_obs_batchr9metricsrVs @restimatezFeatureImportance.estimatesP177%L _R( Xt{J788  K77 57QQ Qt{## T TA:&& T T!%y!9!9N!4444*.+*E*E"E+F++'!1a$&7262Ck2Q+R+R#S#S 1a4   T __Q'' QQQQ%J:P:PQQQr.) n_parallelismr2rec j |tj|z}t d||z}|t|d d}| ddtj j d}tjj|f t!jD]}|t d|z|z fdt!|D}t%j|} tjd| D |< d fd t!t- D} | S) atEstimate the feature importance of the policy given a dataset. For each feature in the dataset, the importance is computed by applying perturbations to each feature and computing the difference between the perturbed prediction and the reference prediction. The importance computation for each feature and each perturbation is repeated `self.repeat` times. If dataset is large the user can initialize the estimator with a `limit_fraction` to limit the dataset to a fraction of the original dataset. The dataset should include a column named `obs` where each row is a vector of D dimensions. The importance is computed for each dimension of the vector. Note (Implementation detail): The computation across features are distributed with ray workers since each feature is independent of each other. Args: dataset: the dataset to use for feature importance. n_parallelism: number of parallel workers to use for feature importance. Returns: A dict mapping each feature index string to its importance. rDr;)r%r#)r4r8rrNc Xg|]&}t|j'S))r2rr3bsizer#)rAremoter3)rTrUbsize_per_taskr#rJ shuffled_dss r z9FeatureImportance.estimate_on_dataset.. sQ   066'#(!- 7   rc8g|]}|dS)r<)r^)rTds rrlz9FeatureImportance.estimate_on_dataset..s"%N%N%N!affWoo%N%N%Nrc(i|]}d||SrPrRrSs rrWz9FeatureImportance.estimate_on_dataset..rXr)r- get_statelimitintrFcountmaxr?r1taker r*rrrYrEr[random_shuffleraygetarrayr^r_)rJr2redsrh actions_dsr`ra remote_fnsds_w_fi_scoresrcrjrVr#rks` @@@@restimate_on_datasetz%FeatureImportance.estimate_on_datasets6{,,.. ]]3t2W]]__DEE F FArxxzz]233^^ + ,$  WWQZZ];?39"= Xt{J788 t{## P PA$3355K [%6%6%8%8:%E-$WXXN       z**   J!WZ00NH%N%N~%N%N%NOOJqMM__Q'' QQQQ%J:P:PQQQr)__name__ __module__ __qualname__r rr!rrrfloatrpd DataFramerIrrstrrrdrr~ __classcell__)rKs@rrCrC|s. X #BM 4-4-4-4- 4- blC0",>? 4-4-4-4-4- 4-l!o!$sCx.!!!!FX8;BBBB25B c3hBBB BBBBBrrC)r"r")$r\typingrrrnumpyrr6rrwray.datar#ray.rllib.offline.offline_evaluatorrray.rllib.policyrray.rllib.policy.sample_batchr r ray.rllib.utils.annotationsr r r ray.rllib.utils.typingrndarrayrrrrr!rr1ridatarArCrRrrrs7 &&&&&&&&&& @@@@@@######WWWWWWWWOOOOOOOOOO2222220rz0#0000 r|C %% <%sCx.%% %%%%P1 X 1 1",,d23 1  1 sCx. 111 1h^^^^^(^^^^^r