&`iFdZddlZddlmZmZddlmZddlmZm Z ddl m Z m Z ddl mZmZddlmZdd lmZdd lmZdd lmZmZmZmZmZejeZd Zd Z dZ!GddeZ"GddeZ#dS)aOAsynchronous Proximal Policy Optimization (APPO) The algorithm is described in [1] (under the name of "IMPACT"): Detailed documentation: https://docs.ray.io/en/master/rllib-algorithms.html#appo [1] IMPACT: Importance Weighted Asynchronous Architectures with Clipped Target Networks. Luo et al. 2020 https://arxiv.org/pdf/1912.00167 N)OptionalType)Self)DEPRECATED_VALUEdeprecation_warning)AlgorithmConfig NotProvided)IMPALA IMPALAConfig) RLModuleSpec)Policy)override)LAST_TARGET_UPDATE_TSLEARNER_STATS_KEYNUM_AGENT_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLEDNUM_TARGET_UPDATES mean_kl_loss curr_kl_coeffold_action_distceZdZdZdfd Zeeeeeeeeeeeeeeeedde e de e de e de e d e e d e e d e e d e e d e e de e de e de e de ffdZeedfd ZeedZeedefdZeeefdZxZS) APPOConfiga Defines a configuration class from which an APPO Algorithm can be built. .. testcode:: from ray.rllib.algorithms.appo import APPOConfig config = ( APPOConfig() .training(lr=0.01, grad_clip=30.0, train_batch_size_per_learner=50) ) config = config.learners(num_learners=1) config = config.env_runners(num_env_runners=1) config = config.environment("CartPole-v1") # Build an Algorithm object from the config and run 1 training iteration. algo = config.build() algo.train() del algo .. testcode:: from ray.rllib.algorithms.appo import APPOConfig from ray import tune config = APPOConfig() # Update the config object. config = config.training(lr=tune.grid_search([0.001,])) # Set the config object's env. config = config.environment(env="CartPole-v1") # Use to_dict() to get the old-style python config dict when running with tune. tune.Tuner( "APPO", run_config=tune.RunConfig( stop={"training_iteration": 1}, verbose=0, ), param_space=config.to_dict(), ).fit() .. testoutput:: :hide: ... Nc\ddi|_t|ptd|_d|_d|_d|_d|_d|_ d|_ d |_ d |_ d |_ d |_d |_d |_d |_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d |_d |_d|_ d|_!d|_"d|_#tH|_%tH|_&dS)z"Initializes a APPOConfig instance.typeStochasticSampling) algo_classTg?g?Fg{Gz?g@gD@ global_normadamgMb@?gGz?g?g?Nrdi,)'exploration_configsuper__init__APPOvtraceuse_gaelambda_ clip_param use_kl_losskl_coeff kl_targettarget_worker_clippingcircular_buffer_num_batches$circular_buffer_iterations_per_batchnum_env_runnerstarget_network_update_freqbroadcast_interval grad_clip grad_clip_byopt_typelrdecaymomentumepsilon vf_loss_coeff entropy_coefftau lr_scheduleentropy_coeff_schedulenum_gpusnum_multi_gpu_tower_stacksminibatch_buffer_sizereplay_proportionreplay_buffer_num_slotslearner_queue_sizelearner_queue_timeoutrtarget_update_frequency use_critic)selfr __class__s r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/algorithms/appo/appo.pyr'zAPPOConfig.__init__VsH ( #  J$6$777      &)#,-(451 !*+'"#*     ! &*# *+'%&"!$'*$"$%("(8$*)r)r*r+r,r-r.r/r4r?r0r1r2rIrJr)r*r+r,r-r.r/r4r?r0r1r2returnc .| tkrtddd|tkrtdddtjdi||tur||_|tur||_|tur||_|tur||_|tur||_ |tur||_ |tur||_ |tur||_ | tur| |_ | tur| |_| tur| |_| tur| |_|S) u Sets the training related configuration. Args: vtrace: Whether to use V-trace weighted advantages. If false, PPO GAE advantages will be used instead. use_gae: If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf. Only applies if vtrace=False. lambda_: GAE (lambda) parameter. clip_param: PPO surrogate slipping parameter. use_kl_loss: Whether to use the KL-term in the loss function. kl_coeff: Coefficient for weighting the KL-loss term. kl_target: Target term for the KL-term to reach (via adjusting the `kl_coeff` automatically). target_network_update_freq: NOTE: This parameter is only applicable on the new API stack. The frequency with which to update the target policy network from the main trained policy network. The metric used is `NUM_ENV_STEPS_TRAINED_LIFETIME` and the unit is `n` (see [1] 4.1.1), where: `n = [circular_buffer_num_batches (N)] * [circular_buffer_iterations_per_batch (K)] * [train batch size]` For example, if you set `target_network_update_freq=2`, and N=4, K=2, and `train_batch_size_per_learner=500`, then the target net is updated every 2*4*2*500=8000 trained env steps (every 16 batch updates on each learner). The authors in [1] suggests that this setting is robust to a range of choices (try values between 0.125 and 4). target_network_update_freq: The frequency to update the target policy and tune the kl loss coefficients that are used during training. After setting this parameter, the algorithm waits for at least `target_network_update_freq` number of environment samples to be trained on before updating the target networks and tune the kl loss coefficients. NOTE: This parameter is only applicable when using the Learner API (enable_rl_module_and_learner=True). tau: The factor by which to update the target policy network towards the current policy network. Can range between 0 and 1. e.g. updated_param = tau * current_param + (1 - tau) * target_param target_worker_clipping: The maximum value for the target-worker-clipping used for computing the IS ratio, described in [1] IS = min(π(i) / π(target), ρ) * (π / π(i)) circular_buffer_num_batches: The number of train batches that fit into the circular buffer. Each such train batch can be sampled for training max. `circular_buffer_iterations_per_batch` times. circular_buffer_iterations_per_batch: The number of times any train batch in the circular buffer can be sampled for training. A batch gets evicted from the buffer either if it's the oldest batch in the buffer and a new batch is added OR if the batch reaches this max. number of being sampled. Returns: This updated AlgorithmConfig object. rIr4T)oldnewerrorrJzM`use_critic` no longer supported! APPO always uses a value function (critic).)rQhelprS)rrr&trainingr r)r*r+r,r-r.r/r4r?r0r1r2)rKr)r*r+r,r-r.r/r4r?r0r1r2rIrJkwargsrLs rMrVzAPPOConfig.trainingshP #&6 6 6 -0     ) ) )  %      ""6"""  $ $ DK + % %"DL + % %"DL [ ( ((DO k ) )*D  ; & &$DM K ' '&DN %[ 8 8.HD + k ! !DH ! 4 4*@D ' &k 9 9/JD , /{ B B4  5 rNc4t|jrm|jdks |jdkr|d|jdkr|d|jdkr|ddSdSdS)Nrr"a`minibatch_buffer_size/replay_proportion` not valid on new API stack with APPO! Use `circular_buffer_num_batches` for the number of train batches in the circular buffer. To change the maximum number of times any batch may be sampled, set `circular_buffer_iterations_per_batch`.aO`num_multi_gpu_tower_stacks` not supported on new API stack with APPO! In order to train on multi-GPU, use `config.learners(num_learners=[number of GPUs], num_gpus_per_learner=1)`. To scale the throughput of batch-to-GPU-pre-loading on each of your `Learners`, set `num_gpu_loader_threads` to a higher number (recommended values: 1-8).r$aE`learner_queue_size` not supported on new API stack with APPO! In order set the size of the circular buffer (which acts as a 'learner queue'), use `config.training(circular_buffer_num_batches=..)`. To change the maximum number of times any batch may be sampled, set `config.training(circular_buffer_iterations_per_batch=..)`.)r&validateenable_rl_module_and_learnerrDrE _value_errorrCrGrKrLs rMrYzAPPOConfig.validates   , )Q..$2HC2O2O!!>.!33!!&",,!!R+  (-,rNc|jdkrddlm}|S|jdvrtdtd|jd)Ntorchr)APPOTorchLearner)tf2tfzPTensorFlow is no longer supported on the new API stack! Use `framework='torch'`.The framework z+ is not supported. Use `framework='torch'`.) framework_str2ray.rllib.algorithms.appo.torch.appo_torch_learnerr_ ValueError)rKr_s rMget_default_learner_classz$APPOConfig.get_default_learner_class4s   ( (      $ #  = 0 0+  +!3+++ rNcv|jdkrddlm}ntd|jdt |S)Nr^r)APPOTorchRLModulerbz/ is not supported. Use either 'torch' or 'tf2'.) module_class)rc4ray.rllib.algorithms.appo.torch.appo_torch_rl_modulerhrer )rKRLModules rMget_default_rl_module_specz%APPOConfig.get_default_rl_module_specGss   ( (       /!3///  2222rNc4tjddizS)Nvf_share_layersF)r&_model_config_auto_includesr\s rMroz&APPOConfig._model_config_auto_includesUsww26G5OOOrNNrON)__name__ __module__ __qualname____doc__r'rr r rrboolfloatintrrVrYrfr rlpropertyrro __classcell__rLs@rMrr(sS++ZC+C+C+C+C+C+JXl"-"-#.&1&1$/%04?*2=5@>I 0##ssss$ s % s UO sd^s5/sE?s%-SMse_s!)s&.c]s/7sms& 'ssssssjXl      DXl$Xl 3L 3 3 3 3 XoPPPPXPPPPPrNrceZdZfdZeedfd ZeeedefdZ eeede de e e fdZxZS) r(ctj|g|Ri||jjs|jddSdS)zInitializes an APPO instance.c*|Srp update_targetp_s rMzAPPO.__init__..eARARrNN)r&r'configrZ env_runnerforeach_policy_to_train)rKrargsrWrLs rMr'z APPO.__init__\sf1$111&111 {7 T O 3 34R4R S S S S S T TrNrONc:|jjr tSt|jt }|j|jjdkrtnt}|jj |jj z}||z |kro|jtxxdz cc<||jt <|j d|jjrfd}|j |S)N agent_stepsrc*|Srprrs rMrz$APPO.training_step..~rrNcZtvs$Jdtf|vrK|td}| J|f||dStd|dS)Nz'{} should be nested under policy id keyklzNo data for {}, not updating kl)rformatget update_klloggerwarning)pipi_idr train_resultss rMupdatez"APPO.training_step..updates,MAAAAHH-& DAAA  --*512CDHHNN!~~ u/E~~~ R((((('H'O'OPU'V'VWWWWWrN)rrZr& training_step _countersrcount_steps_byrr num_epochsrDrrrr-)rK last_updatecur_tstarget_update_freqrrrLs @rMrzAPPO.training_stepgs0 ; 3 +77((** *--// n%:; ;->>('*  "[3dk6WW K "4 4 4 N- . . .! 3 . . .4:DN0 1 O 3 34R4R S S S{& @ X X X X X"77???rNctSrp)r)clss rMget_default_configzAPPO.get_default_configs||rNrc|ddkrddlm}|S|ddkr|jrtdddlm}|Sddlm}|S) N frameworkr^r)APPOTorchPolicyrazWRLlib's RLModule and Learner API is not supported for tf1. Use framework='tf2' instead.) APPOTF1Policy) APPOTF2Policy)+ray.rllib.algorithms.appo.appo_torch_policyrrZre(ray.rllib.algorithms.appo.appo_tf_policyrr)rrrrrs rMget_default_policy_classzAPPO.get_default_policy_classs + ' ) ) S S S S S S" " K D ( (2  / O N N N N N N N N N N N rNrq)rrrsrtr'rr r classmethodrrrrrr rrzr{s@rMr(r([s T T T T TXf......` Xf:[ Xf!$! $v, !!![!!!!!rNr()$ruloggingtypingrrtyping_extensionsrray._common.deprecationrr%ray.rllib.algorithms.algorithm_configrr "ray.rllib.algorithms.impala.impalar r "ray.rllib.core.rl_module.rl_moduler ray.rllib.policy.policyr ray.rllib.utils.annotationsrray.rllib.utils.metricsrrrrr getLoggerrrrLEARNER_RESULTS_KL_KEY!LEARNER_RESULTS_CURR_KL_COEFF_KEYOLD_ACTION_DIST_KEYrr(rUrNrMrs  !!!!!!!!""""""IIIIIIIINNNNNNNNCCCCCCCC;;;;;;******000000  8 $ $($3!'pPpPpPpPpPpPpPpPf X!X!X!X!X!6X!X!X!X!X!rN