&`i]dZddlZddlmZmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZddlmZmZddlmZdd lmZmZdd lmZmZdd lmZdd lmZmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4erddl5m6Z6ej7e8Z9dZ:dZ;dZGddeZ?GddeZ@dS)a3 Proximal Policy Optimization (PPO) ================================== This file defines the distributed Algorithm class for proximal policy optimization. See `ppo_[tf|torch]_policy.py` for the definition of the policy loss. Detailed documentation: https://docs.ray.io/en/master/rllib-algorithms.html#ppo N) TYPE_CHECKINGAnyDictListOptionalTypeUnion)Self)DEPRECATED_VALUE) Algorithm)AlgorithmConfig NotProvided) RLModuleSpec)standardize_fieldssynchronous_parallel_sample)multi_gpu_train_one_steptrain_one_step)Policy) OldAPIStackoverride) ALL_MODULESENV_RUNNER_RESULTSENV_RUNNER_SAMPLING_TIMERLEARNER_RESULTSLEARNER_UPDATE_TIMERNUM_AGENT_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLEDNUM_ENV_STEPS_SAMPLED_LIFETIME!NUM_MODULE_STEPS_TRAINED_LIFETIME SAMPLE_TIMERSYNCH_WORKER_WEIGHTS_TIMERTIMERS)LEARNER_STATS_KEY) Scheduler) ResultDict)log_once)Learnervf_loss_unclippedvf_explained_var mean_kl_loss curr_kl_coeffcurr_entropy_coeffc!eZdZdZdfd ZeedefdZeede e de ffdZ eee e e e e e e e e e e e e edd eed eed eed eed eedeedeedeedeeee eefdeedeedeedeeee eefdeffdZeedfd Zeeedee efffd ZxZS) PPOConfigaTDefines a configuration class from which a PPO Algorithm can be built. .. testcode:: from ray.rllib.algorithms.ppo import PPOConfig config = PPOConfig() config.environment("CartPole-v1") config.env_runners(num_env_runners=1) config.training( gamma=0.9, lr=0.01, kl_coeff=0.3, train_batch_size_per_learner=256 ) # Build a Algorithm object from the config and run 1 training iteration. algo = config.build() algo.train() .. testcode:: from ray.rllib.algorithms.ppo import PPOConfig from ray import tune config = ( PPOConfig() # Set the config object's env. .environment(env="CartPole-v1") # Update the config object's training parameters. .training( lr=0.001, clip_param=0.2 ) ) tune.Tuner( "PPO", run_config=tune.RunConfig(stop={"training_iteration": 1}), param_space=config, ).fit() .. testoutput:: :hide: ... Ncddi|_t|ptd|_d|_d|_d|_d|_d|_ d |_ d|_ d |_ d|_ d |_d |_d |_d |_d|_d|_d|_d|_d|jd<d|_d|_t2|_t2|_dS)z!Initializes a PPOConfig instance.typeStochasticSampling) algo_classg-C6 ?autoiTg?g?g{Gz?g333333?g$@NFvf_share_layers)exploration_configsuper__init__PPOlrrollout_fragment_lengthtrain_batch_size use_criticuse_gae num_epochsminibatch_sizeshuffle_batch_per_epochlambda_ use_kl_losskl_coeff kl_target vf_loss_coeff entropy_coeff clip_param vf_clip_param grad_clipnum_env_runnersmodelentropy_coeff_schedule lr_scheduler sgd_minibatch_sizer8)selfr2 __class__s p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/algorithms/ppo/ppo.pyr;zPPOConfig.__init__ls ( #  J$5#666'-$ $ !'+$    ! !). $%&*##3/returnct|jdkrddlm}t|St d|jd)Ntorchr)DefaultPPOTorchRLModule) module_classThe framework z/ is not supported. Use either 'torch' or 'tf2'.) framework_str:ray.rllib.algorithms.ppo.torch.default_ppo_torch_rl_modulerZr ValueError)rSrZs rUget_default_rl_module_specz$PPOConfig.get_default_rl_module_specsm   ( (       -DEEE E/!3/// rVr'c|jdkrddlm}|S|jdvrtdtd|jd)NrYr)PPOTorchLearner)tf2tfzPTensorFlow is no longer supported on the new API stack! Use `framework='torch'`.r\z+ is not supported. Use `framework='torch'`.)r]0ray.rllib.algorithms.ppo.torch.ppo_torch_learnerrbr_)rSrbs rUget_default_learner_classz#PPOConfig.get_default_learner_classs   ( (      # "  = 0 0+  +!3+++ rV)r@rArErFrGrHrIrJrPrKrLrMrQr8r@rArErFrGrHrIrJrPrKrLrMrQc tjdi||tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_ |tur||_ | tur| |_ | tur| |_ | tur| |_ | tur| |_| tur| |_|S)a Sets the training related configuration. Args: use_critic: Should use a critic as a baseline (otherwise don't use value baseline; required for using GAE). use_gae: If true, use the Generalized Advantage Estimator (GAE) with a value function, see https://arxiv.org/pdf/1506.02438.pdf. lambda_: The lambda parameter for General Advantage Estimation (GAE). Defines the exponential weight used between actually measured rewards vs value function estimates over multiple time steps. Specifically, `lambda_` balances short-term, low-variance estimates against long-term, high-variance returns. A `lambda_` of 0.0 makes the GAE rely only on immediate rewards (and vf predictions from there on, reducing variance, but increasing bias), while a `lambda_` of 1.0 only incorporates vf predictions at the truncation points of the given episodes or episode chunks (reducing bias but increasing variance). use_kl_loss: Whether to use the KL-term in the loss function. kl_coeff: Initial coefficient for KL divergence. kl_target: Target value for KL divergence. vf_loss_coeff: Coefficient of the value function loss. IMPORTANT: you must tune this if you set vf_share_layers=True inside your model's config. entropy_coeff: The entropy coefficient (float) or entropy coefficient schedule in the format of [[timestep, coeff-value], [timestep, coeff-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated coefficient values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. clip_param: The PPO clip parameter. vf_clip_param: Clip param for the value function. Note that this is sensitive to the scale of the rewards. If your expected V is large, increase this. grad_clip: If specified, clip the global norm of gradients by this amount. Returns: This updated AlgorithmConfig object. )r:trainingrr@rArErFrGrHrIrJrKrLrMrQrP)rSr@rArErFrGrHrIrJrPrKrLrMrQr8kwargsrTs rUrizPPOConfig.trainingsv ""6""" [ ( ((DO + % %"DL + % %"DL k ) )*D  ; & &$DM K ' '&DN  + +!.D   + +!.D  [ ( ((DO  + +!.D  K ' '&DN k ) )*D  ! 4 4*@D ' rVc t||jsG|j|jkr7|d|jd|jd|jd|jd ns|jrl|j}|jp|j}t|trBt|tr-||kr'|d|d|d|d|jd |j s'|j dkr|j s|d|jrT|j|d |j|d t!j|jd d t|jt$r"|jdkr|ddSdSdS)Nz`minibatch_size` (z!) must be <= `train_batch_size` (z.). In PPO, the train batch will be split into zG chunks, each of which is iterated over (used for updating the policy) z times.z-) must be <= `train_batch_size_per_learner` (truncate_episodeszEpisode truncation is not supported without a value function (to estimate the return at the end of the truncated trajectory). Consider setting batch_mode=complete_episodes.zW`lr_schedule` is deprecated and must be None! Use the `lr` setting to setup a schedule.zm`entropy_coeff_schedule` is deprecated and must be None! Use the `entropy_coeff` setting to setup a schedule.rJzentropy coefficient)fixed_value_or_schedule setting_name descriptionr6z`entropy_coeff` must be >= 0.0)r:validate4validate_train_batch_size_vs_rollout_fragment_lengthenable_rl_module_and_learnerrCr? _value_errorrBtrain_batch_size_per_learner isinstanceint in_evaluation batch_moderArQrPr$rJfloat)rSmbstbsrTs rUrpzPPOConfig.validates  AACCC 1 #d&;;;   T%8'+'<'+':DH?     . %C3Lt7LC#s##  3(<(< s!!OOO7:OO+.OO7;oOOO" #666L7   0     , +!!8*6!!C  (,(:,1     d(% 0 0 @T5G#5M5M   > ? ? ? ? ? @ @5M5MrVc4tjddizS)Nr8F)r:_model_config_auto_includes)rSrTs rUr}z%PPOConfig._model_config_auto_includesgsww26G5OOOrVNrWN)__name__ __module__ __qualname____doc__r;rr rr`r rstrrfrr rboolryrrvr rirppropertyrrr} __classcell__)rTs@rUr.r.?s**X/0/0/0/0/0/0bXo L    Xo5i#1E+F$Xo&1"-#.&1$/%0)4)4JU&1)4%0?J(%[[[TN[$ [ % [ d^ [5/[E?[ [ [!)d5e3D.E)F G[UO[ [E?[ d4c5j(9#:; >+ N   ! . .-1-?*# /                     s?B D !D  DD;B=HH H5AJJJc hjt5jjdkr,t jjjjj}n+t jjjjj}|sicdddS|}j txx| z cc<j txx| z cc<t|dg}dddn #1swxYwYjjrt!|}nt#|}t%|}j tfd|Dd}jt(5jdkrd}j|||dddn #1swxYwY|D]\}}|t0d }||jj|t0d z} |t0d } t;d rcjd idr5| dkr/t<d || | |j!|"d|j!|d#} t;drF| jj$kr6d_%t<d| djdd|dj&'||S)Nr)rrr)rrr advantagesc@i|]}|jj|jSrh) env_runner policy_mapnum_grad_updates).0pidrSs rU z4PPO._training_step_old_api_stack..s8,,,T_/4E,,,rV)timestepnum_grad_updates_per_policyr)rr global_varsklvf_loss policy_lossppo_warned_lr_ratiorOr8dzThe magnitude of your value function loss for policy: {} is extremely large ({}) compared to the policy loss ({}). This can prevent the policy from learning. Consider scaling down the VF loss by reducing vf_loss_coeff, or disabling vf_share_layers.rewardsppo_warned_vf_clipTz1The mean reward returned from the environment is z! but the vf_clip_param is set to rLz%. Consider increasing it for policy: z' to improve value function convergence.)(_timersr rrrrrras_multi_agent _countersrrr env_stepsrsimple_optimizerrrlistrr!num_remote_workersritemsr#get get_policy update_klrIr&loggerwarningformatpolicy_batchesset_get_interceptormeanrLwarned_vf_cliprset_global_vars) rS train_batch train_resultspolicies_to_updaterr policy_id policy_info kl_divergencescaled_vf_lossr mean_rewards ` rUrz PPO._training_step_old_api_stacks\, ' J J{)]::9#4$(K$F%)[%A :#4"&+"D%)[%A   J J J J J J J J &4466K N2 3 3 3{7N7N7P7P P 3 3 3 N0 1 1 1[5J5J5L5L L 1 1 1,[<.IIK) J J J J J J J J J J J J J J J, ; ' H*4==MM4T;GGM!-"4"4"6"677'>?,,,,-,,,   \4 5  $7799A==/3,%221M/ +3               '4&9&9&;&;$ $  "I{((9:>>tDDM OOI & & 0 0 ? ? ? )K8I,J9,UU &&78GK.// KOOGR00445FGG #S(('(.vi'U'U   &y 1 E Ed K K K%4Y? JOOQQK-.. $+";;;&*#3 338< O8T33;D333 '' 444s+A+D A=DDD!=G**G.1G.r)rrr classmethodrr r.rr rrrrrrr%rrhrVrUr<r<ms Xi9[ Xi $ $v,    [ "XiOOOObajaaa[aaarVr<)Arloggingtypingrrrrrrr typing_extensionsr ray._common.deprecationr ray.rllib.algorithms.algorithmr %ray.rllib.algorithms.algorithm_configr r"ray.rllib.core.rl_module.rl_modulerray.rllib.execution.rollout_opsrrray.rllib.execution.train_opsrrray.rllib.policy.policyrray.rllib.utils.annotationsrrray.rllib.utils.metricsrrrrrrrrrr r!r"$ray.rllib.utils.metrics.learner_infor##ray.rllib.utils.schedules.schedulerr$ray.rllib.utils.typingr%ray.util.debugr&ray.rllib.core.learner.learnerr' getLoggerrr%LEARNER_RESULTS_VF_LOSS_UNCLIPPED_KEY$LEARNER_RESULTS_VF_EXPLAINED_VAR_KEYLEARNER_RESULTS_KL_KEY!LEARNER_RESULTS_CURR_KL_COEFF_KEY&LEARNER_RESULTS_CURR_ENTROPY_COEFF_KEYr.r<rhrVrUrs  HHHHHHHHHHHHHHHHHH""""""444444444444NNNNNNNN;;;;;;+*****========                            CBBBBB999999------######7666666  8 $ $(;%'9$'$3!)=&kPkPkPkPkPkPkPkP\ MMMMM)MMMMMrV