&`ilzddlZddlmZmZmZmZmZmZddlm Z ddl m Z m Z ddl mZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)e#\Z*Z+Z,e$Z-ej.e/Z0GddeZ1GddeZ2dS)N)AnyDictOptionalTupleTypeUnion)Self)DEPRECATED_VALUEdeprecation_warning)AlgorithmConfig NotProvided)DQN) SACTFPolicy)"AddObservationsFromEpisodesToBatch)+AddNextObservationsFromEpisodesToTrainBatch)Learner) RLModuleSpec)Policy) deep_update)override) try_import_tftry_import_tfp)EpisodeReplayBuffer)LearningRateOrScheduleRLModuleSpecTypec,VeZdZdZd$fd Zeeeeeeeeeeeeeeeeeeeeeeddee dee e e fdee e e fdee d ee d eee e fd eeeeeeffd ee d ee e e fdee dee dee dee e e fdeedeedeedeedee dee deedef*fdZeed%fd Zeed&dedefdZeedefdZeedeed e ffd!Zee d$fd" Zefd#ZxZS)' SACConfiga Defines a configuration class from which an SAC Algorithm can be built. .. testcode:: config = ( SACConfig() .environment("Pendulum-v1") .env_runners(num_env_runners=1) .training( gamma=0.9, actor_lr=0.001, critic_lr=0.002, train_batch_size_per_learner=32, ) ) # Build the SAC algo object from the config and run 1 training iteration. algo = config.build() algo.train() NcXddi|_t|ptd|_ddgdgddid|_ddgdgddid|_d|_d |_d |_ d |_ d |_ d tdddd|_ d|_d|_dddd|_d|_d|_d|_d|_d|_d|_d |_d|_d|_d|_d |_d|_d|_d|_t>|_ t>|_!dS)NtypeStochasticSampling) algo_classTrelu) fcnet_hiddensfcnet_activationpost_fcnet_hiddenspost_fcnet_activation custom_modelcustom_model_configFg{Gzt?g?autoPrioritizedEpisodeReplayBufferg.Ag333333?g?)rcapacityalphabetaga2U0*3?)actor_learning_ratecritic_learning_rateentropy_learning_rategiUMu>rid)"exploration_configsuper__init__SACtwin_qq_model_configpolicy_model_config clip_actionstau initial_alphatarget_entropyn_stepintreplay_buffer_configstore_buffer_in_checkpointstraining_intensity optimizationactor_lr critic_lralpha_lrlr grad_cliptarget_network_update_freqrollout_fragment_lengthtrain_batch_size_per_learnertrain_batch_size(num_steps_sampled_before_learning_startsmin_time_s_per_iteration"min_sample_timesteps_per_iteration_deterministic_loss_use_beta_distributionr use_state_preprocessorworker_side_prioritization)selfr! __class__s p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/algorithms/sac/sac.pyr6zSACConfig.__init__4s ( #  J$5#666  !3Z &"$%) #%   "3Z &"$%) #% $ $  " $ 5C% % !,1("&#'$(%)     *+'(.$-0) #8<5)*%25/$) &+#&6#*:''')r8r9r:r<r=r>r?rBrArCr;rIoptimization_configrErFrGrJrQrRrNr8r9r:r<r=r>r?rBrArCr;rIrYrErFrGrJrQrRrNreturnc Ntjdi||tur||_|tur|j||tur|j||tur||_|tur||_|tur||_ |tur||_ |tur||_ | tur+td|j id| iddgdg}|d|_ | tur| |_| tur| |_| tur| |_| tur| |_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|S)u *Sets the training related configuration. Args: twin_q: Use two Q-networks (instead of one) for action-value estimation. Note: Each Q-network will have its own target network. q_model_config: Model configs for the Q network(s). These will override MODEL_DEFAULTS. This is treated just as the top-level `model` dict in setting up the Q-network(s) (2 if twin_q=True). That means, you can do for different observation spaces: `obs=Box(1D)` -> `Tuple(Box(1D) + Action)` -> `concat` -> `post_fcnet` obs=Box(3D) -> Tuple(Box(3D) + Action) -> vision-net -> concat w/ action -> post_fcnet obs=Tuple(Box(1D), Box(3D)) -> Tuple(Box(1D), Box(3D), Action) -> vision-net -> concat w/ Box(1D) and action -> post_fcnet You can also have SAC use your custom_model as Q-model(s), by simply specifying the `custom_model` sub-key in below dict (just like you would do in the top-level `model` dict. policy_model_config: Model options for the policy function (see `q_model_config` above for details). The difference to `q_model_config` above is that no action concat'ing is performed before the post_fcnet stack. tau: Update the target by au * policy + (1- au) * target_policy. initial_alpha: Initial value to use for the entropy weight alpha. target_entropy: Target entropy lower bound. If "auto", will be set to `-|A|` (e.g. -2.0 for Discrete(2), -3.0 for Box(shape=(3,))). This is the inverse of reward scale, and will be optimized automatically. n_step: N-step target updates. If >1, sars' tuples in trajectories will be postprocessed to become sa[discounted sum of R][s t+n] tuples. An integer will be interpreted as a fixed n-step value. If a tuple of 2 ints is provided here, the n-step value will be drawn for each sample(!) in the train batch from a uniform distribution over the closed interval defined by `[n_step[0], n_step[1]]`. store_buffer_in_checkpoints: Set this to True, if you want the contents of your buffer(s) to be stored in any saved checkpoints as well. Warnings will be created if: - This is True AND restoring from a checkpoint that contains no buffer data. - This is False AND restoring from a checkpoint that does contain buffer data. replay_buffer_config: Replay buffer config. Examples: { "_enable_replay_buffer_api": True, "type": "MultiAgentReplayBuffer", "capacity": 50000, "replay_batch_size": 32, "replay_sequence_length": 1, } - OR - { "_enable_replay_buffer_api": True, "type": "MultiAgentPrioritizedReplayBuffer", "capacity": 50000, "prioritized_replay_alpha": 0.6, "prioritized_replay_beta": 0.4, "prioritized_replay_eps": 1e-6, "replay_sequence_length": 1, } - Where - prioritized_replay_alpha: Alpha parameter controls the degree of prioritization in the buffer. In other words, when a buffer sample has a higher temporal-difference error, with how much more probability should it drawn to use to update the parametrized Q-network. 0.0 corresponds to uniform probability. Setting much above 1.0 may quickly result as the sampling distribution could become heavily “pointy” with low entropy. prioritized_replay_beta: Beta parameter controls the degree of importance sampling which suppresses the influence of gradient updates from samples that have higher probability of being sampled via alpha parameter and the temporal-difference error. prioritized_replay_eps: Epsilon parameter sets the baseline probability for sampling so that when the temporal-difference error of a sample is zero, there is still a chance of drawing the sample. training_intensity: The intensity with which to update the model (vs collecting samples from the env). If None, uses "natural" values of: `train_batch_size` / (`rollout_fragment_length` x `num_env_runners` x `num_envs_per_env_runner`). If not None, will make sure that the ratio between timesteps inserted into and sampled from th buffer matches the given values. Example: training_intensity=1000.0 train_batch_size=250 rollout_fragment_length=1 num_env_runners=1 (or 0) num_envs_per_env_runner=1 -> natural value = 250 / 1 = 250.0 -> will make sure that replay+train op will be executed 4x asoften as rollout+insert op (4 * 250 = 1000). See: rllib/algorithms/dqn/dqn.py::calculate_rr_weights for further details. clip_actions: Whether to clip actions. If actions are already normalized, this should be set to False. grad_clip: If not None, clip gradients during optimization at this value. optimization_config: Config dict for optimization. Set the supported keys `actor_learning_rate`, `critic_learning_rate`, and `entropy_learning_rate` in here. actor_lr: The learning rate (float) or learning rate schedule for the policy in the format of [[timestep, lr-value], [timestep, lr-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated learning rate values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. Note: It is common practice (two-timescale approach) to use a smaller learning rate for the policy than for the critic to ensure that the critic gives adequate values for improving the policy. Note: If you require a) more than one optimizer (per RLModule), b) optimizer types that are not Adam, c) a learning rate schedule that is not a linearly interpolated, piecewise schedule as described above, or d) specifying c'tor arguments of the optimizer that are not the learning rate (e.g. Adam's epsilon), then you must override your Learner's `configure_optimizer_for_module()` method and handle lr-scheduling yourself. The default value is 3e-5, one decimal less than the respective learning rate of the critic (see `critic_lr`). critic_lr: The learning rate (float) or learning rate schedule for the critic in the format of [[timestep, lr-value], [timestep, lr-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated learning rate values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. Note: It is common practice (two-timescale approach) to use a smaller learning rate for the policy than for the critic to ensure that the critic gives adequate values for improving the policy. Note: If you require a) more than one optimizer (per RLModule), b) optimizer types that are not Adam, c) a learning rate schedule that is not a linearly interpolated, piecewise schedule as described above, or d) specifying c'tor arguments of the optimizer that are not the learning rate (e.g. Adam's epsilon), then you must override your Learner's `configure_optimizer_for_module()` method and handle lr-scheduling yourself. The default value is 3e-4, one decimal higher than the respective learning rate of the actor (policy) (see `actor_lr`). alpha_lr: The learning rate (float) or learning rate schedule for the hyperparameter alpha in the format of [[timestep, lr-value], [timestep, lr-value], ...] In case of a schedule, intermediary timesteps will be assigned to linearly interpolated learning rate values. A schedule config's first entry must start with timestep 0, i.e.: [[0, initial_value], [...]]. Note: If you require a) more than one optimizer (per RLModule), b) optimizer types that are not Adam, c) a learning rate schedule that is not a linearly interpolated, piecewise schedule as described above, or d) specifying c'tor arguments of the optimizer that are not the learning rate (e.g. Adam's epsilon), then you must override your Learner's `configure_optimizer_for_module()` method and handle lr-scheduling yourself. The default value is 3e-4, identical to the critic learning rate (`lr`). target_network_update_freq: Update the target network every `target_network_update_freq` steps. num_steps_sampled_before_learning_starts: Number of timesteps (int) that we collect from the runners before we start sampling the replay buffers for learning. Whether we count this in agent steps or environment steps depends on the value of `config.multi_agent(count_steps_by=...)`. _deterministic_loss: Whether the loss should be calculated deterministically (w/o the stochastic action sampling step). True only useful for continuous actions and for debugging. _use_beta_distribution: Use a Beta-distribution instead of a `SquashedGaussian` for bounded, continuous action spaces (not recommended; for debugging only). Returns: This updated AlgorithmConfig object. rAF)r5trainingr r8r9updater:r<r=r>r?rBrrArCr;rIrDrErFrGrJrQrRrN)rUr8r9r:r<r=r>r?rBrArCr;rIrYrErFrGrJrQrRrNkwargsnew_replay_buffer_configrVs rWr]zSACConfig.trainings@ ""6"""  $ $ DK  , ,   & &~ 6 6 6 k 1 1  $ + +,? @ @ @ k ! !DH  + +!.D   , ,"0D   $ $ DK &k 9 9/JD , { 2 2(3')BC')=>'('( (( $)AAW(XD % [ 0 0&8D # { * * ,D  K ' '&DN k 1 1 3D  ; & &$DM K ' '&DN ; & &$DM %[ 8 8.HD + k 1 1':D $ ! 4 4*@D ' 3; F F8  9 rXc *tt|jtr|jd}n|j}|jsL|jdkrA|j|kr6td|jd|jd|jd|jdd |jtkrtdd t|_|j |j d krtd |j d vrHtAtdt r t jnddt%d|jr|jddvr||jrft|jt,s:t|jt.r2t|jdt,r|jdkr|jstd|js~t|jdt,rd|jdvs@t|jdt2r/t5|jdt6rtd|jr2|jtdtddSdS)Nr+r*z Your `rollout_fragment_length` (z') is smaller than needed for `n_step` (zB)! If `n_step` is an integer try setting `rollout_fragment_length=z@`. If `n_step` is a tuple, try setting `rollout_fragment_length=z`.z config['use_state_preprocessor']F)olderrorgz `grad_clip` value must be > 0.0!)tftf2zYou need `tensorflow_probability` in order to run SAC! Install it via `pip install tensorflow_probability`. Your tf.__version__=z5.Trying to import tfp results in the following error:T)rcr)rr,MultiAgentEpisodeReplayBuffer(MultiAgentPrioritizedEpisodeReplayBufferrsamplerz[When using the new `EnvRunner API` the replay buffer must be of type `EpisodeReplayBuffer`.EpisodeadWhen using the old API stack the replay buffer must not be of type `EpisodeReplayBuffer`! We suggest you use the following config to run SAC on the old API stack: `config.training(replay_buffer_config={'type': 'MultiAgentPrioritizedReplayBuffer', 'prioritized_replay_alpha': [alpha], 'prioritized_replay_beta': [beta], 'prioritized_replay_eps': [eps], })`.zBasic learning rate parameter `lr` is not `None`. For SAC use the specific learning rate parameters `actor_lr`, `critic_lr` and `alpha_lr`, for the actor, critic, and the hyperparameter `alpha`, respectively and set `config.lr` to None.aaYou are running SAC on the new API stack! This is the new default behavior for this algorithm. If you don't want to use the new API stack, set `config.api_stack(enable_rl_module_and_learner=False, enable_env_runner_and_connector_v2=False)`. For a detailed migration guide, see here: https://docs.ray.io/en/master/rllib/new-api-stack-migration-guide.html)r5validate isinstancer?tuple in_evaluationrK ValueErrorrSr r rI frameworktfploggerwarningrd __version__r"enable_env_runner_and_connector_v2rAinput_strlistenable_rl_module_and_learnerr issubclassrrH)rUmin_rollout_fragment_lengthrVs rWrjzSACConfig.validates,  dk5 ) ) 6*.+a. ' '*.+ '" ,66,)**?43O??59[??CG;??-1KN ???   &*: : : 6    +;D ' > %$.C*?*??@@ @ >] * *s{ NNG46"@"..DGGG     & & & &  3/ )&1 t{C00#4;55't{1~s;;$K9,,5-) 8 44Vtjd|jizS)Nr8)r5_model_config_auto_includesr8)rUrVs rWrz%SACConfig._model_config_auto_includes,sww2h 5LLLrXN)rZN)r)__name__ __module__ __qualname____doc__r6rr r rboolrrvrfloatrr@rrr r]rjr}rrrrrpropertyr __classcell__rVs@rWrrs(U;U;U;U;U;U;nXo"-3>8C*)46A8C6A9D.9'2%08C5@6A5@4?.91rs::::::::::::::::""""""IIIIIIIINNNNNNNN,,,,,,>>>>>>+*****;;;;;;******''''''000000CCCCCCCCTTTTTTKKKKKKKK} Rn  8 $ $OMOMOMOMOMOMOMOMd#rX