&`iqdZddlZddlmZmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZdd lmZmZmZdd lmZdd lmZdd lmZddl m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8ddl9m:Z:ddl;mZ>ej?e@ZAGddeZBGdde ZCdS)a [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf [2] Mastering Atari with Discrete World Models - 2021 D. Hafner, T. Lillicrap, M. Norouzi, J. Ba https://arxiv.org/pdf/2010.02193.pdf N)AnyDictOptionalUnion)Self) Algorithm)AlgorithmConfig NotProvided)DreamerV3Catalog) do_symlog_obs)AddIsFirstsToBatch))report_dreamed_eval_trajectory_vs_samplesreport_predicted_vs_sampled_obs!report_sampling_and_replay_buffer)AddStatesFromEpisodesToBatch)DEFAULT_MODULE_ID)Columns) RLModuleSpec)INPUT_ENV_SINGLE_SPACES)synchronous_parallel_sample) SampleBatch) deep_update) PublicAPIoverride) ENV_RUNNER_RESULTSLEARN_ON_BATCH_TIMERLEARNER_RESULTSNUM_ENV_STEPS_SAMPLED_LIFETIMENUM_ENV_STEPS_TRAINED_LIFETIMENUM_GRAD_UPDATES_LIFETIMENUM_SYNCH_WORKER_WEIGHTSREPLAY_BUFFER_RESULTS SAMPLE_TIMERSYNCH_WORKER_WEIGHTS_TIMERTIMERS)one_hot)EpisodeReplayBuffer)LearningRateOrSchedulec.deZdZdZd'fd ZeefdZedZ eee e e e e e e e e e e e e e e e e e e e e dde e de e d e ed e ed e ed e e d e e de e de ede ede e de ede ede ede e de e de e de eee fde ede ede edef,fdZeee e e dde ed e ed!e effd"Zeed(fd# Zeed$Zeedefd%Zeeedee efffd& ZxZS))DreamerV3Configa4Defines a configuration class from which a DreamerV3 can be built. .. testcode:: from ray.rllib.algorithms.dreamerv3 import DreamerV3Config config = ( DreamerV3Config() .environment("CartPole-v1") .training( model_size="XS", training_ratio=1, # TODO model={ "batch_size_B": 1, "batch_length_T": 1, "horizon_H": 1, "gamma": 0.997, "model_size": "XS", }, ) ) config = config.learners(num_learners=0) # Build a Algorithm object from the config and run 1 training iteration. algo = config.build() # algo.train() del algo NcHt|ptd|_d|_dt dd|_d|_d|_d|_ d |_ d |_ d |_ d |_ d |_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_!d|_"d|_#d|_$dS)z'Initializes a DreamerV3Config instance.) algo_classXSir'g.A)typecapacity-C6?giUMu>@gffffff?ga2U0*3?gGz?Tg?g@@gY@autoFNgCl?r)%super__init__ DreamerV3 model_sizetraining_ratiointreplay_buffer_configworld_model_lractor_lr critic_lr batch_size_Bbatch_length_T horizon_H gae_lambda entropy_scalereturn_normalization_decay train_critic train_actorintrinsic_rewards_scale$world_model_grad_clip_by_global_normcritic_grad_clip_by_global_normactor_grad_clip_by_global_norm symlog_obs use_float16 use_curiosity"metrics_num_episodes_for_smoothing"report_individual_batch_item_statsreport_dream_datareport_images_and_videoslrgammatrain_batch_sizenum_env_runnersrollout_fragment_lengthenable_rl_module_and_learner"enable_env_runner_and_connector_v2use_worker_filter_stats)selfr, __class__s |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/algorithms/dreamerv3/dreamerv3.pyr7zDreamerV3Config.__init__XsH J$;)<<< "*C% % !#  !*.' '*$4:1/4,.3+  " 34/27/!&(-% $ '($,0)26/ (-$$$ct|||}|jr'|tt |SN)r6build_env_to_module_connector*add_default_connectors_to_learner_pipeline insert_beforerr )r[envspacesdevice connectorr\s r]raz-DreamerV3Config.build_env_to_module_connectorsWGG99#vvNN  :   # #,"$$   r^c$|j|jpdzS)z`Returns the batch_size_B per Learner worker. Needed by some of the DreamerV3 loss math.r5)r@ num_learners)r[s r]batch_size_B_per_learnerz(DreamerV3Config.batch_size_B_per_learners  T%6%;!<r?rIrJrKrLrMr<rNr9r:r@rArBrCrDrErFrGrHr=r>r?rIrJrKrLrMr<rNreturnc 2|turtdtjdi||tur||_|tur||_|tur||_|tur||_|tur||_|tur||_ |tur||_ |tur||_ | tur| |_ | tur| |_ | tur| |_| tur| |_| tur| |_|tur||_|tur||_|tur||_|tur||_|tur||_|tur||_|tur+t/d|jid|iddgdg}|d|_|S)aSets the training related configuration. Args: model_size: The main switch for adjusting the overall model size. See [1] (table B) for more information on the effects of this setting on the model architecture. Supported values are "XS", "S", "M", "L", "XL" (as per the paper), as well as, "nano", "micro", "mini", and "XXS" (for RLlib's implementation). See ray.rllib.algorithms.dreamerv3.utils. __init__.py for the details on what exactly each size does to the layer sizes, number of layers, etc.. training_ratio: The ratio of total steps trained (sum of the sizes of all batches ever sampled from the replay buffer) over the total env steps taken (in the actual environment, not the dreamed one). For example, if the training_ratio is 1024 and the batch size is 1024, we would take 1 env step for every training update: 1024 / 1. If the training ratio is 512 and the batch size is 1024, we would take 2 env steps and then perform a single training update (on a 1024 batch): 1024 / 2. batch_size_B: The batch size (B) interpreted as number of rows (each of length `batch_length_T`) to sample from the replay buffer in each iteration. batch_length_T: The batch length (T) interpreted as the length of each row sampled from the replay buffer in each iteration. Note that `batch_size_B` rows will be sampled in each iteration. Rows normally contain consecutive data (consecutive timesteps from the same episode), but there might be episode boundaries in a row as well. horizon_H: The horizon (in timesteps) used to create dreamed data from the world model, which in turn is used to train/update both actor- and critic networks. gae_lambda: The lambda parameter used for computing the GAE-style value targets for the actor- and critic losses. entropy_scale: The factor with which to multiply the entropy loss term inside the actor loss. return_normalization_decay: The decay value to use when computing the running EMA values for return normalization (used in the actor loss). train_critic: Whether to train the critic network. If False, `train_actor` must also be False (cannot train actor w/o training the critic). train_actor: Whether to train the actor network. If True, `train_critic` must also be True (cannot train actor w/o training the critic). intrinsic_rewards_scale: The factor to multiply intrinsic rewards with before adding them to the extrinsic (environment) rewards. world_model_lr: The learning rate or schedule for the world model optimizer. actor_lr: The learning rate or schedule for the actor optimizer. critic_lr: The learning rate or schedule for the critic optimizer. world_model_grad_clip_by_global_norm: World model grad clipping value (by global norm). critic_grad_clip_by_global_norm: Critic grad clipping value (by global norm). actor_grad_clip_by_global_norm: Actor grad clipping value (by global norm). symlog_obs: Whether to symlog observations or not. If set to "auto" (default), will check for the environment's observation space and then only symlog if not an image space. use_float16: Whether to train with mixed float16 precision. In this mode, model parameters are stored as float32, but all computations are performed in float16 space (except for losses and distribution params and outputs). replay_buffer_config: Replay buffer config. Only serves in DreamerV3 to set the capacity of the replay buffer. Note though that in the paper ([1]) a size of 1M is used for all benchmarks and there doesn't seem to be a good reason to change this parameter. Examples: { "type": "EpisodeReplayBuffer", "capacity": 100000, } Returns: This updated AlgorithmConfig object. zd`DreamerV3Config.curiosity` is not fully supported and tested yet! It thus remains disabled for now.r<F)r ValueErrorr6trainingr9r:r@rArBrCrDrErFrGrHr=r>r?rIrJrKrLrMrr<)r[r9r:r@rArBrCrDrErFrGrHr=r>r?rIrJrKrLrMr<rNkwargsnew_replay_buffer_configr\s r]rozDreamerV3Config.trainingsD  + +4  ""6""" [ ( ((DO  , ,"0D  { * * ,D   , ,"0D  K ' '&DN [ ( ((DO  + +!.D  %[ 8 8.HD + { * * ,D  k ) )*D  "+ 5 5+BD (  , ,"0D  ; & &$DM K ' '&DN /{ B B4  5 ++ = =3RD 0 ) < <2PD / [ ( ((DO k ) )*D  { 2 2(3')BC')=>'('( (( $)AAW(XD % r^)rPrQrRrPrQrRc tjdi||tur||_|tur||_|tur||_|S)aSets the reporting related configuration. Args: report_individual_batch_item_stats: Whether to include loss and other stats per individual timestep inside the training batch in the result dict returned by `training_step()`. If True, besides the `CRITIC_L_total`, the individual critic loss values per batch row and time axis step in the train batch (CRITIC_L_total_B_T) will also be part of the results. report_dream_data: Whether to include the dreamed trajectory data in the result dict returned by `training_step()`. If True, however, will slice each reported item in the dream data down to the shape. (H, B, t=0, ...), where H is the horizon and B is the batch size. The original time axis will only be represented by the first timestep to not make this data too large to handle. report_images_and_videos: Whether to include any image/video data in the result dict returned by `training_step()`. **kwargs: Returns: This updated AlgorithmConfig object. rm)r6 reportingr rPrQrR)r[rPrQrRrpr\s r]rszDreamerV3Config.reportingNs`> ##F### -[ @ @6XD 3 K / /%6D " #; 6 6,DD ) r^c(t|jr|d|js|d|jdkr9|j|jzdkr&|d|jd|jd|jr|js|d|j |d |j d d kr|d dSdS) Nz2DreamerV3 does NOT support multi-agent setups yet!zQDreamerV3 must be run with `config.api_stack(enable_rl_module_and_learner=True)`!r5rzYour `batch_size_B` (z() must be a multiple of `num_learners` (zZ) in order for DreamerV3 to be able to split batches evenly across your Learner processes.zCannot train actor network (`train_actor=True`) w/o training critic! Make sure you either set `train_critic=True` or `train_actor=False`.zV`train_batch_size` should NOT be set! Use `batch_size_B` and `batch_length_T` instead.r.r'zPDreamerV3 must be run with the `EpisodeReplayBuffer` type! None other supported.) r6validateis_multi_agent _value_errorrXrir@rGrFrUr<getr[r\s r]ruzDreamerV3Config.validatexsz    T   R S S S0    7     q d&7$:K&Kq&P&P   (9#'#4      D$5    W     ,   ,     $ ( ( 0 04I I I   #      J Ir^cX|jdkrddlm}|Std|jd)Ntorchr)DreamerV3TorchLearnerThe framework  is not supported.) framework_strray.rllib.algorithms.dreamerv3.torch.dreamerv3_torch_rl_modulerrnrr )r[modules r]get_default_rl_module_specz*DreamerV3Config.get_default_rl_module_specsf   ( (        Td.@TTTUU U?OPPPPr^cztj|j|j|j|j|j|jdzS)N)rTrBr9rLrMrA)r6_model_config_auto_includesrTrBr9rLrMrArys r]rz+DreamerV3Config._model_config_auto_includessCww2Z//+"1 6 6   r^r`rkN)__name__ __module__ __qualname____doc__r7rr rapropertyrjr rstrfloatr;boolr(rdictrrorsrurrrrrr __classcell__r\s@r]r*r*:s:;-;-;-;-;-;-~Xo     ==X= Xo%0*5&1(3#.&1)46A'2&13>;F5@6A@K;F:E1<&1/:(3/^^^SM^! ^ sm ^ ! ^C=^UO^ ^%-UO^tn^d^^"*%^!!78^12^ 23!^"/7uo#^$*2%%^&)1'^(U49-.)^*d^+^,'tn-^. ~/^2 3^^^^^^@Xo>I,73> '''-5TN'$D> ' #+4. ''''''RXo******XXoVVVXo QL Q Q Q Q Xo T#s(^     X     r^r*ceZdZdZeedZeeedefdZ eede ffd Z eed dZ e defd Zed fd ZxZS) r8zJImplementation of the model-based DreamerV3 RL algorithm described in [1].c td)NzDreamerV3 does not support the `compute_single_action()` API. Refer to the README here (https://github.com/ray-project/ray/tree/master/rllib/algorithms/dreamerv3) to find more information on how to run action inference with this algorithm.)NotImplementedError)r[argsrps r]compute_single_actionzDreamerV3.compute_single_actions! -   r^rkctSr`)r*)clss r]get_default_configzDreamerV3.get_default_configs   r^configct|t|jjd|jj|jj|_dS)Nr/)r/r@rA)r6setupr'rr<r@rA replay_buffer)r[rr\s r]rzDreamerV3.setupsU  f1[5jA1;5   r^Nc  |jdkr5td|jj|jjzdd}|jttf5|j |jj|jjzks|j |jj ks|st|j|jj|jjz|jjdd\}}|j|t&|j |d}t+d |D}|}|jt&t.fd dkrt|j|jj|jjz|z |jjddd \}}|j|t&|j ||t+d |Dz }|j |jj|jjzk|j |jj k|dddn #1swxYwYt1|j|j |j}|j|gt6|jt:\} } dx} } | |z |jj kr|jtt<f5td|jd| d|j |jj|jj} |jj|jjz}| |z } tA| tBjj"r>| tFj$| d<tK| d| j&| tFj$<|j'(tS| *t.|jt&t.fd i}|j|tV| dz } |j,tZdddddn #1swxYwY| |z |jj kt]|j| |jj|jjt_| |jj0|jj1o |jdzdkte|j| d|jj3dz|j4j5j6t_| |jj0|jj7o |jdzdk|jj8|jttrf5|j,ttdd|j;|j'ddddn #1swxYwY|j,d|j ddS)Nrz.Filling replay buffer so it contains at least z/ timesteps (required for a single train batch).FT) worker_setmax_agent_stepssample_timeout_s_uses_new_env_runners_return_metrics)key)episodesc34K|]}t|VdSr`len.0epss r] z*DreamerV3.training_step..s(3Q3QCHH3Q3Q3Q3Q3Q3Qr^default)rrrrandom_actionsrrc34K|]}t|VdSr`rrs r]rz*DreamerV3.training_step..5s((G(GcS(G(G(G(G(G(Gr^)metricsrz Sub-iteration /))r@rA actions_ints)depth)batch timestepsr5 lifetime_sum)reduced)rsampler@rArL do_report)rr burn_in_T dreamed_T dreamer_modelrLr frameworksum)from_worker_or_learner_groupinference_onlyactual_training_ratio)window)<training_iterationloggerinforr@rArlog_timer%r#rget_num_timestepsr:renv_runner_grouprWnum_envs_per_env_runnerr aggregateraddrpeekrrlocal_replay_buffer get_metricsr"rerrr isinstancegymDiscreterACTIONSr&n learner_groupupdateras_multi_agentr log_valuer rr rLrRrrB env_runnerrrrQrr$r! sync_weights)r[ have_sampledrenv_runner_resultsenv_steps_last_regular_sample total_sampled _episodes_env_runner_resultsreplay_buffer_resultssingle_observation_spacesingle_action_spacereplayed_steps_this_itersub_iterrreplayed_stepslearner_resultss r] training_stepzDreamerV3.training_stepsu  "a ' ' KK7;+dk.HH777    \ " "FL#9 : :; H; H "4466;+dk.HHJJ&$+*DDD#E 0K#4 ;+=>&*[%A*.$( 0 0 0,, &&'9?Q&RRR "&&&999# 143Q3Q3Q3Q3Q0Q0Q- = L%%+-KL !&  6Q#'#8 K4t{7QQ;<*.)E'+.2(, 6 6 62I2L**+>DV*WWW&**I*>>>!S(G(GY(G(G(G%G%GGMk"4466;+dk.HHJJ&$+*DDD#E; H; H; H; H; H; H; H; H; H; H; H; H; H; H; H| *L0B    !% 8 D D F F  56*.5~.13///F7?+ #'"4";";%f--<<>> 7 8I8I/1OP$%9J99 #< # # &&O&LLLA  &&-q'G% % % % % % % % % % % % % % %  %'D D K &''^ (L1;5$( &  47+c1Q6    * 2Lk+a//0>$( &  -T$2IC2OST2Tk/    $\ " "F,F#G H H   L " "#;Qu " M M M  ! . .-1-?# /                    68KTUVVVVVs9)G6I++I/2I/FR--R14R1AXXXcd}|jtd|jttf|p|z S)aReturns the actual training ratio of this Algorithm (not the configured one). The training ratio is copmuted by dividing the total number of steps trained thus far (replayed from the buffer) over the total number of actual env steps taken thus far. r0rr)rrrrr)r[rs r]r:zDreamerV3.training_ratios\|  !? KK !!')GH"   r^c@t||jjrnt |jjt |jjjtksJ|jjjt|j_dSdS)aSts the algorithm to the provided state Args: state: The state dictionary to restore this `DreamerV3` instance to. `state` may have been returned by a call to an `Algorithm`'s `__getstate__()` method. )stateN) r6 __setstate__r+share_module_between_env_runner_and_learneridrrr_learnerr)r[rr\s r]rzDreamerV3.__setstate__s 5))) ; B do,--"+23DE22&*%7%@%G!&DO " " "   r^r)rrrrrrr classmethodr*rr rrrrr:rrrrs@r]r8r8sHTTXi    Xi!?!!![!Xi O      $XiAWAWAWAWF    X &Yr^r8)Drloggingtypingrrrr gymnasiumrtyping_extensionsrray.rllib.algorithms.algorithmr%ray.rllib.algorithms.algorithm_configr r 0ray.rllib.algorithms.dreamerv3.dreamerv3_catalogr $ray.rllib.algorithms.dreamerv3.utilsr ;ray.rllib.algorithms.dreamerv3.utils.add_is_firsts_to_batchr .ray.rllib.algorithms.dreamerv3.utils.summariesrrrray.rllib.connectors.commonrray.rllib.corerray.rllib.core.columnsr"ray.rllib.core.rl_module.rl_moduler ray.rllib.envrray.rllib.execution.rollout_opsrray.rllib.policy.sample_batchrray.rllib.utilsrray.rllib.utils.annotationsrrray.rllib.utils.metricsrrrrrr r!r"r#r$r%ray.rllib.utils.numpyr&4ray.rllib.utils.replay_buffers.episode_replay_bufferr'ray.rllib.utils.typingr( getLoggerrrr*r8rmr^r]rs------------""""""444444NNNNNNNNMMMMMM>>>>>> EDDDDD,,,,,,******;;;;;;111111GGGGGG555555'''''';;;;;;;;                          *)))))TTTTTT999999  8 $ $L L L L L oL L L ^ SSSSS SSSSSr^