&`i9dZddlZddlmZmZddlmZddlm Z ddl m Z ddl m Z mZddlmZe \ZZ dd Zd dd d d dZd d ddedefdZd d d ddededdfdZdZdZdZdZdS)a [1] Mastering Diverse Domains through World Models - 2023 D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap https://arxiv.org/pdf/2301.04104v1.pdf [2] Mastering Atari with Discrete World Models - 2021 D. Hafner, T. Lillicrap, M. Norouzi, J. Ba https://arxiv.org/pdf/2010.02193.pdf N)create_cartpole_dream_imagecreate_frozenlake_dream_image)DEFAULT_MODULE_ID)Columns)try_import_torch)LEARNER_RESULTSREPLAY_BUFFER_RESULTS)inverse_symlogtorchc v|j}|d}|d}|dkr%tt|jjj}|jt| ||zdf |t| ||zf|jddz | } n[|j| ||zdf| ||zf|jddz} tj | ||f|z} | S)Returnsrr N)hz)shapenextiter world_modeldecoder parametersdevicer from_numpyreshapetodetachcpunumpynp) h_t0_to_H z_t0_to_H dreamer_modelobs_dims_shape frameworkrTBr!reconstructed_obs_distr_means_TxBreconstructed_obs_T_Bs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/algorithms/dreamerv3/utils/summaries.pyreconstruct_obs_from_h_and_zr+s OE aA aA Gd=4<GGIIJJKKR  % - -""9--55q1ubkBBEEfMM""9--!a%IOABB$7788F .  VXX SUU UWW *)-:,E,M,MQ ,,Q9?122+> >??-N- - )J)Aq6N+B ! )rT) batch_indicesdescinclude_imagesr%c|sdS|d}t|d|d|||} |drtnt} |D].} g} t t | dz D]} | | | | | |d| | |d| | |d | dz| d |vr|d | | nd|d | dz| |d | | |d| | d  |d|rd|zndd| tj | di0dS)N dream_datah_states_t0_to_H_BxTz_states_prior_t0_to_H_BxTr!r"r#r$r%CartPolervalues_dreamed_t0_to_H_BxT actions_ints_dreamed_t0_to_H_BxTrewards_dreamed_t0_to_H_BxT DISAGREE_intrinsic_rewards_H_BxTcontinues_dreamed_t0_to_H_BxTVALUE_TARGETS_H_BxTT) dreamed_obs dreamed_V dreamed_a dreamed_r_tp1dreamed_ri_tp1 dreamed_c_tp1 value_target initial_h as_tensordreamed_trajectories__Baxis) r+ startswithrrrangelenappendrupdater concatenate)resultsenvr#r$r-r.r/r%r1dreamed_obs_H_Bfuncbimagests r*report_dreamed_trajectoryrXHs &J2349:#% O >>* % % +## *    s?++a/00  A MM / 21 5()EFqI!L)*LMaPQRS#-.K#LQQRU#STU#V >HH BCAFqII!##BCAEJ1M!()>!?!B1!E()?@CAF"#$%'''    . HT'ADrHHQHHN6222     5   r,) symlog_obs do_reportrYrZc ttdf}||dgd}||dd|d}|s||ddSt |t j|d |f|tjj d dz|tjd d || dS) aSummarizes sampled data (from the replay buffer) vs world-model predictions. World model predictions are based on the posterior states (z computed from actual observation encoder input + the current h-states). Observations: Computes MSE (sampled vs predicted/recreated) over all features. For image observations, also creates direct image comparisons (sampled images vs predicted (posterior) ones). Rewards: Compute MSE (sampled vs predicted). Continues: Compute MSE (sampled vs predicted). Args: metrics: The MetricsLogger object of the DreamerV3 algo. sample: The sampled data (dict) from the replay buffer. Already torch-tensor converted. batch_size_B: The batch size (B). This is the number of trajectories sampled from the buffer. batch_length_T: The batch length (T). This is the length of an individual trajectory sampled from the buffer. do_report: Whether to actually log the report (default). If this is set to False, this function serves as a clean-up on the given metrics, making sure they do NOT contain anymore any (spacious) data relevant for producing the report/videos. /WORLD_MODEL_fwd_out_obs_distribution_means_b0xTNdefaultrF key_error.WORLD_MODEL_sampled_vs_predicted_posterior_b0x_videosrrrmetricscomputed_float_obs_B_T_dimssampled_obs_B_T_dims metrics_keyrY) rrpeekdelete _report_obsr rrOBSr) rdsample batch_size_Bbatch_length_TrYrZfwd_output_key*predicted_observation_means_single_examplefinal_result_keys r*report_predicted_vs_sampled_obsrrsD 9N 292>222 . NN>UN333 QPPP '5999$&J 6  &"5";ABB"? ?% % $GK015$      r,)rYrZr%returnc`|ttdfi}|ttddd|d} d|d} d|d} |sG|| d|| d|| dd St |d d |d d ||t jjd d |} |} | |z}t|tj | d dd d|t jd d| |f| |t||dd |t j d d | |f| t||dd d|dz d d | |f| d S)aLogs dreamed observations, rewards, continues and compares them vs sampled data. For obs, we'll try to create videos (side-by-side comparison) of the dreamed, recreated-from-prior obs vs the sampled ones (over dreamed_T timesteps). Args: metrics: The MetricsLogger object of the DreamerV3 algo. sample: The sampled data (dict) from the replay buffer. Already torch-tensor converted. burn_in_T: The number of burn-in timesteps (these will be skipped over in the reported video comparisons and MSEs). dreamed_T: The number of timesteps to produce dreamed data for. dreamer_model: The DreamerModel to use to create observation vectors/images from dreamed h- and (prior) z-states. symlog_obs: Whether to inverse-symlog the computed observations or not. Set this to True for environments, in which we should symlog the observations. do_report: Whether to actually log the report (default). If this is set to False, this function serves as a clean-up on the given metrics, making sure they do NOT contain anymore any (spacious) data relevant for producing the report/videos. r1r]Fr_%EVALUATION_sampled_vs_dreamed_prior_H_obs _rewards_MSE_continues_MSENh_states_t0_to_H_Bx1rz_states_prior_t0_to_H_Bx1rr4rrcrewards_dreamed_t0_to_H_Bx1)rdcomputed_rewardssampled_rewardsrgcontinues_dreamed_t0_to_H_Bx1? is_terminated)rdcomputed_continuessampled_continuesrg)rhrrrir+rrkrrjr swapaxes_report_rewardsREWARDS_report_continues)rdrl burn_in_T dreamed_Tr#rYrZr%r1final_result_key_obsfinal_result_key_rewfinal_result_key_contrSt0tHs r*)report_dreamed_eval_trajectory_vs_samplesrs+@ +\:J NN?$5|uNUUUR9RRRG GGG J III +u===+u===,>>>334Q79:1=#gk*04 O B iB$&KA$F$F aC% $GK01be<(    #$AB1Ew/2b59( %&EFqI!88!!!RU(C) r,c|}|}|}|}||j||||dt ddS)N)capacitysize_num_episodessize_timestepsreplayed_steps added_stepsr)keywindow)get_num_episodesget_num_timestepsget_sampled_timestepsget_added_timestepslog_dictrr )rd replay_bufferepisodes_in_buffer ts_in_bufferrrs r*!report_sampling_and_replay_bufferr!s&7799 2244L"88::N3355K %.!3*,&    "     r,ct|jdvr/t|jdkrdnd}|rt|}|sX|dzdz}|dzdz}tj|ddtj}tj||}tj|ddtj}tj||}tj||gd }t|jd krtj |d }| ||d d dSdS)aSummarizes computed- vs sampled observations: MSE and (if applicable) images. Args: metrics: The MetricsLogger object of the DreamerV3 algo. computed_float_obs_B_T_dims: Computed float observations (not clipped, not cast'd). Shape=(B, T, [dims ...]). sampled_obs_B_T_dims: Sampled observations (as-is from the environment, meaning this could be uint8, 0-255 clipped images). Shape=(B, T, [dims ...]). metrics_key: The metrics key (or key sequence) under which to log ths resulting video sequence. symlog_obs: Whether to inverse-symlog the computed observations or not. Set this to True for environments, in which we should symlog the observations. )r)rrrr)rrrrrggo@rrIr item_seriesr)reducerN) rMrr r clipastypeuint8 transposerP expand_dims log_value)rdrerfrgrYtranspose_axescomputed_imagessampled_vs_computed_imagess r*rjrj5s.  %&&&00 ##7#=>>!CCOO   V*89T*U*U ' V+F+LPS*S '$83$>##E #%7+?e#L#L#S#S$$ $&<0Dn#U#U '"=sEJJQQ H  ,GG&(^ 2 3& & & " #) * *e 3 3)+8RTV)W)W &  &       A10r,ctjtj||z }tj|}|||ddSNr)r)r meansquarer)rdr|r}rgmse_sampled_vs_computed_rewardss r*rrtsl')g "_455''#')g.M&N&N# 'r,ctjtj|||jz }|||ddSr)r rrrdtyper)rdrrrg!mse_sampled_vs_computed_continuess r*rrsr)+ !2!9!9:L:R!S!S S  ))%  )r,)r )__doc__rr .ray.rllib.algorithms.dreamerv3.utils.debuggingrrray.rllib.corerray.rllib.core.columnsrray.rllib.utils.frameworkrray.rllib.utils.metricsrr ray.rllib.utils.torch_utilsr r rFr+rXboolrrrrrjrrr,r*rs-,,,,,******666666766666    q )!)!)!)!d < < < < < J===  =====NWWWWW WWWWt(< < < ~$r,