&`i%dZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZdd lmZed d d ZedededdedddedkrjddlmZeZdZdZdZejdkrejddnejdeeej !"ddd d!d"d#$#ej$rdned%&%ej$rdned'd(ej&pdzd)d*+'ed#gd,d-d.g/0Z(ej d1kre(%d2d3d4d5d6d78ejdkr0e()d9e*ejDd:;ee(edSdS)Example using 2 ConnectorV2 for observation frame-stacking in Atari environments. An RLlib Algorithm has 3 distinct connector pipelines: - An env-to-module pipeline in an EnvRunner accepting a list of episodes and producing a batch for an RLModule to compute actions (`forward_inference()` or `forward_exploration()`). - A module-to-env pipeline in an EnvRunner taking the RLModule's output and converting it into an action readable by the environment. - A learner connector pipeline on a Learner taking a list of episodes and producing a batch for an RLModule to perform the training forward pass (`forward_train()`). Each of these pipelines has a fixed set of default ConnectorV2 pieces that RLlib adds/prepends to these pipelines in order to perform the most basic functionalities. For example, RLlib adds the `AddObservationsFromEpisodesToBatch` ConnectorV2 into any env-to-module pipeline to make sure the batch for computing actions contains - at the minimum - the most recent observation. On top of these default ConnectorV2 pieces, users can define their own ConnectorV2 pieces (or use the ones available already in RLlib) and add them to one of the 3 different pipelines described above, as required. This example: - shows how the `FrameStackingEnvToModule` ConnectorV2 piece can be added to the env-to-module pipeline. - shows how the `FrameStackingLearner` ConnectorV2 piece can be added to the learner connector pipeline. - demonstrates that using these two pieces (rather than performing framestacking already inside the environment using a gymnasium wrapper) increases overall performance by about 5%. How to run this script ---------------------- `python [script file name].py --num-frames=4 --env=ALE/Pong-v5` Use the `--num-frames` option to define the number of observations to framestack. If you don't want to use Connectors to perform the framestacking, set the `--use-gym-wrapper-framestacking` flag to perform framestacking already inside a gymnasium observation wrapper. In this case though, be aware that the tensors being sent through the network are `--num-frames` x larger than if you use the Connector setup. For debugging, use the following additional command line options `--no-tune --num-env-runners=0` which should allow you to set breakpoints anywhere in the RLlib code and have the execution stop there for inspection and debugging. For logging to your WandB account, use: `--wandb-key=[your WandB API key] --wandb-project=[some project name] --wandb-run-name=[optional: WandB run name (within the defined project)]` Results to expect ----------------- With `--num-frames=4` and using the two extra ConnectorV2 pieces (in the env-to-module and learner connector pipelines), you should see something like this using: `--env ALE/Pong-v5 --num-learners=4 --num-gpus-per-learner=1 --num-env-runners=95` +---------------------------+------------+--------+------------------+... | Trial name | status | iter | total time (s) | | | | | | |---------------------------+------------+--------+------------------+... | PPO_atari-env_2fc4a_00000 | TERMINATED | 200 | 335.837 | +---------------------------+------------+--------+------------------+... Note that the time to run these 200 iterations is about ~5% faster than when performing framestacking already inside the environment (using a `gymnasium.wrappers.ObservationWrapper`), due to the additional network traffic needed (sending back 4x[obs] batches instead of 1x[obs] to the learners). Thus, with the `--use-gym-wrapper-framestacking` option (all other options being equal), the output looks like this: +---------------------------+------------+--------+------------------+... | Trial name | status | iter | total time (s) | | | | | | |---------------------------+------------+--------+------------------+... | PPO_atari-env_2fc4a_00000 | TERMINATED | 200 | 351.505 | +---------------------------+------------+--------+------------------+... N)FrameStackingEnvToModule)FrameStackingLearner)DefaultModelConfig)wrap_atari_for_new_api_stack)make_multi_agent)add_rllib_example_script_args#run_rllib_example_script_experiment)get_trainable_clsi@KLg4@)default_timestepsdefault_reward default_iterszale_py:ALE/Pong-v5)envz --num-framesz*The number of observation frames to stack.)typedefaulthelpz--use-gym-wrapper-framestacking store_truezWhether to use RLlib's Atari wrapper's framestacking capabilities (as opposed to doing it via a specific ConenctorV2 pipeline).)actionr__main__)tunecTttjtjdkSNr) num_frames multi_agent)rargsr num_agents)rspacesdevices /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/examples/connectors/frame_stacking.py_make_env_to_module_connectorr!ys*(!+    cTttjtjdkSr)rrrr)input_observation_spaceinput_action_spaces r _make_learner_connectorr&s(#!+    r"cttjtjfi|dditjr tjndS)N render_mode rgb_array) framestack)rgymmakerruse_gym_wrapper_framestackingrcfgs r _env_creatorr0sM+ HTX E E E (D E E$(#EO4     r"z atari-envcjttt|fidtjiS)Nr)rr0dictrrr.s r r3s66(66S<<\4?;<<r"Fg) frameskipfull_action_spacerepeat_action_probabilityT) env_config clip_rewards)env_to_module_connectornum_envs_per_env_runnerg{Gz?ga2U0*#?gY@ global_norm)learner_connector entropy_coefflr grad_clip grad_clip_by))rr:) rr:)@rr:)rr:relu)vf_share_layers conv_filtersconv_activationhead_fcnet_hiddens) model_configPPO rEgffffff?g?g?g$@) num_epochsminibatch_sizelambda_kl_coeff clip_param vf_clip_paramch|]}d|S)p).0is r r[s>>>!g!gg>>>r"c d|S)NrWrX)aidakws r r3r3s IIIr")policiespolicy_mapping_fn)+__doc__ gymnasiumr+1ray.rllib.connectors.env_to_module.frame_stackingr+ray.rllib.connectors.learner.frame_stackingr-ray.rllib.core.rl_module.default_model_configr%ray.rllib.env.wrappers.atari_wrappersr+ray.rllib.examples.envs.classes.multi_agentrray.rllib.examples.utilsrr ray.tune.registryr parser set_defaults add_argumentint__name__rayr parse_argsrr!r&r0r register_envalgoget_default_config environment env_runnersr-training num_learners rl_module base_configrrangerXr"r r|sNN^VVVVVVLLLLLLLLLLLLNNNNNNHHHHHH0///// ' &d#    5  %  @ z     D                   +|444 $)$$      %*-0     532$%    :W@W$+0q1&   ++ $NNN &$'5    CX yE     >>uuT_'='=>>>==    (' T:::::ar"