&`ipddlZddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZmZddlmZmZmZmZddlmZddlmZddlmZdd l m!Z!erdd l"m#Z#ej$e%Z& d"de ej'de(de)de)dej'f dZ* d#de e+efdede,de,fdZ- d$dddddddddddde ej.de e de e de e de e d e,de,de eej/j0ffd!Z1dS)%N) TYPE_CHECKINGAnyDictListOptionalTypeUnion)tune) WANDB_ENV_VARWandbLoggerCallback)ENV_RUNNER_RESULTSEPISODE_RETURN_MEANEVALUATION_RESULTSNUM_ENV_STEPS_SAMPLED_LIFETIME)"convert_numpy_to_python_primitives) ResultDict) CLIReporter)TRAINING_ITERATION)AlgorithmConfigY@順parserdefault_reward default_itersdefault_timestepsreturnc:|tj}|dtdd|dgddd |d tdd |d tdd|dtdd|dtdd|dtdd|dtdd|dddd|dtdddgd |d!d"d#$|d%tdd&|d'tdgd(d) |d*d"d+$|d,td-d.|d/tdd0|d1td2d3|d4tdd5|d6d"d7$|d8tdd9|d:tdd;|dt |d?|d@t|dA|dBt|dC|dDd"dE$|dFd"dG$|dHtddI|dJt ddK|dLt ddM|dNtddO|dPtdQ|dRd"dS$|dTtddU|dVd"dW$|dXd"Y|S)ZacAdds RLlib-typical (and common) examples scripts command line args to a parser. TODO (sven): This function should be used by most of our examples scripts, which already mostly have this logic in them (but written out). Args: parser: The parser to add the arguments to. If None, create a new one. default_reward: The default value for the --stop-reward option. default_iters: The default value for the --stop-iters option. default_timesteps: The default value for the --stop-timesteps option. Returns: The altered (or newly created) parser object. Nz--algoPPOz&The RLlib-registered algorithm to use.)typedefaulthelpz --framework)tftf2torchr%zThe DL framework specifier.)choicesr!r"z--envz2The gym.Env identifier to run the experiment with.z--num-env-runnersz 0, will run as multi-agent with the environment simply cloned n times and each agent acting independently at every single timestep. The overall reward for this experiment is then the sum over all individual agents' rewards.z--evaluation-num-env-runnerszGThe number of evaluation (remote) EnvRunners to use for the experiment.z--evaluation-intervalz`Every how many iterations to run one round of evaluation. Use 0 (default) to disable evaluation.z--evaluation-durationc0|dkr|nt|S)Nauto)int)vs l/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/examples/utils.pyz/add_rllib_example_script_args..usAKKqqSVV zThe number of evaluation units to run each evaluation round. Use `--evaluation-duration-unit` to count either in 'episodes' or 'timesteps'. If 'auto', will run as many as possible during train pass (`--evaluation-parallel-to-training` must be set then).z--evaluation-duration-unitepisodes timestepszThe evaluation duration unit to count by. One of 'episodes' or 'timesteps'. This unit will be run `--evaluation-duration` times in each evaluation round. If `--evaluation-duration=auto`, this setting does not matter.)r r!r&r"z!--evaluation-parallel-to-training store_truezWhether to run evaluation parallel to training. This might help speed up your overall iteration time. Be aware that when using this option, your reported evaluation results are referring to one iteration before the current one.)actionr"z--outputz\The output directory to write trajectories to, which are collected by the algo's EnvRunners.z --log-level)INFODEBUGWARNERRORz-The log-level to be used by the RLlib logger.z --no-tunezWhether to NOT use tune.Tuner(), but rather a simple for-loop calling `algo.train()` repeatedly until one of the stop criteria is met.z --num-sampleszMHow many (tune.Tuner.fit()) experiments to execute - if possible in parallel.z--max-concurrent-trialsz1How many (tune.Tuner) trials to run concurrently.z --verbosezBThe verbosity level for the `tune.Tuner()` running the experiment.z--checkpoint-freqzThe frequency (in training iterations) with which to create checkpoints. Note that if --wandb-key is provided, all checkpoints will automatically be uploaded to WandB.z--checkpoint-at-endzWhether to create a checkpoint at the very end of the experiment. Note that if --wandb-key is provided, all checkpoints will automatically be uploaded to WandB.z --wandb-keyz/The WandB API key to use for uploading results.z--wandb-projectzThe WandB project name to use.z--wandb-run-namezThe WandB run name to use.z --stop-rewardz0Reward at which the script should stop training.z --stop-itersz"The number of iterations to train.z--stop-timestepsz8The number of (environment sampling) timesteps to train.z --as-testzWhether this script should be run as a test. If set, --stop-reward must be achieved within --stop-timesteps AND --stop-iters, otherwise this script will throw an exception at the end.z--as-release-testzWhether this script should be run as a release test. If set, all that applies to the --as-test option is true, plus, a short JSON summary will be written into a results file whose location is given by the ENV variable `TEST_OUTPUT_JSON`.z--num-learnerszLThe number of Learners to use. If `None`, use the algorithm's default value.z--num-cpus-per-learnerzTThe number of CPUs per Learner to use. If `None`, use the algorithm's default value.z--num-gpus-per-learnerzThe number of GPUs per Learner to use. If `None` and there are enough GPUs for all required Learners (--num-learners), use a value of 1, otherwise 0.z#--num-aggregator-actors-per-learnerzaThe number of Aggregator actors to use per Learner. If `None`, use the algorithm's default value.z --num-cpus)r r!z --local-modez,Init Ray in local mode for easier debugging.z --num-gpusz6The number of GPUs to use (only on the old API stack).z--old-api-stackz.Run this script on the old API stack of RLlib.z--enable-new-api-stack)r2)argparseArgumentParser add_argumentstrr)float)rrrrs r+add_rllib_example_script_argsr>'s=(~(** sE0X &&& *     A     K   # S   J  &  V    1   3 3A  $ [)     +  !   222 <  K     !  @     Q     2      2    >     -     )    ?     1    ! G   5 '          - %   3:::  ;   E    =   Mr-Fstopresults keep_ray_upc|D]\}}|}|dD]:}|}t|tr ||vr ||}8d}|Z t j|s4||kr.td|d|d|stj dS#t$rYwxYwdS)aChecks stopping criteria on `ResultDict` Args: stop: Dictionary of stopping criteria. Each criterium is a mapping of a metric in the `ResultDict` of the algorithm to a certain criterium. results: An RLlib `ResultDict` containing all results from a training step. keep_ray_up: Optionally shutting down the runnin Ray instance. Returns: True, if any stopping criterium is fulfilled. Otherwise, False. /NzStop criterion (= ) fulfilled!TF) itemssplitstrip isinstancedictnpisnanprintrayshutdown TypeError)r?r@rAkey thresholdvalks r+ should_stoprUDs**,,Y3  A A#t$$ c!f ;  8C== SI%5%5FFFyFFFGGG"#LNNNtt    H  5s0AB99 CC)r?success_metric trainabletune_callbacks keep_configrA schedulerprogress_reporter base_configrargsrVrWrXrYc '|"t} | }|jrtd|jrd|_t j|jpd|j d|:tdt|j tdt|jt|ji}|} |s| |j|j!| j| |j|jr| dd|j| |j|j| |j | jrz|j|jd krtd t jd d } |j|jn| jpd } |jpd | z}|j|jnd | z}| d |j| |j|j!| |j!|j4| |kr| d n[| d nD| |kr#td|jd|jd| d| |j|j"| |j"n"|j| |j|j#d kr3| $|j%|j#|j&|j'|j(|j)| *|j)|j+| ,|j+|j-r|js|jrJ| .}t_|t|jD]f}|0}t|vrG|tttbj2}tgd|d|dth|vrKt|thvr7|thtt}tgd|dtg|5D]\}}|}|6dD]} ||} #tn$rd}YnwxYw|Ktcj8|s7||kr1tgd|d |d!|st j9|ccSh|st j9|S|pg}tu|d"r|j;txtzj>vr|j;ptzj>tx}|j?pV|j@Ad#ztjCd$d#t| jAz}|Etd@||dd%|jGr d&|jGini| O|jHd krDtitd'd(d)td*tdtd+id,| jJD-} d.tzj>d/<tjK}tjM|p| jN| tjO||jP|tjQ|jR|jS0| 1tjT|jU|jV|23W}tjK|z }|st j9|jXr#d4|jXD}td5|d}|jr|WthdtdttdtfD]}||vr |||i}n|tdt|j i}tt|5\'} t'fd6|]^D}!|!| krd}tgd7'd8| d9|jr|j_j`d }"|"ja}#|#bd:dt||"jdgttjKt|#|g| g|st|"d inid;}$tzj>d5}&tjh|$|&dddn #1swxYwY|std7'd8| d?|S)Aav Given an algorithm config and some command line args, runs an experiment. There are some constraints on what properties must be defined in `args`. It should ideally be generated via calling `args = add_rllib_example_script_args()`, which can be found in this very module here. The function sets up an Algorithm object from the given config (altered by the contents of `args`), then runs the Algorithm via Tune (or manually, if `args.no_tune` is set to True) using the stopping criteria in `stop`. At the end of the experiment, if `args.as_test` is True, checks, whether the Algorithm reached the `success_metric` (if None, use `env_runners/ episode_return_mean` with a minimum value of `args.stop_reward`). See https://github.com/ray-project/ray/tree/master/rllib/examples for an overview of all supported command line options. Args: base_config: The AlgorithmConfig object to use for this experiment. This base config will be automatically "extended" based on some of the provided `args`. For example, `args.num_env_runners` is used to set `config.num_env_runners`, etc.. args: A argparse.Namespace object, ideally returned by calling `args = add_rllib_example_script_args()`. It must have the following properties defined: `stop_iters`, `stop_reward`, `stop_timesteps`, `no_tune`, `verbose`, `checkpoint_freq`, `as_test`. Optionally, for WandB logging: `wandb_key`, `wandb_project`, `wandb_run_name`. stop: An optional dict mapping ResultDict key strings (using "/" in case of nesting, e.g. "env_runners/episode_return_mean" for referring to `result_dict['env_runners']['episode_return_mean']` to minimum values, reaching of which will stop the experiment). Default is: { "env_runners/episode_return_mean": args.stop_reward, "training_iteration": args.stop_iters, "num_env_steps_sampled_lifetime": args.stop_timesteps, } success_metric: Only relevant if `args.as_test` is True. A dict mapping a single(!) ResultDict key string (using "/" in case of nesting, e.g. "env_runners/episode_return_mean" for referring to `result_dict['env_runners']['episode_return_mean']` to a single(!) minimum value to be reached in order for the experiment to count as successful. If `args.as_test` is True AND this `success_metric` is not reached with the bounds defined by `stop`, will raise an Exception. trainable: The Trainable sub-class to run in the tune.Tuner. If None (default), use the registered RLlib Algorithm class specified by args.algo. tune_callbacks: A list of Tune callbacks to configure with the tune.Tuner. In case `args.wandb_key` is provided, appends a WandB logger to this list. keep_config: Set this to True, if you don't want this utility to change the given `base_config` in any way and leave it as-is. This is helpful for those example scripts which demonstrate how to set config settings that are otherwise taken care of automatically in this function (e.g. `num_env_runners`). Returns: The last ResultDict from a --no-tune run OR the tune.Tuner.fit() results. Nz`--enable-new-api-stack` flag no longer supported (it's the default behavior now)! To switch back to the old API stack on your scripts, use the `--old-api-stack` flag.T)num_cpus local_modeignore_reinit_errorrCF)enable_rl_module_and_learner"enable_env_runner_and_connector_v2)num_env_runners)num_envs_per_env_runnerrz--num-gpus is not supported on the new API stack! To train on GPUs, use the command line options `--num-gpus-per-learner=1` and `--num-learners=[your number of available GPUs]`, instead.GPUr7)num_gpus) num_learners)!num_aggregator_actors_per_learner)num_gpus_per_learnerz0You are running your script with --num-learners=z and --num-gpus-per-learner=z, but your cluster only has z GPUs!)num_cpus_per_learner)evaluation_num_env_runnersevaluation_intervalevaluation_durationevaluation_duration_unitevaluation_parallel_to_training) log_level)outputziter=z R=)endz R(eval)=zStop criterium (rDrE wandb_key-z\W+)api_keyprojectupload_checkpointsnameiter time_total_sztotal time (s)tszcombined returnc.i|]}td|d|S)z/module_episode_returns_mean/zreturn )r ).0pids r+ z7run_rllib_example_script_experiment..sB.UUPSUU&r-)metric_columns0RAY_AIR_NEW_OUTPUT)checkpoint_frequencycheckpoint_at_end)r?verbose callbackscheckpoint_configr[) num_samplesmax_concurrent_trialsrZ) param_space run_config tune_configcg|]n}|jrVt|jddr;t|jdjdkr|jdjdnt|oS)rr]r8)r]hasattrlenrepr)res r+ z7run_rllib_example_script_experiment..s   v !!&)V44 9z6run_rllib_example_script_experiment..s>  (.3C" #      r-`z` of z reached! okconfig) time_taken trial_states last_updatestatspassed not_passedfailuresTEST_OUTPUT_JSONz/tmp/learning_test.jsonwtz not reached!r)ir> parse_argsenable_new_api_stack ValueErroras_release_testas_testrNinitr_r`r r stop_rewardrstop_timestepsr stop_iters frameworkenv environment old_api_stack api_stackrd env_runnersrerbrgcluster_resourcesgetrhrj resourceslearnersrirkrm evaluationrlrnrorprq debuggingrr offline_datano_tunebuildrangetrainrKnanrMrrFrGKeyErrorrLrOrrur osenviron wandb_projectalgolowerresubr<appendr wandb_run_name num_agentsrpoliciestimer Tuner algo_class RunConfigrCheckpointConfigcheckpoint_freqr TuneConfigrrfiterrors RuntimeErrornextr{max get_dataframeiterrows_experiment_analysistrials last_resultpopr=statusropenjsondump)(r\r]r?rVrWrXrYrArZr[rrnum_gpus_availablenum_actual_learnersnum_gpus_requestednum_gpus_needed_if_availablerir@ mean_returnRevalrQrRrSrTrurx start_timerr test_passedtry_itsuccess_metric_value best_valuetrialr json_summaryfilenamefrs( @r+#run_rllib_example_script_experimentrus P |.00  ""    *     H&$?  |! 9 9$7 9 94;K! D D$B D D#   F j4((( 8 FJ$6   tx ( ( (      -238       +   t/C  D D D  ' 3   t7S  T T T  .@ 5}(T]Q->-> Q"%!6!8!8!???5A> (0%)EEEOOO;;;;OOO;;;;#&888 2(22022*222T5NOOO(4T5NOOO] &   dm  4 4 4  #a ' '   +/+J$($<$($<)-)F040T     > %   t~  6 6 6 ; "   t{  3 3 3 |#<<(<<<<||~~txx 2DODDEE # #AjjllG!W,,%&89==' 1a11K11r::::"g--&'2D*EEE 234FG')%))r2222 GGG"&**,, # #Y3A!!f#"?28C==?SI=M=MJSJJ9JJJKKK&' "NNNNN #  LNNN $)rNt[!!  "mrz&A&AN?bj&? $ IOO   #bfVS#fj//:O:O:Q:Q&R&R R    !#'  594GOFD/00R       T_q%8%8' &"$42D)AA,?AACT   &     &(+BJ#$Jj&V&>L$"3%)%9"&"80    O("&"<   $ cee% &z)J   ~    ^      R& R R   K |-  !%RR(:RR=PRR%==(;==  T>>&,d6l%;NE"%)AA,?AA4CS" 48^=Q=Q=S=S8T8T3U3U00    292G2G2I2I2R2R2T2T      - - -K Q(QQ/CQQQ R R R   +07:E%E IIh % % %#J//!&$TY[[11;EBB&-#./3>FSZZOOBLz~~&8:STTHh%% + ,*** + + + + + + + + + + + + + + + P&PP-APPP  Ns$ S S& %S& 4eee)Nrrr)Fr)2r9rloggingrrrtypingrrrrrrr numpyrKrNr ray.air.integrations.wandbr r ray.rllib.utils.metricsr rrrray.rllib.utils.serializationrray.rllib.utils.typingrray.tunerray.tune.resultrray.rllib.algorithmsr getLogger__name__loggerr:r=r)r>r<boolrU Namespace result_grid ResultGridrrr-r+rs   IIIIIIII MLLLLL------ ......5444444  8 $ $15!# XX X, -XXX X  XXXX|DI)) sCx.)#-)<@) ))))f*.C %) $%)CCC"C 8% &C 4. C TN C ~ CTNCCC :t'2 23CCCCCCr-