&`ivF ddlZddlmZmZmZddlZddlmZm Z ddl m Z m Z m Z ddlmZddlmZddlmZmZddlmZdd lmZmZmZmZmZdd lmZmZm Z m!Z!m"Z"dd l#m$Z$dd l%m&Z&ddl'Z'ej(e)Z*e&d edeee"fddfdZ+ed edede!de ddf dZ,e&d ede-de.dee!fdZ/e&de0ddfdZ1e&de!de-ddfdZ2dede!ddfdZ3dS)N)AnyDictOptional)DEPRECATED_VALUEdeprecation_warning)MultiAgentBatch SampleBatchconcat_samples) OldAPIStack) from_config) ALL_MODULES TD_ERROR_KEY)LEARNER_STATS_KEY)EpisodeReplayBuffer!MultiAgentPrioritizedReplayBufferMultiAgentReplayBufferPrioritizedEpisodeReplayBuffer ReplayBuffer)AlgorithmConfigDictModuleID ResultDictSampleBatchType TensorType)log_once) DeveloperAPI replay_buffer td_errorsreturncpt|tr|D]\}}|dtfvrt|vs |tAt d|rtd|dh| |t|dSdS)N__all__z+no_td_error_in_train_results_from_module_{}z0Trying to update priorities for module with ID `z` in prioritized episode replay buffer without providing `td_errors` in train_results. Priority update for this policy is being skipped.) isinstanceritemsr rrformatloggerwarningupdate_priorities)rr module_idtd_errors x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/utils/replay_buffers/utils.py*update_priorities_in_episode_replay_bufferr*"s-!?@@O$-??#4#4 O O IxY 4448++x /E/MAHHSSNN8%888   + +H\,BI N N N N5OO O Oconfig train_batch train_resultsc4t|tri}|D]R\}}|d|td}|j|}|d|d} tj|vrg} |j rt||j gz} n'|tjdt|} d} | D]"} | | | | | z } #tj| } |Qtd|r-t"d|V| Qtd|r-t"d|t| t|kr|j}t| t|krt| |zdksJ| d |gdddf} t| t|ksJ| |f||<T||dSdS) aUpdates the priorities in a prioritized replay buffer, given training results. The `abs(TD-error)` from the loss (inside `train_results`) is used as new priorities for the row-indices that were sampled for the train batch. Don't do anything if the given buffer does not support prioritized replay. Args: replay_buffer: The replay buffer, whose priority values to update. This may also be a buffer that does not support priorities. config: The Algorithm's config dict. train_batch: The batch used for the training update. train_results: A train results dict, generated by e.g. the `train_one_step()` utility. r(N batch_indexesrz+no_td_error_in_train_results_from_policy_{}zTrying to update priorities for policy with id `{}` in prioritized replay buffer without providing td_errors in train_results. Priority update for this policy is being skipped.z.no_batch_indices_in_train_result_for_policy_{}zTrying to update priorities for policy with id `{}` in prioritized replay buffer without providing batch_indices in train_batch. Priority update for this policy is being skipped.)r!rr"getrpolicy_batchesset_get_interceptorr SEQ_LENS zero_paddedlen max_seq_lenappendnparrayrr#r$r%replay_sequence_lengthreshaper&)rr,r-r. prio_dict policy_idinfor( policy_batch batch_indices_batch_indicesseq_lens sequence_sumseq_lenTs r)"update_priorities_in_replay_bufferrHFs.-!BCCJ3 ,2244C =C =OIt xx D1B,C,G,G ,S,STTH&5i@L  , ,T 2 2 2(,,_==M#|33"$  +S"8}} 0H/IIHH+K,@A/CMM/RH ! ',,G"))- *EFFF G+LL " 8 8 AHHSSNN#$*6)#4#4  $DKKIVVNN#$*6)#4#4  =!!S]]22!8 &&X663};M;MPQ;QUV;V;V;VV - 5 5r1g > >qqq!t D =))S]]::::$18#>LU###I% Y & &!//K r+c|ddi|d<|dttkrtddd|dt}|tkrtd d d |d t}|tkr!|dd t}|tkrtd dd|dt}|tkr||dd<tdd|dt}|tkr!|ddt}|tkrtddd gd}|D]}||ttkr^td|d|d |d|||d|<|d|didt}|tkrtddd ||d<|dd}|||dd<td d!d |d} d"| vs Jd#|dd"} t | t r'| d$d%krd&| z|dd"<t| |d} t| |dd"<t| d'r`|dd(d)d*krtd+|ddd,d-krtd.dS|ddrtd/dS)0aChecks and fixes values in the replay buffer config. Checks the replay buffer config for common misconfigurations, warns or raises error in case validation fails. The type "key" is changed into the inferred replay buffer class. Args: config: The replay buffer config to be validated. Raises: ValueError: When detecting severe misconfiguration. replay_buffer_configNworker_side_prioritizationz$config['worker_side_prioritization']zPrioritized replay is not supported when replay_mode=lockstep.rrLzDPrioritized replay is not supported when replay_sequence_length > 1.zJWorker side prioritization is not supported when prioritized_replay=False.) r2rrr#r!strfindr rjhasattr ValueError) r,r\capacityr`rakeys_with_deprecated_positionskrhr<rW buffer_type dummy_buffers r)validate_buffer_configrxszz($//7)+%& zz.0@AAEUUU6N     $8:JKK---$  zz-)9::H###0155mEUVV###<     ZZ +;<> !1  ,,,#5    &&&", > > ::a) * *.> > > "))!,,>EEaHH    zz011=4:1I-.q1jj )2..223DFVWWO***@E     >M9: $ZZ(@$GG) # %& $  2=     ""89&&&&I '&&/08K+s## (8(8(=(=(C(C . ; %&v. {F3I,JKKL-1,-?-?F !"6*|011 ) * . .}m L L  P * + / /0H! L Lq P P. Q P ( ) - -.J K K ,   r+itemrsc~tdr|}tj}|jdz }||zdz }d|||j||}||krt||d|zkrt |dSt |dSdS)a@Warn if the configured replay buffer capacity is too large for machine's memory. Args: item: A (example) item that's supposed to be added to the buffer. This is used to compute the overall memory footprint estimate for the buffer. capacity: The capacity value of the buffer. This is interpreted as the number of items (such as given `item`) that will eventually be stored in the buffer. Raises: ValueError: If computed memory footprint for the buffer exceeds the machine's RAM. warn_replay_buffer_capacitygeAz~Estimated max memory usage for replay buffer is {} GB ({} batches of size {}, {} bytes each), available system memory is {} GBg?N) r size_bytespsutilvirtual_memorytotalr#countrrr$r%r@)ryrs item_size psutil_memtotal_gbmem_sizemsgs r)r{r{ss -..OO%% *,, #c)i'#- //5v(DJ 800  h  S// ! h & & NN3      KK     #r+bufferfake_sample_outputct|tr6tts!tddt dt tffd }||_dS)aPatch a ReplayBuffer such that we always sample fake_sample_output. Transforms fake_sample_output into a MultiAgentBatch if it is not a MultiAgentBatch and the buffer is a MultiAgentBuffer. This is useful for testing purposes if we need deterministic sampling. Args: buffer: The buffer to be patched fake_sample_output: The output to be sampled N_rcS)aPAlways returns a predefined batch. Args: _: dummy arg to match signature of sample() method __: dummy arg to match signature of sample() method ``**kwargs``: dummy args to match signature of sample() method Returns: Predefined MultiAgentBatch fake_sample_output )rkwargsrs r) fake_samplez;patch_buffer_with_fake_sampling_method..fake_samples "!r+)N) r!rrr as_multi_agentrrrrN)rrrs ` r)&patch_buffer_with_fake_sampling_methodrs&011N*O;;N));<<KKMM " "s "0I " " " " " " FMMMr+)4loggingtypingrrrnumpyr:ray._common.deprecationrrray.rllib.policy.sample_batchrr r ray.rllib.utils.annotationsr ray.rllib.utils.from_configr ray.rllib.utils.metricsr r$ray.rllib.utils.metrics.learner_inforray.rllib.utils.replay_buffersrrrrrray.rllib.utils.typingrrrrrray.utilrray.util.annotationsrr} getLogger__name__r$r*rHintboolrUdictrxr{rrr+r)rs&&&&&&&&&&IIIIIIIIVVVVVVVVVV333333333333========BBBBBB------  8 $ $ O& OHj() O O O O OF `3`3 `3!`3 `3  `3`3`3 `3F,/GK oDb4bDbbbbJ  C D    F! ! .=!  ! ! ! ! ! ! r+