&`i68\ddlZddlZddlmZddlmZmZmZmZm Z ddl Z ddl Z ddl mZmZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZddlZd Zej e!Z"eGd deZ#edede$ddfdZ%eGddeeZ&dS)N)Enum)AnyDictListOptionalUnion) SampleBatchconcat_samples)FaultAwareApply)override) WindowStat)ReplayBufferInterface)SampleBatchType) DeveloperAPI)log_once__all__c"eZdZdZdZdZdZdZdS) StorageUnitzSpecifies how batches are structured in a ReplayBuffer. timesteps: One buffer slot per timestep. sequences: One buffer slot per sequence. episodes: One buffer slot per episode. fragemts: One buffer slot per incoming batch. timesteps sequencesepisodes fragmentsN)__name__ __module__ __qualname____doc__ TIMESTEPS SEQUENCESEPISODES FRAGMENTS/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/utils/replay_buffers/replay_buffer.pyrrs.IIHIIIr"ritem num_itemsreturnc~tdr|}tj}|jdz }||zdz }d|||j||}||krt||d|zkrt |dSt |dSdS)z;Warn if the configured replay buffer capacity is too large.replay_capacitygeAz~Estimated max memory usage for replay buffer is {} GB ({} batches of size {}, {} bytes each), available system memory is {} GBg?N) r size_bytespsutilvirtual_memorytotalformatcount ValueErrorloggerwarninginfo)r$r% item_size psutil_memtotal_gbmem_sizemsgs r#warn_replay_capacityr8+s!""OO%% *,, #c)y(3. //5v)TZH00  h  S// ! h & & NN3      KK     #r"ceZdZdZ ddedeeeffdZe e defdZ e e d e dd fd Z ed e dd fd Ze e  ddeedee fdZeddedefdZe e deeeffdZe e deeefdd fdZedeede fdZd S) ReplayBufferaQThe lowest-level replay buffer interface used by RLlib. This class implements a basic ring-type of buffer with random sampling. ReplayBuffer is the base class for advanced types that add functionality while retaining compatibility through inheritance. The following examples show how buffers behave with different storage_units and capacities. This behaviour is generally similar for other buffers, although they might not implement all storage_units. Examples: .. testcode:: from ray.rllib.utils.replay_buffers.replay_buffer import ReplayBuffer from ray.rllib.utils.replay_buffers.replay_buffer import StorageUnit from ray.rllib.policy.sample_batch import SampleBatch # Store any batch as a whole buffer = ReplayBuffer(capacity=10, storage_unit=StorageUnit.FRAGMENTS) buffer.add(SampleBatch({"a": [1], "b": [2, 3, 4]})) buffer.sample(1) # Store only complete episodes buffer = ReplayBuffer(capacity=10, storage_unit=StorageUnit.EPISODES) buffer.add(SampleBatch({"c": [1, 2, 3, 4], SampleBatch.T: [0, 1, 0, 1], SampleBatch.TERMINATEDS: [False, True, False, True], SampleBatch.EPS_ID: [0, 0, 1, 1]})) buffer.sample(1) # Store single timesteps buffer = ReplayBuffer(capacity=2, storage_unit=StorageUnit.TIMESTEPS) buffer.add(SampleBatch({"a": [1, 2], SampleBatch.T: [0, 1]})) buffer.sample(1) buffer.add(SampleBatch({"a": [3], SampleBatch.T: [2]})) print(buffer._eviction_started) buffer.sample(1) buffer = ReplayBuffer(capacity=10, storage_unit=StorageUnit.SEQUENCES) buffer.add(SampleBatch({"c": [1, 2, 3], SampleBatch.SEQ_LENS: [1, 2]})) buffer.sample(1) .. testoutput:: True `True` is not the output of the above testcode, but an artifact of unexpected behaviour of sphinx doctests. (see https://github.com/ray-project/ray/pull/32477#discussion_r1106776101) 'rcapacity storage_unitc |dtjfvrtj|_n|dtjfvrtj|_n|dtjfvrtj|_nh|dtjfvrtj|_nFt dtjdtjdtjdtjd| g|_|d kr"t d |||_ d |_ tj |j |_ d |_d |_d |_d |_t%d d |_d |_d|_dS)aInitializes a (FIFO) ReplayBuffer instance. Args: capacity: Max number of timesteps to store in this FIFO buffer. After reaching this number, older samples will be dropped to make space for new ones. storage_unit: If not a StorageUnit, either 'timesteps', 'sequences' or 'episodes'. Specifies how experiences are stored. ``**kwargs``: Forward compatibility kwargs. rrrrzstorage_unit must be either 'z', 'z' or 'z ', but is rzHCapacity of replay buffer has to be greater than zero but was set to {}.F evicted_hitiN)rrr=rrr r/_storager-r< _next_idxnpzeros _hit_count_eviction_started_num_timesteps_added_num_timesteps_added_wrap_num_timesteps_sampledr _evicted_hit_stats_est_size_bytes batch_size)selfr<r=kwargss r#__init__zReplayBuffer.__init__zs" K)>? ? ? + 5D   k;+@A A A + 5D   j+*>? ? ? + 4D   k;+@A A A + 5D  G 0EGG)GG/:/CGG",GG8DGG   q==%%+VH%5%5 ! (4=11"' %&!)*&'(#",]D"A"A r"r&c*t|jSN)lenr@)rLs r#__len__zReplayBuffer.__len__s4=!!!r"batchNc |jdksdSt||j|jz |jtjkr*|d}|D]}|j|fi|dS|jtjkrEd}| tj D]!}|}||z}|j|||fi||}"dS|jtj kr| D]} | tjdgddkr]| tjdgds'| tjdgdr|j| fi|t#drt$d dS|jtjkr|j|fi|dSdS) avAdds a batch of experiences or other data to this buffer. Splits batch into chunks of timesteps, sequences or episodes, depending on `self._storage_unit`. Calls `self._add_single_batch` to add resulting slices to the buffer storage. Args: batch: The batch to add. ``**kwargs``: Forward compatibility kwargs. rN)r$r%TFonly_full_episodeszThis buffer uses episodes as a storage unit and thus allows only full episodes to be added to it (starting from T=0 and ending in `terminateds=True` or `truncateds=True`. Some samples may be dropped.)r.r8r<r=rr timeslices_add_single_batchrgetr SEQ_LENSrsplit_by_episodeT TERMINATEDS TRUNCATEDSrr0r2r ) rLrSrMrXttimestep_countseq_len start_seqend_seqepss r#addzReplayBuffer.adds){Q F%4=5;3NOOOO   5 5 5))!,,J 4 4&&q33F3333 4 4 +"7 7 7N 99[%9:: ) )* (72&&uYw->'?JJ6JJJ!(  ) )  +"6 6 6--//  77;=1#..q1Q66GGK3dV<S// / / M  & & &  DOO$5$5 5 !%t~!>   $6$A$A$C$C C ,0DM$. )  DOO$5$5 5   ! 0  # ( ()H I I I./DODN +  )T] : :%)D "-.D *DNNN NNa NNNNr"r%c tdkrtdfdt|D}|}xj|jz c_|S)a6Samples `num_items` items from this buffer. The items depend on the buffer's storage_unit. Samples in the results may be repeated. Examples for sampling results: 1) If storage unit 'timesteps' has been chosen and batches of size 5 have been added, sample(5) will yield a concatenated batch of 15 timesteps. 2) If storage unit 'sequences' has been chosen and sequences of different lengths have been added, sample(5) will yield a concatenated batch with a number of timesteps equal to the sum of timesteps in the 5 sampled sequences. 3) If storage unit 'episodes' has been chosen and episodes of different lengths have been added, sample(5) will yield a concatenated batch with a number of timesteps equal to the sum of timesteps in the 5 sampled episodes. Args: num_items: Number of items to sample from this buffer. ``**kwargs``: Forward compatibility kwargs. Returns: Concatenated batch of items. rz&Trying to sample from an empty buffer.cZg|]'}tjdtdz (S)rrU)randomrandintrQ).0_rLs r# z'ReplayBuffer.sample..;s/LLLa3t99q=11LLLr")rQr/range_encode_samplerHr.)rLr%rMidxessamples` r#ruzReplayBuffer.sampless@ t99>>EFF FLLLL5;K;KLLL$$U++ ##v|3## r"Fdebugc|j|j|j|j|jt |jd}|r,||j |S)zReturns the stats of this buffer. Args: debug: If True, adds sample eviction statistics to the returned stats dict. Returns: A dictionary of stats about this buffer. ) added_countadded_count_wrappedeviction_started sampled_countest_size_bytes num_entries) rFrGrErHrJrQr@updaterIstats)rLrvdatas r#rzReplayBuffer.stats@sk 4#'#A $ 6!8"2t}--     9 KK/5577 8 8 8 r"cv|j|jd}||d|S)N)r@rAF)rv)r@rAr~rrLstates r# get_statezReplayBuffer.get_stateWs8!]HH TZZeZ,,--- r"rc|d|_|d|_|d|_|d|_|d|_|d|_|d|_dS)Nr@rArxryrzr{r|)r@rArFrGrErHrJrs r# set_statezReplayBuffer.set_state]sfj) {+$)-$8!)./D)E&!&'9!:&+O&<#$%56r"rtcg}|D]7}|j|xxdz cc<||j|8|rt|}nt }||S)z?Fetches concatenated samples at given indices from the storage.rU)rDrhr@r r decompress_if_needed)rLrtsamplesiouts r#rszReplayBuffer._encode_sampleis - -A OA   ! #    NN4=+ , , , ,   ))CC--C   """ r")r;rrP)F)rrrrintrstrrrNr rrRrrfrrYrrubooldictrrrrrrrsr!r"r#r:r:Bs144p0;@@@C,-@@@@DX#$$""""%$"X#$$1414t141414%$14f" o" D" " " \" HX#$$)-$$!#$ / "$$$%$$L4D\,X#$$4S>%$ X#$$ 7tCH~ 7$ 7 7 7%$ 7 DI /   \   r"r:)'loggingrmenumrtypingrrrrrnumpyrBrayray.rllib.policy.sample_batchr r ray.rllib.utils.actor_managerr ray.rllib.utils.annotationsr #ray.rllib.utils.metrics.window_statr #ray.rllib.utils.replay_buffers.baserray.rllib.utils.typingrray.util.annotationsrray.util.debugrr* _ALL_POLICIES getLoggerrr0rrr8r:r!r"r#rs 33333333333333 EEEEEEEE999999000000::::::EEEEEE222222------######   8 $ $     $   /cd,ttttt(/tttttr"