§ &`ƒiºãó°—ddlZddlZddlZddlmZddlmZddlmZm Z ddl m Z ddl m Z ddlmZddlmZmZdd lmZe Gd „d ¦«¦«ZdS) éN)ÚOptional©ÚSimpleReplayBuffer)ÚDEFAULT_POLICY_IDÚconcat_samples)Ú OldAPIStack)Ú ReplayMode)Ú _ALL_POLICIES)ÚPolicyIDÚSampleBatchType)Ú_Timercóp—eZdZdZejfdededefd„Zde ddfd „Z e fd e de e fd „Zdefd „ZdS) ÚMixInMultiAgentReplayBufferažThis buffer adds replayed samples to a stream of new experiences. - Any newly added batch (`add()`) is immediately returned upon the next `replay` call (close to on-policy) as well as being moved into the buffer. - Additionally, a certain number of old samples is mixed into the returned sample according to a given "replay ratio". - If >1 calls to `add()` are made without any `replay()` calls in between, all newly added batches are returned (plus some older samples according to the "replay ratio"). .. testcode:: from ray.rllib.execution.buffers.mixin_replay_buffer import ( MixInMultiAgentReplayBuffer) from ray.rllib.policy.sample_batch import SampleBatch # replay ratio 0.66 (2/3 replayed, 1/3 new samples): buffer = MixInMultiAgentReplayBuffer(capacity=100, replay_ratio=0.66) A, B, C = (SampleBatch({"obs": [1]}), SampleBatch({"obs": [2]}), SampleBatch({"obs": [3]})) buffer.add(A) buffer.add(B) buffer.add(B) print(buffer.replay()["obs"]) .. testoutput:: :hide: ... ÚcapacityÚ replay_ratioÚ replay_modecó:‡—‰|_||_d|_|jdkr|jd|jz z |_|dtjfvrtj|_nD|dtjfvrtj|_n"td |¦«¦«‚ˆfd„}tj |¦«|_ t¦«|_ t¦«|_t¦«|_d|_tj t"¦«|_dS)aèInitializes MixInReplay instance. Args: capacity: Number of batches to store in total. replay_ratio: Ratio of replayed samples in the returned batches. E.g. a ratio of 0.0 means only return new samples (no replay), a ratio of 0.5 means always return newest sample plus one old one (1:1), a ratio of 0.66 means always return the newest sample plus 2 old (replayed) ones (1:2), etc... Nçð?ÚlockstepÚ independentzUnsupported replay mode: {}có$•—t‰¬¦«S)N)Ú num_slotsr)rs€úƒ/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/execution/buffers/mixin_replay_buffer.pyÚ new_bufferz8MixInMultiAgentReplayBuffer.__init__..new_bufferNsø€Ý%°Ð9Ñ9Ô9Ð 9ór)rrÚreplay_proportionr ÚLOCKSTEPrÚ INDEPENDENTÚ ValueErrorÚformatÚ collectionsÚ defaultdictÚreplay_buffersr Úadd_batch_timerÚ replay_timerÚupdate_priorities_timerÚ num_addedÚlistÚlast_added_batches)Úselfrrrrs ` rÚ__init__z$MixInMultiAgentReplayBuffer.__init__1sø€ð !ˆŒ Ø(ˆÔØ!%ˆÔØ Ô  Ò #Ð #Ø%)Ô%6¸#ÀÔ@QÑ:QÑ%RˆDÔ "à ˜:¥zÔ':Ð;Ð ;Ð ;Ý)Ô2ˆDÔ Ð Ø ˜]­JÔ,BÐCÐ CÐ CÝ)Ô5ˆDÔ Ð åÐ:×AÒAÀ+ÑNÔNÑOÔOÐ Oð :ð :ð :ð :ð :õ*Ô5°jÑAÔAˆÔõ &™xœxˆÔÝ"™HœHˆÔÝ'-¡x¤xˆÔ$ðˆŒõ#.Ô"9½$Ñ"?Ô"?ˆÔÐÐrÚbatchÚreturnNcó<—| ¦«}| ¦«}|j5|jtjkrK|jt |¦«|j t  |¦«n_|j   ¦«D]E\}}|j| |¦«|j |  |¦«ŒFddd¦«n #1swxYwY|xj |jz c_ dS)a2Adds a batch to the appropriate policy's replay buffer. Turns the batch into a MultiAgentBatch of the DEFAULT_POLICY_ID if it is not a MultiAgentBatch. Subsequently adds the individual policy batches to the storage. Args: batch: The batch to be added. N)ÚcopyÚas_multi_agentr$rr rr#r Ú add_batchr)ÚappendÚpolicy_batchesÚitemsr'Úcount)r*r,Ú policy_idÚ sample_batchs rÚaddzMixInMultiAgentReplayBuffer.add^sY€ð— ’ ‘ ” ˆØ×$Ò$Ñ&Ô&ˆà Ô !ð Lð LØÔ¥:Ô#6Ò6Ð6ðÔ#¥MÔ2×<Ò<¸UÑCÔCÐCØÔ'­ Ô6×=Ò=¸eÑDÔDÐDÐDð05Ô/C×/IÒ/IÑ/KÔ/KðLðLÑ+I˜|ØÔ'¨ Ô2×<Ò<¸\ÑJÔJÐJØÔ+¨IÔ6×=Ò=¸lÑKÔKÐKÐKð Lð Lð Lñ Lô Lð Lð Lð Lð Lð Lð Løøøð Lð Lð Lð L𠈌˜%œ+Ñ%ˆŒˆˆs°CC<Ã<DÄDr6cóB—|jtjkr2|tkr't d t¦«¦«‚|j|}t|¦«dks)t|j|¦«dkr |j dkrdS|j 5|j|}g|j|<|j dkrt|¦«cddd¦«S|j dkr |  ¦«cddd¦«St|¦«}|j }tj¦«||zkrF|dz}| |  ¦«¦«tj¦«||zk°Ft|¦«cddd¦«S#1swxYwYdS)Nz¼Trying to sample from single policy's buffer in lockstep mode. In lockstep mode, all policies' experiences are sampled from a single replay buffer which is accessed with the policy id `{}`rrgé)rr rr rr r#Úlenr)rr%rÚreplayrÚrandomr2)r*r6ÚbufferÚoutput_batchesÚnum_newrs rr<z"MixInMultiAgentReplayBuffer.replay{s €ð Ô zÔ2Ò 2Ð 2°yÅMÒ7QÐ7QÝð*÷+1ª&µÑ*?Ô*?ñ ôð ðÔ$ YÔ/ˆõ ˆv‰;Œ;˜!Ò Ð Ý Ô'¨ Ô2Ñ 3Ô 3°qÒ 8Ð 8¸TÔ=NÐQTÒ=TÐ=Tà4ðÔ ð 2ð 2Ø!Ô4°YÔ?ˆNØ13ˆDÔ # IÑ .ðÔ  CÒ'Ð'Ý% nÑ5Ô5ð  2ð 2ð 2ð 2ñ 2ô 2ð 2ð 2ðÔ" cÒ)Ð)Ø—}’}‘”ð 2ð 2ð 2ð 2ñ 2ô 2ð 2ð 2õ˜.Ñ)Ô)ˆGØ $Ô 6Ð Ý”-‘/”/ GÐ.?Ñ$?Ò?Ð?Ø! QÑ&Ð!Ø×%Ò% f§m¢m¡o¤oÑ6Ô6Ð6õ”-‘/”/ GÐ.?Ñ$?Ò?Ð?õ" .Ñ1Ô1ð' 2ð 2ð 2ð 2ñ 2ô 2ð 2ð 2ð 2ð 2ð 2ð 2øøøð 2ð 2ð 2ð 2ð 2ð 2sÂ1FÃFÄBFÆFÆFcó(—tj¦«S)zµReturns the computer's network name. Returns: The computer's networks name or an empty string, if the network name could not be determined. )ÚplatformÚnode)r*s rÚget_hostz$MixInMultiAgentReplayBuffer.get_host¦s€õŒ}‰ŒÐr)Ú__name__Ú __module__Ú __qualname__Ú__doc__r rÚintÚfloatr+r r8rr rr<ÚstrrD©rrrrsÖ€€€€€ðððH#-Ô"8ð +@ð+@àð+@ðð+@ð ð +@ð+@ð+@ð+@ðZ&˜ð&¨Tð&ð&ð&ð&ð<%6ð)2ð)2Ø!ð)2à /Ô "ð)2ð)2ð)2ð)2ðV˜#ððððððrr)r!rBr=ÚtypingrÚray.rllib.execution.replay_opsrÚray.rllib.policy.sample_batchrrÚray.rllib.utils.annotationsrÚ8ray.rllib.utils.replay_buffers.multi_agent_replay_bufferr Ú,ray.rllib.utils.replay_buffers.replay_bufferr Úray.rllib.utils.typingr r Úray.util.timerr rrLrrúrUsðØÐÐÐØ€€€Ø € € € ØÐÐÐÐÐà=Ð=Ð=Ð=Ð=Ð=ØKÐKÐKÐKÐKÐKÐKÐKØ3Ð3Ð3Ð3Ð3Ð3ØOÐOÐOÐOÐOÐOØFÐFÐFÐFÐFÐFØ<Ð<Ð<Ð<Ð<Ð<Ð<Ð<Ø!Ð!Ð!Ð!Ð!Ð!ð ð]ð]ð]ð]ð]ñ]ô]ñ „ð]ð]ð]r