&`i) ddlmZmZmZddlZddlmZmZm Z ddl m Z ddl m Z ddlmZddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZm Z e\Z!Z"Z#GddZ$ede%de&de&de'de&f dZ(edej)de'de'dej)fdZ*eGddeZ+dS))ListOptionalUnionN)BoxDiscreteSpace)ActionDistribution) ModelCatalog)ModelV2) SampleBatch) OldAPIStackoverride) Exploration) try_import_tf) from_config)get_placeholder)FromConfigSpecModelConfigDict TensorTypeceZdZdZddedeeefdZde j de j fd Z d ed ed eddfd Z e defdZdS)_MovingMeanStdz!Track moving mean, std and count.-C6?Nepsilonshapec|sg}tj|tj|_tj|tj|_||_dS)zInitialize object. Args: epsilon: Initial count. shape: Shape of the trackables mean and std. )dtypeN)npzerosfloat32meanonesvarcount)selfrrs ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/utils/exploration/random_encoder.py__init__z_MovingMeanStd.__init__sL EHU"*555 75 333 inputsreturnctj|d}tj|d}|jd}||||tj||jz dzS)zNormalize input batch using moving mean and std. Args: inputs: Input batch to normalize. Returns: Logarithmic scaled normalized output. raxis)rr r"r update_paramslogstd)r$r( batch_mean batch_var batch_counts r%__call__z_MovingMeanStd.__call__$skWV!,,, F6*** l1o  :y+>>>vftx'!+,,,r'r1r2r3c||jz }|j|z}|j|z||z z|_|j|jz}||z}||ztj|d|jz|z|z z}||z |_||_dS)zUpdate moving mean, std and count. Args: batch_mean: Input batch mean. batch_var: Input batch variance. batch_count: Number of cases in the batch. N)r r#r"rpower) r$r1r2r3delta tot_countm_am_bM2s r%r.z_MovingMeanStd.update_params3sTY&J, I% i(?? h#+% 3Y%++dj8;FR R > r'c4tj|jS)zhGet moving standard deviation. Returns: Returns moving standard deviation. )rsqrtr")r$s r%r0z_MovingMeanStd.stdHswtx   r')rN)__name__ __module__ __qualname____doc__floatrrintr&rndarrayr4r.propertyr0r'r%rrs++   Xd3i5H     -rz -bj - - - -,1@E *!U!!!X!!!r'r beta_schedulebetarhostepr)c(|dkr |d|z |zzS|S)aUpdate beta based on schedule and training step. Args: beta_schedule: Schedule for beta update. beta: Initial beta. rho: Schedule decay parameter. step: Current training iteration. Returns: Updated beta as per input schedule. linear_decay?rG)rHrIrJrKs r% update_betarORs(&&c d*++ Kr' obs_embeds embed_dimk_nnc>tj|d|g}tj|dddddf|dddddfz d}|dddd|fdddftjS)a<Compute states entropy using K nearest neighbour method. Args: obs_embeds: Observation latent representation using encoder model. embed_dim: Embedding vector dimension. k_nn: Number of nearest neighbour for K-NN estimation. Returns: Computed states entropy. Nr+)rreshapelinalgnormargsortastyper)rPrQrR obs_embeds_dists r%compute_states_entropyr\ds*Z"i99K 9>>+aaaqqqj1Kaaa 4KKRT> U UD <T ! !*j*DL "//!!;?DL"ABB1E  D  &1'*n,*(* & & & """r'T)exploreaction_distributiontimesteprc<|j|||S)N)rrr)rget_exploration_action)r$rrrs r%rzRE3.get_exploration_actions-)@@ 3hPWA   r'cj|jdkr||||}ntd|S)zCalculate states' latent representations/embeddings. Embeddings are added to the SampleBatch object such that it doesn't need to be calculated during each training step. rjzNot implemented for Torch.)rg_postprocess_tfr})r$policy sample_batchtf_sesss r%postprocess_trajectoryzRE3.postprocess_trajectory s? >W $ $// gNNLL9:: :r'cd|jdkr4||j|j|tji}nat |tj|tjid }||tj <|S)z7Calculate states' embeddings and add it to SampleBatch.ru) feed_dictr) rgrunrrr rrurrnumpy OBS_EMBEDS)r$rrrrPs r%rzRE3._postprocess_tfs >T ! !  <ko)FG%JJ ))!!;?L4Q"RSSTUVegg 0: [+,r')N)r?r@rArBrstrr rDrrrCrr&rrr rrboolrrr __classcell__)rs@r%r^r^xs.8<' %48r r r r  r  r  r %_5r r r r r r ".1r r r r r r hXk     0  Z(        Xk           r'r^),typingrrrrrgymnasium.spacesrrrray.rllib.models.action_distr ray.rllib.models.catalogr ray.rllib.models.modelv2r ray.rllib.policy.sample_batchr ray.rllib.utils.annotationsr r'ray.rllib.utils.exploration.explorationrray.rllib.utils.frameworkrray.rllib.utils.from_configrray.rllib.utils.tf_utilsrray.rllib.utils.typingrrrtf1rutfvrrrCrDrOrEr\r^rGr'r%rs&((((((((((1111111111;;;;;;111111,,,,,,555555========??????333333333333444444NNNNNNNNNN} R;!;!;!;!;!;!;!;!| s%e35 " E E'*E25EZEEE E& lllll+ll lllr'