&`i!:ddlmZddlmZddlmZddlmZddlm Z ddl m Z e \Z Z eGddZeGd d ZeGd d ZeGd dZeGddZdS)) PolicyState) SampleBatch) TorchPolicy) OldAPIStack)try_import_torch)PiecewiseSchedulec*eZdZdZddZfdZxZS)LearningRateSchedulez9Mixin for TorchPolicy that adds a learning rate schedule.NcTd|_d|_|||_nBt||ddd|_|jd|_| ||_dSt||ddd|_|jd|_dS)N outside_value frameworkr) _lr_schedule _lr2_schedulecur_lrrvaluecur_lr2)selflr lr_schedulelr2 lr2_schedules q/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/policy/torch_mixins.py__init__zLearningRateSchedule.__init__s !  DKK 1;r?2+>$!!!D +11!44DK  DLLL!2L,>{KKK  ' 3!%!=!C!CJ'""D    4 3rr*r0s@rr2r22sP>>GGG.rr2cHeZdZdZdZdZdeffd Zdeddffd ZxZ S) KLCoeffMixinzAssigns the `update_kl()` method to a TorchPolicy. This is used by Algorithms to update the KL coefficient after each learning step based on `config.kl_target` and the measured KL value (from the train_batch). c:|d|_|d|_dS)Nkl_coeff kl_target)r>r?)rconfigs rrzKLCoeffMixin.__init__^sz*  ,rc|d|jzkr|xjdzc_n|d|jzkr|xjdzc_|jS)Ng@g?g?)r?r>)r sampled_kls r update_klzKLCoeffMixin.update_kldsP dn, , , MMS MMM #. . . MMS MM}rreturnc\t}|j|d<|S)Ncurrent_kl_coeff)r! get_stater>rstater)s rrGzKLCoeffMixin.get_statems*!!##$(M ! rrINc|d|jd|_t|dS)NrFr>)popr@r>r! set_staterHs rrLzKLCoeffMixin.set_statess> "4dk*6MNN  %     r) r+r,r-r.rrCrrGrLr/r0s@rr<r<Us--- ; !{!t!!!!!!!!!!rr<ceZdZdZdZdZdS)ValueNetworkMixinaAssigns the `_value()` method to a TorchPolicy. This way, Policy can call `_value()` to get the current VF estimate on a single(!) observation (as done in `postprocess_trajectory_fn`). Note: When doing this, an actual forward pass is being performed. This is different from only calling `model.value_function()`, where the result of the most recent forward pass is being used to return an already calculated tensor. c||ds|drfd}nd}|_dS)Nuse_gaevtracect|}|}|\}}jdS)Nr)r_lazy_tensor_dictmodelvalue_functionitem) input_dict model_out_rs rrz)ValueNetworkMixin.__init__..values](44 !33J?? #zz*55 1z002215::<<.valuessr)get_value)rr@rs` rrzValueNetworkMixin.__init__sf ::i  FJJx$8$8   = = = = = =    rcBtj|iS)aDefines extra fetches per action computation. Args: input_dict (Dict[str, TensorType]): The input dict used for the action computing forward pass. state_batches (List[TensorType]): List of state tensors (empty for non-RNNs). model (ModelV2): The Model object of the Policy. action_dist: The instantiated distribution object, resulting from the model's outputs and the given distribution class. Returns: Dict[str, TensorType]: Dict with extra tf fetches to perform per action computation. )rVF_PREDSrU)rrW state_batchesrT action_dists rextra_action_outz"ValueNetworkMixin.extra_action_outs"*  %"6"6"8"8  rN)r+r,r-r.rrdr[rrrNrNzs<.     rrNc&eZdZdZdZddZdZdS)TargetNetworkMixinaMixin class adding a method for (soft) target net(s) synchronizations. - Adds the `update_target` method to the policy. Calling `update_target` updates all target Q-networks' weights from their respective "main" Q-networks, based on tau (smooth, partial updating). ch|jdd}||dS)Ntau?)rh)r@r^ update_target)rrhs rrzTargetNetworkMixin.__init__s4kooeS)) s#####rNcp|jdd|jt t |j}fd|D|jD]}| dS)Nrhric@i|]\}}||zdz |zzS)r r[).0kvmodel_state_dictrhs r z4TargetNetworkMixin.update_target..sG   1 s%a((AGq=8   r) r@r^rT state_dictnextiter target_modelsvaluesitemsload_state_dict)rrhtarget_state_dicttargetrps ` @rrjz TargetNetworkMixin.update_targets0T[__UC00:0022 !d&8&?&?&A&A!B!BCCNNPP     )//11    (//11 5 5F  " "#3 4 4 4 4 5 5rcXtj|||dSN)r set_weightsrj)rweightss rr}zTargetNetworkMixin.set_weightss/ g... rr|)r+r,r-r.rrjr}r[rrrfrfsP$$$ 5555(rrfN)ray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.policy.torch_policyrray.rllib.utils.annotationsrray.rllib.utils.frameworkrray.rllib.utils.schedulesrtorchnnr r2r<rNrfr[rrrs//////555555555555333333666666777777     r #'#'#'#'#'#'#' #'L  D !!!!!!!!!!!!!! !!H 8 8 8 8 8 8 8  8 v &&&&&&& &&&r