&`i'B+ddlmZmZmZmZmZmZmZddlZ ddl m Z m Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZm Z dd l!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(e"\Z)Z*Z+edddddddddddddddddde dddde,deeeeeegee(ee(ffdeedge&fdeeeegee,e(ffdeeee&gdfdeeede(ge'fdeeede'gdfdeeeee'gee,e(ffdeeegee,e(ffdeeegee,e(ffdeeee j-e j-e&gdfdeeee j-e j-e&gdfdeeee j.j-e j.j-e&gdfd eeee j-e j-e&gdfd!eeee j.j-e j.j-e&gefd"eee(ee(gee(e(ffd#eeeee(e(e(gee(e/ee(ffd$eee/d%eeege0fd&eef(d'Z1dS)()CallableDictListOptionalTupleTypeUnionN)DEPRECATED_VALUEdeprecation_warning)ModelV2)TFActionDistribution)eager_tf_policy)DynamicTFPolicy)Policy) SampleBatch)TFPolicy) add_mixins force_list) OldAPIStackoverride) try_import_tf)LEARNER_STATS_KEY)AlgorithmConfigDictModelGradients TensorType)get_default_configpostprocess_fnstats_fn optimizer_fncompute_gradients_fnapply_gradients_fn grad_stats_fnextra_action_out_fnextra_learn_fetches_fnvalidate_spaces before_initbefore_loss_init after_init make_modelaction_sampler_fnaction_distribution_fnmixinsget_batch_divisibility_reqobs_include_prev_action_rewardextra_action_fetches_fn gradients_fnnameloss_fnrrrztf.keras.optimizers.Optimizerr r!z tf.Operationr"r#r$r%r&r'r(r)r*r+r,r-returnc ttt||tkrt dd|t ddd|t dd dG   fd d }fd }fd }t ||_t ||_||_ ||_ |S)aSHelper function for creating a dynamic tf policy at runtime. Functions will be run in this order to initialize the policy: 1. Placeholder setup: postprocess_fn 2. Loss init: loss_fn, stats_fn 3. Optimizer init: optimizer_fn, gradients_fn, apply_gradients_fn, grad_stats_fn This means that you can e.g., depend on any policy attributes created in the running of `loss_fn` in later functions such as `stats_fn`. In eager mode, the following functions will be run repeatedly on each eager execution: loss_fn, stats_fn, gradients_fn, apply_gradients_fn, and grad_stats_fn. This means that these functions should not define any variables internally, otherwise they will fail in eager mode execution. Variable should only be created in make_model (if defined). Args: name: Name of the policy (e.g., "PPOTFPolicy"). loss_fn (Callable[[ Policy, ModelV2, Type[TFActionDistribution], SampleBatch], Union[TensorType, List[TensorType]]]): Callable for calculating a loss tensor. get_default_config (Optional[Callable[[None], AlgorithmConfigDict]]): Optional callable that returns the default config to merge with any overrides. If None, uses only(!) the user-provided PartialAlgorithmConfigDict as dict for this Policy. postprocess_fn (Optional[Callable[[Policy, SampleBatch, Optional[Dict[AgentID, SampleBatch]], Episode], None]]): Optional callable for post-processing experience batches (called after the parent class' `postprocess_trajectory` method). stats_fn (Optional[Callable[[Policy, SampleBatch], Dict[str, TensorType]]]): Optional callable that returns a dict of TF tensors to fetch given the policy and batch input tensors. If None, will not compute any stats. optimizer_fn (Optional[Callable[[Policy, AlgorithmConfigDict], "tf.keras.optimizers.Optimizer"]]): Optional callable that returns a tf.Optimizer given the policy and config. If None, will call the base class' `optimizer()` method instead (which returns a tf1.train.AdamOptimizer). compute_gradients_fn (Optional[Callable[[Policy, "tf.keras.optimizers.Optimizer", TensorType], ModelGradients]]): Optional callable that returns a list of gradients. If None, this defaults to optimizer.compute_gradients([loss]). apply_gradients_fn (Optional[Callable[[Policy, "tf.keras.optimizers.Optimizer", ModelGradients], "tf.Operation"]]): Optional callable that returns an apply gradients op given policy, tf-optimizer, and grads_and_vars. If None, will call the base class' `build_apply_op()` method instead. grad_stats_fn (Optional[Callable[[Policy, SampleBatch, ModelGradients], Dict[str, TensorType]]]): Optional callable that returns a dict of TF fetches given the policy, batch input, and gradient tensors. If None, will not collect any gradient stats. extra_action_out_fn (Optional[Callable[[Policy], Dict[str, TensorType]]]): Optional callable that returns a dict of TF fetches given the policy object. If None, will not perform any extra fetches. extra_learn_fetches_fn (Optional[Callable[[Policy], Dict[str, TensorType]]]): Optional callable that returns a dict of extra values to fetch and return when learning on a batch. If None, will call the base class' `extra_compute_grad_fetches()` method instead. validate_spaces (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable that takes the Policy, observation_space, action_space, and config to check the spaces for correctness. If None, no spaces checking will be done. before_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable to run at the beginning of policy init that takes the same arguments as the policy constructor. If None, this step will be skipped. before_loss_init (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], None]]): Optional callable to run prior to loss init. If None, this step will be skipped. after_init (Optional[Callable[[Policy, gym.Space, gym.Space, AlgorithmConfigDict], None]]): Optional callable to run at the end of policy init. If None, this step will be skipped. make_model (Optional[Callable[[Policy, gym.spaces.Space, gym.spaces.Space, AlgorithmConfigDict], ModelV2]]): Optional callable that returns a ModelV2 object. All policy variables should be created in this function. If None, a default ModelV2 object will be created. action_sampler_fn (Optional[Callable[[TensorType, List[TensorType]], Tuple[TensorType, TensorType]]]): A callable returning a sampled action and its log-likelihood given observation and state inputs. If None, will either use `action_distribution_fn` or compute actions by calling self.model, then sampling from the so parameterized action distribution. action_distribution_fn (Optional[Callable[[Policy, ModelV2, TensorType, TensorType, TensorType], Tuple[TensorType, type, List[TensorType]]]]): Optional callable returning distribution inputs (parameters), a dist-class to generate an action distribution object from, and internal-state outputs (or an empty list if not applicable). If None, will either use `action_sampler_fn` or compute actions by calling self.model, then sampling from the so parameterized action distribution. mixins (Optional[List[type]]): Optional list of any class mixins for the returned policy class. These mixins will be applied in order and will have higher precedence than the DynamicTFPolicy class. get_batch_divisibility_req (Optional[Callable[[Policy], int]]): Optional callable that returns the divisibility requirement for sample batches. If None, will assume a value of 1. Returns: Type[DynamicTFPolicy]: A child class of DynamicTFPolicy based on the specified args. r.T)olderrorNr/r#)r5newr6r0r c JeZdZ d  f d Zee d fd ZeefdZeefdZ eefdZ eefdZ ee fdZ dS) #build_tf_policy..policy_clsNc r|||| r |||| fd}tj|||||||  r ||||d|_dS)Ncr|||||jri}n |}t|dr|j|dS||_dS)N_extra_action_fetches) _is_towerhasattrr<update)policy obs_space action_spaceconfigextra_action_fetchesr'r#s w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/policy/tf_policy_template.pybefore_loss_init_wrapperzNbuild_tf_policy..policy_cls.__init__..before_loss_init_wrappers#N$$VY fMMM&.&2B.+-((+>+>v+F+F(6#:;;H0778LMMMMM3GF000) rArBrCr2rr"r'r)r*r+existing_inputsexisting_modelr-r)r__init__global_timestep)selfrArBrCrIrHrFr+r*r(r&r'r#r-r"r2r)rr%s rErJz,build_tf_policy..policy_cls.__init__s GivFFF C D)\6BBB H H H H H H  $#)!+!9%"3'= /-+E    " B 4L&AAA$%D rGcRtj||}r||||S|SN)rpostprocess_trajectory)rL sample_batchother_agent_batchesepisoders rErOz:build_tf_policy..policy_cls.postprocess_trajectory s= "8|LLL X%~dL:MwWWW rGcr||j}n|}t|}|jr|j|}|sdS|jdr|S|dSN%_tf_policy_handles_more_than_one_lossr)rC optimizerr explorationget_exploration_optimizer)rL optimizersbasers rErVz-build_tf_policy..policy_cls.optimizers 2)\$ << !^^D11 #J//J T!-GG SS  %tDE %!!"!}$rGct|}t|}r3|jdr |||S||d|dS|||SrT)rrC gradients)rLrVlossrYlossesrZr s rEr\z-build_tf_policy..policy_cls.gradients)s#I..J%%F# @;FGP//j&III0/jmVAYOOO~~dJ???rGcPr |||S|||SrN)build_apply_op)rLrVgrads_and_varsr!rZs rEr`z2build_tf_policy..policy_cls.build_apply_op9s9! L))$ >JJJ**4NKKKrGcNt|fi|jSrN)dictextra_compute_action_fetchesr<)rLrZs rErdz@build_tf_policy..policy_cls.extra_compute_action_fetches@s611$77;?;U rGclrttiifi|S|SrN)rcrextra_compute_grad_fetches)rLrZr$s rErfz>build_tf_policy..policy_cls.extra_compute_grad_fetchesFsI% =.3TT7M7Md7S7STTT66t<<> MySpecialDQNPolicyClass = DQNTFPolicy.with_updates( .. name="MySpecialDQNPolicyClass", .. loss_function=[some_new_loss_function], .. ) )build_tf_policyrc) overridesoriginal_kwargss rE with_updatesz%build_tf_policy..with_updatesTs)"DDo!C!C!C!CDDDrGc&tjdiS)Nrl)r_build_eager_tf_policy)rosrEas_eagerz!build_tf_policy..as_eagergs5HHHHHrG) localscopyrrr r staticmethodrprsrgri)r1r2rrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0rjrprsrZros ` ``````````````` ` @@rErmrmsVhhmmooO ov . .D%)999 @MMMM*)/DD    4JRVWWWW|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=|=T|=|=|=|EEEEE&IIIII+<88J&x00JJ"J rG)2typingrrrrrrr gymnasiumgymray._common.deprecationr r ray.rllib.models.modelv2r "ray.rllib.models.tf.tf_action_distr ray.rllib.policyr"ray.rllib.policy.dynamic_tf_policyrray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.policy.tf_policyrray.rllib.utilsrrray.rllib.utils.annotationsrrray.rllib.utils.frameworkr$ray.rllib.utils.metrics.learner_inforray.rllib.utils.typingrrrtf1tftfvstrSpacespacestypeintrmrlrGrErs5EEEEEEEEEEEEEEEEEE-,,,,,CCCCCC,,,,,,>>>>>>******555555//////22222222========333333BBBBBB } R KOQU   OSRV    #'DH#3 wPPP P $34kB j$z**+ -P!4&2E*E!FGPx 5tCO7L LMNP&-.0OOPP#&9:FVWP ! 4n E~ U !P*&+~6S*_8MMN+P0"(F8T#z/5J+J"KL1P2%XvhS*_8M.M%NO3P4&#)SY0CDdJK5P:&#)SY0CDdJK;P@ SZ%sz'79L Mt S APJ&#)SY0CDdJKKPP SZ%sz'79L Mw V QPZ *d:./z:7M1NNO[P`% Wj*j A *dD$44 5 7 aPl T$Z mPn!)6(C-)@ AoPx /yPPP PPPrG