&`iF ddlZddlmZddlmZmZmZmZddlZ ddl Z ddl m Z ddlmZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&e\Z'Z(Z)e \Z*Z+eGddZ,edede-defdZ.ee(fdede dede&fdZ/iZ0ee(fdede dede&fdZ1dS)N) OrderedDict)AnyDictListUnion)Space) Deprecated)RepeatedValuesPreprocessorget_preprocessor)RepeatedValues) SampleBatch)ViewRequirementNullContextManager) OldAPIStack) TensorType try_import_tftry_import_torch)Repeated)ModelConfigDictModelInputDictTensorStructTypec eZdZdZdedededededef dZd e e fd Z d e ee fd e e d e d e e e ffdZ d e fdZde de ee fd ee e e ffdZd e ee ffdZ d d eeefd e ed e d e e e ffdZd e fdZd ejfdZ d!ded ee e e ee fffdZ d!ded ee e e ee fffdZd efdZeddZdS)"ModelV2aDefines an abstract neural network model for use with RLlib. Custom models should extend either TFModelV2 or TorchModelV2 instead of this class directly. Data flow: obs -> forward() -> model_out \-> value_function() -> V(s) obs_space action_space num_outputs model_configname frameworkc||_||_||_||_|pd|_||_d|_|jd|_tj td|ji|_ dS)aInitializes a ModelV2 instance. This method should create any variables used by the model. Args: obs_space: Observation space of the target gym env. This may have an `original_space` attribute that specifies how to unflatten the tensor into a ragged tensor. action_space: Action space of the target gym env. num_outputs: Number of output units of the model. model_config: Config for the model, documented in ModelCatalog. name: Name (scope) for the model. framework: Either "tf" or "torch". default_modelN _time_majorr)shiftspace) rrrrrr _last_outputget time_majorr OBSrview_requirements)selfrrrrrr s l/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/models/modelv2.py__init__zModelV2.__init__$s~4!*#/ +-90 ' +// >> O_1DNKKK" returncgS)a~Get the initial recurrent state values for the model. Returns: List of np.array (for tf) or Tensor (for torch) objects containing the initial hidden state of an RNN, if applicable. .. testcode:: :skipif: True import numpy as np from ray.rllib.models.modelv2 import ModelV2 class MyModel(ModelV2): # ... def get_initial_state(self): return [ np.zeros(self.cell_size, np.float32), np.zeros(self.cell_size, np.float32), ] r+s r,get_initial_statezModelV2.get_initial_stateKs ( r. input_dictstateseq_lensct)aVCall the model with the given input tensors and state. Any complex observations (dicts, tuples, etc.) will be unpacked by __call__ before being passed to forward(). To access the flattened observation tensor, refer to input_dict["obs_flat"]. This method can be called any number of times. In eager execution, each call to forward() will eagerly evaluate the model. In symbolic execution, each call to forward creates a computation graph that operates over the variables of this model (i.e., shares weights). Custom models should override this instead of __call__. Args: input_dict: dictionary of input tensors, including "obs", "obs_flat", "prev_action", "prev_reward", "is_training", "eps_id", "agent_id", "infos", and "t". state: list of state tensors with sizes matching those returned by get_initial_state + the batch dimension seq_lens: 1d tensor holding input sequence lengths Returns: A tuple consisting of the model output tensor of size [BATCH, num_outputs] and the list of new RNN state(s) if any. .. testcode:: :skipif: True import numpy as np from ray.rllib.models.modelv2 import ModelV2 class MyModel(ModelV2): # ... def forward(self, input_dict, state, seq_lens): model_out, self._value_out = self.base_model( input_dict["obs"]) return model_out, state NotImplementedError)r+r4r5r6s r,forwardzModelV2.forwardas V"!r.ct)aeReturns the value function output for the most recent forward pass. Note that a `forward` call has to be performed first, before this methods can return anything and thus that calling this method does not cause an extra forward pass through the network. Returns: Value estimate tensor of shape [BATCH]. r8r2s r,value_functionzModelV2.value_functions "!r. policy_loss loss_inputsc|S)aOverride to customize the loss function used to optimize this model. This can be used to incorporate self-supervised losses (by defining a loss over existing input and output tensors of this model), and supervised losses (by defining losses over a variable-sharing copy of this model's layers). You can find an runnable example in examples/custom_loss.py. Args: policy_loss: List of or single policy loss(es) from the policy. loss_inputs: map of input placeholders for rollout data. Returns: List of or scalar tensor for the customized loss(es) for this model. r1)r+r=r>s r, custom_losszModelV2.custom_losss (r.ciS)aOverride to return custom metrics from your model. The stats will be reported as part of the learner stats, i.e., info.learner.[policy_id, e.g. "default_policy"].model.key1=metric1 Returns: The custom metrics for this model. r1r2s r,metricszModelV2.metricss  r.Ncvt|tr|d}n|}|seg}d}d||vrJ||d||dz }d||vJ||tj}|jdr |d|d <nt|d|j |j |d< t|dj d krt|d|j |d <n |d|d <n#t$r|d|d <YnwxYw|5|||pg|}dddn #1swxYwYt|tr,|jd hz |_|j|_|jd hz |_t|t(st|t*rt|d kr"t-d ||\}}t|t(s"t-d |||_|t|dkr|n|pgfS) aBCall the model with the given input tensors and state. This is the method used by RLlib to execute the forward pass. It calls forward() internally after unpacking nested observation tensors. Custom models should override forward() instead of __call__. Args: input_dict: Dictionary of input tensors. state: list of state tensors with sizes matching those returned by get_initial_state + the batch dimension seq_lens: 1D tensor holding input sequence lengths. Returns: A tuple consisting of the model output tensor of size [BATCH, output_spec.size] or a list of tensors corresponding to output_spec.shape_list, and a list of state tensors of [BATCH, state_size_i] if any. T)shallowrz state_in_{}N_disable_preprocessor_apiobsobs_flatz@forward() must return a tuple of (output, state) tensors, got {}zState output is not a list: {}) isinstancer copyformatappendr'SEQ_LENSrrestore_original_dimensionsrr lenshapeflattenAttributeErrorcontextr: accessed_keys deleted_keys added_keyslisttuple ValueErrorr&) r+r4r5r6restorediresoutputs state_outs r,__call__zModelV2.__call__s,@ j+ . . )!t44HH!((H EA&&q))Z77 Z (<(4>HUO 9z%(.//!33+2:e3Ddn+U+UHZ((+5e+.get_values6923''#HW_27Nr.c&g|] }|Sr1r1).0rrs r, z_unpack_obs..s!???1))A,,???r.N.r).rrE)lengthsmax_len)rJgymspacesrTuplerrXrYdictid_cacher rPrQrZrLr} preprocessorszipsizerMrreshaperitemsr child_preprocessorr child_spacer _obs_space)rGr%rprepoffset batch_dimsupr obs_slicek child_sizerwith_repeat_dimrs @r,rr|sb%#*/3:+;XFGGO ucj. / / JsT5M4R4R  ucjo . . 3=c43H3H J e99  "U))$DD*#E**511D6{{S  $(r%yy! sy>>A  2$*Q-!?!?CJJJqM39   ?? # # #@??? #2#???JJcin--J eSZ- . .( Wt)**c%,.?.????"BBU\""B#???AD. ==  1Vfqvo%= => !& !)))Z$qw--5OPP"+ sz / / Wt)**c%,.?.????"BBU\""B#??? A !3U\5G5G5I5IJJ   6AqVfqvo%= => !& "%%id17mm1KLL'! d$>?? E E E E?05J&kG'//CG jEM:+FFOOU->)TTTA!!Wdo>UVVV V r.)2rx collectionsrtypingrrrr gymnasiumrrrgymnasium.spacesrray._common.deprecationr ray.rllib.models.preprocessorsr r ray.rllib.models.repeated_valuesr ray.rllib.policy.sample_batchr !ray.rllib.policy.view_requirementrray.rllib.utilsrray.rllib.utils.annotationsrray.rllib.utils.frameworkrrrray.rllib.utils.spaces.repeatedrray.rllib.utils.typingrrrrr}tfvr~_rrwrRrOrrr1r.r,rs######))))))))))))""""""......WWWWWWWW;;;;;;555555======......333333QQQQQQQQQQ444444TTTTTTTTTT} R    q m m m m m m m  m `  888 888 8 8:AA A %A25AAAA AF  @B[[Z[[#[GW[[[ [[[r.