&`i>dZddlZddlZddlmZmZmZmZmZm Z ddl Z ddl m Z ddlmZddlmZmZmZmZerddlmZejeZeGdd ZeGd d ejZeGd d eZeGddeZeGddejZdS)zIThis file defines base types and common structures for RLlib connectors. N) TYPE_CHECKINGAnyDictListTupleUnion)ViewRequirement) OldAPIStack)ActionConnectorDataTypeAgentConnectorDataTypeAlgorithmConfigDict TensorType)Policyc eZdZdZ ddedeedejdejde e e fd e f d Z eddZdS)ConnectorContextzData bits that may be needed for running connectors. Note(jungong) : we need to be really careful with the data fields here. E.g., everything needs to be serializable, in case we need to fetch them in a remote setting. NFconfiginitial_statesobservation_space action_spaceview_requirementsis_policy_recurrentcb|pi|_|pg|_||_||_||_||_dS)aConstruct a ConnectorContext instance. Args: initial_states: States that are used for constructing the initial input dict for RNN models. [] if a model is not recurrent. action_space_struct: a policy's action space, in python data format. E.g., python dict instead of DictSpace, python tuple instead of TupleSpace. Nrrrrrr)selfrrrrrrs r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/connectors/connector.py__init__zConnectorContext.__init__%sB$l ,2!2(!2#6   policyrreturnc t|j||j|j|j|S)zBuild ConnectorContext from a given policy. Args: policy: Policy Returns: A ConnectorContext instance. r)rrget_initial_staterrr is_recurrent)rs r from_policyzConnectorContext.from_policy>sP =!3355$6,$6 & 3 3 5 5     r)NNNNNF)rrrr)__name__ __module__ __qualname____doc__r rrgymSpacerstrr boolr staticmethodr#rrrrs'++/'+"&8<$)77#7Z(79 7 i 7  _ 45 7"77772   \   rrcxeZdZdZdefdZdZdZddefdZ d e e e ffd Z eded e d dfd Zd S) ConnectoraConnector base class. A connector is a step of transformation, of either envrionment data before they get to a policy, or policy output before it is sent back to the environment. Connectors may be training-aware, for example, behave slightly differently during training and inference. All connectors are required to be serializable and implement to_state(). ctxcd|_dSNT _is_training)rr0s rrzConnector.__init___s rcd|_dSr2r3rs r in_trainingzConnector.in_trainingcs rcd|_dS)NFr3r6s rin_evalzConnector.in_evalfs!rr indentationc&d|z|jjzS)N ) __class__r$rr:s r__str__zConnector.__str__is[ 4>#:::rrctS)aSerialize a connector into a JSON serializable Tuple. to_state is required, so that all Connectors are serializable. Returns: A tuple of connector's name and its serialized states. String should match the name used to register the connector, while state can be any single data structure that contains the serialized state of the connector. If a connector is stateless, state can simply be None. NotImplementedErrorr6s rto_statezConnector.to_statels #"rparamsctS)aLDe-serialize a JSON params back into a Connector. from_state is required, so that all Connectors are serializable. Args: ctx: Context for constructing this connector. params: Serialized states of the connector to be recovered. Returns: De-serialized connector. rA)r0rDs r from_statezConnector.from_state{s #"rNr)r$r%r&r'rrr7r9intr?rr*rrCr,rFr-rrr/r/Rs  !,!!!!!!!""";;3;;;; #%S/ # # # # #( ## #+ # # #\ # # #rr/cbeZdZdZdefdZdefdZdee dee fdZ d e de fd Z d S) AgentConnectora Connector connecting user environments to RLlib policies. An agent connector transforms a list of agent data in AgentConnectorDataType format into a new list in the same AgentConnectorDataTypes format. The input API is designed so agent connectors can have access to all the agents assigned to a particular policy. AgentConnectorDataTypes can be used to specify arbitrary type of env data, Example: Represent a list of agent data from one env step() call. .. testcode:: import numpy as np ac = AgentConnectorDataType( env_id="env_1", agent_id=None, data={ "agent_1": np.array([1, 2, 3]), "agent_2": np.array([4, 5, 6]), } ) Or a single agent data ready to be preprocessed. .. testcode:: ac = AgentConnectorDataType( env_id="env_1", agent_id="agent_1", data=np.array([1, 2, 3]), ) We can also adapt a simple stateless function into an agent connector by using register_lambda_agent_connector: .. testcode:: import numpy as np from ray.rllib.connectors.agent.lambdas import ( register_lambda_agent_connector ) TimesTwoAgentConnector = register_lambda_agent_connector( "TimesTwoAgentConnector", lambda data: data * 2 ) # More complicated agent connectors can be implemented by extending this # AgentConnector class: class FrameSkippingAgentConnector(AgentConnector): def __init__(self, n): self._n = n self._frame_count = default_dict(str, default_dict(str, int)) def reset(self, env_id: str): del self._frame_count[env_id] def __call__( self, ac_data: List[AgentConnectorDataType] ) -> List[AgentConnectorDataType]: ret = [] for d in ac_data: assert d.env_id and d.agent_id, "Skipping works per agent!" count = self._frame_count[ac_data.env_id][ac_data.agent_id] self._frame_count[ac_data.env_id][ac_data.agent_id] = ( count + 1 ) if count % self._n == 0: ret.append(d) return ret As shown, an agent connector may choose to emit an empty list to stop input observations from being further prosessed. env_idcdS)zReset connector state for a specific environment. For example, at the end of an episode. Args: env_id: required. ID of a user environment. Required. Nr-)rrKs rresetzAgentConnector.resets  routputcdS)aCallback on agent connector of policy output. This is useful for certain connectors, for example RNN state buffering, where the agent connect needs to be aware of the output of a policy forward pass. Args: ctx: Context for running this connector call. output: Env and agent IDs, plus data output from policy forward pass. Nr-)rrNs ron_policy_outputzAgentConnector.on_policy_outputs  racd_listrclt|ttfs Jdfd|DS)aTransform a list of data items from env before they reach policy. Args: ac_data: List of env and agent IDs, plus arbitrary data items from an environment or upstream agent connectors. Returns: A list of transformed data items in AgentConnectorDataType format. The shape of a returned list does not have to match that of the input list. An AgentConnector may choose to derive multiple outputs for a single piece of input data, for example multi-agent obs -> multiple single agent obs. Agent connectors may also choose to skip emitting certain inputs, useful for connectors such as frame skipping. z=Input to agent connectors are list of AgentConnectorDataType.c:g|]}|Sr- transform).0drs r z+AgentConnector.__call__.. s%444aq!!444r) isinstancelisttuple)rrQs` r__call__zAgentConnector.__call__sW" tUm   K K J K K 544484444rac_datact)aTransform a single agent connector data item. Args: data: Env and agent IDs, plus arbitrary data item from a single agent of an environment. Returns: A transformed piece of agent connector data. rArr]s rrUzAgentConnector.transform s "!rN) r$r%r&r'r*rMr rPrr r\rUr-rrrJrJsMM^ C      '>     5345 $ %5555. "!7 " Args: name: name of the connector to be removed. rzRemoved connector z from .z*Trying to remove a non-existent connector N) enumeraterfr=r$loggerinfowarning)rrmidxirks rremovezConnectorPipeline.remove^s do..  DAq{#t++, !88$ KKSTSS9PSSS T T T T T NNOOOO P P P P Pr connectorc 8d}t|jD]\}}|jj|krn|dkrt d||j||t d|jjd|d|jjddS) zInsert a new connector before connector Args: name: name of the connector before which a new connector will get inserted. connector: a new connector to be inserted. rorCan not find connector Inserted z before  to rpNrqrfr=r$ ValueErrorinsertrrrsrrmrxrurks r insert_beforezConnectorPipeline.insert_beforeos00  FC{#t++, 77=t==>> > sI...  - +4 - -d - -.) - - -     rc >d}t|jD]\}}|jj|krn|dkrt d||j|dz|t d|jjd|d|jjdd S) zInsert a new connector after connector Args: name: name of the connector after which a new connector will get inserted. connector: a new connector to be inserted. rorrzr{z after r|rpNr}rs r insert_afterzConnectorPipeline.insert_afters00  FC{#t++, 77=t==>> > sQw 222  - +4 - -T - -.) - - -     rc|jd|td|jjd|jjddS)zAppend a new connector at the beginning of a connector pipeline. Args: connector: a new connector to be appended. rAdded z to the beginning of rpN)rfrrrrsr=r$rrxs rprependzConnectorPipeline.prependsj q),,,  *Y(1 * *~& * * *     rc|j|td|jjd|jjddS)zAppend a new connector at the end of a connector pipeline. Args: connector: a new connector to be appended. rz to the end of rpN)rfappendrrrsr=r$rs rrzConnectorPipeline.appendsh y)))  *Y(1 * *~& * * *     rrr:cxddz|jjzgfd|jDzS)N r<c@g|]}|dzS))r?)rVrkr:s rrXz-ConnectorPipeline.__str__..s)CCCaqyyq))CCCr)joinr=r$rfr>s `rr?zConnectorPipeline.__str__sMyy ; !8 8 9CCCC4?CCC D   rkeyct|tst|trtdt|tr|j|gSt|t r8g}|jD],}t|j|r| |-|Std t |g}|jD]'}|jj |kr| |(|S)aReturns a list of connectors that fit 'key'. If key is a number n, we return a list with the nth element of this pipeline. If key is a Connector class or a string matching the class name of a Connector class, we return a list of all connectors in this pipeline matching the specified class. Args: key: The key to index by Returns: The Connector at index `key`. z8Slicing of ConnectorPipeline is currently not supported.z*Indexing by {} is currently not supported.) rYr*slicerBrHrftype issubclassr=rformatr$)rrresultsrks r __getitem__zConnectorPipeline.__getitem__s#s## #u%% )NC%% ,--C&& **A!!+s33*q))))@GGS RR " "A{#s**q!!!rNrG)r$r%r&r'rrr/rr7r9r*rwrrrrrHr?rrrr-rrrereOs?GG%,%$y/%%%%Q3QQQQ" # )    *      *               3    %uS#t^4%%%%%%rre) r'abcloggingtypingrrrrrr gymnasiumr(!ray.rllib.policy.view_requirementr ray.rllib.utils.annotationsr ray.rllib.utils.typingr r r rray.rllib.policy.policyr getLoggerr$rrrABCr/rJrarer-rrrs ????????????????======333333/......  8 $ $ 5 5 5 5 5 5 5  5 p 6#6#6#6#6#6#6# 6#r H"H"H"H"H"YH"H" H"V 3"3"3"3"3"i3"3" 3"l NNNNNNN NNNr