&`i+ ddlZddlmZmZmZmZmZmZmZm Z ddl Z ddl Z ddlZddlmZddlmZddlmZddlmZddlmZddlmZddlmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)erdd lm*Z*ej+e,Z-e\Z.Z/Z0e d-d e1dedde$de j2de j2de3deegdfdee3fdZ4ede1dee$ee1efee1e%fffdZ5e d.ddde1de1de&dee6d ee7d!ee7d"eed#e7d$ee3de&fd%Z8eddd&e eee1e&fffd'Z9ed(d)*d/d+Z:d,Z;eZi|]\}}|tj|Sr-)r deserialize).0idspecs r; z6parse_policy_specs_from_checkpoint..s6-5RJ "4 ( ()openpickleloadloadsr1items)r=fcheckpoint_dictwrBrCserialized_policy_specsrEs r;"parse_policy_specs_from_checkpointrVos dD  )Q +a..)))))))))))))))  _X.//Ao&MEE/1W:66M/9P9V9V9X9XL , 55s 266policyenv_idagent_idobsreward terminated truncatedinfoexploretimestepc |js Jdt||j|jtj|i} ||| tj<||| tj<||| tj<||| tj <t||| g} || } g} | D]}| |j j || }tjd|}t!|||j j|}|jr||}|j}n|d}| ||j|| S)aRun a connector enabled policy using environment observation. policy_inference manages policy and agent/action connectors, so the user does not have to care about RNN state buffering or extra fetch dictionaries. Note that connectors are intentionally run separately from compute_actions_from_input_dict(), so we can have the option of running per-user connectors on the client side in a server-client deployment. Args: policy: Policy object used in inference. env_id: Environment ID. RLlib builds environments' trajectories internally with connectors based on this, i.e. one trajectory per (env_id, agent_id) tuple. agent_id: Agent ID. RLlib builds agents' trajectories internally with connectors based on this, i.e. one trajectory per (env_id, agent_id) tuple. obs: Environment observation to base the action on. reward: Reward that is potentially used during inference. If not required, may be left empty. Some policies have ViewRequirements that require this. This can be set to zero at the first inference step - for example after calling gmy.Env.reset. terminated: `Terminated` flag that is potentially used during inference. If not required, may be left None. Some policies have ViewRequirements that require this extra information. truncated: `Truncated` flag that is potentially used during inference. If not required, may be left None. Some policies have ViewRequirements that require this extra information. info: Info that is potentially used durin inference. If not required, may be left empty. Some policies have ViewRequirements that require this. explore: Whether to pick an exploitation or exploration action (default: None -> use self.config["explore"]). timestep: The current (sampling) time step. Returns: List of outputs from policy forward pass. zz(local_policy_inference..s QqTrLr)agent_connectors__check_atari_obs_spacein_evalaction_connectorsrNEXT_OBSREWARDS TERMINATEDS TRUNCATEDSINFOSrcompute_actions_from_input_dictdata sample_batchtree map_structurerraw_dictoutputappendon_policy_output)rWrXrYrZr[r\r]r^r_r` input_dictacd_list ac_outputsoutputsac policy_outputaction_connector_dataacdactionss r;local_policy_inferencersd FFEFFC    ##%%% $$&&&&,J *0 ;&'.8 ;*+-6 ;)* (, ;$% vx<<.H/5.E.Eh.O.OJGHH>> G ?  *>>=II 7 Hbg. ! !   # '**+@AACjGG#A&Gw 001FGGGG NrLbatchc d}D]}|dr|dz }dt|D}|tjtjfd|Dtjtj |j dd}|S) aReturns log likelihood for actions in given batch for policy. Computes likelihoods by passing the observations through the current policy's `compute_log_likelihoods()` method Args: batch: The SampleBatch or MultiAgentBatch to calculate action log likelihoods from. This batch/batches must contain OBS and ACTIONS keys. Returns: The probabilities of the actions in the batch, given the observations and the policy. r state_in_c8g|]}d|S)z state_in_{})format)rHis r; z;compute_log_likelihoods_from_input_dict.. s&KKKa-&&q))KKKrLc g|] }| Sr-r-)rHkrs r;rz;compute_log_likelihoods_from_input_dict..s444AuQx444rLactions_in_input_normalizedF)r obs_batch state_batchesprev_action_batchprev_reward_batchactions_normalized) keys startswithrangecompute_log_likelihoodsrACTIONSOBSr1 PREV_ACTIONS PREV_REWARDSr,)rWrnum_state_inputsr state_keyslog_likelihoodss ` r;'compute_log_likelihoods_from_input_dictrs$ ZZ\\"" << $ $ "  ! KK59I3J3JKKKJ"("@"@k)* (4444444))K$<==))K$<==!=,,-JERR #A##O rLz8Policy.from_checkpoint([checkpoint path], [policy IDs]?)T)newerrorcdSNr-)r= policy_idss r;load_policies_from_checkpointrsDrLctdtj|Dr+tdrtddSdSdS)Nc3nK|]0}t|tjr|jtkndV1dS)FN)r/npndarrayshaper )rHos r; z*__check_atari_obs_space..sS   '1BJ&?&?J?""U      rL)warn_about_possibly_non_wrapped_atari_enva(The observation you fed into local_policy_inference() has dimensions (210, 160, 3), which is the standard for atari environments. If RLlib raises an error including a related dimensionality mismatch, you may need to use ray.rllib.env.wrappers.atari_wrappers.wrap_deepmind to wrap you environment.)anyrqflattenrloggerwarning)rZs r;rfrfs   c""     ? @ @  NN#         rL)rNN)NNNNNNr)=loggingtypingrrrrrrr r gymnasiumgymnumpyrrqray.cloudpickle cloudpicklerNray._common.deprecationr ray.rllib.core.rl_moduler ray.rllib.models.preprocessorsr ray.rllib.policy.policyrray.rllib.policy.sample_batchrray.rllib.utils.annotationsrray.rllib.utils.frameworkrray.rllib.utils.typingrrrrrrrray.utilrr getLogger__name__rr2r'tfvstrSpaceintr<rVfloatboolrrrrfvalidate_policy_idr-rLr;rs#                     ......777777::::::......555555333333333333/......  8 $ $} R =A?L?L?Lx.?L.?Ly ?L ) ?L  ?Lhr='89:?L 3-?L?L?L ?LD 6 6 %tCO'T TU666 68 #!% $"ee e ee  e UO e  e~e 4.eesmeeee eP  ";S:J5J0K#KL B JRVWWW   XW &(rL