&`i7<ddlmZddlmZmZddlZddlZddlm Z ddl m Z ddl m Z mZe deejefdefd Ze d Ze Gd d ejZe Gd dejZe GddejZe GddejZe GddejZe GddejZe GddejZe GddejZe GddejZe GddejZ e Z!e d*dejd"e"d#e"d$ee"d%edejf d&Z#e d+d)Z$dS),)deque)OptionalUnionN)spaces) PublicAPI)resizergb2grayenvreturnc"t|tsQt|jdr+|jjt |jjdkrdSdt|vS|dp|dS)aReturns, whether a given env object or env descriptor (str) is an Atari env. Args: env: The gym.Env object or a string descriptor of the env (for example, "ale_py:ALE/Pong-v5"). Returns: Whether `env` is an Atari environment. shapeNFz AtariEnvr8r?r9r:r;s rr=zFireResetEnv.resetms     +/8==+;+;(Q Iq  % % DHN $ $V $ $ $.2hmmA.>.>+Q It  % % DHN $ $V $ $ $Dyrc6|j|Sr!r r4r%acs rr4zFireResetEnv.stepwx}}R   rNr*r+r,r#r=r4r-rrrBrBcsA===!!!!!rrBc.eZdZdZddddZdZdZdS) FrameStackctj||||_t g||_|jj}tj tj |jj |dtj |jj |d|d|d|d|zf|jj|_dS) zStack k last frames.)maxlen)repeatsaxisrrDrlowhighr dtypeN)rrr#krframesrr rBoxr'repeatrWrXrY)r%r rZshps rr#zFrameStack.__init__}s T3'''Bq))) #)!' #/3QRHHH305qrJJJq63q63q6A:.'- " " " rNseedoptionsc|j||\}}t|jD]}|j|||fS)Nr_)r r=rangerZr[append_get_ob)r%r`raobinfosr?s rr=zFrameStack.resets`HNNgN>> Etv # #A K  r " " " "||~~u$$rc|j|\}}}}}|j||||||fSr!)r r4r[rdre)r%r7rfr)r9r:r;s rr4zFrameStack.stepsO26(--2G2G/FJ 4 2||~~vz9dBBrcvt|j|jksJtj|jdS)NrrU)rr[rZr' concatenater%s rrezFrameStack._get_obs64;46))))~dk2222r)r*r+r,r#r=r4rer-rrrPrP{sd    !$%%%%% CCC 33333rrPceZdZdZdZdS)FrameStackTrajectoryViewctj|||jj}|ddksJt jdd|d|df|jj|_dS)z4No stacking. Trajectory View API takes care of this.rrDrrVN)rrr#rr rr\rY)r%r r^s rr#z!FrameStackTrajectoryView.__init__sq T3'''#)1v{{{{!'CFCF#33;P;V" " " rc.tj|dS)NrSrj)r'squeezer% observations rrtz$FrameStackTrajectoryView.observationsz+B////rN)r*r+r,r#rtr-rrrnrns2   00000rrnc"eZdZddZdZdZdS) MaxAndSkipEnvctj||tjd|jjz|jj|_||_ dS)z!Return only every `skip`-th frame)r)rYN) rrr#r'zerosrr rY _obs_buffer_skip)r%r skips rr#zMaxAndSkipEnv.__init__sV T3'''8 3(. .c6K6Q    rc:d}dx}x}}t|jD]]}|j|\}}}}}||jdz kr ||jd<||jdz kr ||jd<||z }|s|rn^|jd} | ||||fS)z:Repeat action, sum reward, and max over last observations.gNrrrDrj)rcr{r r4rzmax) r%r7 total_rewardr9r:r;ir8r) max_frames rr4zMaxAndSkipEnv.steps (,, ,Ytz""  A7;x}}V7L7L 4CYDJN""&) #DJN""&) # F "L Y  $((a(00 , ItCCrc &|jjdi|S)Nr-)r r=)r%r>s rr=zMaxAndSkipEnv.resetstx~'''''rN)rwr@r-rrrvrvsIDDD&(((((rrvc:eZdZd dZdZdZdZdZdZdZ dS) MonitorEnvNctj||d|_d|_d|_g|_g|_d|_d|_ dS)z4Record episodes stats prior to EpisodicLifeEnv, etc.Nr) rrr#_current_reward _num_steps _total_steps_episode_rewards_episode_lengths _num_episodes _num_returnedr$s rr#zMonitorEnv.__init__sW T3'''#  " "rc :|jjdi|\}}|jt|j|_|jN|j|j|j|j|xj dz c_ d|_d|_||fS)NrDrr-) r r=rsumrrrrdrr)r%r>r8r;s rr=zMonitorEnv.resets"DHN,,V,, T   $ #D$9 : :D    +  ! ( ()= > > >  ! ( ( 9 9 9   ! #   Dyrc|j|\}}}}}|xj|z c_|xjdz c_|xjdz c_|||||fS)NrD)r r4rrr)r%r7r8rewr9r:r;s rr4zMonitorEnv.stepsi04 f0E0E-S*i # 1 QCY44rc|jSr!)rrls rget_episode_rewardszMonitorEnv.get_episode_rewards $$rc|jSr!)rrls rget_episode_lengthszMonitorEnv.get_episode_lengthsrrc|jSr!)rrls rget_total_stepszMonitorEnv.get_total_stepss   rc#Kt|jt|jD]}|j||j|fVt|j|_dSr!)rcrrrr)r%rs rnext_episode_resultszMonitorEnv.next_episode_resultsslt)3t/D+E+EFF G GA(+T-B1-EF F F F F !677rr!) r*r+r,r#r=r4rrrrr-rrrrs     555%%%%%%!!!88888rrc"eZdZddZdZdZdS) NoopResetEnvctj||||_d|_d|_|jddksJdS)zsSample initial states by taking random number of no-ops on reset. No-op is assumed to be action 0. NrNOOP)rrr#noop_maxoverride_num_noops noop_actionr5rG)r%r rs rr#zNoopResetEnv.__init__s] T3'''  "&}002215??????rc |jjdi||j|j}nb |jjd|jdz}n8#t$r+|jjd|jdz}YnwxYw|dksJd}t|D]@}|j |j \}}}}}|s|r|jjdi|\}}A||fS)z7Do no-op action for a number of steps in [1, noop_max].NrDrr-) r r=rr5 np_randomintegersrAttributeErrorrandintrcr4r)r%r>noopsr8r?r9r:r;s rr=zNoopResetEnv.reset s      " .+EE O099!T]Q=NOO! O O O088DMA __class__s rr#zNormalizedImageEnv.__init__%sV$)&)))!$  (.* "0" " rcL|tjdz dz S)Ng`@r)astyper'rrss rrtzNormalizedImageEnv.observation0s"""2:..6#==r)r*r+r,r#rt __classcell__)rs@rrr#sG     >>>>>>>rrc"eZdZddefdZdZdS)GrayScaleAndResizeT grayscalectj||||_||_||_t jdd|j|j|rdndftj |_ dS)z.Warp frames to the specified size (dim x dim).rrprDrFrVN) rObservationWrapperr#widthheightrrr\r'uint8r)r%r dimrs rr#zGrayScaleAndResize.__init__6sq ''c222  "!'; ,AAAB( " " " rc|jr:t|}t||j|j}|dddddfSt||j|jS)N)rr)rr rrr)r%frames rrtzGrayScaleAndResize.observationCsb > GUOOE5DJGGGEAAAt$ $% 4:FFF FrN)T)r*r+r,boolr#rtr-rrrr4sH   D     GGGGGrr@rwTr frameskip framestackrctj|d}t|||}t |}|dkr|jJt ||}t|d}t|}d |j vrt|}|rt|| }|S) aeWraps `env` for new-API-stack-friendly RLlib Atari experiments. Note that we assume reward clipping is done outside the wrapper. Args: env: The env object to wrap. dim: Dimension to resize observations to (dim x dim). frameskip: Whether to skip n frames and max over them (keep brightest pixels). framestack: Whether to stack the last n (grayscaled) frames. Note that this step happens after(!) a possible frameskip step, meaning that if frameskip=4 and framestack=2, we would perform the following over this trajectory: actual env timesteps: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -> ... frameskip: ( max ) ( max ) ( max ) ( max ) framestack: ( stack ) (stack ) Returns: The wrapped gym.Env. i)max_episode_steps)rrrDNr|rrrE)rZ) rwrappers TimeLimit WarpFramerspecrvrr/r5rGrBrP)r rrrrs rwrap_atari_for_new_api_stackrOs< ,  ? ?C CSI 6 6 6C S ! !C1}}x###Ci000 sR ( ( (C #  C 2244443, +++ JrTFc8t|}t|d}|j|durt|d}t |}d|jvrt|}t||}|durt|d}|S)a Configure environment for DeepMind-style Atari. Note that we assume reward clipping is done outside the wrapper. Args: env: The env object to wrap. dim: Dimension to resize observations to (dim x dim). framestack: Whether to framestack observations. rrNTrwrrE) rrrrvr/r5rGrBrrP)r rr noframeskips r wrap_deepmindrs S//C sR ( ( (C x t 3 3Ca((( #  C 2244443 C  CTa   Jr)rrwNT)rTF)% collectionsrtypingrr gymnasiumrnumpyr'rray.rllib.utils.annotationsrray.rllib.utils.imagesrr Envrrrrr"rrr/rBrPrrnrvrrrrrintrrr-rrrs""""""""11111133333333 C% %C$CCC C2      C%  """""ck"" "J !!!!!3;!! !. 3333333 3<  0 0 0 0 0s5 0 0  0 (((((CK(( (B /8/8/8/8/8/8/8 /8d !!!!!!!!!!3;!!!! !!H  > > > > >/ > >  >  GGGGG/GG G.    $ 22 2 22 2  2 W222 2j  r