&`iYdZddlmZmZmZmZddlZddlZ ddl Z ddl m Z m Z mZddlmZddlmZddlmZmZmZddlmZdd lmZdd lmZdd lmZdd lm Z m!Z!dd l"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.e#\Z/Z0Z1e Gdde0re0j2j3j4ne5Z6e GddeZ7GddeZ8GddeZ9dS)a [1] - Attention Is All You Need - Vaswani, Jones, Shazeer, Parmar, Uszkoreit, Gomez, Kaiser - Google Brain/Research, U Toronto - 2017. https://arxiv.org/pdf/1706.03762.pdf [2] - Stabilizing Transformers for Reinforcement Learning - E. Parisotto et al. - DeepMind - 2019. https://arxiv.org/pdf/1910.06764.pdf [3] - Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. Z. Dai, Z. Yang, et al. - Carnegie Mellon U - 2019. https://www.aclweb.org/anthology/P19-1285.pdf )AnyDictOptionalUnionN)BoxDiscrete MultiDiscrete)deprecation_warning)ModelV2)GRUGateRelativeMultiHeadAttentionSkipConnection)RecurrentNetwork) TFModelV2) SampleBatch)ViewRequirement) OldAPIStackoverride) try_import_tf)get_base_struct_from_space)flatten_inputs_to_1d_tensorone_hot)ListModelConfigDict TensorType)log_oncecPeZdZdZ d dededeeffd Zdedefd Z xZ S) PositionwiseFeedforwardzA 2x linear layer with ReLU activation in between described in [1]. Each timestep coming from the attention head will be passed through this layer separately. Nout_dim hidden_dimoutput_activationc Ftjdi|tjj|tjj|_tjj|||_ tdrtddSdS)N activationpositionwise_feedforward_tfz5rllib.models.tf.attention_net.PositionwiseFeedforwardold) super__init__tfkeraslayersDensennrelu _hidden_layer _output_layerrr )selfrr r!kwargs __class__s u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/models/tf/attention_net.pyr*z PositionwiseFeedforward.__init__/s ""6"""X_22 uz3    X_22  13   1 2 2  K        inputsreturnc X~||}||SN)r1r2)r3r8r4outputs r6callzPositionwiseFeedforward.callEs, ##F++!!&)))r7r;) __name__ __module__ __qualname____doc__intrrr*rr= __classcell__r5s@r6rr's,0 $C= ,*:*J********r7rceZdZdZdejjdejjdedede deded ed ed effd Z e e d e dee de de ee ffdZe e deejfdZxZS)TrXLNetz"A TrXL net Model described in [1].observation_space action_space num_outputs model_confignamenum_transformer_units attention_dim num_headshead_dimposition_wise_mlp_dimc tdrtdt|||||||_||_||_| |_|d|_|j d|_ tj j |j|j fd} tj j || } t!|jD]} t#t%||| dd d | }t#t'|| |} tj j d | } tj j |jtj jjd| }tj j| g|g|_d S)arInitializes a TrXLNet object. Args: num_transformer_units: The number of Transformer repeats to use (denoted L in [2]). attention_dim: The input and output dimensions of one Transformer unit. num_heads: The number of attention heads to use in parallel. Denoted as `H` in [3]. head_dim: The dimension of a single(!) attention head within a multi-head attention unit. Denoted as `d` in [3]. position_wise_mlp_dim: The dimension of the hidden layer within the position-wise MLP (after the multi-head attention block within one Transformer unit). This is the size of the first of the two layers within the PositionwiseFeedforward. The second layer always has size=`attention_dim`. trxl_net_tfz%rllib.models.tf.attention_net.TrXLNetr& max_seq_lenrr8shaperKFNrrNrOinput_layernormr!) fan_in_layeraxislogitsr$rK)rr r)r*rLrMrNrOrSrUobs_dimr+r,r-Inputr.rangerr rLayerNormalizationrI activationslinearmodelsModel base_model)r3rGrHrIrJrKrLrMrNrOrPr8E_out_MHA_outr\r5s r6r*zTrXLNet.__init__Os< M " "  ;      |[,   &;"*"  ' 6(.q1 &&#T\2'  %%m44V<<t122 G GA n*)'%$)&* "     GN' 7LMMEHO66B6??FFEE&&  )=)D8'  (///6(CCr7r8stateseq_lensr9c|d}t||fddd|j df}||g}t|d}|dd| df}||gfS)NrrZ)r+concatrSrfrU)r3r8rjrk observationsr\Ts r6 forward_rnnzTrXLNet.forward_rnnsQx yy,!7ay@@TEUDUDWDWAWX ,00 HHV  Q A233 ~%%r7cZtj|j|jftjgSr;)npzerosrSr^float32r3s r6get_initial_statezTrXLNet.get_initial_states% $*DL92:FFGGr7)r>r?r@rAgymspacesSpacerBrstrr*rrrrrqrsndarrayrwrCrDs@r6rFrFKsn,,GD:+GDj&GD GD & GD  GD #GDGDGDGD #GDGDGDGDGDGDRX& &)-j)9&EO& d:& '&&& &"XH4 #3HHH HHHHHr7rFcdeZdZdZdddddddddd ejjd ejjd eed e d e dededededededede ffdZ e edeededeeeffdZe edeejfdZe edefdZxZS)GTrXLNetasA GTrXL net Model described in [2]. This is still in an experimental phase. Can be used as a drop-in replacement for LSTMs in PPO and IMPALA. To use this network as a replacement for an RNN, configure your Algorithm as follows: Examples: >> config["model"]["custom_model"] = GTrXLNet >> config["model"]["max_seq_len"] = 10 >> config["model"]["custom_model_config"] = { >> num_transformer_units=1, >> attention_dim=32, >> num_heads=2, >> memory_inference=100, >> memory_training=50, >> etc.. >> } rm@2 g@)rLrMrNmemory_inferencememory_trainingrOrPinit_gru_gate_biasrGrHrIrJrKrLrMrNrrrOrPrc t||||||_|_|_| _| _| _|d_|j d_ tj j dj fd}fdtjD}tj j j|}|g}tjD](}t#t%j|| dtjjt+| d |d z ||| }t#tj tj j d t3j| tjjft+| d|d z |}||*d_d_|rtj j jdd|_tj j d dd|}j|g}n|g}j_tj |g|z||dd z_j tjD]}tCddjf}tEd|djj|j#d|<tE|dj#d|<dS)aInitializes a GTrXLNet instance. Args: num_transformer_units: The number of Transformer repeats to use (denoted L in [2]). attention_dim: The input and output dimensions of one Transformer unit. num_heads: The number of attention heads to use in parallel. Denoted as `H` in [3]. memory_inference: The number of timesteps to concat (time axis) and feed into the next transformer unit as inference input. The first transformer unit will receive this number of past observations (plus the current one), instead. memory_training: The number of timesteps to concat (time axis) and feed into the next transformer unit as training input (plus the actual input sequence of len=max_seq_len). The first transformer unit will receive this number of past observations (plus the input sequence), instead. head_dim: The dimension of a single(!) attention head within a multi-head attention unit. Denoted as `d` in [3]. position_wise_mlp_dim: The dimension of the hidden layer within the position-wise MLP (after the multi-head attention block within one Transformer unit). This is the size of the first of the two layers within the PositionwiseFeedforward. The second layer always has size=`attention_dim`. init_gru_gate_bias: Initial bias values for the GRU gates (two GRUs per Transformer unit, one after the MHA, one after the position-wise MLP). rSrNr8rTc g|]M}tjjdjftjd|NS)Nz memory_in_{})rUdtyperK)r+r,r-r_rMruformat.0ir3s r6 z%GTrXLNet.__init__..se    HO ! !T/0j#**1-- "     r7TrVzmha_{}rm)rXrK)memoryrYrZ)rr r!zpos_wise_mlp_{}r\r]values)r8outputsgg?rUz state_out_{}-{}:-1)shiftbatch_repeat_valuespace state_in_{}F)rused_for_training)$r)r*rLrMrNrrrOrSrUr^r+r,r-r_r`r.rr r/r0r r Sequentialrarappend_logits _value_outrIre trxl_modelsummaryrrview_requirements)r3rGrHrIrJrKrLrMrNrrrOrPr input_layer memory_insrg memory_outsrri values_outoutsrr5s` r6r*zGTrXLNet.__init__s\  |[,   &;"*" 0.  ' 6(.q1 ho++42FX+VV     4566    %%d&899+FFg t122 & &A n* .'%$(&(ej %%788__QU++   JqM + + +G N##:::CC/$($6'<.0ej  %%788&--a!e44     E"   u % % % %   "8?00 T1DL..qT.QQRWXXJL*-DD7D#1D (..=:-tk#2#>N7N)   !!!t122  Ac$*<)>???E>M%%a((ood&;<<#'#3 ???D "=#7#7#:#: ;@Ou@@@D ">#8#8#;#; < <  r7rjrkr9c |Jt|d}|tj}t|}|d|z}t|td|g|ddgd}|g|z}j@t|ddjg} |d_ |dd} n2t|ddj g} |dd} | fd| DfS)NrrYrmrZrcTg|]$}t|djg%S)rY)r+reshaperM)rmr3s r6rz$GTrXLNet.forward..rs/RRRRZZB(:#;<<RRRr7) r+rUrOBSrrnrrrIrrM) r3 input_dictrjrkBrorUrpall_outoutrs ` r6forwardzGTrXLNet.forwardZs-### HHX  q !!+/2 && !HMzz, B7E!""I:NUV 0W0WXX //<.5"899 < #**WQZ"d.>)?@@C%ajDO!!""+KK**WQZ"d.@)ABBC!!""+KRRRRkRRRRRr7cDfdtjDS)Ncg|]D}tjd|jjESr)r+rtrrrrUrs r6rz.GTrXLNet.get_initial_state..vsQ    HHT+M,@,@,C,CDJP Q Q   r7)r`rLrvs`r6rwzGTrXLNet.get_initial_statets9    4566    r7cDt|jdgS)NrY)r+rrrvs r6value_functionzGTrXLNet.value_function{szz$/B4000r7)r>r?r@rArxryrzrrBrr{floatr*rr rrrrrsr|rwrrCrDs@r6r~r~s:&' "!%'$'OOO:+Oj&Oc] O & O  O #OOOOOO #O"OOOOOOdXgS!%j!1S=GS d:& 'SSSS2X 4 #3     Xg1 11111111r7r~c TeZdZdZdejjdejjdedede f fd Z e e de e efd eed ed eeeffd Ze ed efd Ze ed eeejeeffdZxZS)AttentionWrapperzGGTrXL wrapper serving as interface for ModelV2s that set use_attention. obs_spacerHrIrJrKc4tdrtdt||d|||d|_|d|_t |j|_d|_ tj |jD]}t|tr|xj |jz c_ -t|tr(|xj t!j|jz c_ j|j5|xj t)t!j|jz c_ |xj t)t-|z c_ |jr|xj|j|j zz c_|jr|xj|jz c_|}|d|_|jNt2jt9dt9d |jft j }n|}t=||d|d |d |j|d |d|d|d|d|d |_t@j!j"#|jjf} ||_tAj!j"$|jd| } t@j!j%&| g| g|_'tAj!j"$dd| } t@j!j%&| g| g|_(|jj)|_)|j*|j)d_+|jrKtYtZj.|jd/|j|j)tZj0<|jrGtYtZj1d/|j|j)tZj2<dSdS)N attention_wrapper_tf_deprecationz2ray.rllib.models.tf.attention_net.AttentionWrapperr&attention_use_n_prev_actionsattention_use_n_prev_rewardsrrMz-infinf)rUrgtrxlattention_num_transformer_unitsattention_num_headsattention_head_dimattention_memory_inferenceattention_memory_trainingattention_position_wise_mlp_dimattention_init_gru_gate_bias)rLrMrNrOrrrPrrr#rmobsr)rr)r)3rr r)r*use_n_prev_actionsuse_n_prev_rewardsrrHaction_space_struct action_dimtreeflatten isinstancernr rssumnvecrUrBprodlenrIrMrxryrrrur~rr+r,r-r_r.rdre_logits_branch _value_branchrrrrrACTIONSr PREV_ACTIONSREWARDS PREV_REWARDS) r3rrHrIrJrKrcfgin_spaceinput_rr5s r6r*zAttentionWrapper.__init__s 6 7 7  H     L$ dKKK"./M"N"./M"N#=d>O#P#P \$":;; 3 3E%** 357*E=11 326%*#5#55(3rwu{';';#<#<<3s5zz??2  " J    7$/ I I    " 8    7 7   1   'z~~f uU||D4D3Fbj&HH!H      "%&G"H,/0-. !=> ;<"%&G"H"#AB    $&&dj.D-F&GG'ho##D$4#FFvNN ho33VHseDDho##A$#77??X_22F8cUCC!%!=.2nu%+  " ?N#'ood&=>>@@@D ";#; <  " ?N#8??4;R+S+S@@@D ";#; < < <  r7rrjrkr9c |J||gd\}}g}|jr|tj}|jdrit ||jd}t|t |ddg}| |ntt|j trKt|jD]4} | t|dd| f|j 5nt|j t rtd|j|j j dD]m} | tt|dd| | |j j dzftj|j nn`| tt|tjd|j|jzg|jrh| tt|tjtjd|jg|r t|g|zd}|x|d <|d <||||\|_} ||j} | | fS) N_disable_action_flatteningT) spaces_struct time_axisrrY)rrmrZobs_flatr)_wrapped_forwardrrrrJrrr+rrUrrrHrr`rr castrurrrrnr _featuresr) r3rrjrk wrapped_outrhprev_a_rprev_n_actionsflatrr model_outs r6rzAttentionWrapper.forwards###..z2tDD Q  "- ' (@AN !=>( 2""&":" zz$$):B(?@@%%%% d/::"4#:;; #N111a4$8$:KLL  1=AA"42D4E4KA4N  !# "$2()1q43D3J13M/M+M(M%&%'J !"!" '+&7     OO GGNBJ??!84?!JK  "  OO GGJ{'?@"*MM01     F))[MH$<1)EEK6A@ :E!2&*jjUH&M&M# ''77 +%%r7c|j Jdt||jdgS)NzMust call forward() first!rY)rr+rrrvs r6rzAttentionWrapper.value_function4s?~))+G)))zz$,,T^<.;sQ    HTZ1-2F2Fq2I2IJPV W W   r7)r`rrLrvs`r6rwz"AttentionWrapper.get_initial_state9s;    4:;<<    r7)r>r?r@rArxryrzrBrr{r*rrrrrrr rrrsr|rwrCrDs@r6rrs{QQ^:#^j&^ ^ & ^  ^^^^^^@XN&j)N&JN& N& d:& ' N&N&N& N&`XgD DDDDXg 5bj)94 ;K)K#L        r7r):rAtypingrrrr gymnasiumrxnumpyrsrgymnasium.spacesrrr ray._common.deprecationr ray.rllib.models.modelv2r ray.rllib.models.tf.layersr r r!ray.rllib.models.tf.recurrent_netrray.rllib.models.tf.tf_modelv2rray.rllib.policy.sample_batchr!ray.rllib.policy.view_requirementrray.rllib.utils.annotationsrrray.rllib.utils.frameworkr"ray.rllib.utils.spaces.space_utilsrray.rllib.utils.tf_utilsrrray.rllib.utils.typingrrrray.utilrtf1r+tfvr,r-LayerobjectrrFr~rr(r7r6rs  .----------- 9999999999777777,,,,,, ?>>>>>444444555555==============333333IIIIIIIIIIIIIIDDDDDDDDDD} R  * * * * *rEbho33v * *  *F cHcHcHcHcHcHcH cHLK1K1K1K1K1K1K1K1\~ ~ ~ ~ ~ y~ ~ ~ ~ ~ r7