&`i8NddlmZddlmZddlmZmZddlZddl Z ddl m Z m Z ddl mZddlmZddlmZmZdd lmZmZmZmZe\ZZZe\ZZd Zd Zd Z d Z!eeddd=de"de"dej#fdZ$eedd d>deej#dee%dej#fdZ&ed?dede%defdZ'e d@dej#dej#deej#dee(dej#f d Z)e dAd!ed"eed#e%d$e%def d%Z*ed&Z+edBdej#d(e,dej#fd)Z-edej#dej#fd*Z.e dCdej#deej#d+eej#de%d,e,f d-Z/edd'd.ej0fdeee"fd/e"d0e,d1e,d2e1dej#f d3Z2eee"fd4Z3edDdej#d5e,dej#fd6Z4edEdej#d7e%dej#fd8Z5e dFdeej#e6fd:e"d;ee,dej#fd<Z7dS)G) OrderedDict)MappingProxyType)ListOptionalN)Discrete MultiDiscrete) Deprecated) PublicAPI) try_import_tftry_import_torch) SpaceStructTensorStructType TensorTypeUniongư>iz)RLlib itself has no use for this anymore.F)helperror@sizealignreturnc||jz}tj||dz ztj}|jj|z}|dkrdn||z }|dkr)|||dzdd|}n ||||z|}t||ksJt||jj|zdksJ|jj|S)aReturns an array of a given size that is 64-byte aligned. The returned array can be efficiently copied into GPU memory by TensorFlow. Args: size: The size (total number of items) of the array. For example, array([[0.0, 1.0], [2.0, 3.0]]) would have size=4. dtype: The numpy dtype of the array. align: The alignment to use. Returns: A np.ndarray with the given specifications. dtyper)itemsizenpemptyuint8ctypesdataviewlen)rrrnr data_alignoffsetoutputs i/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/utils/numpy.py aligned_arrayr*s& u~A HQ%!)_BH 5 5 5E"U*J//QQ (:FAvvv *+AaC055e<<v *+0077 v;;$   F    =  % * * *FM,> * * * Mitems time_majorct|dkrgSt|dkr|dSt|dtjr|djtjtjtjfvrO|dj}ttd|D|}||durEtd|D}|dj d|f|dj ddz}nwtd|D}||dj df|dj ddz}n2td |D}|f|dj ddz}| |}|j j d zdksJ|j j tj|||rdnd |Stj||rdnd S) a Concatenate arrays, ensuring the output is 64-byte aligned. We only align float arrays; other arrays are concatenated as normal. This should be used instead of np.concatenate() to improve performance when the output array is likely to be fed into TensorFlow. Args: items: The list of items to concatenate and align. time_major: Whether the data in items is time-major, in which case, we will concatenate along axis=1. Returns: The concat'd and aligned array. rrc3$K|] }|jV dSN)r.0ss r) z!concat_aligned..as$ 7 7A 7 7 7 7 7 7r+NTc30K|]}|jdVdS)rNshaper1s r)r4z!concat_aligned..d(::q ::::::r+rc30K|]}|jdVdSrNr6r1s r)r4z!concat_aligned..ir8r+c30K|]}|jdVdSr:r6r1s r)r4z!concat_aligned..ns(661AGAJ666666r+r)outaxisr=)r$ isinstancerndarrayrfloat32float64r r*sumr7reshaper!r" concatenate)r,r-rflat batch_dim new_shaper(s r)concat_alignedrI=s 0 5zzQ UqQx E!Hbj ) )BeAhn   A// aS 7 7 7 7 777??  !T!!::E::::: "1X^A. ;e??  ::E::::: &aq(9;e?? 6666666I" uQx~abb'99Ii((}!B&!+++V]-?+++ u&J/EqqAFFFF ~ez*@!!qAAAAr+Tx reduce_typec8fd}tj||S)aConverts values in `stats` to non-Tensor numpy or python types. Args: x: Any (possibly nested) struct, the values in which will be converted and returned as a new struct with all torch/tf tensors being converted to numpy types. reduce_type: Whether to automatically reduce all float64 and int64 data into float32 and int32 data, respectively. Returns: A new struct with the same structure as `x`, but with all values converted to numpy arrays (on CPU). cltrt|tjrt|dkr&|n7|}notrft|tjtj fr@t|dr0t sJ|}n|}rt|tjrtj|jtjr |tj}n>tj|jt(r|tj}|S)Nrnumpy)torchr?Tensorr$rcpuitemdetachrNtfVariablehasattrexecuting_eagerlyrr@ issubdtyperfloatingastyperAintint32)rRretrKs r)mappingz!convert_to_numpy..mappings[  Zel33 tyy{{##q(( !!![[]]&&((..00 C  dRY $<== BI$PWBXBX '')) ) ))**,,CCC  +:c2:66 +}SY 44 +jj,,sy#.. +jj** r+)tree map_structure)rJrKr^s ` r)convert_to_numpyraxs1$*  gq ) ))r+weightsbiases frameworkcd d}||}|dko7|jd|jdko|jd|jdk}|||}||}tj|||dn|zS) aCalculates FC (dense) layer outputs given weights/biases and input. Args: x: The input to the dense layer. weights: The weights matrix. biases: The biases vector. All 0s if None. framework: An optional framework hint (to figure out, e.g. whether to transpose torch weight matrices). Returns: The dense layer's output. FctrRt|tjr8|}t rGt r.t|t jr|}|rtj |}|Sr0) rOr?rPrQrSrNrTrWrUr transpose)r"rgs r)map_zfc..map_s  3$ -- 3xxzz((**0022  $"&&(( $$ ,, $zz||  &<%%D r+rOrr)rgNF)r7rmatmul)rJrbrcrdrhrgs r)fcrls(     QAW$  gmA&&I171:q9I+Id7i000G T&\\F 9Q 6>CCv FFr+inputs spaces_struct time_axis batch_axisc 4 |r|sJtj|}|tj|ndgt|z}d}d}g}t||D] \ } ||r jd}|r jd}t | t rb|rtj ||zg | t | j  tj t | tr|rtj ||zdg |rI| tj fdt!| jDd| tj fdt!| jDdbt t$rtj g |rtj ||zdg n0|rtj |dg ntj dg | tj  tj|d} |rtj| ||dg} | S) a Flattens arbitrary input structs according to the given spaces struct. Returns a single 1D tensor resulting from the different input components' values. Thereby: - Boxes (any shape) get flattened to (B, [T]?, -1). Note that image boxes are not treated differently from other types of Boxes and get flattened as well. - Discrete (int) values are one-hot'd, e.g. a batch of [1, 0, 3] (B=3 with Discrete(4) space) results in [[0, 1, 0, 0], [1, 0, 0, 0], [0, 0, 0, 1]]. - MultiDiscrete values are multi-one-hot'd, e.g. a batch of [[0, 2], [1, 4]] (B=2 with MultiDiscrete([2, 5]) space) results in [[1, 0, 0, 0, 1, 0, 0], [0, 1, 0, 0, 0, 0, 1]]. Args: inputs: The inputs to be flattened. spaces_struct: The (possibly nested) structure of the spaces that `inputs` belongs to. time_axis: Whether all inputs have a time-axis (after the batch axis). If True, will keep not only the batch axis (0th), but the time axis (1st) as-is and flatten everything from the 2nd axis up. batch_axis: Whether all inputs have a batch axis. If True, will keep that batch axis as-is and flatten everything from the other dims up. Returns: A single 1D tensor resulting from concatenating all flattened/one-hot'd input components. Depending on the time_axis flag, the shape is (B, n) or (B, T, n). .. testcode:: :skipif: True # B=2 from ray.rllib.utils.tf_utils import flatten_inputs_to_1d_tensor from gymnasium.spaces import Discrete, Box out = flatten_inputs_to_1d_tensor( {"a": [1, 0], "b": [[[0.0], [0.1]], [1.0], [1.1]]}, spaces_struct=dict(a=Discrete(2), b=Box(shape=(2, 1))) ) print(out) # B=2; T=2 out = flatten_inputs_to_1d_tensor( ([[1, 0], [0, 1]], [[[0.0, 0.1], [1.0, 1.1]], [[2.0, 2.1], [3.0, 3.1]]]), spaces_struct=tuple([Discrete(2), Box(shape=(2, ))]), time_axis=True ) print(out) .. testoutput:: [[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]] # B=2 n=4 [[[0.0, 1.0, 0.0, 0.1], [1.0, 0.0, 1.0, 1.1]], [[1.0, 0.0, 2.0, 2.1], [0.0, 1.0, 3.0, 3.1]]] # B=2 T=2 n=4 Nrrdepthcg|]=\}}tdd|f|tj>S)Nrrone_hotrZrrAr2ir%input_s r) z/flatten_inputs_to_1d_tensor..-sV $1$F111a4L:::AA"*MMr+r>c~g|]9\}}t||tj:S)rrrvrxs r)r{z/flatten_inputs_to_1d_tensor..7sN $1$F1IQ777>>rzJJr+)r_flattenr$zipr7r?rrrDappendrwr%rZrArrE enumeratenvecfloatarray) rmrnrorp flat_inputs flat_spacesBTr<spacemergedrzs @r)flatten_inputs_to_1d_tensorrsD-*-- -,v&&K  $ ]###Vc+&& & A A C[+660202  99 QA $LO eX & &( 2 5FQUG44 JJwvUW555<r) r7rzerosrangerErksigmoidmultiplytanhadd)rJrbrcrr-rsequence_length batch_sizeunitsc_statesh_statesunrolled_outputst input_matrixinput_matmul_matrix sigmoid_1 sigmoid_2tanh_3 sigmoid_4s r)lstmrs~6g:4aa15Oj/a0J M!  !E&8:u"56668:u"5666*1-*1-P8?J*NOOO8:*NOOO? # #11%/?qAAAqqqzzQqqq!QQQwZ ~|X&>QGGG  i g>>G/519uqy3H0HIKWXX ;x33/1U7 ;<< ,QQQ 0A-ABCC6(BK 6$B$BCC/519uqy3H0HIJJ ;y"'(*;*;<<  1(0 Q111W % %(0 QQQ111W % % h1 11r+rirson_value off_valuerct|tr!tj|tj}n5t r.t|t jr|}|jtj kr!| tj }d}|dkrtj |dz}tj ||ks0Jd tj |||j}tjg||R|z}g}t!|jD]}dg|jz} dg|jz} d| |<tj||| } |dkr&||dz | |dz <tj| | } || ||||t-|<| |S)aOne-hot utility function for numpy. Thanks to qianyizhang: https://gist.github.com/qianyizhang/07ee1c15cad08afb03f5de69349efc30. Args: x: The input to be one-hot encoded. depth: The max. number to be one-hot encoded (size of last rank). on_value: The value to use for on. Default: 1.0. off_value: The value to use for off. Default: 0.0. Returns: The one-hot encoded equivalent of the input array. rrrrz.sl   1 CJJ!OOAaDD111a4 B B B I I"* U U   r+rtr>)rOr?rPrNr7rrEr)rJdepthsr7s` @r)one_hot_multidiscreters Au|,, GGII GE >     !&))       r+alphac4tj|||z|S)zImplementation of the leaky ReLU function. y = x * alpha if x < 0 else x Args: x: The input values. alpha: A scaling ("leak") factor to use for negative x. Returns: The leaky ReLU output for x. )rmaximum)rJrs r)relur"s :aUA & &&r+ derivativecL|r|d|z zSddtj| zz S)aY Returns the sigmoid function applied to x. Alternatively, can return the derivative or the sigmoid function. Args: x: The input to the sigmoid function. derivative: Whether to return the derivative or not. Default: False. Returns: The sigmoid function (or its derivative) applied to x. r)rexp)rJrs r)rr2s3$AE{Ar N##r+rtr=epsilonc|pt}tj|}tj|tj||dz |S)a{Returns the softmax values for x. The exact formula used is: S(xi) = e^xi / SUMj(e^xj), where j goes over all elements in x. Args: x: The input to the softmax function. axis: The axis along which to softmax. epsilon: Optional epsilon as a minimum value. If None, use `SMALL_NUMBER`. Returns: The softmax over x. T)keepdims) SMALL_NUMBERrrrrC)rJr=rx_exps r)softmaxrFsE$%G F1IIE :ebfUD4@@@@' J JJr+)rr0)T)NN)NFT)r)NNFr)rirj)rtN)8 collectionsrtypesrtypingrrrNrr_gymnasium.spacesrrray._common.deprecationr ray.rllib.utils.annotationsr ray.rllib.utils.frameworkr r ray.rllib.utils.typingr rrrtf1rTtfvrO_r LARGE_INTEGERMIN_LOG_NN_OUTPUTMAX_LOG_NN_OUTPUTr[r@r*boolrIrastrrlrrrrrrrAtyperwrrrlistrr+r)rs######""""""!!!!!!!! 44444444......111111EEEEEEEESSSSSSSSSSSS} R    q     4 3    :   4  ;?3B3B  3B)1$3BZ3B3B3B   3Bl &*&*(&*t&*GW&*&*&* &*R $(# &G&G z&G Z&G RZ &G} &G Z &G&G&G &GR ,0 CC CK(CC C  CCC CL ## #L "*URZ   &rz &bj & & &  & $(48 B2B2 ZB2 RZ B2&bj1 B2  B2  B2B2B2 B2J * 99 Z_9 99 9  9 Z 999 9x $(I       ' 'BJ 'u 'rz ' ' '  ' $$rz$t$ $$$ $& KOKK RZ K&)K9A%KZKKK KKKr+