&`ix BddlZddlZddlZddlmZmZmZmZmZddl Z ddl Z ddl Z ddlmZmZddlmZddlmZddlmZmZmZddlmZddlmZdd lmZmZm Z m!Z!m"Z"erdd l#m$Z$m%Z%dd l&m'Z'dd l(m)Z)ej*e+Z,e\Z-Z.d Z/dZ0e-rej1dZ2n e3dZ2edddede"dee4e"ffdZ5eddddddee6de4de"fdZ7eddde"fd Z8eded!dee4e"ffd"Z9e dWd$ee4d%e:d&e:d'eed(fd)Z;edXd*e!d$ee4fd+Zed3ee"de"fd4Z?edZd*e"d6e6de"fd7Z@ed*e"de"fd8ZAed*e"d9e jBde"fd:ZCedXd*e"d;eeDde"fd<ZEe d[d=e"d>eeDd?e:de"fd@ZFedAedBedCe6ddfdDZGedddEe"ddfdFZHedXdGeeDddfdHZIedIe"dJe"de"fdKZJed\dMZKed]dNZLe d^ddLdReDdSe6dTe6d$ee4f dUZMdVZNdS)_N) TYPE_CHECKINGDictListOptionalUnion)Discrete MultiDiscrete)version)RepeatedValues) DeveloperAPI OldAPIStack PublicAPI)try_import_torch) SMALL_NUMBER)LocalOptimizer NetworkType SpaceStructTensorStructType TensorType) ParamDict ParamList) TorchPolicy) TorchPolicyV2gߌ3gߌ3Gz2.0.0zFtorch is not installed. TORCH_COMPILE_REQUIRED_VERSION is not defined.policyr optimizerlossreturncd}|jd|jd}n tj}d}|jD]}t t d|d}|rtt j||}t|tj r&| }|t||z }|dz }|t|jkriSd|iS)aUApplies gradient clipping to already computed grads inside `optimizer`. Note: This function does NOT perform an analogous operation as tf.clip_by_global_norm. It merely clips by norm (per gradient tensor) and then computes the global norm across all given tensors (but without clipping by that global norm). Args: policy: The TorchPolicy, which calculated `loss`. optimizer: A local torch optimizer object. loss: The torch loss tensor. Returns: An info dict containing the "grad_norm" key and the resulting clipped gradients. r grad_clipNc|jduSN)grad)ps o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/rllib/utils/torch_utils.pyz%apply_grad_clipping..Ks qvT'9params grad_gnorm)confignpinf param_groupslistfilternnutilsclip_grad_norm_ isinstancetorchTensorcpunumpyminlen) rrrr) clip_valuenum_none_grads param_groupr' global_norms r$apply_grad_clippingr>-s(J }[!-];/ V N -  f99;x;PQQRR  (226:FFK+u|44 8)oo//5577 #k:66 6JJ a NNY34444 * %%r&value)r grad_clip_bygradients_dictrrr@c2|dS|dvrtd|d|dkr@|D])\}}|dnt|| |||<*dS|dkrf|D]O\}}|H|ddd }||kr|||z PdSt|}t|}t|d kr|S|t |d z| z }t |d} |D]C} | ?| | | jD|S)a?Performs gradient clipping on a grad-dict based on a clip value and clip mode. Changes the provided gradient dict in place. Args: gradients_dict: The gradients dict, mapping str to gradient tensors. grad_clip: The value to clip with. The way gradients are clipped is defined by the `grad_clip_by` arg (see below). grad_clip_by: One of 'value', 'norm', or 'global_norm'. Returns: If `grad_clip_by`="global_norm" and `grad_clip` is not None, returns the global norm of all tensors, otherwise returns None. N)r?normr=z`grad_clip_by` (z*) must be one of [value|norm|global_norm]!r?rCeeAneginfposinfrgư>)r8?)max) ValueErroritemsr4cliprC nan_to_nummul_r.valuescompute_global_normr9clampdetachtodevice) rArr@kvrCgradients_list total_norm clip_coeffclip_coeff_clampedgs r$clip_gradientsr^`s,;;; W| W W W   w"((**  DAq uzz!iZ'K'K 1       "((** - -DAq}vvayy++5+FF)##FF9t+,,,  - -n335566(88 ~  ! # # Z$->I!N!NN #[[[== A AA}  2 5 5ah ? ?@@@r&rYrcdt|dkrtdSttfd|Ddd}|S)zComputes the global norm for a gradients dict. Args: gradients_list: The gradients list containing parameters. Returns: Returns the global norm of all tensors in `gradients_list`. @rcg|]F}|t|ddGS)NrErFrG)r4rCrTrO).0r] norm_types r$ z'compute_global_norm..sQ    =  188::y11566 ==r&rErFrG)r9r4tensorrCstackrO)rYrZrds @r$rRrRsI >a||C         (        jdj++ r&)rrctfdjDd}|_|t|dS)azConcatenates multi-GPU (per-tower) TD error tensors given TorchPolicy. TD-errors are extracted from the TorchPolicy via its tower_stats property. Args: policy: The TorchPolicy to extract the TD-error values from. Returns: A dict mapping strings "td_error" and "mean_td_error" to the corresponding concatenated and mean-reduced values. cg|]N}|jdtdgjOS)td_errorra) tower_statsgetr4rfrUrV)rctrs r$rez.concat_multi_gpu_td_errors..sY    M  j%,,u*=*= > > A A&- P P   r&rdim)rj mean_td_error)r4catmodel_gpu_towersrjmean)rrjs` r$concat_multi_gpu_td_errorsrtsuyy    ,     HFOH--  r&FrV pin_memory use_streamstream)ztorch.cuda.Streamztorch.cuda.classes.Streamc8tntdjdkotjr|rrQt tjjtjjjfsJdtdnBtjn#tjndfdtj |S)aj Converts any (possibly nested) structure to torch.Tensors. Args: x: The input structure whose leaves will be converted. device: The device to create the tensor on (e.g. "cuda:0" or "cpu"). pin_memory: If True, calls `pin_memory()` on the created tensors. use_stream: If True, uses a separate CUDA stream for `Tensor.to()`. stream: An optional CUDA stream for the host-to-device copy in `Tensor.to()`. Returns: A new structure with the same layout as `x` but with all leaves converted to torch.Tensors. Leaves that are None are left unchanged. Nr6cudaz-`stream` must be a torch.cuda.Stream but got .rVcf||St|tr3ttj|j|j|jSt|r|}nt|tj r|j tks|j j tjur|S|jjsZt#j5t#jdt|}dddn #1swxYwYnGt|}n,ttj|}|r)|j tjkr|}rr|}riOtj5|d}dddn #1swxYwYn-|d}n|}|S)NignoreT) non_blocking)r3r tree map_structurerQlengthsmax_lenr4 is_tensorr+ndarraydtypeobjecttypestr_flags writeablewarningscatch_warnings simplefilter from_numpyasarrayis_floating_pointfloat16floatruryrwrU)itemrfrVis_cudamappingrurws r$rz(convert_to_torch_tensor..mappings <K dN + + !"7DK88    ??4  8FF bj ) ) 8zV##tz"''A'A :' 0,..44)(333"--d33F444444444444444))$//%%bj&6&677F  # # % % $&,%-*G*G\\^^F  )' )&&((F  '!Z&&v..BB#YYvDYAAFBBBBBBBBBBBBBBB 6==YYv&&F s$/D  D D G88G<?G<) r4rVrry is_availabler3Streamclassesdefault_streamrr)xrVrurvrwrrs `` `@@r$convert_to_torch_tensorrs;0&,%7U\\& ! ! !U\\%=P=PF{f$C%**A*A*C*CG   >!!UZ. 0B0IJSSR4<<RRRSSS**,,Z..f.==FF222222222h  gq ) ))r&rc8fd}tj||S)aCreates a copy of `x` and makes deep copies torch.Tensors in x. Also moves the copied tensors to the specified device (if not None). Note if an object in x is not a torch.Tensor, it will be shallow-copied. Args: x : Any (possibly nested) struct possibly containing torch.Tensors. device : The device to move the tensors to. Returns: Any: A new struct with the same structure as `x`, but with all torch.Tensors deep-copied and moved to the specified device. ct|tjrU,t|n&|S|Sr!)r3r4r5clonerTrU)rrVs r$rz#copy_torch_tensors..mappingXs_ dEL ) ) > DKKMM***[[]]%%f--  Kr&)rr)rrVrs ` r$copy_torch_tensorsrFs1$  gq ) ))r&ypredc|}t|d}t||z d}tdg|j}t|d||tzz z dS)a,Computes the explained variance for a pair of labels and predictions. The formula used is: max(-1.0, 1.0 - (std(y - pred)^2 / std(y)^2)) Args: y: The labels. pred: The predictions. Returns: The explained variance given a pair of labels and predictions. rrngr()squeezer4varrfrUrVrKr)rr squeezed_yy_vardiff_varmin_s r$explained_varianceresJ IIjaI ( (Eyydllnn4!y<.s@PPP1%))EIIeii3&7&788#>>PPPr&c3LK|]}t|dV dS)r`N)r4r)rcl2s r$ zglobal_norm..s0AA2s++AAAAAAr&rr)r single_l2ss r$r=r=sCQPPPPJ 99SAAjAAAAA3 G GGr&rJdeltactt||kt|ddz|t|d|zz zS)aComputes the huber loss for a given term and delta parameter. Reference: https://en.wikipedia.org/wiki/Huber_loss Note that the factor of 0.5 is implicitly included in the calculation. Formula: L = 0.5 * x^2 for small abs x (delta threshold) L = delta * (abs(x) - 0.5*delta) for larger abs x (delta threshold) Args: x: The input term, e.g. a TD error. delta: The delta parmameter in the above formula. Returns: The Huber loss resulting from `x` and `delta`. r`r)r4whereabsr)rrs r$ huber_lossrs]$ ;; ! u !SC 1e +,  r&cndtt|dzS)zComputes half the L2 norm over a tensor's values without the sqrt. output = 0.5 * sum(x ** 2) Args: x: The input tensor. Returns: 0.5 times the L2 norm over the given tensor's values (w/o sqrt). rr`)r4rrrs r$l2_lossrs) 599Q,,-- --r&rcBt|tr7tj|jSt|trt|jdtj r;tj |j} j ddn|j}tfdt!|DdSt#d|)aReturns a one-hot tensor, given and int tensor and a space. Handles the MultiDiscrete case as well. Args: x: The input tensor. space: The space to use for generating the one-hot tensor. Returns: The resulting one-hot tensor. Raises: ValueError: If the given space is not a discrete one. .. testcode:: import torch import gymnasium as gym from ray.rllib.utils.torch_utils import one_hot x = torch.IntTensor([0, 3]) # batch-dim=2 # Discrete space with 4 (one-hot) slots per batch item. s = gym.spaces.Discrete(4) print(one_hot(x, s)) x = torch.IntTensor([[0, 1, 2, 3]]) # batch-dim=1 # MultiDiscrete space with 5 + 4 + 4 + 7 = 20 (one-hot) slots # per batch item. s = gym.spaces.MultiDiscrete([5, 4, 4, 7]) print(one_hot(x, s)) .. testoutput:: tensor([[1, 0, 0, 0], [0, 0, 0, 1]]) tensor([[1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0]]) rrcg|]A\}}tjdd|f|BSr!)r0 functionalrlong)rcinrs r$rezone_hot..VsC R R R$!QR] " "1QQQT7<<>>1 5 5 R R Rr&rnz#Unsupported space for `one_hot`: {})r3rr0rrrrr nvecr+rravelrrr4rq enumeraterLformat)rrrs` r$rr(sJ%"" N}$$QVVXXuw777 E= ) ) N ejmRZ 0 0 8EJ''D !'!*b))AA:Dyy R R R R)D// R R R   >EEeLLMMMr&axiscLt|td}t||t|}t||t||z S)zSame as torch.mean() but ignores -inf values. Args: x: The input tensor to reduce mean over. axis: The axis over which to reduce. None for all axes. Returns: The mean reduced inputs, ignoring inf values. z-inf)r4nerr zeros_liker)rrmaskx_zeroeds r$reduce_mean_ignore_infr]so 88AuV}} % %D{{4E$4$4Q$7$788H 99Xt $ $uyyt'D'D DDr&rmaxlen time_majorc||}tt|jt |fz}||jd |k}|s| }| |p tj |S)alOffers same behavior as tf.sequence_mask for torch. Thanks to Dimitris Papatheodorou (https://discuss.pytorch.org/t/pytorch-equivalent-for-tf-sequence-mask/ 39036). Args: lengths: The tensor of individual lengths to mask by. maxlen: The maximum length to use for the time axis. If None, use the max of `lengths`. dtype: The torch dtype to use for the resulting mask. time_major: Whether to return the mask as [B, T] (False; default) or as [T, B] (True). Returns: The sequence mask resulting from the given input and parameters. Nr(rn) rKr4onestuplerintrUrVcumsumrmrbool)rrrrrs r$ sequence_maskrms2~ ::eGM**c&kk^; < z)update_target_network..sE Aq 3A !c'Q .r&N)rrMload_state_dict)rrrnew_state_dictrs ` @r$update_target_networkrsx($$&&J))++1133N ~.....r& kl_divergencec|r0|rtddSdSdS)Na}KL divergence is non-finite, this will likely destabilize your model and the training process. Action(s) in a specific state have near-zero probability. This can happen naturally in deterministic environments where the optimal policy has zero mass for a specific action. To fix this issue, consider setting the coefficient for the KL loss term to zero or increasing policy entropy.)loss_initializedisinfloggerwarning)rrs r$warn_if_infinite_kl_divergencersd    ]%8%8%:%:  *         r&seedc| [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5, 0.5, 0.0, 0.0] -> [-5 -4 -3 -2 -1 0 1 2 3 4 5] (0.5*2 + 0.5*3=2.5) Example: num_buckets=5 lower_bound=-1 upper_bound=1 value=0.1 -> [0.0, 0.0, 0.8, 0.2, 0.0] -> [-1 -0.5 0 0.5 1] (0.2*0.5 + 0.8*0=0.1) Args: value: The input tensor of shape (B,) to be two-hot encoded. num_buckets: The number of buckets to two-hot encode into. lower_bound: The lower bound value used for the encoding. If input values are lower than this boundary, they will be encoded as `lower_bound`. upper_bound: The upper bound value used for the encoding. If input values are higher than this boundary, they will be encoded as `upper_bound`. Returns: The two-hot encoded tensor of shape (B, num_buckets). rr{r(rJr`rrnN) r4rSarangerrfloorceilreqrgrqrzeros)r?rrr rV batch_indices bucket_deltaidxrWkp1values_k values_kp1 weights_k weights_kp1 indices_k indices_kp1indicesupdatesoutputs r$two_hotr4sb KK{K 8 8ELLEKN6LBBHHJJM+-+/BL <% < /C CA **S//C ++add3iisC 0 0C ++cff[))39c : :CQ--Hs\11J#:(=>I /K ]A.B 77I++}c2+;;KiiK0ai88==??GiiK0ai88G[[QV[ D DF+2F7111a4='!!!Q$- '( Mr&c6 ddlm}dS#t$rYdSwxYw)NrTF) torch._dynamo_dynamo ImportError)dynamos r$_dynamo_is_availabler:ns@&&&&&&t uus  )NFFNr!)NF)rJ)NNF)rrrr)rrrr)rrrN)Ologgingrrtypingrrrrr gymnasiumgymr7r+rgymnasium.spacesrr packagingr ray.rllib.models.repeated_valuesr ray.rllib.utils.annotationsr r rray.rllib.utils.frameworkrray.rllib.utils.numpyrray.rllib.utils.typingrrrrrray.rllib.core.learner.learnerrrray.rllib.policy.torch_policyr ray.rllib.policy.torch_policy_v2r getLogger__name__rr4r0 FLOAT_MIN FLOAT_MAXparseTORCH_COMPILE_REQUIRED_VERSIONrLstrr>rr^rRrtrrrrrr=rrSpacerrrrrrr rrrr4r:rr&r$rQsz ============== 44444444;;;;;;LLLLLLLLLL666666......?CCCCCCCC999999>>>>>>  8 $ $     r   %2W]7%;%;""%/ZP&&"  /& /&&4/&