PiRddlZddlmcmZddlmZGddejZdS)N)nncVeZdZdZd dededdffd Zdejdejfd Z xZ S) RMSNormz Root Mean Square Normalization in fp32. See: https://pytorch.org/docs/stable/generated/torch.nn.RMSNorm.html Args: dim (int): embedding size eps (float): small value to avoid division by zero. Default: 1e-6 ư>dimepsreturnNct|f|_||_t jt j||_dS)N) super__init__normalized_shaperr Parametertorchonesscale)selfrr __class__s n/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/rms_norm.pyr zRMSNorm.__init__sG !$\%*S//22 xctj||j|j|j|jS)z Args: x (torch.Tensor): input tensor to normalize Returns: torch.Tensor: The normalized and scaled tensor having the same shape as ``x``. )r weightr)Frms_normfloatr rrtodtype)rrs rforwardzRMSNorm.forwardsGz GGII!2:    "QW++  r)r) __name__ __module__ __qualname____doc__intrr rTensorr __classcell__)rs@rrr s33C3e3t333333 %,rr)rtorch.nn.functionalr functionalrModulerrrr*ss bir