PiLddlmZddlZddlmZGddejZdS))OptionalN)nnc eZdZdZdejddejdejdeejdejffdZd e j d e j fd Z xZ S) FeedForwardaThis class implements the feed-forward network derived from Llama2. Args: gate_proj (nn.Module): Projection from input dim to hidden dim, fed through activation and multiplied by up_proj. down_proj (nn.Module): Final projection to output dim. up_proj (Optional[nn.Module]): Projection from input dim to hidden dim, multiplied by activation(gate_proj). activation (nn.Module): Activation function to use. Default is nn.SiLU(). N)up_proj activation gate_proj down_projrrct||_||_||_||_dS)N)super__init__w1w2w3r)selfr r rr __class__s r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/feed_forward.pyr zFeedForward.__init__s; $xreturnc|||}|j|||z}||}|S)ad Args: x (torch.Tensor): input tensor with shape ``(..., in_dim)``, where ``in_dim`` is the input dimension of both ``gate_proj`` and ``up_proj``. Returns: torch.Tensor: output tensor with shape ``(..., out_dim)``, where ``out_dim`` is the output dimension of ``down_proj``. )rrrr)rrhs rforwardzFeedForward.forward(sN OODGGAJJ ' ' 7 DGGAJJA GGAJJr) __name__ __module__ __qualname____doc__rSiLUModulerr torchTensorr __classcell__)rs@rrrs   (, ' % % %9 %9 % ")$ % I % % % % % %%,rr)typingrr rrrrrr%sj (((((")(((((r