PiLddlmZddlZddlmZGddejZdS))OptionalN)nnc eZdZdZ ddedededdffd Zd Zddeddfd Z dd e j d e e j de j fdZ xZ S)Qwen2RotaryPositionalEmbeddingsau RoPE Embeddings used in the Qwen2 model. Ref: https://huggingface.co/Qwen/Qwen2-7B-Instruct This class is not numerically equivalent to the RoPE Embedding module used by Llama2 and Llama3. Args: dim (int): Embedding dimension. This is usually set to the dim of each head in the attention module computed as ``embed_dim`` // ``num_heads`` max_seq_len (int): Maximum expected sequence length for the model, if exceeded the cached freqs will be recomputed base (float): The base for the geometric progression used to compute the rotation angles .Adim max_seq_lenbasereturnNct||_||_||_|dSN)super__init__r r r rope_init)selfr r r __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/qwen2/_positional_embeddings.pyrz(Qwen2RotaryPositionalEmbeddings.__init__sG  & c d|jtjd|jdd|jdz|jz zz }|d|d||jdS)Ng?rthetaF persistent)r torcharanger floatregister_bufferbuild_rope_cacher )rrs rrz)Qwen2RotaryPositionalEmbeddings.rope_init+s I Q!,,->A ->?EEGG$(R T  We>>> d./////rctj||jj|jj}tjd||j}tj||gd}tj|| gd}| d|ddS)N)dtypedevicez i, j -> ijr cacheFr) rrrr!r"einsumrcatcossinr)rr seq_idx idx_thetafreqsr%s rrz0Qwen2RotaryPositionalEmbeddings.build_rope_cache3s, tz/ 8I    Lw CCIIKK  9i0b999 599;; 4"=== We>>>>>rx input_posc2|d}|d}||jd|n |j|}|d|d|dz}|dd|f|j}|d|df|j}|dd|jddzf}|d|jddzdf} t j| |fd} ||z| |zz} | |S)ae Args: x (torch.Tensor): input tensor with shape [b, s, n_h, h_d] input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b, s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Returns: Tensor: output tensor with RoPE applied Notation used for tensor shapes: - b: batch size - s: sequence length - n_h: num heads - h_d: head dim TODO: The implementation below can be made more efficient for inference. r#Nr.r$) sizer%viewtor!shaperr'type_as) rr-r.seq_lenhead_dim rope_cacher(r)x1x2rotatedx_outs rforwardz'Qwen2RotaryPositionalEmbeddings.forwardDs+4&&))66":: %.$5DJxx 4:i;P  __R!X\BB ixi(++AG44hii(++AG44 s&agbkQ&&& ' sAGBK1$&&& ')bS"I2...SWs]+}}Qr)rr)rr)__name__ __module__ __qualname____doc__intrrrrrTensorrr= __classcell__)rs@rrrs& !             000??C?4????$DH3 3 3 *25<*@3 3 3 3 3 3 3 3 3 rr)typingrrrModulerrrrHst i i i i i bii i i i i r