PiLddlmZddlZddlmZGddejZdS))OptionalN)nnc eZdZdZ ddedededdffd Zd Zddeddfd Z dd ej d e ej dej fdZ xZ S)Phi3RotaryPositionalEmbeddingsa| RoPE Embeddings used in the Phi3 model. Ref: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct This class is not numerically equivalent to the RoPE Embedding module used by Llama2 and Llama3. Args: dim (int): Embedding dimension. This is usually set to the dim of each head in the attention module computed as ``embed_dim`` // ``num_heads`` max_seq_len (int): Maximum expected sequence length for the model, if exceeded the cached freqs will be recomputed base (int): The base for the geometric progression used to compute the rotation angles 'dim max_seq_lenbasereturnNct||_||_||_|dSN)super__init__r r r rope_init)selfr r r __class__s ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/phi3/_position_embeddings.pyrz'Phi3RotaryPositionalEmbeddings.__init__sG  & c d|jtjd|jdd|jdz|jz zz }|d|d||jdS)Ng?rthetaF persistent)r torcharanger floatregister_bufferbuild_rope_cacher )rrs rrz(Phi3RotaryPositionalEmbeddings.rope_init+s I Q!,,->A ->?EEGG$(R T  We>>> d./////rctj||jj|jj}tjd||j}tj||gd}tj|| gd}| d|ddS)N)dtypedevicez i, j -> ijr cacheFr) rrrr!r"einsumrcatcossinr)rr seq_idx idx_thetafreqsr%s rrz/Phi3RotaryPositionalEmbeddings.build_rope_cache3s, tz/ 8I    Lw CCIIKK  9i0b999 599;; 4"=== We>>>>>rx input_posc|d}|d}||jd|n |j|}|d|d|dz}|dd|f}|d|df}|dd|jddzf}|d|jddzdf} t j| |fd} ||z| |zz} | |S)ae Args: x (torch.Tensor): input tensor with shape [b, s, n_h, h_d] input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b, s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Returns: Tensor: output tensor with RoPE applied Notation used for tensor shapes: - b: batch size - s: sequence length - n_h: num heads - h_d: head dim TODO: The implementation below can be made more efficient for inference. r#Nr.r$)sizer%viewshaperr'type_as) rr-r.seq_lenhead_dim rope_cacher(r)x1x2rotatedx_outs rforwardz&Phi3RotaryPositionalEmbeddings.forwardDs 4&&))66":: %.$5DJxx 4:i;P  __R!X\BB ixi(hii( s&agbkQ&&& ' sAGBK1$&&& ')bS"I2...SWs]+}}Qr)rr)rr) __name__ __module__ __qualname____doc__intrrrrTensorrr< __classcell__)rs@rrrs&              000??C?4????$DH3 3 3 *25<*@3 3 3 3 3 3 3 3 3 rr)typingrrrModulerrrrGst i i i i i RYi i i i i r