Pi*vddlmZmZddlZddlmZGddejZGddejZdS))AnyOptionalN)nnc eZdZdZ ddedededdffd Zd Zddeddfd Zdd d ej de ej dej fdZ xZ S)RotaryPositionalEmbeddingsa, This class implements Rotary Positional Embeddings (RoPE) proposed in https://arxiv.org/abs/2104.09864. Reference implementation (used for correctness verfication) can be found here: https://github.com/meta-llama/llama/blob/main/llama/model.py#L80 In this implementation we cache the embeddings for each position upto ``max_seq_len`` by computing this during init. Args: dim (int): Embedding dimension. This is usually set to the dim of each head in the attention module computed as ``embed_dim // num_heads`` max_seq_len (int): Maximum expected sequence length for the model, if exceeded the cached freqs will be recomputed base (int): The base for the geometric progression used to compute the rotation angles 'dim max_seq_lenbasereturnNct||_||_||_|dSN)super__init__r r r rope_init)selfr r r __class__s y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/position_embeddings.pyrz#RotaryPositionalEmbeddings.__init__"sG  & c d|jtjd|jdd|jdz|jz zz }|d|d||jdSNg?rthetaF persistent)r torcharanger floatregister_bufferbuild_rope_cacher rrs rrz$RotaryPositionalEmbeddings.rope_init.s I Q!,,->A ->?EEGG$(R T  We>>> d./////rc^tj||jj|jj}tjd||j}tjtj|tj |gd}| d|ddS)Ndtypedevice i, j -> ijr cacheFr) rrrr%r&einsumrstackcossinr )rr seq_idx idx_thetar*s rr!z+RotaryPositionalEmbeddings.build_rope_cache6s, tz/ 8I    Lw CCIIKK  UYy1159Y3G3GHbQQQ We>>>>>r) input_posxr1cL|d}||jd|n |j|}|jg|jddddR}|d|dd|dd}t j|d|dz|d|dzz |d|dz|d|dzzgd}|d}| |S)a# Args: x (torch.Tensor): input tensor with shape ``[b, s, n_h, h_d]`` input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b, s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Returns: torch.Tensor: output tensor with shape ``[b, s, n_h, h_d]`` Notation used for tensor shapes: - b: batch size - s: sequence length - n_h: num heads - h_d: head dim Nr(r.r.r4) sizer*rreshapeshapeviewrr,flattentype_as)rr2r1seq_len rope_cachexshapedx_outs rforwardz"RotaryPositionalEmbeddings.forwardEs..&&))%.$5DJxx 4:i;P $!''))#9QWSbS\929q999  __Ra!W\\!__aPP  *V"44&/Jv$667*V"44&/Jv$667     a  }}Qr)rr )r) __name__ __module__ __qualname____doc__intrrr!rTensorrrB __classcell__rs@rrr s.              000 ? ?C ?4 ? ? ? ? GK5 5 5 5 -5el-C5 5 5 5 5 5 5 5 5 rrc|eZdZdZ ddededededed ed d ffd Zd ZddZde j de d e j fdZ xZ S) VisionRotaryPositionalEmbeddingsa This class implements two-dimensional Rotary Positional Embeddings (RoPE) for images based on the axial frequency 2D RoPE described in https://arxiv.org/pdf/2403.13298. The position embedding is simply applied to the x-axis and y-axis separately, encoding the x and y position of each patch within every tile.. The embedding is applied to each tile identically. Note: This module assumes the CLS token embedding is appended at the end of the sequence. Args: patch_size (int): The size of each patch. Used to divide the tiles into patches. E.g. for ``patch_size=40``, a tile of shape (400, 400) will have 10x10 grid of patches. tile_size (int): The size of your image tiles, if the image was tile-cropped in advance. Otherwise, the size of the full input image. In this case, the function will consider your image as a single tile. max_num_tiles (int): The maximum number of tiles in the image. This is used to unfold the input sequence length into sequence length per tile so RoPE can be applied to each tile separately. dim (int): Embedding dimension. Unlike :class:`~torchtune.modules.RotaryPositionalEmbeddings`, this is usually set to the dim of each head in the attention module divided by 2, computed as ``embed_dim // num_heads // 2``. The divide by 2 accounts for x and y positions. base (int): The base for the geometric progression used to compute the rotation angles append_cls_token (bool): Set to True if CLS token embedding is at the end of the sequence in the vision transformer, False if is in the beginning of the sequence. RoPE is zeroed out for the CLS token. Default is True. r T patch_size tile_size max_num_tilesr r append_cls_tokenr Nct||z|_||_||_||_||_|dSr)rrpatch_grid_sizerOr r rPr)rrMrNrOr r rPrs rrz)VisionRotaryPositionalEmbeddings.__init__s\ (J6* 0 rcd|jtjd|jdd|jdz|jz zz }|d|d|dSr)r rrr rr r!r"s rrz*VisionRotaryPositionalEmbeddings.rope_inits I Q!,,->A ->?EEGG$(R T  We>>> rc z|jdz}tj||jj|jj}|jr9tj|dtjd|j|jzg}n8tjdtjd|j|jz|g}||jz}||jz}tj d|dz|j }tj d|dz|j }tj||gd}| | ddkd}tj tj|tj|gd}|d|d dS) Nrr$r(r4r'r)rr*Fr)rRrrrr%r&rPcatonesr+r masked_fill unsqueezer,r-r.r ) rpatches_per_tile patch_idx patch_x_pos patch_y_posx_thetay_thetafreqsr*s rr!z1VisionRotaryPositionalEmbeddings.build_rope_caches/2L DJ$4TZ=N      AY_YEUVVVVII AY_YEUVVVVI $"66 4#77 ,|[1_djIIOOQQ,|[1_djIIOOQQ 7G,"555!!)"5"5b"9"9A"=qAA UYu--uy/?/?@bIII We>>>>>rr2kwargsc |j\}}}}|||jd||dzd}|d}||jjdkr&t d|d|jjdd|jdd|d|dzd} tj |d| dz|d | d zz |d | dz|d| d zzgd} | ||j|z||} | |S) a Args: x (torch.Tensor): input tensor with shape ``[b, s, n_h, h_d]`` **kwargs (Any): additional keyword arguments. This is kept to match the forward signature of :class:`~torchtune.modules.RotaryPositionalEmbeddings`. Returns: torch.Tensor: output tensor with shape ``[b, s, n_h, h_d]`` Raises: ValueError: if sequence length of input tensor does not match the 2D RoPE cache's sequence length Notation used for tensor shapes: - b: batch size - s: sequence length - n_h: num heads - h_d: head dim r(rrzInput sequence length z. does not match 2D RoPE cache sequence length .r4r6r7) r:rr9rOr8r* ValueErrorr;rr,r=) rr2r`bsz_n_hh_dr@r>r?rAs rrBz(VisionRotaryPositionalEmbeddings.forwards].7QS ''))##C);RcQhPQRR,,q// dj&q) ) )vvv`d`j`pqr`svvv  Z__Q7AsaxCC  *V"44&/Jv$667*V"44&/Jv$667     c4#5#?cJJ}}Qr)r T)r N)rCrDrErFrGboolrrr!rrHrrBrIrJs@rrLrL}s@!%      "   &?&?&?&?P5 <5 5   5 5 5 5 5 5 5 5 rrL)typingrrrrModulerrLrrrls! m m m m m m m m `Q Q Q Q Q ryQ Q Q Q Q r