Pi^kddlZddlmZmZmZddlZddlmcmZ ddlmZddl m Z m Z Gddej ZGddej ZGd d ej ZdS) N)AnyDictTuple)nn)distribute_tensorDTensorcheZdZdZdedededdffd Zdejd ee dejfd Z xZ S) TokenPositionalEmbeddinga Token positional embedding for images, different for every token in an image. Notice that tile is different from patch (token). For details, please check the documentation of :class:`torchtune.modules.vision_transformer.VisionTransformer`. Args: embed_dim (int): The dimensionality of each token embedding. tile_size (int): The size of your image tiles, if the image was tile-cropped in advance. Otherwise, the size of the input image. In this case, the function will consider your image as a single tile. patch_size (int): The size of each patch. Used to divide the tiles into patches. E.g. for ``patch_size=40``, a tile of shape (400, 400) will have 10x10 grid of patches with shape (40, 40) each. embed_dim tile_size patch_sizereturnNct||z}|dzdz}|dz}tj|t j||fz|_dSN)super__init__r Parametertorchrandnpositional_embedding)selfr r r patch_grid_sizen_tokens_per_tilescale __class__s ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/clip/_position_embeddings.pyrz!TokenPositionalEmbedding.__init__ sn #z1+Q.24$&L EK!2I >?? ?% % !!!xargsc||jzS)z Args: x (torch.Tensor): torch.Tensor with shape (..., n_tokens_per_tile, embed_dim) *args (Tuple[Any]): Optional args. Returns: torch.Tensor: The input tensor with added positional embeddings. )r)rr!r"s rforwardz TokenPositionalEmbedding.forward)s4,,,r ) __name__ __module__ __qualname____doc__intrrTensorrrr$ __classcell__rs@rr r s   # # 3 4       - -eCj -U\ - - - - - - - -r r c PeZdZdZdededededdf fd Zejd ee e fd e d e e d ee e fddf d Z e dejdedejfdZe dejdededejfdZdejdejdejfdZxZS)TiledTokenPositionalEmbeddinga Token positional embedding for tiled images, different for every tile, different for every token. There are two positional embeddings in this module: * local_token_positional_embedding: same for every tile, different for every token. Equivalent to :class:`torchtune.models.clip._position_embeddings.TokenPositionalEmbedding`, but gated. * global_token_positional_embedding: different for every tile, different for every token. Notice that tile is different from patch (token). For details, please check the documentation of :class:`torchtune.modules.vision_transformer.VisionTransformer`. Args: max_num_tiles (int): The maximum number of tiles an image can be divided into. embed_dim (int): The dimensionality of each token embedding. tile_size (int): The size of your image tiles, if the image was tile-cropped in advance. Otherwise, the size of the input image. In this case, the function will consider your image as a single tile. patch_size (int): The size of each patch. Used to divide the tiles into patches. E.g. for ``patch_size=40``, a tile of shape (400, 400) will have 10x10 grid of patches with shape (40, 40) each. max_num_tilesr r r rNc t||z}|dzdz|_|dz}tj|t j|j|fz|_tj|t j|||j|z|_tjt j d|_ | |j dSr) rrrrrrr local_token_positional_embedding!global_token_positional_embeddingzerosgate"_register_load_state_dict_pre_hook_load_state_dict_hook)rr/r r r rrrs rrz&TiledTokenPositionalEmbedding.__init__Ms #z1!0!!3a!7413 EK!7 CDD D1 1 - 24 k&  2 2 .LQ00  //0JKKKKKr state_dictprefixr"kwargsc ||dz}|t|tr%d}|j}|j}|}nd}|j\} } tj| dz dzdkrtd| d|j j\} } | |ttj| dz  }|rt||| }|||dz<|j|j jkr%td |jd |j jd ||dz} | t| tr%d}| j}| j}| } nd}| j\}}} }tj| dz dzdkrtd| d|jj\}}} } || |ttj| dz } |rt| || } | ||dz<| j|jjkr'td| jd |jjd dSdS)a_ Interpolates positional embeddings to accomodate different number of tiles and tokens per tile, in case the model was instantiated with different settings than the one you are loading the state dict from. For more info, please check self._resize_local_position_embedding and self._resize_global_position_embedding functions. Args: state_dict (Dict[str, Any]): The state dict to load. prefix (str): The prefix of the state dict. *args (Tuple[Any]): Additional positional arguments. **kwargs (Dict[str, Any]): Additional keyword arguments. Raises: ValueError: If loaded local or global embedding n_tokens_per_tile is not derived from a squared grid, **or** if after interpolation, the shape of the loaded local embedding is not compatible with the current embedding, **or** if after interpolation, the shape of the loaded global embedding is not compatible with the current embedding. r1NTFrrzCLoaded local positional embedding has shape inpt_n_tokens_per_tile=zS, which indicates a grid_size that is not squared. This is currently not supported.)local_pos_embedtgt_patch_grid_size device_mesh placementsz,Loaded local positional embedding has shape z&, after interpolation. Expected shape .r2)global_pos_embedtgt_max_num_tilesr<z-Loaded global positional embedding has shape )get isinstancerr>r? full_tensorshapemathsqrt ValueErrorr1 _resize_local_position_embeddingr)rr2!_resize_global_position_embedding)rr7r8r"r9inpt_local_pos_embedlocal_embed_is_shardedlocal_embed_device_meshlocal_embed_placementsinpt_n_tokens_per_tileinpt_embed_dimtgt_n_tokens_per_tile tgt_embed_diminpt_global_pos_embedglobal_embed_is_shardedglobal_embed_device_meshglobal_embed_placements_tgt_max_num_tiles_xtgt_max_num_tiles_ys rr6z3TiledTokenPositionalEmbedding._load_state_dict_hookjsB *~~ 7 7   + .88 /)-&*>*J')=)H&';'G'G'I'I$$).&6J5O 2 "Ny/!344q8A== iCYiii5; %$(#H#H 4$' 2G!2K(L(L$M$M$I$$ & '8( 75((($% ;; %*8>??!jCWC]jj;?;`;fjjj !+ 8 8! !  ! , /99 0*.'+@+L(*?*J'(=(I(I(K(K%%*/'.C.I +Aq(!y/!344q8A== iCYiii6<  ##%%)$J$J!6"5$' 2G!2K(L(L$M$M%K%% ! ' (9) 86)))%& << &+9?@@!kDYD_kk;?;a;gkkki - ,b@@r r;r<c|j\}}ttj|dz }|dg|dd}}|d||ddddd}t j|||gdd }|dddddd|}|d}tj ||gd }| S) a Interpolates the local position embedding for a vision encoder to accommodate a different number of tokens per tile. This is the only dimension that changes during interpolation. Args: local_pos_embed (torch.Tensor): The position embeddings tensor to be resized. It has shape [n_tokens_per_tile, emb_dim], where the first token is the CLS token and n_tokens_per_tile = patch_grid_size**2 + 1. tgt_patch_grid_size (int): The target size of each patch grid, i.e., the square root of the number of tokens per tile, excluding the class token. Returns: torch.Tensor: The resized position embeddings tensor of shape [tgt_n_tokens_per_tile, dim], where tgt_n_tokens_per_tile = tgt_patch_grid_size**2 + 1. Example: >>> import torch >>> import math >>> local_pos_embed = torch.randn((10*10+1, 64)) # Example input tensor >>> tgt_patch_grid_size = 20 # Target number of tokens per tile >>> resized_pos_embed = _resize_local_position_embedding(local_pos_embed, tgt_patch_grid_size) >>> print(resized_pos_embed.shape) torch.Size([20*20+1, 64]) rrNrbilinearTsizemode align_cornersdim) rFr)rGrHreshapepermuteF interpolatesqueezercat contiguous)r;r<rPrQinpt_patch_grid_size cls_tokens rrJz>TiledTokenPositionalEmbedding._resize_local_position_embeddings&<2A1F."49-Ca-G#H#HII QC ABB # *11 #%92  '!Q1   - %':;    *11!Q1==EE r>   *11!44 )Y$@aHHH))+++r rArBcX|ddddddddf}|dddddgddf}|j\}}}}ttj|} |||| | |}|ddddd}||| z|| z|}|d}t||zt||zf} |dddd}tj || dd }|dddd d}| |||||}|ddddd}| ||t|dz|}|dddd}tj |||fdd } | dddd}tj ||gd }|S) af Interpolates the global position embedding for a vision encoder to accommodate new grid dimensions. The embedding dimension is not changed during interpolation, only max_num_tiles and num_tokens_per_tile. Args: global_pos_embed (torch.Tensor): The input global position embeddings tensor of shape [max_num_tiles_x, max_num_tiles_y, num_tokens_per_tile, embed_dim], where num_tokens_per_tile = inpt_patch_grid_size * inpt_patch_grid_size + 1 (CLS token), and max_num_tiles_x == max_num_tiles_y. tgt_max_num_tiles (int): The target maximum number of tiles along one dimension (assumed square grid). tgt_patch_grid_size (int): The target size of each patch grid, i.e., the square root of the number of tokens per tile, excluding the class token. Returns: torch.Tensor: The resized global position embeddings tensor of shape [tgt_max_num_tiles, tgt_max_num_tiles, tgt_patch_grid_size * tgt_patch_grid_size + 1, embed_dim]. Example: >>> import torch >>> global_pos_embed = torch.arange(3*3*(2*2+1)*4).reshape((3, 3, 2*2+1, 4)) # Example input tensor >>> tgt_max_num_tiles = 2 # Target maximum number of tiles >>> tgt_patch_grid_size = 3 # Target patch grid size >>> resized_global_pos_embed = ( >>> _resize_global_position_embedding(global_pos_embed, tgt_max_num_tiles, tgt_patch_grid_size)) >>> print(resized_global_pos_embed.shape) torch.Size([2, 2, 3*3+1, 4]) Nrrrr]r^Tr_rc)rFr)rGrHrerfrk unsqueezergrhriviewrrj) rArBr< pos_embed cls_embedmax_num_tiles_xmax_num_tiles_yrr rltgt_sizecls_embed_resizeds rrKz?TiledTokenPositionalEmbedding._resize_global_position_embeddingFsH%QQQ122qqq[1 $QQQA3\2  O      #49->#?#?@@%%      %%aAq!44??AA %% 2 2 2 2   ''**  !$77 8 8 !$77 8 8  %%aAq11 M     %%aAq1199!<< NN        %%aAq!44??AA NN   #Q& ' '    %%aAq11 M #%67     &--aAq99 !9i%;CCC**,,,r r! aspect_ratioc|j\}}}}||jd|jz zz}|||||}t |D]\}\}} t || z} |jd|d| ddddf} | | |j |} | |jz} ||d| ddddfxx| z cc<|S)a Args: x (torch.Tensor): torch.Tensor with shape (bsz * n_imgs, n_tiles, n_tokens_per_tile, embed_dim). aspect_ratio (torch.Tensor): torch.Tensor with shape (bsz * n_imgs, 2), where aspect_ratio[k] represents the aspect ratio of the k^th image of the batch before tile-cropping, e.g. aspect_ratio[k] = (2,1). Returns: torch.Tensor: The input tensor with added positional embeddings. rN) rFr1r4tanhrq enumerater)r2rer) rr!rxbsz_and_n_imgsn_tilesrr batch_idx n_tiles_h n_tiles_wn_non_padded_tilesrrs rr$z%TiledTokenPositionalEmbedding.forwards.AB=!2I 6!dinn>N>N:NO P FF>7,=y I I1:<1H1H A A -I- 9"%Y%:!;!; > JYJ111,I "))"D$:II"DINN$4$44I i,,,aaa2 3 3 3y @ 3 3 3 3r )r%r&r'r(r)rrno_gradrstrrrr6 staticmethodr*rJrKr$r+r,s@rr.r.5s.L L-0L=@LNQL LLLLLL:U]__UcNUUSz U sCx. U  UUU_Un@,@,r?rErrFrI_resize_position_embeddingr)rr7r8r"r9rembedding_is_shardedr>r?rYrZtgt_num_tokenstgt_embinpt_max_num_tiles_xinpt_max_num_tiles_yinpt_num_tokensinpt_emb embedding_news rr6z-TilePositionalEmbedding._load_state_dict_hooks8NN6K#788  )W-- -'+$'3 &1 %1133 ',$$  ##6K/06  $$.00H4G4G m)-)=mmDNvXcOcDdDjmm $';;; aJN.J^aa !;;-@<M$  1! +)!!! 0=Jv + ,"dn&::: Z)-)=ZZDQDWZZs ! p;:r rrBrc|dddd}tj|||fdd}|dddd}|S)a Interpolates positional embeddings to accomodate a different max_num_tiles. These are the only dimensions that changes during interpolation. Args: embedding (torch.Tensor): torch.Tensor with shape (max_num_tiles, max_num_tiles, 1, embed_dim tgt_max_num_tiles (int): The number of tiles to resize to. Returns: torch.Tensor: The resized embedding. Example: >>> import torch >>> # create dummy embedding >>> embedding = torch.arange(2*2*2*2).reshape(2, 2, 2, 2).float() >>> resized_embed = _dynamic_resize(embedding, tgt_max_num_tiles=1) >>> print(resized_embed.shape) >>> torch.Size([1, 1, 2, 2]) rr]rrr^Tr_)rfrgrhrk)rrBs rrz2TilePositionalEmbedding._resize_position_embedding\sr0%%aAq11 M #%67    %%aAq11 ##%%%r r!rxc,t|D]\}\}}t||z}|jd|d|ddddf}||d|j}||d|ddddfxx||jzz cc<|S)a args: x (torch.Tensor): torch.Tensor with shape (bsz * n_imgs, n_tiles, n_tokens_per_tile, embed_dim). aspect_ratio (torch.Tensor): torch.Tensor with shape (bsz * n_imgs, 2), representing the aspect ratio of the image before tile-cropping, e.g. (2,1). returns: torch.Tensor: The input tensor with added positional embeddings. Nr)r{r)rrer r4rz)rr!rxr~rrrrrs rr$zTilePositionalEmbedding.forwards2;<1H1H T T -I- 9"%Y%:!;!; z z:I:qqq!!!'CDI"))*>CSCS7S S 3 3 3 3r )r%r&r'r(r)rrrrrrrr6rr*rr$r+r,s@rrrs8  LLLLLLLL$U]__YcNYYSz Y sCx. YYY_Yv!&<!&47!& !&!&!&\!&FU\elr r)rGtypingrrrrtorch.nn.functionalr functionalrgtorch.distributed._tensorrrModuler r.rr rrs ########## @@@@@@@@"-"-"-"-"-ry"-"-"-JjjjjjBIjjjZ vvvvvbivvvvvr