PiW*ddlmZddlmZmZmZddlmZddlm Z m Z m Z ddl m Z mZddlmZmZmZmZmZmZddlmZddlmZmZmZdd lmZmZejd d d d d ddd d f de de de de de dede de!de!deee de!de de de!de!def d Z" d?de de de d$e d%e d&e#f d'Z$ d@d(e d)e d*e dej%d+e!def d,Z& d@ejd d d d ddd-d.d/d d d0 d1eed2e!de de de de de dede de!deee de!de de d3e d4e#d5e#d6e!d+e!def(d7Z'd/d d/d d d8d1eede d9e de d:e d;e#de!d3e d4e#d5e#d6e!d+e!defd<Z(d/d d d=d(e d)e d*e dej%d3e d4e#d5e#d6e!d+e!defd>Z)d S)A)partial)CallableListOptional)nn)TiledTokenPositionalEmbeddingTilePositionalEmbeddingTokenPositionalEmbedding)CLIPTextEncoder QuickGELU) FeedForward Fp32LayerNormFrozenNF4LinearMultiHeadAttentionTransformerSelfAttentionLayer VisionRotaryPositionalEmbeddings)+reparametrize_as_dtype_state_dict_post_hook) DoRALinearLORA_ATTN_MODULES LoRALinear) CLSProjectionVisionTransformeriTFN tile_size patch_size embed_dim num_layers num_heads activationcls_output_dim attn_biasuse_rope out_indicesoutput_cls_projection max_num_tiles in_channelsappend_cls_tokenuse_tile_pos_embedreturnc||zdkrtd|d|||z}| rt||nd}|rt||| |dzd| nd}t||||t j||| t j||| t j||| t j||| |d d }t |d |z||}t||t|dt|ddd}|r<| dkr6t| |}t| |}t| |||}nd}d}t|||}t||||||| |||| |  S)a1 Builds the vision encoder associated with the clip model. This includes: - TransformerEncoderLayer - positional embeddings - CLS projection (optional) For details, please check the documentation of :class:`torchtune.modules.vision_transformer.VisionTransformer`. Args: tile_size (int): The size of your image tiles, if the image was tile-cropped in advance. Otherwise, the size of the input image. In this case, the function will consider your image as a single tile. patch_size (int): The size of each patch. Used to divide the tiles into patches. E.g. for ``patch_size=40``, a tile of shape (400, 400) will have 10x10 grid of patches with shape (40, 40) each. embed_dim (int): The dimensionality of each patch embedding (token). num_layers (int): The number of transformer layers. num_heads (int): The number of attention heads in each transformer layer. activation (Callable): The activation function to use in the MLP layer. cls_output_dim (int): The dimensionality of the output tensor from the CLS projection module. attn_bias (bool): Boolean for if to use bias in the attention module. Default True. use_rope (bool): If True, include 2D rope in attention in each transformer layer. Default: False out_indices (Optional[List[int]]): The indices of hidden layers to return. If provided, it will return the intermediate results of the transformer layers before they go through a next layer. For example, ``out_indices=[0,3]`` will return the tokens before they go through the first and fourth layers. output_cls_projection (bool): If True, only the CLS token projection will be outputted, instead of all tokens. Defaults to False. max_num_tiles (int): The maximum number of tiles that can be processed. This is used to determine the size of the positional embeddings. in_channels (int): The number of image input channels. append_cls_token (bool): If True, adds CLS token embedding to the end of the sequence in the vision transformer. Default is False, which adds CLS token to the beginning of the sequence. use_tile_pos_embed (bool): If True, use pre-tile, post-tile, and tiled token positional embeddings, if max_num_tiles > 1. If False, only use standard token positional embeddings. Returns: A `VisionTransformer` object. Raises: AssertionError: If ``embed_dim`` is not divisible by ``num_heads``. rz.embed_dim must be divisible by num_heads, got z and rr!Ni')rrr&dimbaser(biasF) rr num_kv_headshead_dimq_projk_projv_proj output_projpos_embeddings attn_dropout is_causalr)in_dim hidden_dimout_dimr h㈵>epsattnmlpsa_normmlp_normsa_scale mlp_scaler&rr&rrrrrr) rlayertoken_pos_embeddingpre_tile_pos_embedpost_tile_pos_embedcls_projectionr$rrrr'r() ValueErrorrrrrLinearclip_mlprrr rr r)rrrrrr r!r"r#r$r%r&r'r(r)r4rQrope self_attnrDtransformer_layerrOrPrNs }/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/clip/_component_builders.pyclip_vision_encoderrYslx9!! XY X XY X X   I%H !  .IIII   (!'A -      #yII>>>yII>>>yII>>>IiCCC   I y=:<<    C 6  iT222yd333  ma//4'9   6'9   <'!    ""6J)    /-/%)    Mr? vocab_size max_seq_lennorm_epsc t|||||ztj||tj||tj||tj||}t|||dzt }t ||tj||tj||}tj||} t|| ||||S)ap Text encoder for CLIP. CLIP is a model that encodes text and images into a shared vector space. Blog post: https://openai.com/index/clip/ Paper: https://arxiv.org/abs/2103.00020 Args: embed_dim (int): embedding/model dimension size num_heads (int): number of attention heads num_layers (int): number of transformer layers vocab_size (int): size of the vocabulary, default 49408 max_seq_len (int): context size, default 77 norm_eps (float): small value added to denominator for numerical stability, default 1e-5 Returns: CLIPTextEncoder )rrr3r4r5r6r7r8r)r<r>r=r r@)rCrDrErF)rM final_normr]r^rr)rrrSrTr r LayerNormr ) rrrr]r^r_rCrD encoder_layerras rXclip_text_encoderrds4 i'yI..yI..yI..Ii33   D q=;;    C 2   YH555iX666 M iX666J     rZr<r>r= quantize_basec |stj||nt||fddi|}|stj||nt||fddi|}t||d|S)z= Build the MLP layer associated with the clip model. r1TN gate_proj down_projup_projr )rrSrr )r<r>r=r requantization_kwargsrhris rXrTrTs S &*%%% VZ R Rd R>Q R R T *g&&& Z S St S?R S S y$:   rZr2) r r!r"r$r%r&r' lora_rank lora_alpha lora_dropoutuse_dorare lora_modulesapply_lora_to_mlprnrorprqc ||zdks Jd| rt||nd}td||||||zd|||||| d |}|r"td|d|z|||||||d |}ntd|d|z|||d |}t ||t |d t |d dd }| d krd}d}t |||}n5t| |}t| |}t| |||}t||||||| ||||  }|r)| ttd|S)a Build a LoRA implementation of the CLIP vision encoder. Args: lora_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False tile_size (int): The size of your image tiles, if the image was tile-cropped in advance. Otherwise, the size of the input image. In this case, the function will consider your image as a single tile. patch_size (int): The size of each patch. Used to divide the tiles into patches. E.g. for ``patch_size=40``, a tile of shape (400, 400) will have 10x10 grid of patches with shape (40, 40) each. embed_dim (int): The dimensionality of each patch embedding (token). num_layers (int): The number of transformer layers. num_heads (int): The number of attention heads in each transformer layer. activation (Callable): The activation function to use in the MLP layer. cls_output_dim (int): The dimensionality of the output tensor from the CLS projection module. attn_bias (bool): Boolean for if to use bias in the attention module. Default False. out_indices (Optional[List[int]]): The indices of hidden layers to return. If provided, it will return the intermediate results of the transformer layers before they go through a next layer. For example, ``out_indices=[0,3]`` will return the tokens before they go through the first and fourth layers. output_cls_projection (bool): If True, only the CLS token projection will be outputted, instead of all tokens. Defaults to False. max_num_tiles (int): The maximum number of tiles that can be processed. This is used to determine the size of the positional embeddings. in_channels (int): The number of image input channels. lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): LoRA dropout probability. Default: 0.0 use_dora (bool): Whether to use DoRA layers instead of LoRA layers. Default is ``False``. quantize_base: (bool): Whether to quantize base model weights or not. Only applied to base weights within linear layers LoRA is applied to. The final output linear projection is not supported for quantization currently. Returns: VisionTransformer: Instantiation of VisionTransformer model. rz(embed_dim must be divisible by num_headsr,Nr2) rrrrr3r4r:rnrorprqrer"r) r<r=r>r rnrorerprq)r<r=r>r rer?r@rBrIrLrJrK) rrMrNrOrPrQr$rrrr'T)offload_to_cpu) rlora_clip_attention lora_clip_mlprTrrr r rr_register_state_dict_hookrr)rrrsrrrrrr r!r"r$r%r&r'rnrorprqrerkrQrVrDrWrOrPrNmodels rXlora_clip_vision_encoderr{srD y A % % %'Q % % % !  .IIII $!i'!# I   9}!z||!'%   "    9}!z||'   "   6  iT222yd333 !"6J)   5'9   6'9   <'!     /-/%   E  '' ?PT U U U    LrZ)r:r"rprqrer4r3r:c |stdtd| rtnt} d|vr| |||zf||| | d| n.| st j|||z|nt |||zfd|i| }d|vr| |||zf||| | d| n.| st j|||z|nt |||zfd|i| }d|vr| |||zf||| | d| n.| st j|||z|nt |||zfd|i| }d |vr| ||f||| | d| n(| st j|||nt ||fd|i| }t||||||||d | }|S) a Return an instance of :func:`~torchtune.modules.MultiHeadAttention` with LoRA applied to a subset of its linear layers Args: lora_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. embed_dim (int): embedding dimension for self-attention head_dim (int): dimension of each head in the multihead attention. Usually computed as ``embed_dim // num_heads``. num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure `num_heads` % `num_kv_heads` == 0. For standard MHA set `num_kv_heads` == `num_heads`, for GQA `num_kv_heads` < `num_heads`, and for MQA set `num_kv_heads` == 1. attn_dropout (float): dropout value passed onto scaled_dot_product_attention. Default: 0.0 lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): LoRA dropout probability. Default: 0.0 use_dora (bool): Whether to use DoRA layers instead of LoRA layers. Default is ``False``. quantize_base (bool): Whether to quantize base model parameters for linear layers LoRA is being applied to. Default is ``False``. Returns: MultiHeadAttention: instantiation of self-attention module with LoRA applied to a subset of Q, K, V, output projections. Raises: ValueError: If lora_modules arg is an empty list zMust pass one or more of z as lora_modulesr5)rankalphadropoutrer0r1r6r7r8N) rrr3r4r5r6r7r8r9r:)rRrrrrrSrr)rrrr4rr3r:r"rnrorprqrerk adapter_clsr5r6r7r8rVs rXrwrwsd   K(9 K K K   !)8**jK | # #      '   "   ! BIiX!5I F F F F 9x/6?CV : | # #   8 #  '   "   ! BIi!8y I I I I x'&  @ | # #   8 #  '   "   ! BIi!8y I I I I x'&  @ L ( (     '   "   ! BIi ; ; ; ; 9+48K(#!!   I rZ)rprqrec |rtnt} | d||||||dd| } | d||||||dd| } t| | d|S)zQ Build the MLP layer with LoRA applied to the gate and down projections. T)r<r>r}r~rreuse_biasNrgrv)rrr ) r<r>r=r rnrorprqrerkrrhris rXrxrxOs !)8**jK  #     I  #     I y$:   rZ)r[r\r?)F)* functoolsrtypingrrrtorchr*torchtune.models.clip._position_embeddingsrr r #torchtune.models.clip._text_encoderr r torchtune.modulesr rrrrrtorchtune.modules.common_utilsrtorchtune.modules.peftrrr$torchtune.modules.vision_transformerrrSiLUintboolrYfloatrdModulerTr{rwrxrvrZrXrs}++++++++++ KJJJJJJJWVVVVVLLLLLLLLLLQQQQQQQQ7'+"'"#TTTTT T  T  TTTT$s)$T TTTTT !TTTTv 88888 8  8  8888@      >$d7'+"'-ddd()dd  d  ddddddd$s)$d dd !d$%d&'d()d*+d,-d01dddd^WWW()W W  W  WWWWWWWWW"#WWWWD''' '' '  '  '''''''''''rZ