Pi90zddlZddlmZddlmZddlmZmZddlmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZmZmZdd lmZmZ Gd d ejZGddejZ d5dedededededededededed ed!ed"ed#ed$eed%ef d&Z d6ddddddddd'd'd( d)eed*e dedededededededededed ed!ed"ed#ed$eed+ed,ed-ed.e d/e d%ef.d0Z!dddddd'd'd1d2eededededededed ed#eed3eed$eed+ed,ed-ed.e d/e d%e f"d4Z"dS)7N)nn)(_register_reparametrize_state_dict_hooks)ListOptional)FrozenNF4LinearRotaryPositionalEmbeddingsTransformerSelfAttentionLayer)Gemma2Attention) GemmaRMSNorm)TransformerDecoder TiedLinear)GemmaNormEmbeddings) DoRALinearLORA_ATTN_MODULES LoRALinear) gemma_mlplora_gemma_mlpc.eZdZdeddffd ZdZxZS)TanhSoftCapping capping_valuereturnNcVt||_dSN)super__init__r)selfr __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/gemma2/_component_builders.pyrzTanhSoftCapping.__init__&s) *cV||jz }tj|}||jz}|Sr)rtorchtanh)r attn_weightss rforwardzTanhSoftCapping.forward-s1#d&88 z,// #d&88 r)__name__ __module__ __qualname__floatrr$ __classcell__rs@rrr%s^++ ++++++rrc:eZdZdZdedededdffd ZdZxZS) Gemma2FinalNormz* Combines RMSNorm and SoftCapping r embed_dimepsrNct||_t|||_t ||_dS)Nr.)rrrr rms_normr logit_capping)rrr-r.rs rrzGemma2FinalNorm.__init__7sM *$YC888 ,];;rcZ||}||}|Sr)r1r2)rxs rr$zGemma2FinalNorm.forwardBs+ MM!     q ! !r) r%r&r'__doc__r(intrr$r)r*s@rr,r,3s| < < < <  < < < < < <rr,ư>'I@>@ vocab_size num_layers num_headshead_dim num_kv_headsr-intermediate_dim max_seq_len attn_dropoutnorm_eps rope_basehidden_capping_valuefinal_capping_valuesliding_window_sizequery_pre_attn_scalarrct||| }tj}t |D]}t ||}t ||||tj|||zdtj|||zdtj|||zdtj||z|d|d|||dzdkr| nd| |}t||t|| t|| t|| t||  }| |t||}t|}t||||||t| ||  }|S) a Build the decoder associated with the gemma2 model. This includes: - Token embeddings - num_layers number of TransformerSelfAttentionLayer blocks - RMS Norm layer applied to the output of the transformer - Final projection into token space Args: vocab_size (int): number of tokens in vocabulary. num_layers (int): number of layers in the transformer decoder. num_heads (int): number of query heads. For MHA this is also the number of heads for key and value head_dim (int): dimension of head num_kv_heads (int): number of key and value heads. embed_dim (int): embedding dimension for self-attention intermediate_dim (int): intermediate dimension for MLP max_seq_len (int): maximum sequence length the model will be run with, attn_dropout (float): dropout value passed onto scaled_dot_product_attention. Default: 0.0 norm_eps (float): epsilon in RMS norms Default: 1e-6 rope_base (int): base for the rotary positional embeddings. Default: 10_000 Returns: TransformerDecoder: Instantiation of gemma model. dimrCbase)rM hidden_dimFbiasNrr-r?rAr@q_projk_projv_proj output_projpos_embeddingskv_cacherCrDrI softcappingrJr0attnmlpsa_normmlp_normsa_scale mlp_scaletok_embeddingslayersrCr?outputr@norm)rr!r ModuleListrangerr Linearr r appendrr r r,)r=r>r?r@rAr-rBrCrDrErFrGrHrIrJroperd layer_idxr]self_attlayerrcrWmodels rgemma2rpHsV &( R[ \ \ \D X " "F:&& I2BCCC"%9Y H(<5III9Y x(?eLLL9Y x(?eLLL )h"6 NNN#%8AA 7I7I 3 3t,"7!   &. 999!):::!):::"9(;;;      e(Y??N^,,K % 0) J J J   E LrF) rDrErFrGrHrIrJ lora_dropoutuse_dora quantize_baselora_attn_modulesapply_lora_to_mlp lora_rank lora_alpharqrrrsc t||}t|}tj}t |D]}|rt |||||||}nt |||}tdid|d|d|d|d|d| d | d | d |d zd kr|ndd| d|d|d|d|d|d|}t||t|| t|| t|| t|| }| |t||| |||t||| }|rt||jj|S)ad Return a version of Gemma with LoRA applied based on the passed in configuration. Note: output projection lora is not supported because it is tied to token embeddings Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False vocab_size (int): number of tokens in vocabulary. num_layers (int): number of layers in the transformer decoder. num_heads (int): number of query heads. For MHA this is also the number of heads for key and value head_dim (int): dimension of head num_kv_heads (int): number of key and value heads. embed_dim (int): embedding dimension for self-attention intermediate_dim (int): intermediate dimension for MLP max_seq_len (int): maximum sequence length the model will be run with, attn_dropout (float): dropout value passed onto scaled_dot_product_attention. Default: 0.0 norm_eps (float): epsilon in RMS norms Default: 1e-6 rope_base (int): base for the rotary positional embeddings. Default: 10_000 lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): LoRA dropout probability. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base: (bool): Whether to quantize base model weights or not. Only applied to base weights within linear layers LoRA is applied to. The final output linear projection is not supported for quantization currently. Returns: TransformerDecoder: Instantiation of Gemma model with LoRA applied to a subset of the attention projections in each layer. )rMrOrvrwrqrrrs)rMrOrs lora_modulesr-r?rAr@rFrCrDrIrRrNrZrJrvrwrqrrrsr0r[rb)dtype)rr rrgrhrrlora_gemma2_self_attentionr r rjr r,rweightrz)rtrur=r>r?r@rAr-rBrCrDrErFrGrHrIrJrvrwrqrrrsrcrWrdrlr]rmrnros r lora_gemma2r~sM@)Y??N^,,K ]__F:&&))  e +#%)!+CC 6FVcdddC-   ** i  i &  X   i  $  & 9BA 7I7I 3 3t -, #8"7  i "z &  x! "*M# (. 999!):::!):::"9(;;;      e % 0) J J J   E[ 1n>S>YZZZZ Lr)rDrFrIrZrqrrrsryrZc|stdtd|r|n|}|rtnt}d|vr||||z| | | |n0|st j|||zdnt |||zd}d|vr||||z| | | |n0|st j|||zdnt |||zd}d|vr||||z| | | |n0|st j|||zdnt |||zd}d |vr|||z|| | | |n0|st j||z|dnt ||z|d}t||| }t|||||||||d|||| | }|S) NzMust pass one or more of z as lora_modulesrT)rankalphadropoutrsFrPrUrVrWrLrS) ValueErrorrrrrrirrr )ryr-r?r@rArCrDrFrIrZrJrvrwrqrrrs adapter_clsrTrUrVrWrkrms rr|r|)s,   K(9 K K K   $0><rs SSSSSS!!!!!!!! ?>>>>>888888<<<<<<<<KKKKKKLLLLLLLLLLPPPPPPPP      bi   bi<"%!$#,0ZZZZZ Z  Z  ZZZZZZ ZZZ%SMZ !ZZZZ@$@"%!$#,03@@@-.@@  @  @@@@@@@@@@  !@"#@$%@&%SM'@*+@,-@./@01@23@45@@@@X)-#& 'rrr()r r  r  rrrrr"#r%r$C=rr !r"#r$%r&'r*+rrrrrrr