Pi'ddlmZddlmZddlmZmZddlmZddl m Z  defdZ dd ee d e de dedede de defdZeedZde_defdZ dd ee d e de dedede de defdZeedZde_defdZ dd ee d e de dedede de defdZeedZde_dS))partial)List)gemma2 lora_gemma2)TransformerDecoder)LORA_ATTN_MODULESreturnc:tddddddddd d d d d  S)z Builder for creating a Gemma2 2B model initialized w/ the default 2b parameter values from: https://github.com/google/gemma_pytorch/blob/main/gemma/config.py Returns: TransformerDecoder: Instantiation of Gemma2 2B model  $ ư>>@I@ vocab_size num_layers num_headshead_dim num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_epshidden_capping_valuefinal_capping_valuesliding_window_sizer{/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/gemma2/_model_builders.py gemma2_2br*sB !     r(Fr rlora_attn_modulesapply_lora_to_mlp lora_rank lora_alpha lora_dropoutuse_dora quantize_basectd"id|d|ddddddd d d d d dddddddddddddddd|d|d|d |d!|S)#a Builder for creating a Gemma2 2B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_2b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma2 2B model with LoRA applied r,r-rr rr rr rrrrrrrrr rr!rr"rr#rr$rr%rr.r/r0r1r2r'rr,r-r.r/r0r1r2s r)lora_gemma2_2br6-s @    ++ ++ 7 2   !     Q $  D S  "T !D !D  )! ":# $"\% &' ($m) r(T)r2z Builder for creating a Gemma2 model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma2_2b` for full API arguments. c:tddddddddd d d d d  S)z Builder for creating a Gemma2 9B model initialized w/ the default 9b parameter values from: https://github.com/google/gemma_pytorch/blob/main/gemma/config.py Returns: TransformerDecoder: Instantiation of Gemma 9B model r *r+rr 8rrrrrrrr&r'r(r) gemma2_9br;nsB !     r(ctd"id|d|ddddddd d d d d dddddddddddddddd|d|d|d |d!|S)#a Builder for creating a Gemma 9B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma2 9B model with LoRA applied r,r-rr rr8rr+rrrr rr9rr:r rr!rr"rr#rr$rr%rr.r/r0r1r2r'r4r5s r)lora_gemma2_9br=s @    ++ ++ 7 2   "     Q $  D S  "T !D !D  )! ":# $"\% &' ($m) r(z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma2_9b` for full API arguments. c<tddddddddd d d d d dS)z Builder for creating a Gemma2 27B model initialized w/ the default 27b parameter values from: https://github.com/google/gemma_pytorch/blob/main/gemma/config.py Returns: TransformerDecoder: Instantiation of Gemma2 27B model r . r+rrrrrr)rrrrrrrr r!r"r#r$r%query_pre_attn_scalarr&r'r(r) gemma2_27brFsE !  !   r(ctd$id|d|ddddddd d d d d dddddddddddddddddd|d |d!|d"|d#|S)%a Builder for creating a Gemma2 27B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma2 27B model with LoRA applied r,r-rr rr?rr@rrArr+rrBrrCr rr!rr"rr#rr$rr%rrErDr.r/r0r1r2r'r4r5s r)lora_gemma2_27brHs@    ++ ++ 7 2   "     R $  D S  "T !D !D  "c! ")# $:% &"\' () *$m+ r(z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma2_27b` for full API arguments. N)Fr r+rFF) functoolsrtypingr+torchtune.models.gemma2._component_buildersrrtorchtune.modulesrtorchtune.modules.peftrr*boolintfloatr6qlora_gemma2_2b__doc__r;r=qlora_gemma2_9brFrHqlora_gemma2_27br'r(r)rUs KKKKKKKK000000444444 %6$55-.555 5  5  555555p'.===%6$55-.555 5  5  555555p'.===&8$66-.666 6  6  666666r7?$???r(