PiddlmZmZddlmZmZddlmZddlm Z ddl m Z ddl m Z ddl mZddlmZ d efd Zd"d ed eedee d e fdZ d#dee dedededededed efdZeedZde_d efdZ d#dee dedededededed efd ZeedZd!e_d S)$)ListOptional)gemma lora_gemma)TransformerDecoder)GemmaTokenizer)LORA_ATTN_MODULES) _TemplateType)_get_prompt_template)partialreturnc 4tddddddddd d S) z Builder for creating a Gemma 2B model initialized w/ the default 2b parameter values from: https://blog.google/technology/developers/gemma-open-models/ Returns: TransformerDecoder: Instantiation of Gemma 2B model @ ư> vocab_size num_layers num_headshead_dim num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_epsrz/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/gemma/_model_builders.pygemma_2br(s9     r&Npathr!prompt_templatecHt|||t|ndS)a Tokenizer for Gemma. Args: path (str): path to the tokenizer max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None. prompt_template (Optional[_TemplateType]): optional specified prompt template. If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface` class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Returns: GemmaTokenizer: Instantiation of the Gemma tokenizer Nr)r!r*)rr r,s r'gemma_tokenizerr-.sG" txGxSNbcrNsNsNsY] ^ ^ ^^r&Frrlora_attn_modulesapply_lora_to_mlp lora_rank lora_alpha lora_dropoutuse_dora quantize_basec|tdid|d|ddddddd d d d d dddddddddd|d|d|d|d|S)a Builder for creating a Gemma 2B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_2b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma 2B model with LoRA applied r/r0rrrrrrrrrrrrr rr!rr"rr#rr1r2r3r4r5r%rr/r0r1r2r3r4r5s r' lora_gemma_2br9Bs@    ++ ++ 7 2   !     Q $  D S  ) : "\  ! "$m# r&T)r5z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma_2b` for full API arguments. c 4tdddddddddd S) z Builder for creating a Gemma 7B model initialized w/ the default 7b parameter values from: https://blog.google/technology/developers/gemma-open-models/ Returns: TransformerDecoder: Instantiation of Gemma 7B model rr.r `rrrrr$r%r&r'gemma_7br>s9     r&c|tdid|d|ddddddd d d dd d ddddddddd|d|d|d|d|S)a Builder for creating a Gemma 7B model with LoRA enabled. The Gemma defaults are the same as in :func:`~torchtune.models.gemma.gemma_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Gemma 7B model with LoRA applied r/r0rrrr;rr.rrrrr<r r=r!rr"rr#rr1r2r3r4r5r%r7r8s r' lora_gemma_7br@s@    ++ ++ 7 2   "     R $  D S  ) : "\  ! "$m# r&z Builder for creating a Gemma model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_gemma_7b` for full API arguments. )NN)Frr.rFF)typingrr*torchtune.models.gemma._component_buildersrrtorchtune.modulesr!torchtune.models.gemma._tokenizerrtorchtune.modules.peftr torchtune.data._prompt_templatesr r functoolsr r(strintr-boolfloatr9qlora_gemma_2b__doc__r>r@qlora_gemma_7br%r&r'rOst "!!!!!!!HHHHHHHH000000<<<<<<444444::::::AAAAAA $,^^#^HSM^S[\iSj^wE^^^^,$22-.222 2  2  222222hd;;;$0$22-.222 2  2  222222hd;;;r&