Pi*ddlmZmZddlmZmZddlmZmZddl m Z m Z ddl m Z ddlmZddlmZ de fd Zde fd Zde fd Z d"d ededeedeedeede f dZ d#deedededededededede fdZ d$deededededededede fd Z d$deededededededede fd!Zd S)%)ListOptional)_get_prompt_template _TemplateType) lora_qwen2qwen2)QWEN2_SPECIAL_TOKENSQwen2Tokenizer)TransformerDecoder)LORA_ATTN_MODULES)parse_hf_tokenizer_jsonreturnc 4tdddddddddd S) z Builder for creating a Qwen2 model initialized w/ the default 7B parameter values from https://huggingface.co/Qwen/Qwen2-7B-Instruct Returns: TransformerDecoder: Instantiation of Qwen2 7B model RJư>.A) vocab_size num_layers num_heads num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_eps rope_baserz/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/qwen2/_model_builders.pyqwen2_7br's9     r%c 6tddddddddd d d S) a Builder for creating a Qwen2 model initialized w/ the default 0.5B parameter values from https://huggingface.co/Qwen/Qwen2-0.5B-Instruct Returns: TransformerDecoder: Instantiation of Qwen2 0.5B model Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. QrrrrT rrrrrrrr r!r"tie_word_embeddingsr#r$r%r& qwen2_0_5br1-s<      r%c 6tddddddddd d d S) a Builder for creating a Qwen2 model initialized w/ the default 1.5B parameter values from https://huggingface.co/Qwen/Qwen2-1.5B-Instruct Returns: TransformerDecoder: Instantiation of Qwen2 1.5B model Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. r)r r,#rrrrTr/r#r$r%r& qwen2_1_5br6Hs<      r%Npath merges_filespecial_tokens_pathrprompt_templatec ||t|nt}|t|nd}td|||||d|S)a Tokenizer for Qwen2. Args: path (str): path to the vocab.json file. merges_file (str): path to the merges.txt file. special_tokens_path (Optional[str]): Path to ``tokenizer.json`` from Hugging Face model files that contains all registered special tokens, or a local json file structured similarly. Default is None to use the canonical Qwen2 special tokens. max_seq_len (Optional[int]): A max sequence length to truncate tokens to. Default: None prompt_template (Optional[_TemplateType]): optional specified prompt template. If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface` class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Default is None. Returns: Qwen2Tokenizer: Instantiation of the Qwen2 tokenizer N)r7r8special_tokensrr:r$)r r rr )r7r8r9rr:kwargsr<templates r&qwen2_tokenizerr?csy:  * 3444 ! 2A1L_---RV    %       r%Frlora_attn_modulesapply_lora_to_mlpapply_lora_to_output lora_rank lora_alpha lora_dropoutuse_dora quantize_basectdid|d|d|ddddddd d d d d dddddddddd|d|d|d|d|S)a Builder for creating a Qwen2 7B model with LoRA enabled. The Qwen2 defaults are the same as in :func:`~torchtune.models.qwen2.qwen2_7b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False apply_lora_to_output (bool): whether to apply LoRA to the model's final output projection. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Qwen2 7B model with LoRA applied rBrCrDrrrrrrrrrrrrrr rr!rr"rrErFrGrHrIr$r)rBrCrDrErFrGrHrIs r& lora_qwen2_7brLsB    ++ ++ 21 6   2   "  Q $  E S  ) ) :  "\! "# $$m% r%ctd id|d|ddddddd d d d d dddddddddddddd|d|d|d|d|S)!a Builder for creating a Qwen2 0.5B model with LoRA enabled. The Qwen2 defaults are the same as in :func:`~torchtune.models.qwen2.qwen2_0_5b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Qwen2 0.5B model with LoRA applied Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. rBrCrDFrr)rr*rr+rr,rr-rr.rrr rr!rr"rr0TrErFrGrHrIr$rKrBrCrErFrGrHrIs r&lora_qwen2_0_5brOsD    ++ ++ #U 6   2   "  Q #  E S  ) !D )  :! ""\# $% &$m' r%ctd id|d|ddddddd d d d d dddddddddddddd|d|d|d|d|S)!a Builder for creating a Qwen2 1.5B model with LoRA enabled. The Qwen2 defaults are the same as in :func:`~torchtune.models.qwen2.qwen2_1_5b`, while LoRA default params are based on https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Qwen2 1.5B model with LoRA applied Note: Qwen2 0.5B and Qwen2 1.5B model builders will enable `tie_word_embeddings` by default and returns an instance of `TransformerDecoder`. rBrCrDFrr)rrrr3rr,rr4rr5rrr rr!rr"rr0TrErFrGrHrIr$rKrNs r&lora_qwen2_1_5brQsD    ++ ++ #U 6   2   "  Q $  E S  ) !D )  :! ""\# $% &$m' r%)NNNN)FFr@rArFF)Fr@rArFF)typingrr torchtune.data._prompt_templatesrr*torchtune.models.qwen2._component_buildersrr!torchtune.models.qwen2._tokenizerr r torchtune.modulesr torchtune.modules.peftr 'torchtune.modules.transforms.tokenizersr r'r1r6strintr?boolfloatrLrOrQr$r%r&r]s "!!!!!!!PPPPPPPPHHHHHHHHRRRRRRRR000000444444KKKKKK$,&6&:)-!%/3 ** **"#*# * m, *****^$!&44-.444 4  4  4444444r$66-.666 6  6  666666v$66-.666 6  6  66666666r%