Pi;@ddlmZddlmZmZddlmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZ d efd Z d"d ed edeedeedeed e f dZ d#deedededededededed efdZeed Zd!e_d S)$)partial)ListOptional)_get_prompt_template _TemplateType) lora_phi3phi3) Phi4Tokenizer)TransformerDecoder)LORA_ATTN_MODULES)parse_hf_tokenizer_jsonreturnc 2tddddddddd S) z Builder for creating the Phi4 (14B) Instruct Model. Returns: TransformerDecoder: Instantiation of Phi4 (14B) Instruct Model ( F@h㈵>) vocab_size num_layers num_heads num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_eps)r y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/phi4/_model_builders.pyphi4_14br$s6     r"N vocab_path merges_pathspecial_tokens_pathrprompt_templatecv|t|nd}|t|nd}t|||||S)a Phi4 tokenizer. Args: vocab_path (str): Path to vocab.json. merges_path (str): Path to merges.txt. special_tokens_path (Optional[str]): Path to ``tokenizer.json`` from Hugging Face model files that contains all registered special tokens, or a local json file structured similarly. Default is None to use the canonical Phi4 special tokens. max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None. prompt_template (Optional[_TemplateType]): optional specified prompt template. If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface` class. If a dictionary, it is assumed to be a custom prompt template mapping role to the prepend/append tags. Returns: Phi4Tokenizer: Instantiation of the Phi-4 (14B) tokenizer. N)r%r&special_tokensrr()r rr )r%r&r'rr(r*templates r#phi4_tokenizerr,(sj8  * 3444  2A1L_---RV  %    r"Frlora_attn_modulesapply_lora_to_mlpapply_lora_to_output lora_rank lora_alpha lora_dropoutuse_dora quantize_basec|tdid|d|d|ddddddd d d d d dddddddd|d|d|d|d|S)a Builder for creating a Phi4 (14b) model with LoRA enabled. The Phi4 defaults are the same as in :func:`~torchtune.models.phi4.phi4`. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False apply_lora_to_output (bool): whether to apply LoRA to the model's final output projection. Default: False lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model weights Returns: TransformerDecoder: Instantiation of Phi4 (14B) model with LoRA applied r/r0r1rrrrrrrrrrrrrrrr rr2r3r4r5r6r!)r)r/r0r1r2r3r4r5r6s r# lora_phi4_14br8SsB    ++ ++ 21 7   2   "  R $  E S  ) : "\  ! "$m# r"T)r6z Builder for creating a Phi4 (14B) model with QLoRA enabled. Base model weights in linear layers that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314. Please see `lora_phi4_14b` for full API arguments. )NNNNN)FFr-r.rFF) functoolsrtypingrr torchtune.data._prompt_templatesrr)torchtune.models.phi3._component_buildersrr torchtune.models.phi4._tokenizerr torchtune.modulesr torchtune.modules.peftr torchtune.modules.tokenizersr r$strintr,boolfloatr8qlora_phi4_14b__doc__r!r"r#rGs!!!!!!!!PPPPPPPPEEEEEEEE::::::000000444444@@@@@@ $*)-!%/3 (((("#(# ( m, (  ((((Z$!&33-.333 3  3  3333333ld;;;r"