Pic : d dl mZmZ d dlmZmZ d dlmZ d dlm Z d dl
m Z d dl m
Z
d dlmZ d dlmZ d d lmZ d
e fd Zd"d
edee dee dee d
ef
dZ d#dee dededededed ed ed
e fd Z e
ed Zd!e_ d S )$ )ListOptional)phi3 lora_phi3)Phi3MiniTokenizer)TransformerDecoder)LORA_ATTN_MODULES)partial)parse_hf_tokenizer_json)
_TemplateType)_get_prompt_templatereturnc 2 t ddddddddd S ) a`
Builder for creating the Phi3 Mini 4K Instruct Model.
Ref: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
Note:
This model does not currently support 128K context length nor optimizations
such as sliding window attention.
Returns:
TransformerDecoder: Instantiation of Phi3 Mini 4K Instruct Model
@} h㈵>)
vocab_size
num_layers num_heads num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_eps)r y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/phi3/_model_builders.py phi3_minir# s6
r! Npathspecial_tokens_pathr prompt_templatec t |t | nd}|t | nd}t | ||| S )a Phi-3 Mini tokenizer.
Ref: https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/tokenizer_config.json
Args:
path (str): Path to the SPM tokenizer model.
special_tokens_path (Optional[str]): Path to ``tokenizer.json`` from Hugging Face
model files that contains all registered special tokens, or a local json file
structured similarly. Default is None to use the canonical Phi3 special tokens.
max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages,
after which the input will be truncated. Default is None.
prompt_template (Optional[_TemplateType]): optional specified prompt template.
If a string, it is assumed to be the dotpath of a :class:`~torchtune.data.PromptTemplateInterface`
class. If a dictionary, it is assumed to be a custom prompt template mapping role to the
prepend/append tags.
Note:
This tokenizer includes typical LM EOS and BOS tokens like
, , and . However, to support chat completion,
it is also augmented with special tokens like
and .
Returns:
Phi3MiniSentencePieceBaseTokenizer: Instantiation of the SPM tokenizer.
N)r$ special_tokensr r& )r r
r )r$ r% r r& r( templates r" phi3_mini_tokenizerr* , sR 2 FYEd,-@AAAjnN8G8S#O444Y]H $~S^px y y yyr! F r lora_attn_modulesapply_lora_to_mlpapply_lora_to_output lora_rank
lora_alpha lora_dropoutuse_dora
quantize_basec | t di d| d|d|ddddddd dd
d d d
ddddddd|d|d|d|d|S )a_
Builder for creating a Phi3 Mini (3.8b) model with LoRA enabled.
The Phi3 defaults are the same as in :func:`~torchtune.models.phi3.phi3_mini`,
while LoRA default params are based on
https://github.com/tloen/alpaca-lora/blob/8bb8579e403dc78e37fe81ffbb253c413007323f/finetune.py#L41-L43.
Args:
lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers
LoRA should be applied to in each self-attention block. Options are
``{"q_proj", "k_proj", "v_proj", "output_proj"}``.
apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer.
Default: False
apply_lora_to_output (bool): whether to apply LoRA to the model's final output projection.
Default: False
lora_rank (int): rank of each low-rank approximation
lora_alpha (float): scaling factor for the low-rank approximation
lora_dropout (float): dropout probability for the low-rank approximation. Default: 0.0
use_dora (bool): Decompose the LoRA weight into magnitude and direction, as
introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353).
quantize_base (bool): Whether to quantize base model weights
Returns:
TransformerDecoder: Instantiation of Phi3 Mini model with LoRA applied
r- r. r/ r r r r r r r r r r r r r r r r r0 r1 r2 r3 r4 r )r )r- r. r/ r0 r1 r2 r3 r4 s r" lora_phi3_minir6 J s F ++ ++ 2 1 6
2 "
R $ D S ) : "\ ! " $m# r! T)r4 z
Builder for creating a Phi3 mini model with QLoRA enabled. Base model weights in linear layers
that LoRA is applied to are quantized per the QLoRA paper: https://arxiv.org/abs/2305.14314.
Please see `lora_phi3_mini` for full API arguments.
)NNN)FFr+ r, r FF) typingr r )torchtune.models.phi3._component_buildersr r torchtune.models.phi3._tokenizerr torchtune.modulesr torchtune.modules.peftr functoolsr
'torchtune.modules.transforms.tokenizersr torchtune.data._prompt_templatesr r
r# strintr* boolfloatr6 qlora_phi3_mini__doc__r r! r" rE s ! ! ! ! ! ! ! ! E E E E E E E E > > > > > > 0 0 0 0 0 0 4 4 4 4 4 4 K K K K K K : : : : : : A A A A A A % 0z zc z
z[cdg[h z CK LY CZ z fw z z z z@ $!& 5 5 -.55 5 5
5
5 5 5 5 5 5 5p '.=== r!