Pic*tddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZmZmZ d-d ed ed ed ed edededededede fdZd.dedededefdZ d/ddddddddeededed ed ed ed ed edededededededed ed!edede f&d"Zdddddd#d$eed ed ed edededededed ed!ededefd%Zdddd&dedededed ed!ededefd'Zdddd(d)ed ed ed ed ed edededededede fd*Z d/ddddddddeededed)ed ed ed ed ed edededededededed ed!edede f(d+Zd,S)0)List)nn) FeedForwardFrozenNF4LinearMultiHeadAttentionRMSNormRotaryPositionalEmbeddingsTransformerDecoderTransformerSelfAttentionLayer)(_register_reparametrize_state_dict_hooks) DoRALinearLORA_ATTN_MODULES LoRALinearh㈵>' vocab_size num_layers num_heads num_kv_heads embed_dimintermediate_dim max_seq_len attn_dropoutnorm_eps rope_basereturnc ||z} |r|n|}t| || } tj} t|D]} t |||| tj||| zdtj||| zdtj||| zdtj||d| d|| }t ||}t||t||t||}| |tj ||}tj||d}t|| ||| t|| | S) a Build the decoder associated with the mistral model. This includes: - Token embeddings - num_layers number of TransformerSelfAttentionLayer blocks - RMS Norm layer applied to the output of the transformer - Final projection into token space This does NOT currently include inference-time optimizations such as sliding-window attention Args: vocab_size (int): number of tokens in vocabulary. num_layers (int): number of layers in the transformer decoder. num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure `num_heads` % `num_kv_heads` == 0. For standard MHA set `num_kv_heads` == `num_heads`, for GQA `num_kv_heads` < `num_heads`, and for MQA set `num_kv_heads` == 1. embed_dim (int): embedding dimension for self-attention intermediate_dim (int): intermediate dimension for MLP max_seq_len (int): maximum sequence length the model will be run with, attn_dropout (float): dropout value passed onto scaled_dot_product_attention. Default: 0.0 norm_eps (float): epsilon in RMS norms rope_base (int): base for the rotary positional embeddings. Default: 10_000 Returns: TransformerDecoder: Instantiation of mistral model. dimrbaseFbiasN rrrhead_dimq_projk_projv_proj output_projpos_embeddingskv_cacherrr hidden_dimr epsattnmlpsa_normmlp_normr/tok_embeddingslayersrrr%normoutput r r ModuleListrangerLinear mistral_mlpr rappend Embeddingr )rrrrrrrrrrr%roper8_ self_attnr2layerr7r)s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/mistral/_component_builders.pymistralrG%sRI%H#/><>>K % YH - - -   Fr r- quantize_basec(|stj||dnt||d}|stj||dnt||d}|stj||dnt||d}t|||S)z@ Build the MLP layer associated with the Mistral model. Fr" gate_proj down_projup_proj)rr>rr)r r-rIrLrMrNs rFr?r?zs  : #z.... S*5 9 9 9 : *c.... Z5 9 9 9 : #z.... S*5 9 9 9 i Q Q QQrH)rrr lora_dropoutuse_dorarIlora_attn_modulesapply_lora_to_mlpapply_lora_to_output lora_rank lora_alpharOrPctj}t|D]}t|||||| | | |||| }|rt || | ||||}nt || |}t ||t|| t|| }||tj ||}|rtnt}|r|||| |ntj ||d}t||||||zt|| | }|rt||jj |S) ag Return a version of Mistral (an instance of :func:`~torchtune.modules.TransformerDecoder`) with LoRA applied based on the passed in configuration. Args: lora_attn_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to in each self-attention block. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. apply_lora_to_mlp (bool): whether to apply LoRA to the MLP in each transformer layer. Default: False apply_lora_to_output (bool): whether to apply LoRA to the model's final output projection. Default: False vocab_size (int): number of tokens in vocabulary. num_layers (int): number of layers in the transformer decoder. num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure `num_heads` % `num_kv_heads` == 0. For standard MHA set `num_kv_heads` == `num_heads`, for GQA `num_kv_heads` < `num_heads`, and for MQA set `num_kv_heads` == 1. embed_dim (int): embedding dimension for self-attention max_seq_len (int): maximum sequence length the model will be run with intermediate_dim (int): intermediate dimension for MLP. attn_dropout (float): dropout value passed onto scaled_dot_product_attention. Default: 0.0 norm_eps (float): epsilon in RMS norms. rope_base (int): base for the rotary positional embeddings. Default: 10_000 lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): LoRA dropout probability. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base: (bool): Whether to quantize base model weights or not. Only applied to base weights within linear layers LoRA is applied to. The final output linear projection is not supported for quantization currently. Returns: TransformerDecoder: Instantiation of Mistral model with LoRA applied to a subset of the attention projections in each layer. lora_modulesrrrrrrrTrUrOrPrIr r-rTrUrOrPrI)r r-rIr.r0)rankalphaFr"r5r6dtyperr<r=lora_mistral_self_attentionlora_mistral_mlpr?r rr@rAr rr>r r weightr])rQrRrSrrrrrrrrrrrTrUrOrPrIr8rCrDr2rEr7 adapter_clsr)models rF lora_mistralrds~]__F :  %%/*%#%!%'      "+#%)!+CC*:-C. x888999      e\*i88N!)8**jK  : Iz LLLL Yy*5 9 9 9 %y( YH - - -   E[ 1n>S>YZZZZ LrH)rrrOrPrIrXc |stdtd||z} |r|n|}| rtnt} d|vr| ||| z||| | n0| st j||| zdnt ||| zd}d|vr| ||| z||| | n0| st j||| zdnt ||| zd}d|vr| ||| z||| | n0| st j||| zdnt ||| zd}d |vr| ||||| | n*| st j||dnt ||d}t| || }t|||| ||||||| }|S) a Return an instance of :func:`~torchtune.modules.MultiHeadAttention` with LoRA applied to a subset of its linear layers Args: lora_modules (List[LORA_ATTN_MODULES]): list of which linear layers LoRA should be applied to. Options are ``{"q_proj", "k_proj", "v_proj", "output_proj"}``. embed_dim (int): embedding dimension for self-attention num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure `num_heads` % `num_kv_heads` == 0. For standard MHA set `num_kv_heads` == `num_heads`, for GQA `num_kv_heads` < `num_heads`, and for MQA set `num_kv_heads` == 1. max_seq_len (int): maximum sequence length the model will be run with attn_dropout (float): dropout value passed onto scaled_dot_product_attention. Default: 0.0 rope_base (int): base for the rotary positional embeddings. Default: 10_000 lora_rank (int): rank of each low-rank approximation lora_alpha (float): scaling factor for the low-rank approximation lora_dropout (float): LoRA dropout probability. Default: 0.0 use_dora (bool): Decompose the LoRA weight into magnitude and direction, as introduced in "DoRA: Weight-Decomposed Low-Rank Adaptation" (https://arxiv.org/abs/2402.09353). quantize_base (bool): Whether to quantize base model parameters for linear layers LoRA is being applied to. Default is ``False``. Returns: MultiHeadAttention: instantiation of self-attention module with LoRA applied to a subset of Q, K, V, output projections. Raises: ValueError: If lora_modules arg is an empty list zMust pass one or more of z as lora_modulesr&)rZr[dropoutrIFr"r'r(r)r) rrrr%r&r'r(r)r*rr) ValueErrorrr rrr>rr r)rXrrrrrrrTrUrOrPrIr%rbr&r'r(r)rBrDs rFr_r_sd   K(9 K K K   I%H#/><<S>YZZZZ LrHN)rrr)F)FF)typingrtorchrtorchtune.modulesrrrrr r r torchtune.modules.common_utilsr torchtune.modules.peftr rrintfloatrGboolr?rdr_r`rmrorHrFrysTSSSSSLLLLLLLLLL *RRRRR R  R  RRRRRRRRRjRRSRcR$R;RRRR0$!&A+AAA-.AAA  AAAAAAAAAA"#A$%A&'A()A*+A,-AAAAXKKK()K K  K  KKKKKKKKK !KKKKh''' '' '  '  '''''''jQQQQ Q  Q  QQQQQQQQQQQQl$!&H/HHH-.HHH  HHHHHHHHH !H"#H&'H()H*+H,-H./H01HHHHHHrH