`ihddlmZmZddlZddlmZddlmZddlmZddl m Z ddl m Z dd l mZd d lmZmZmZmZd d lmZe jeZGddedZGddejZGddeZGddeZGddeZGddeZgdZ dS))Optional TypedDictN)nn)ACT2FN)Cache)Unpack)logging)deprecate_kwarg)GraniteMoeDecoderLayerGraniteMoeForCausalLMGraniteMoeModelGraniteMoePreTrainedModel)GraniteMoeSharedConfigcdeZdZUdZejed<ejed<eed<eed<ejed<dS)GraniteFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensor/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/granitemoeshared/modular_granitemoeshared.pyrr&sb"######## _r$rF)totalcLeZdZdZdeffd ZdejdejfdZxZ S)GraniteMoeSharedMLPz~ MLP layer for shared experts Args: config: Configuration object with model hyperparameters. configcDt|j|_|j|_t |j|_tj |j|jdzd|_ tj |j|jd|_ dS)Nr F)bias) super__init__ hidden_size input_sizeshared_intermediate_sizer hidden_act activationrLinear input_linear output_linearselfr) __class__s r%r-zGraniteMoeSharedMLP.__init__Hs  ,!: !23Idot7G!7KRWXXXYt'7uUUUr$ hidden_statesreturnc||}|dd}||d|dz}||}|S)Nr )dimrr)r4chunkr2r5)r7r9chunked_hidden_statess r%forwardzGraniteMoeSharedMLP.forwardQsj))-88 - 3 3A2 3 > >(=a(@AADYZ[D\\ **=99 r$) rrrrrr-rTensorr@ __classcell__r8s@r%r(r(?s|V5VVVVVVU\elr$r(ceZdZdedeffd Zeddd dd ejd e ejd e ej de e d e e de e de ej de e de e ejejfdeede eje e ejejfffdZxZS)GraniteMoeSharedDecoderLayerr) layer_idxct|||jdkrdnt||_dS)Nr)r,r-r0r( shared_mlp)r7r)rFr8s r%r-z%GraniteMoeSharedDecoderLayer.__init__ZsC +++"("AQ"F"F$$L_`fLgLgr$past_key_valuepast_key_valuesz4.58)new_nameversionNFr9attention_mask position_idsoutput_attentions use_cachecache_positionoutput_router_logitsposition_embeddingskwargsr:c d|} ||}|jd|||||||| d| \}} | ||jzz}|} ||}||\} }|j| }n| ||z}~ | ||jzz}|f}|r|| fz }|r||fz }|S)a1 Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_values (`Cache`, *optional*): cached past key and value projection states cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence output_router_logits (`bool`, *optional*): Whether or not to return the logits of all the routers. They are useful for computing the router loss, and should not be returned during inference. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `GraniteFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. )r9rMrNrJrOrPrQrSNr#)input_layernorm self_attnresidual_multiplierpost_attention_layernormblock_sparse_moerH)r7r9rMrNrJrOrPrQrRrSrTresidualself_attn_weightsmoe_hidden_states router_logitsoutputss r%r@z$GraniteMoeSharedDecoderLayer.forward^sN!,,];; ,:4> , ')%+/) 3 , ,  , , ( (!=43K#KK !55mDD +/+@+@+O+O(= ? "-MM- 0N0NNM  =43K#KK  "  , )+ +G  (  ' 'Gr$)NNNFFNFN)rrrrr!r-r rrArrrbooltupler r FloatTensorr@rBrCs@r%rErEYsh5h#hhhhhh_%0A6RRR2637+/,1$)59/4KOOO|O!.Ou/0 O "% O $D> OD>O!!12O'tnO&eEL%,,F&GHO45O u (51BEDU1U+V"WW XOOOSROOOOOr$rEc eZdZUeed<dgZdS)GraniteMoeSharedPreTrainedModelr)rEN)rrrrr _no_split_modulesr#r$r%rdrds' """"78r$rdc$eZdZdeffd ZxZS)GraniteMoeSharedModelr)cttjfdt jD|_dS)Nc0g|]}t|Sr#)rE).0rFr)s r% z2GraniteMoeSharedModel.__init__..s$ n n n )&) < < n n nr$)r,r-r ModuleListrangenum_hidden_layerslayersr6s `r%r-zGraniteMoeSharedModel.__init__sV    m n n n neTZTlNmNm n n n   r$)rrrrr-rBrCs@r%rgrgsE 5          r$rgc*eZdZdgZdeffd ZxZS)GraniteMoeSharedForCausalLMzlm_head.weightr)ct|t||_|dS)N)r,r-rgmodel post_initr6s r%r-z$GraniteMoeSharedForCausalLM.__init__s@    *622  r$)rrr_tied_weights_keysrr-rBrCs@r%rqrqsN*+5r$rq)rqrgrd)!typingrrrr activationsr cache_utilsrprocessing_utilsr utilsr utils.deprecationr granitemoe.modeling_granitemoer rrrconfiguration_granitemoesharedr get_loggerrloggerrModuler(rErdrgrq__all__r#r$r%rs '&&&&&&& !!!!!! &&&&&&000000 CBBBBB  H % %)52")4UUUUU#9UUUp99999&?999      O   "7 f e er$