`if,dZddlmZmZmZddlZddlmZddlmZddl m Z m Z ddl m Z mZmZmZmZmZddlmZmZmZmZd d lmZd d lmZmZd d lmZd d lm Z d dl!m"Z"m#Z#m$Z$d dl%m&Z&d dl'm(Z(m)Z)ddl*m+Z+e)rddl,m-Z-ddl.m/Z/m0Z0ndZ-e(r ddl1m2Z2m3Z3nd\Z3Z2e4e-e2e3fZ5e$j6e7Z8GddedZ9Gdde Z GddeZ:d4dZ;Gd d!e Z<Gd"d#eZ=d$Z>Gd%d&ej?Z@Gd'd(eZAGd)d*eZBGd+d,e ZCe"Gd-d.eZDe"Gd/d0eDZEGd1d2eZFgd3ZGdS)5zPyTorch Bamba model.)Optional TypedDictUnionN)nn)ACT2FN) HybridMambaAttentionDynamicCacheJambaAttentionDecoderLayer)LlamaAttentionLlamaForCausalLMLlamaMLP LlamaRMSNormLlamaRotaryEmbedding rotate_half)MambaRMSNormGatedpad_tensor_by_sizereshape_into_chunks segment_sum)AttentionMaskConverter)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack)auto_docstringcan_return_tuplelogging)deprecate_kwarg)is_causal_conv1d_availableis_mamba_2_ssm_available) BambaConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_update)NNcdeZdZUdZejed<ejed<eed<eed<ejed<dS)BambaFlashAttentionKwargsa Keyword arguments for advanced Flash Attention, causal-conv1d, and mamba_ssm kernel usage. Use cases include padding-free training and fewer `torch.compile` graph breaks. Attributes: cu_seq_lens_q (`torch.LongTensor`) Gets cumulative sequence length for query state. cu_seq_lens_k (`torch.LongTensor`) Gets cumulative sequence length for key state. max_length_q (`int`): Maximum sequence length for query state. max_length_k (`int`): Maximum sequence length for key state. seq_idx (`torch.IntTensor): Index of each packed sequence. cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kseq_idxN) __name__ __module__ __qualname____doc__torch LongTensor__annotations__int IntTensor{/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/bamba/modular_bamba.pyr(r(Ksb"######## _r8r(F)totalc.eZdZdZejdfdefdZdS)ra A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. Nconfigc $|j|_d|_|j}|j}g|_g|_g|_t|jD]}|j|dkrw|xjtj |j |j zd|j z|zz||gz c_|xjtj |j|j||gz c_|xjtjggzgz c_|xjtjggzgz c_|j|fdt|jD|_fdt|jD|_dS)NFmambadevicedtyperAcDg|]}tjggzSrCr2tensor.0_ batch_sizerAs r9 z=HybridMambaAttentionDynamicCache.__init__..s/rrrQ%,tj'8HHHrrrr8cDg|]}tjggzSrErFrHs r9rLz=HybridMambaAttentionDynamicCache.__init__..s/tttqEL" ):6JJJtttr8)layers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr2zeros mamba_expand hidden_sizemamba_n_groups mamba_n_heads mamba_d_headrGappend key_cache value_cache)selfr<rKrBrAconv_kernel_sizessm_state_sizeis ` ` r9__init__z)HybridMambaAttentionDynamicCache.__init__ss!'!9"'!.-"$v/00 2 2A%a(G33  K",v/AAAH]D]`nDnn(%# %  K",+&%#  $   U\2$2CF%S%S%S$TT  EL" 1B6$R$R$R#SS'..q1111rrrrrRWX^XpRqRqrrrtttttTYZ`ZrTsTstttr8)r.r/r0r1r2float16r!rdr7r8r9rresQ  ?DmTX$u$u{$u$u$u$u$u$ur8rceZdZdS)BambaRotaryEmbeddingNr.r/r0r7r8r9rgrgDr8rgc||}||}|jd}|dd|f|d|df}}|dd|f|d|df} } ||zt||zz} | |zt| |zz} tj| |gd} tj| | gd} | | fS)aApplies Rotary Position Embedding to the query and key tensors. Removes the interleaving of cos and sin from GLM Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. .Ndim) unsqueezeshaperr2cat) qkcossin position_ids unsqueeze_dim rotary_dimq_rotq_passk_rotk_passq_embedk_embeds r9apply_rotary_pos_embr~s, -- & &C -- & &C2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{{511C78Gs{{511C78Gi&)r222Gi&)r222G G r8ceZdZdS)BambaAttentionNrhr7r8r9rrrir8rceZdZdS)BambaRMSNormGatedNrhr7r8r9rrrir8rc|N|jddkr=|jddkr,|j}||dddddfz|}|S)zm Tunes out the hidden states for padding tokens, see https://github.com/state-spaces/mamba/issues/66 Nr r)rorBto) hidden_statesattention_maskrBs r9apply_mask_to_padding_statesrsi!n&:1&=&A&AnFZ[\F]`aFaFa#&111d )CCGGNN r8c neZdZdZdedeffd Z ddejde e de ej d e ejd e ej f d Z dde e de ej d e ejfd Z dde e de ej d e ejd e ej fd ZxZS) BambaMixeruP Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) The are a few differences between this and Mamba2Mixer: - The variable use_precomputed_states is slightly different due to the hybrid cache structure - There's a few non-obvious bugs fixed with batching in the slow path that exist in main - Some extra variables that our layer doesn't need have been removed - We ported most of the refactors in https://github.com/huggingface/transformers/pull/35154, which is (as of Dec 18, 2024) unmerged r< layer_idxc t|j|_|j|_|j|_|j|_t|j |jz|_ ||_ |j |_|j|_t"|j|_|j|_|j|_|j|_|j|_|j|_dt;df|_d|_d|_ |j d|jz|jzz|_!tEj#|j!|j!|j |j|j!|jdz |_$|j |j!z|jz}tEj%|j||j|_&tEj'tQj)|j|_*tQj+d|jdz}tEj'tQj,||_-t]|j |j |_/tEj'tQj)|j|_0tEj%|j |j|j|_1tdstf4d dStf4d dS) NinfgMbP?g?r?r ) in_channels out_channelsbias kernel_sizegroupspadding)repsaThe fast path is not available because one of `(selective_state_update, causal_conv1d_fn, causal_conv1d_update)` is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1dzDThe fast path for Bamba will be used when running the model on a GPU)5superrdr[ num_headsrYrQrbrPrar5rXintermediate_sizermamba_conv_bias use_conv_bias hidden_act activationractmamba_proj_biasuse_bias rms_norm_epslayer_norm_epsilonrZn_groupsr\head_dimmamba_chunk_size chunk_sizefloattime_step_limit time_step_min time_step_maxconv_dimrConv1dconv1dLinearin_proj Parameterr2onesdt_biasarangelogA_logrnormDout_projis_fast_path_availablelogger warning_once)r`r<rprojection_sizeA __class__s r9rdzBambaMixer.__init__st -!-$2 & 3!$V%84;K%K!L!L"#3 +&+,. "("5- +  1!$U5\\2" .T]1BTEX1XX i '-=)A-     04=@4>Qy       |EJt~$>$>??  LDNQ. / /\%)A,,// %d&<$BYZZZ ej8899 $"8$:JQUQ^___ % h   >         f g g g g gr8Nr cache_paramscache_positionrr-c t||}||}|j\}}} |j|jz} |duob|jo[|dkoU|j|jjd|j|jjdcxko|knco|duo |ddk} | r| d |j |j |j gd\} } }t| |j|j|jj d|jj|j} t'j | |j | | gd\}}}t'j|j }|ddddfdddddfd|j|jt&j}|dddddfdd|j}|jddddfd|j}|jddddfd|j}|||j|jd|jz}|||j|jd|jz}|||j |j}t=|j|j||||||d|d }|||j |jz}||| }| |ddddf}nlt'j|j }|j!d t-d fkrind |j!i}|j"r|tG||jj d|jj|j|f|j|j$||j|jj|jj%|j j|j j|j|jd d d |}n| |j |j |j gd\} } }|p| &dd}tNj()||j*|jdz df}|j|j+||jdvr[|,|| &dddd|f&dd} nht[| &dd|jj d|jj|j|&dd} t| |} t'j | |j | | gd\}}}t]|||d|j||||||jd||||jdf|j$|jd|d|jdd|\}}|'|%|j|j+||||d}||| }| |}|S)Nr rrkrl.rBT)zr dt_softplusrrdt_limitF) rrr-rrmsnorm_weight rmsnorm_epsoutproj_weight outproj_biasheaddimngroupsnorm_before_gatereturn_final_statesr?)siluswish)xweightrrr-)rrrr-rrr)/rrrorrbrOrRrrSsqueezesplitrrrr&rrrrr2exprrexpandrrfloat32rrviewr"rrrtrainingr$rvariance_epsilon transposer functionalpadracopy_rr%r#)r`rrrrr-projected_statesrKseq_lenrJgroups_time_state_sizeuse_precomputed_statesgatehidden_states_B_CdtBCrrrhidden_states_reshapedoutdt_limit_kwargshidden_states_B_C_transposedrR scan_output ssm_states r9cuda_kernels_forwardzBambaMixer.cuda_kernels_forward*sJ5]NSS << 66"/!4 GQ!%1D!D  $ &/ &1  &(8>qA&t~6>Az4=!'!* 2MNNAz4=!'!* 2MNNA%2%7%7 DNTXTa%b%b "2'7&    M*..z4>DM;YZZM IImT::M-- ..qqq$|.]s)%z%z%z\]&9!Xt&W&W%z%z%zr8r)r r)8rorBrrrrrrrOrRrrSrollrrArrr2sumrrrrrrrrrarrrbrrrrrrsoftplusclamprrreshape contiguousrbmmrrepeat_interleaverrpermutecumsumr zeros_likerprr)4r` input_statesrrrrKrrJrBrrrrrrRrrrrr cache_devicerdAdBdBxrSssm_states_reshaped C_reshapedyr D_residualA_cumsumLG_intermediateGM_intermediateMY_diag decay_statesB_decaystatesprevious_states decay_chunk new_statesrstate_decay_outC_times_statesstate_decay_out_permutedY_offrcontextualized_statesrs4` @r9 torch_forwardzBambaMixer.torch_forwards( ".!3 GQ"4L.QQ << 55&6&<&<'GR'=' ' #  $ &/ &1  &(8>qA&t~6'24>BEET[M_MfEggK % dk088;;;!!! ! I$5 8H$H! $): ; ;  '/@/J/J1a/P/P, m//043HKgKmnpKq3qst2u (8>>{KKK $5F5P5PQRTU5V5V)W)WX[]e^e]eXe)f)p)pqrtu)v)v w w 89JN[[#k   #T]T5H%H$-Z^ZmJm n    q!Ytz'')) * * * !F I'24>BILAAAq!!!GQQQc\*Ba##**:rx|T]SSBl9-44T\5G5JDMZZG$--b7::bh3G3G.GHHBR!5a!8$:Nq:QRRB/"))$.$-I\]]``glgt`uuA)ByMA-..22,2GGB  *dmR88dAAAFAT]DNdm4SUVU\]_U`aallnnA *b!'"+66AI3aaa<0B*11*b$-PPM i0044L4IIC  #DN 3 9 9'7"4=AAAy!((a$-HHA]Q&&**1733A *b))!!!T3,7AA''T\(9::BR!5a!8$:Nq:QRRB)11*gr4=YY__aaM *gr43FGGMMOOA *gr43FGGMMOOA##DNdm$CX\Xf#ggA##DNdm$CX\Xf#ggA'DO*CCtVH *-? x-X-XXJ*ByM9M]())B.A&{%z%z%z%zboqrtuwxay%z%z%z "M1a !Q1%%A|A2...H +a..))Aqqq!!!QQQaaa23a111dAAAqqq!!!8K6LLN""r"**Ay\AIIaAq!,D,DY,OON""r"**A l]111aaa:%>>CCCJJF!9XaaaAAArssl%;h%FGGL,..q"b!<'>q!Q'J'J $#''++.Fy.QQEA *b$.$-HHAJA!||aaa'111aaa'( *gr22A$)A'7==iHHHii4(( !% knnU.C.C D D$$r8c dtr1d|jjjjvr||||||S|t d|j}|G|jddkr6|jddkr%||dddddfz |}| ||||S)Ncudaz\`seq_idx` support requires fast path support. Please install `mamba_ssm` and `causal_conv1d`r r) rrrrAtyperNotImplementedErrorrBrorr)r`rrrrr-kwargsrBs r9forwardzBambaMixer.forwards " sf 0C0J0O&O&O,,]L.Zhjqrr r  %n #  %.*>q*AA*E*E.J^_`JadeJeJe*^AAAqqq$J-GGKKERRM!!-~~^^^r8)NNNN)NNN)r.r/r0r1r!r5rdr2Tensorrrr3r6rrr" __classcell__rs@r9rrs  ?h{?hs?h?h?h?h?h?hHDH5915-1 gg|g?@g!!12 g !. g %/* ggggZDH5915 L%L%?@L%!!12 L% !. L%L%L%L%dDH5915-1 __?@_!!12 _ !. _ %/* ________r8rceZdZdS)BambaMLPNrhr7r8r9r'r'rir8r'ceZdZdS) BambaRMSNormNrhr7r8r9r)r)rir8r)ceZdZddededeffd Zeddd  dd ej d e ej de ej de e de e de e de ej de eej ej fdeedeeje eejejfffdZxZS)BambaDecoderLayerr>r<r layer_typec0t|||`d}|dkrtnd}|||_||_|dkrt |||_dS|dkrt|||_dStd)Nr r>)r<r attentionzInvalid layer_type) rrd self_attnr' feed_forwardr,rr>r ValueError)r`r<rr, num_expertsffn_layer_classrs r9rdzBambaDecoderLayer.__init__s +++ N &1Q&6&6((D+OF33$  #6YGGGDJJJ ; & &+FI>>DNNN122 2r8past_key_valuepast_key_valuesz4.58)new_nameversionNFrrruoutput_attentions use_cacherposition_embeddingsr!returnc >|} ||}|jdkr|jd||||d| }d} n$|jdkr|jd||||||||d| \}} | |z}|} ||}||}| |z}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`HybridMambaAttentionDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. kwargs (`dict`, *optional*): Arbitrary kwargs. Can be used to provide `BambaFlashAttentionKwargs` for padding-free training and/or improve torch.compile performance. r>)rrrrNr.)rrrur5r8r9rr:r7)input_layernormr,r>r/pre_ff_layernormr0) r`rrrur5r8r9rr:r!residualself_attn_weightsoutputss r9r"zBambaDecoderLayer.forwards"F!,,];;  ?g % %&DJ+,--   M!%   _ + +/=t~ 0+-) /"3#-$7 0 0 0 0 ,M,!=0 !--m<< ))-88  =0  "  , )+ +Gr8)r>)NNNFFNN)r.r/r0r!r5strrdrr2r#rr3rbooltuplerr( FloatTensorr"r$r%s@r9r+r+s33{3s3333333"_%0A6RRR2637FJ,1$)59KOKK|K!.Ku/0 K ""BC K $D> KD>K!!12K&eEL%,,F&GHK23K u (51BEDU1U+V"WW XKKKSRKKKKKr8r+cHeZdZUeed<dZdZdgZdZdZ dZ dZ fdZ xZ S)BambaPreTrainedModelr<modelTr+r5cft|t|try|jjdtjtj d|j dz|j _|j jddSdS)Ng?r ) r _init_weights isinstancerrdatafill_r2rrrrr)r`modulers r9rJz"BambaPreTrainedModel._init_weights/s f%%% fj ) ) % N  % %c * * * % %,q&:JQ:N*O*O P PFL  HM   $ $ $ $ $ % %r8)r.r/r0r!r4base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulrJr$r%s@r9rGrG#sq &*#,-"3NL%%%%%%%%%r8rGceZdZdeffd Zee ddeej deej deej dee deej d ee d ee d ee d eej d eedefdZdej dej d ej de d e f dZedej dededejd ej def dZdZxZS) BambaModelr<c Jt||j|_|j|_t j|j|j|j|_g}t|j D]2}| t|||j |3t j||_|j|_t#|j|j|_t)||_d|_|dS)N)rr,r)r<F)rrd pad_token_id padding_idx vocab_sizer EmbeddingrY embed_tokensrUrVr]r+rN ModuleListlayers_attn_implementationr)rfinal_layernormrg rotary_embgradient_checkpointing post_init)r`r<decoder_layersrcrs r9rdzBambaModel.__init__9s     !. +L):FFDWXXX.f===&+# r8N input_idsrrur5 inputs_embedsr9r8output_hidden_statesrr!r;c ||n |jj}||n |jj}||n |jj}|du|duz rt d|jr%|jr|rtdd}|| |}|} |r|td| &tj | j d| j } || d}|||| ||} ||| } || |}|rdnd}|rdnd}|jD]H}|jd kr| n| }|r|| fz }|| f|||||| |d | }|d} |r|d ||dfz }I|| } |r|| fz }|r|jsd |_|sdn|}t-| ||| S) Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.FzBamba requires an initialized `HybridMambaAttentionDynamicCache` to return a cache. None was provided, so no cache will be returned.r rCrr7r>)rrur5r8r9rr:T)last_hidden_stater5r attentions)r<r8rhr9r1rcrrrr]r2rrorArn_update_causal_mask_update_mamba_maskrbr_r,rarOr)r`rfrrur5rgr9r8rhrr!r causal_mask mamba_maskr:all_hidden_statesall_self_attns decoder_layer layer_mask layer_outputs next_caches r9r"zBambaModel.forwardLs2C1N--TXT_Tq$8$D $+Jj "+!6IIDK?L]  ,,^^LL #oom\JJ"6@BBD0:d![ : :M'4'?7'J'JP[J# 6!m%55!)M )) /"3#-$7    M*!,M  : #/"}Q'7&99N,,];;   2 -!1 1   6?#E 615O .!*?TT &+&+%     r8 input_tensorc >|jjdkr |d|vr|SdS||nd}|jjdkr!|stj||||jrdS|j}|jd}t|tj r |jdn||zdz} | ||| |||jd} |jjdkr@|>|j j d vr0|s.tj|j} tj| | } | S) Nflash_attention_2rrsdpa)rgpast_key_values_length is_trainingr rk)sequence_length target_lengthrBrrK)rxpunpu)r<r`get_seq_lengthr_ignore_causal_mask_sdparrBrorKr2r#5_prepare_4d_causal_attention_mask_with_cache_positionrArfinfomin_unmask_unattended) r`rrvrr5r8past_seen_tokensrBr|r}rn min_dtypes r9rlzBambaModel._update_causal_masksy ; +/B B B)c^.C.C%%4 @O?Z?99;;;`a ; +v 5 5>O 5%>*'7 M   t"&,Q/.%,77 8N  $ $!O3a7 PP +')#)!, Q   K , 6 6*%*.DDD%E  E**.I0CKQZ[[Kr8r|r}rBrKc ||dkr|}notj|j}tj||f|||j}|dkrtj|d}|tj||j|ddkz}|ddddddf |ddd}|| }|j d} |ddddddf|ddddddfkdddd| dddf |} |ddddddd| f| z} | dk} |ddddddd| f | ||ddddddd| f<|S) aM Creates a causal 4D mask of shape `(batch_size, 1, query_length, key_value_length)` from a 2D mask of shape `(batch_size, key_value_length)`, or if the input `attention_mask` is already 4D, do nothing. Args: attention_mask (`torch.Tensor`): A 2D attention mask of shape `(batch_size, key_value_length)` or a 4D attention mask of shape `(batch_size, 1, query_length, key_value_length)`. sequence_length (`int`): The sequence length being processed. target_length (`int`): The target length: when generating with static cache, the mask should be as long as the static cache, to account for the 0 padding, the part of the cache that is not filled yet. dtype (`torch.dtype`): The dtype to use for the 4D attention mask. cache_position (`torch.Tensor`): Indices depicting the position of the input sequence tokens in the sequence. batch_size (`torch.Tensor`): Batch size. Nr) fill_valuerBrAr )diagonalrCrkr)rmr2rrfullrAtriurrrcloneror masked_fill) rr|r}rBrrKr!rnr mask_lengthpadding_attention_mask padding_masks r9rz@BambaModel._prepare_4d_causal_attention_mask_with_cache_positions<  %.*<*<*>*>!*C*C(KK E**.I* -0Ye\j\qK!###jqAAA 5< n>STTTWeWmWmnprsWtWtt tK%dD!!!QQQ&67>>z1bRTUUK))//11 ,226 *8D$9I*Jn]^]^]^`dfgfgfgim]mNn*nAAqqq?*++QQQ.*"U))' +111aaaL[L+@ADZZ +q0 5@AAAqqq,;,AV5W5c5c )66 AAAqqq!!!\k\12r8cZ|}|ddks|tj|dkrd}|S)zv No need for zeroing states when 1. Cached forward 2. Attending to all inputs rNr )r2all)r`rrros r9rmzBambaModel._update_mamba_mask$s> $ ! q ^%?EIn`aNaDbDb%?Jr8) NNNNNNNNN)r.r/r0r!rdrrrr2r3r#rrErCrr(rr"rl staticmethodr5rBrrmr$r%s@r9rWrW7s{&151537FJ59$(,0/359` ` E,-` !.` u/0 ` ""BC `   12 ` D>` $D>` 'tn` !!12` 23`  !` ` ` ^` D: :l: : : :  ::::x5 555{ 5  5  555\5n       r8rWcPeZdZfdZ ddeejdeejdeejdeedeej d eejd ee d ee d ee d eejde e ejfde fdZ ddZxZS)BambaForCausalLMct||j|_|dS)N)rrdz_loss_coefficientrd)r`r<rs r9rdzBambaForCausalLM.__init__1s>    "("; r8Nrrfrrur5rglabelsr9r8rhrlogits_to_keepr;c n||n |jj}| | n |jj} |jd |||||||| | d | } | j}t | t rt| dn| }||dd|ddf}d}||j d |||jj d| }|j dkra| d |jd}||j |zz}t#||| j| j| j S) aJ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BambaForCausalLM >>> model = BambaForCausalLM.from_pretrained("...") >>> tokenizer = AutoTokenizer.from_pretrained("...") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rfrrur5rgr9r8rhr)logitsrr[rrkrlrr?)lossrr5rrkr7)r<r8rhrHrjrKr5slicelm_head loss_functionr[r logsumexprrBpowmeanrr5rrk)r`rfrrur5rgrr9r8rhrrr!rAr slice_indicesrrz_losss r9r"zBambaForCausalLM.forward8sJ2C1N--TXT_Tq$8$D $+Jj  ,64: , )%+'/!5) , ,  , ,  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%pVFt{OeppioppD&**))b)11444:4FFJJ1MMRRTTd5>>%#3!/)     r8Tc |du} | s]||d|jdkr|dd|jd df}nV|jd|jdkr |dd|f}n-t|j|jd|j|j}|b|`|ddz }||dkd| s|dd|jd df}|| rd|i} nd|i} | |||||jj |d| D]\} } | | vr| | | <| S)Nrkr rrCrgrf)rur5r9rrr) rorr<rBrAlongr masked_fill_rupdatenum_logits_to_keepitems) r`rfr5rrgrrur9r! empty_past_kv model_inputskeyvalues r9prepare_inputs_for_generationz.BambaForCausalLM.prepare_inputs_for_generations(4/  )!"%);;;%aaa.*>q*A)A)C)C&CD #~';A'>>>%aaa&78 > Y_Q/DKO  %,*>)..0077;;a?L  % %n&91 = = =  F+AAA 0B/B/D/D,DE   $ $+];LL')=)=)?)?@L ,#2&"0"&+"@"0    !,,.. * *JC,&&$) S!r8) NNNNNNNNNNr)NNNNNT)r.r/r0rdrr2r3r#rrErCrr5rr"rr$r%s@r9rr0sq151537FJ59-1$(,0/35934K K E,-K !.K u/0 K ""BC K   12 K )*K D>K $D>K 'tnK !!12K c5</0K  K K K K `>>>>>>>>r8r)rWrrG)Nr )Hr1typingrrrr2rtransformers.activationsr(transformers.models.jamba.modeling_jambarr (transformers.models.llama.modeling_llamar r r r rr*transformers.models.mamba2.modeling_mamba2rrrrmodeling_attn_mask_utilsrmodeling_outputsrrmodeling_utilsrprocessing_utilsrutilsrrrutils.deprecationrutils.import_utilsrrconfiguration_bambar!+mamba_ssm.ops.triton.selective_state_updater"!mamba_ssm.ops.triton.ssd_combinedr#r$ causal_conv1dr%r&rr get_loggerr.rr(rgr~rrrModulerr'r)r+rGrWr__all__r7r8r9rs(---------- ++++++qqqqqqqq?>>>>>OOOOOOOO------&&&&&& 100000VVVVVVVV,,,,,,"RRRRRRmmmmmmmmm!8DDDDDDDDD-7**46FH\]^^  H % % 42u2u2u2u2u'G2u2u2uj     /   %%%%P     ^        )   \_\_\_\_\_\_\_\_~     x        <   ^^^^^2^^^B%%%%%?%%%&uuuuu%uuupSSSSS'SSSl E D Dr8