`idZddlZddlmZmZmZmZddlZddlmZddl m Z m Z m Z ddl mZddlmZdd lmZdd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddl m!Z!m"Z"ddl#m$Z$ddl%m&Z&m'Z'ddl(m)Z)e'rddl*m+Z+m,Z,ddl-m.Z.nd\Z.Z,Z+e&r ddl/m0Z0m1Z1nd\Z1Z0e2e.e,e0e1e+fZ3e"j4e5Z6Gddej7Z8dej9de:dej9fdZ;Gdd Z< dAd"ej7d#ej9d$ej9d%ej9d&eej9d'e=d(e=fd)Z>Gd*d+ej7Z?Gd,d-ej7Z@Gd.d/ej7ZAGd0d1ej7ZBGd2d3ej7ZCGd4d5ej7ZDe!Gd6d7eZEe!Gd8d9eEZFGd:d;eEeZGe!d<=Gd>d?eEZHgd@ZIdS)BzPyTorch Zamba model.N)AnyCallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)FlashAttentionKwargs)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging)deprecate_kwarg)is_causal_conv1d_availableis_mamba_ssm_available) ZambaConfig)mamba_inner_fnselective_scan_fn)selective_state_update)NNN)causal_conv1d_fncausal_conv1d_updateNNc,eZdZdfd ZdZdZxZS) ZambaRMSNormư>cttjt j||_||_dS)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/zamba/modeling_zamba.pyr)zZambaRMSNorm.__init__@sD l5:k#:#:;; #c|j}|tj}|ddd}|tj||jzz}|j||zS)NT)keepdim) dtypetor+float32powmeanrsqrtr.r-)r/ hidden_states input_dtypevariances r3forwardzZambaRMSNorm.forwardHs|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r4cHt|jjd|jS)Nz, eps=)tupler-shaper.r/s r3 extra_reprzZambaRMSNorm.extra_reprOs& )**II$2GIIIr4)r&)__name__ __module__ __qualname__r)rBrG __classcell__r2s@r3r%r%?sb$$$$$$;;;JJJJJJJr4r%r?n_repreturnc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rEexpandreshape)r?rMbatchnum_key_value_headsslenhead_dims r3 repeat_kvrVTs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr4c eZdZdZdZejdfdZdZde de ej ej ffdZ dd ej d ej de d e eeefde ej ej ff d Zd ejfdZdde e de fdZdS)ZambaHybridDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNc ||_d|_|j|_d|_|j|jz|_|j|_|j |_ |j |_ g|_ g|_ g|_i|_i|_i|_t%|jD]}|xj t)j|j|j |gz c_ |j |j|j z|jf}|xj t)j||gz c_ |j|dkr|j|fdt%|jD|_fdt%|jD|_dS)NFdevicer9hybridcDg|]}tjggzSr[r+tensor.0_ batch_sizer[s r3 z4ZambaHybridDynamicCache.__init__..s/rrrQ%,tj'8HHHrrrr4cDg|]}tjggzSr^r`rbs r3rfz4ZambaHybridDynamicCache.__init__..s/tttqEL" ):6JJJtttr4)r9is_compileablelayers_block_typehas_previous_state mamba_expandr0intermediate_size mamba_d_statessm_state_size mamba_d_convconv_kernel_size n_mamba_heads conv_states ssm_statestransformer_layers_modules _parameters_buffersrangenum_hidden_layersr+zerosappend key_cache value_cache)r/configrer9r[i cache_shapes ` ` r3r)z ZambaHybridDynamicCache.__init__ps #!'!9"'!'!4v7I!I$2 & 3#1"$  v/00 2 2A    J(>@U^dlqrrr!   "&$*<<# K OO Ke T T TU UOO%a(H44'..q111rrrrrRWX^XpRqRqrrrtttttTYZ`ZrTsTstttr4c*t|jSN)lenr|rFs r3__len__zZambaHybridDynamicCache.__len__s4>"""r4 layer_idxrNc6|j||j|fSr)r|r}r/rs r3 __getitem__z#ZambaHybridDynamicCache.__getitem__s~i($*:9*EEEr4 key_states value_states cache_kwargscD|j|jddkr||j|<||j|<nVtj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr7rr6dim)r|rEr}r+cat)r/rrrrs r3updatezZambaHybridDynamicCache.updates >) $ *2 .! 3 3(2DN9 %*6D Y ' '(- 4>)3Lj2Y_`(a(a(aDN9 %*/)T5Ei5PR^4_ef*g*g*gD Y '~i($*:9*EEEr4beam_idxctt|jD];}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<=dS)zDReorders the cache for beam search, given the selected beam indices.rN) rxrr|r[ index_selectr:r}rrrs)r/rrr[s r3 reorder_cachez%ZambaHybridDynamicCache.reorder_cachesEs4>2233 i iI^I.5F(,y(A(N(NqRZR]R]^dReRe(f(fDN9 %%i07F*.*:9*E*R*RSTV^VaVabhViVi*j*jD Y '%i07F*.*:9*E*R*RSTV^VaVabhViVi*j*jD Y '_Y/6F)-)C)P)PQRT\T_T_`fTgTg)h)hDOI & & i ir4rc||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.r)rtrr|rErs r3get_seq_lengthz&ZambaHybridDynamicCache.get_seq_lengthsS3<4CZ2Z2ZD+A..`i t~  ) + +1~i(.r22r4r)r)rHrIrJ__doc__rhr+float16r)rintrDTensorrrdictstrrr LongTensorrrr4r3rXrX`sO  N16tuuuu@###FSFU5<3M-NFFFF26 FFLFlF F tCH~. F u|U\) * FFFF$ ie&6 i i i i33 3c333333r4rXmodulequerykeyvalueattention_maskscalingdropoutc Rt||j}t||j} tj||dd|z} |$|ddddddd|jdf} | | z} t j| dtj  |j } t j | ||j } tj| | } | dd} | | fS)Nr6r rr7)rr9)ptrainingr)rVnum_key_value_groupsr+matmul transposerEr functionalsoftmaxr;r:r9rr contiguous) rrrrrrrkwargsrr attn_weights causal_mask attn_outputs r3eager_attention_forwardrs3 ;<&B\$\!'-'E$ )d2 !'!9i X[_[h>hotuuu i X[_[h>hotuuu i X[_[h>hotuuu i :T] JFL^ejkkk r4past_key_valuepast_key_values4.58new_nameversionNr?rrrNc |jdd}g|d|jR}|||dd}|||dd} |||dd} ||| | |\} } t} |j j dkrt|j j } | ||| | |f|j sdn|j |jd|\} } | jg|dR} || } | | fS)Nr7rr6eagerr)rr)rErUrviewrrrrrr~_attn_implementationrrrrrQrr)r/r?rrrr input_shape hidden_shape query_statesrrattention_interfacerrs r3rBzZambaAttention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT  &'6'='=j,Xa'b'b $J (? ; +w 6 6"9$+:Z"[ $7$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH kk+.. L((r4r)rHrIrJrrrr)rr+rrrXrrrDrBrKrLs@r3rrs   l{lsllllll$_%0A6RRR >B #)#)|#)#)!. #) ""9: #) -. #) u|Xel3XeEL>Q5RR S#)#)#)SR#)#)#)#)#)r4rcbeZdZdZdeffd Z d dejdefdZ d defdZ d defd Z xZ S) ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. r~c t||_||_|j|_|j|_|j|_|j |jz|_ |j |_ |j |_ |j |j z|_|j|_|j|_t'j|j |j |j|j|j |jdz |_|j|_t0|j|_|j|_t'j|j|j dz|j|_t'jt?j |j |j |jdzz|j|_!t'jt?j |j |j|j dz dz|j dzz |_"t'jt?j |j |j|_#t?j$d|jdzt>j%dddf}|&|j d'}t'jt?j(|)|j |jd|_*t'jt?j+|j |j|_,t'j|j |j|j|_-t\st^0ddSdS) Nr) in_channels out_channelsr kernel_sizegroupspaddingr6rg?r9r7aqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)1r(r)r~rr0rmrnrorprkrl mamba_dt_ranktime_step_rankrqmamba_head_dimmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1dhidden_mamba_act activationr actuse_mamba_kernelsuse_fast_kernelsrin_projr*r+rz x_proj_weightdt_proj_weight dt_proj_biasaranger;rPrlogrQA_logr,Dout_projis_fast_path_availablelogger warning_once)r/r~rAr2s r3r)zZambaMambaMixer.__init__.s  "!-$2 & 3!'!4v7I!I$2#1"48JJ#3. i./#-))A-     !1&12 & 8y!143IA3MTXTabbb  \ K"#d&9A&==#    !l [+T-@$BU V VY\ \ !3& '   LT5GI\)])]^^ LD/!35= I I I$PQPQPQ' R HHT+R 0 0 ; ; = =\%)A,,"6"6t7I4K^`b"c"cdd ej);T=PQQRR $"8$:JQUQ^___ %    ^       r4Nr? cache_paramsc |j\}}}|duo |jo|dk}||dd}||dd|dd\}} |d}| d} | ||j d|dd} |j j |j j d|j j d} |r[t|d|j|j| |j j|j}|d}n|0t'j|dks||dz}|Zt*j||j|jdz df} |j|j| t5|| |j j|j}|0t'j|dks||dz}|d|j |j|dd}|jdddddddf|zdd} t'j| |j|j|jgd\} }}|j dddf| ddz}t'j!|j"# }|j$|j$#nd}t'j%|d|f|j&|j'}|rtQ|j D]}tS|j*|jdd|f||d df||d df||||dddf||dddf|j+|| |d df||d d}t'j,||fd}nOt'j%|d|j|jf|j&|j'}tQ|j D]}t[||||||||dd||dd|j+|#| |||d d \}}t'j,||fd}t'j,||dfd}|'|%|j*|j||.|dd}|S) Nrr6r7rr)rrrZ.T) dt_softplus)delta_softplusreturn_last_state)/rErjrrrchunksqueezerrQrqrr-sizer"rrrrr unsqueezer+allrrpadrpcopy_r!rrsplitrrnrexprfloatremptyr[r9rxr rsrrrr)r/r?rrreseq_lenrduse_precomputed_statesprojected_statesgate conv_weightsrrssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r3cuda_kernels_forwardz$ZambaMambaMixer.cuda_kernels_forwardks"/!4 GQ!-T!9!nl>]!nbimnbn << 66@@AFF.33JAwOOUUVW]^U__ t%--a00;;== ||A||J(:BHHRRSTVWXX{)..t{/A/F/Fq/I/I4;K]KbKbcdKeKeff ! L0%%b))(8   M*33B77MM)%)Na>{KKK,]L$+JZgkgvwwwM)%)Narss ;E (8 $])C)CC'M)R S S !-eiRS@S6T6T-$1N111}GZ[]G^F^F`F`C`4a4k4klm4n4n$nM T/1DdFYZ$+I )%)NaN ? E E G GG w 9 9A"111aaaAqqq=1;;AqAAIMPXYZYZYZ\]\]\]_`_`_`bcefefefYfPgPqPqrsuvPwPwwI,y':':1a'@'@'C'CE'J'JAaaaQRQRQRTUWXWXWXjMLcLcdfLgLghhK    AAAqqq!!!QJ 7 8 8 8 8k,B777 !]TVAAAtQQQ=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)r)rrrr[type ValueErrorrr,)r/r?rrs r3rBzZambaMambaMixer.forward)sz   i) V4;M;T;Y-Y-Y i ,,]LYg,hh h   ^ \\\r4r#) rHrIrJrrr)r+rrXrr,rBrKrLs@r3rr!s  ;{;;;;;;|im_%_%"\_%9P_%_%_%_%B[%[%7N[%[%[%[%z ] ]3J ] ] ] ] ] ] ] ]r4rc$eZdZfdZdZxZS)ZambaMLPct||_|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_ t|j |_ dSNFr) r(r)r~r0rlrr gate_projup_proj down_projr hidden_actact_fnr/r~r2s r3r)zZambaMLP.__init__7s  !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r4c||||||z}|Sr)r7r9r5r6)r/xr7s r3rBzZambaMLP.forwardAsANN4;;t~~a/@/@#A#ADLLQROO#STT r4)rHrIrJr)rBrKrLs@r3r2r26sG00000r4r2c8eZdZddedeeffd Zeddd dd ej d ej ded eej dee d ee dee de e deejeeejejfffdZxZS)ZambaAttentionDecoderLayerNr~rctt|||_t ||_t |j|j|_ t |j |j|_ dS)Nr1) r(r)r self_attnr2 feed_forwardr%r rms_norm_epsinput_layernormr0pre_ff_layernormrs r3r)z#ZambaAttentionDecoderLayer.__init__Gsv ' ::$V,,+F,HfNabbb ,V-?VEX Y Y Yr4rrrrFr?original_hidden_statesroutput_attentionsr&rrNc tj||gd}||}|jd||||||d|\}} ||}||}|f} |r| | fz } | S)a Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. r7r)r?rrrrGr&r)r+ concatenaterDrArErB) r/r?rFrrrrGr&rself_attn_weightsoutputss r3rBz"ZambaAttentionDecoderLayer.forwardOs@)=:P*QWYZZZ ,,];; +94>, ')+/ , , , , ( (--m<< ))-88  "  , )+ +Gr4r)NNFF)rHrIrJrrrr)rr+rrXboolrrrD FloatTensorrBrKrLs@r3r>r>Fs>ZZ{Zx}ZZZZZZ_%0A6RRR 26=A,1$)33|3!& 3 3 !. 3 ""9: 3$D>3D>3-.3 u (51BEDU1U+V"WW X333SR33333r4r>ceZdZdedeffd Zeddd dd ejd e ejde ed e ejd e ejde e de e de e de ej de ejde eje e ejejfffdZxZS)ZambaMambaDecoderLayerr~rctt|||_t |j|j|_||_dS)N)r~rr@) r(r)rmambar%r0rCrDrrs r3r)zZambaMambaDecoderLayer.__init__sS $FiHHH +F,>FDWXXX"r4rrrrNFr?rFrrrGr&cache_positiontransformer_hidden_statesrNc  |} | || zn|}||}||||}d} | |z}|f}|r|| fz }|r||fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. N)r?rr)rDrQ)r/r?rFrrrrrGr&rRrSrresidualrJrKs r3rBzZambaMambaDecoderLayer.forwards>! :S9^M5 5 5dq ,,];;  '()#  !!=0  "  , )+ +G  * ) )Gr4) NNNNNFFNN)rHrIrJrrr)rr+rrrXrLrrDrMrBrKrLs@r3rOrOsh#{#s###### _%0A6RRR:>#'15.2=A,1$)59<@::|:!) 6:C= : !. : el+ :""9::$D>:D>:!!12:$,EL#9: u (51BEDU1U+V"WW X:::SR:::::r4rOc|eZdZdedejdeffd Zeddd dd e j d e e j d e e de e j de e j de e de ede ede e jdee je ee je jfffdZxZS)ZambaHybridLayer shared_transflinearrQcrt||_||_||_dSr)r(r)rXrY mamba_decoder)r/rXrYrQr2s r3r)zZambaHybridLayer.__init__s6 * "r4rrrrNFr?rFrrrrGr&rRrNc ||||||||| } | d} |r| d} || } ||| ||||| } |r| d| f| ddz} | S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`ZambaHybridDynamicCache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. )rFrrrrGr&rRrr)rSrrrGr&rRr6N)rXrYr[) r/r?rFrrrrrGr&rR layer_outputsrSrJs r3rBzZambaHybridLayer.forwards@** #9&+/)+   %2!$4!  1 -a 0 $(KK0I$J$J!** &?)+/)+    V*1-/@AMRSRTRTDUUMr4)NNNNNFFN)rHrIrJr>rrrOr)rr+rrrrXrLrrDrMrBrKrLs@r3rWrWs_#&@#")#\r###### _%0A6RRR:>#'15.2=A,1$)59>>|>!) 6>C= > !. > el+ >""9:>$D>>D>>!!12> u (51BEDU1U+V"WW X>>>SR>>>>>r4rWc@eZdZUeed<dZdZddgZdZdZ dZ dZ dZ d S) ZambaPreTrainedModelr~modelTr>rOrFc|jj}t|tjtjfrJ|jjd||j |j j dSdSt|tj rU|jjd||j +|jj|j  dSdSt|tr!|jjddSt|trD|jjd||jjdz}tj|j| ||jj|jjz|jjz}t1jt1j|jj|t7j|jjt7j|jjz zt7j|jjz|jj }|t1jt1j!|  z}|j"j#|t1j$d|j%dzt0j&dddf}|'|j(d)}|j*j#t1j|+|j|j,d|j-jddSdS) Nr)r=stdg?r)minrrr7).r~initializer_rangerrrrr-datanormal_rzero_ Embedding padding_idxr%fill_rrrinituniform_rrkr0rqr+rrandmathr time_step_max time_step_minclamptime_step_floorexpm1rrrrnr;rPrlrrrQrr)r/rrb dt_init_stdrdtinv_dtrs r3 _init_weightsz"ZambaPreTrainedModel._init_weights sk+ fry")4 5 5 % M  & &CS & 9 9 9{&  &&((((('&  - - % M  & &CS & 9 9 9!- "6#56<<>>>>>.-  - - % M  $ $S ) ) ) ) )  0 0 %  % - -3C - @ @ @+3T9K G  V2[L+ N N N![5 8OOSWS^SllN 4;4nEE8DK566$+B[9\9\\^(4;4556e 3e44  %)U["%5%5$5666F   $ * *6 2 2 2 Q 5 9OOOPTVWVWVWPWXA1266AACCA L  # #EIaLL$8$89MvOdfh$i$i j j j HM   $ $ $ $ $% % %r4N) rHrIrJr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulrwrr4r3r_r_s_ &*#57OP"3 NL%%%%%r4r_cBeZdZdZdeffd Ze ddeej deej deej dee d eej d ee d ee d ee d ee deej deeeffdZdZxZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig r~c  t||j|_|j|_t j|j|j|j|_t|}g}g}|j |_ t|j D]}|j |dkr%| t||8|j |dkrb| t j|jj|jjd| t||t#|}t#|}g}g|_t'|j D]\}}|dkred|d gd} g|j fd | D|_| t)|t+|t+|p| t+|t j||_|j|_t3|j|j |_d|_|dS) NrQ)rr\Frzlayers..) z%shared_transf.self_attn.q_proj.weightz%shared_transf.self_attn.k_proj.weightz%shared_transf.self_attn.v_proj.weightz%shared_transf.self_attn.o_proj.weightz+shared_transf.feed_forward.gate_proj.weightz)shared_transf.feed_forward.up_proj.weightz+shared_transf.feed_forward.down_proj.weightz$shared_transf.input_layernorm.weightz%shared_transf.pre_ff_layernorm.weightcg|]}|zSrr)rcr prefix_names r3rfz'ZambaModel.__init__..ls FnFnFn]`{UXGXFnFnFnr4r@)r(r) pad_token_idri vocab_sizerrhr0 embed_tokensr>rirxryr{rOrr~iter_tied_weights_keys enumeraterWnext ModuleListlayersrr%rCfinal_layernormgradient_checkpointing post_init) r/r~block mamba_layers linear_layersrrlayer_id layer_type tied_keysrr2s @r3r)zZambaModel.__init__Jsr    !. +L):FFDWXXX&+# r4N input_idsr position_idsr inputs_embedsr&rGoutput_hidden_states return_dictrRrNc (||n |jj}||n |jj}||n |jj}| | n |jj} |du|duz rt d|jr%|jr|rt dd}|| |}|} tj |} |r|t d| &tj | jd| j} || d}|||| } |rdnd}|rdnd}t%|jD]q\}}|r|| fz }|jr+|jr$||j| | ||| |||| }n|| | ||| |||| }|d} |r|d ||dfz }r|| } |r|| fz }|r|jsd |_t1| |r|nd|| }| r|n|S) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either onezX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fz{Zamba requires an initialized `ZambaHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.rr_rr)rFrrrrrGr&rRT)last_hidden_staterr? attentions)r~rGrr&use_return_dictr0rrrrrr+r rrEr[r_update_causal_maskrr_gradient_checkpointing_func__call__rrjrto_tuple)r/rrrrrr&rGrrrRr?rFrall_hidden_statesall_self_attnsrlayerr]outputs r3rBzZambaModel.forwardys2C1N--TXT_Tq$8$D $+Jj "+!6IIDKk<$D>k<'tnk<d^k<!!12k< u-- .k<k<k<^k<\!!!!!!!r4rceZdZdeffd Ze ddeejdeej deejdee d eej d eejd ee d ee d ee dee deejde eej fde eeffdZ ddZxZS)ZambaForCausalLMr~c&t|t||_dg|jj|_|j|_t j|j|jd|_ | dS)Nzlm_head.weightFr) r(r)rr`rrrrr0lm_headrr:s r3r)zZambaForCausalLM.__init__s    '' #3"Tdj6S"T +y!3V5FUSSS  r4Nrrrrrrlabelsr&rGrrrRlogits_to_keeprNc ||n |jj}| | n |jj} | | n |jj} ||||||||| | |  }|d}t | t rt| dn| }||dd|ddf}d}||j |||j fi| }| s|f|ddz}||f|zn|St|||j |j |jS)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrrrr&rGrrRrrrlosslogitsrr?r)r~rGrrr`rrslicer loss_functionrrrr?r)r/rrrrrrr&rGrrrRrrrKr? slice_indicesrrrs r3rBzZambaForCausalLM.forwardsjP2C1N--TXT_Tq%9$D $+Jj &1%SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%ffdoPPPPD DY,F'+'7D7V##V C%#3!/)     r4Tc |du} | s]||d|jdkr|dd|jd df}nV|jd|jdkr |dd|f}n-t|j|jd|j|j}|b|`|ddz }||dkd| s|dd|jd df}|| rd|i} nd|i} | |||||jj |d| D]\} } | | vr| | | <| S)Nr7rr)r9r[rr)rrr&rrrR) rErXr~r9r[longcumsum masked_fill_rrnum_logits_to_keepitems) r/rrrrrRrr&r empty_past_kv model_inputsrrs r3prepare_inputs_for_generationz.ZambaForCausalLM.prepare_inputs_for_generationjs(4/  )!"%);;;%aaa.*>q*A)A)C)C&CD #~';A'>>>%aaa&78 5 Y_Q/tz$+O  %,*>)..0077;;a?L  % %n&91 = = =  F+AAA 0B/B/D/D,DE   $ $+];LL')=)=)?)?@L ,#2&"0"&+"@"0    !,,.. * *JC,&&$) S!r4) NNNNNNNNNNNr)NNNNNT)rHrIrJrr)rrr+rrrXrMrLrrrDrrBrrKrLs@r3rr s{151537=A59-1$(,0/3&*5934O O E,-O !.O u/0 O ""9: O   12 O )*O D>O $D>O 'tnO d^O !!12O c5</0O  u,, -O O O ^O h????????r4ra The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introcXeZdZfdZe ddeejdeejdeejdee e e ej fdeej deejd ee d ee d ee d ee d e eeffdZxZS)ZambaForSequenceClassificationc t||j|_t||_|jj|_t j|j|jd|_ | dSr4) r(r) num_labelsrr`rrrr0scorerr:s r3r)z'ZambaForSequenceClassification.__init__sv     +'' "&*"?Yv14?OOO  r4Nrrrrrrr&rGrrrNc 4| | n |jj} ||||||||| |  } | d} || } ||jd}n |jd}|jj|dkrt d|jjd}n|}||jjk| jtj }tj |jd| jtj }||z d}n)d}t|jjd| tj || j |f}d}|t|| j}|jjf|jdkr d |j_nN|jdkr7|jtjks|jtjkr d |j_n d |j_|jjd krWt-}|jdkr1|||}n|||}n|jjd krGt1}||d|j|d}n*|jjd krt5}|||}| s|f| ddz}||f|zn|St7||| j| j| j S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). N)rrrrr&rGrrrrz=Cannot handle batch sizes > 1 if no padding token is defined.r7rZz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`r_ regressionsingle_label_classificationmulti_label_classificationr)r~rr`rrErr0r:r[r+int32rargmaxrrr2rH problem_typerr9rrr rr rrrrr?r)r/rrrrrrr&rGrrtransformer_outputsr?rrelast_non_pad_token non_pad_mask token_indices pooled_logitsrloss_fctrs r3rBz&ZambaForSequenceClassification.forwardsJ(&1%!F!Fr!J!J  !#    >*ZZZ    u|Jv}MMMOaab   YYv}--F{'/?a''/;DK,,_q((flej.H.HFL\a\eLeLe/LDK,,/KDK,{'<77"99?a''#8M$9$9$;$;V^^=M=MNNDD#8M6::DD)-JJJ+--x 2 22t G GUWYY)-III,..x v66 F#%(;ABB(??F)-)9TGf$$v E/ /?-;*5     r4r)rHrIrJr)rrr+rrrr listrMrLrDrrBrKrLs@r3rrsV151537KO59-1$(,0/3&*[ [ E,-[ !.[ u/0 [ "%tE4E/F(F"GH [   12 [ )*[ D>[ $D>[ 'tn[ d^[  u66 7[ [ [ ^[ [ [ [ [ r4r)rrrr_)r)Jrrntypingrrrrr+rtorch.nnrr r activationsr cache_utilsr generationrmodeling_attn_mask_utilsrmodeling_flash_attention_utilsrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrutils.deprecationrutils.import_utilsrrconfiguration_zambar&mamba_ssm.ops.selective_scan_interfacerr+mamba_ssm.ops.triton.selective_state_updater causal_conv1dr!r"rr get_loggerrHrModuler%rrrVrXrrrrr2r>rOrWr_rrr__all__rr4r3rs}( 111111111111 AAAAAAAAAA!!!!!! ))))))>>>>>>BBBBBBqqqqqqqqqqFFFFFFFF&&&&&&,,,,,,,,000000TTTTTTTT,,,,,,QXXXXXXXXRRRRRRR@P=-~8DDDDDDDDD-7**.0@BVXfg  H % %JJJJJ29JJJ* UU\ U# U%, U U U U]3]3]3]3]3]3]3]3N%% I% <% % < % U\* %  %%%%%4D)D)D)D)D)RYD)D)D)NQ]Q]Q]Q]Q]biQ]Q]Q]j     ry    ========@BBBBBRYBBBJFFFFFryFFFR)%)%)%)%)%?)%)%)%XGGGGG%GGGV\\\\\+_\\\~    g g g g g %9g g   g T g f fr4