.`i}dZddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z dd l mZdd lmZmZmZdd lmZdd lmZdd lmZddlmZmZmZmZmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6ddl7m8Z8m9Z9m:Z:Gddej;Z<Gddej;Z=Gddej;Z>Gd d!ej;Z?Gd"d#ej;Z@Gd$d%ej;ZAeGd&d'ej;ZBGd(d)ej;e4e5e6ZCdS)*aGPyTorch Zamba2 model implementation for vLLM. This module implements the Zamba2 architecture from https://arxiv.org/abs/2411.15242, which combines Mamba and Transformer architectures in a hybrid model optimized for efficient sequence modeling. The model alternates between state space model layers and attention-based layers. )Iterable)cycle)AnyN)nn) Zamba2Config) Attention)support_torch_compile) CacheConfig ModelConfig VllmConfig)$get_tensor_model_parallel_world_size) GeluAndMul)RMSNorm)ColumnParallelLinearMergedColumnParallelLinearQKVParallelLinearReplicatedLinearRowParallelLinear)LogitsProcessor) MambaMixer2)MambaStateCopyFuncMambaStateCopyFuncCalculatorMambaStateDtypeCalculatorMambaStateShapeCalculator)QuantizationConfig)get_rope)ParallelLMHeadVocabParallelEmbedding)default_weight_loader)IntermediateTensors) HasInnerStateIsHybridSupportsMambaPrefixCaching)AutoWeightsLoader WeightsMapper maybe_prefixc leZdZdZ d dededeeezdedzdef fd Zd e j fd Z xZ S) Zamba2LoRAz}LoRA layer for the Zamba2 model. Implements a LoRA layer that is used in shared attention and gated MLP blocks. N input_dimrank output_dim quant_configprefixc tt||d|d|d|_t |t rt }nt}|||d||d|_dS)zInitialize the attention layer. Args: input_dim: input dimension rank: LoRA rank output_dim: output dimension quant_config: Configuration for model quantization FTz.A)biasr. gather_outputr/z.Br1r.r/N)super__init__rA isinstancelistrB)selfr+r,r-r.r/B_class __class__s u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/zamba2.pyr5zZamba2LoRA.__init__<s %  %===     j$ ' ' +0GG*G  %===     hidden_statescf||\}}||\}}|SN)r6r9)r:r? lora_output_s r=forwardzZamba2LoRA.forwardcs4 .. Q ,, Qr>Nr*) __name__ __module__ __qualname____doc__intr8rstrr5torchTensorrD __classcell__r<s@r=r)r)5s37 % % % % $s)O % )4/ %  % % % % % % N|r>r)ceZdZdZ ddededededzdedzd ed dffd Z d e j d ede j d e j fdZ xZ S)Zamba2AttentionaMulti-head attention mechanism for the Zamba2 model. Implements attention with parallel computation, QKV projections, optional adapters and rotary position embeddings. The attention is computed across distributed blocks for efficient processing. Nr*configbare_block_idxnum_hybrid_layers cache_configr.r/returnc tt}||_||_|j|_|j|_|j|zdksJ|j|z|_|j|_|j|z|_ |jdz dz|_ |j|jz|jkr td|jd|jdt|j|j|jd||d |_ t|j|jd||d  |_t#jg|_||j|jzd z z|jz}t+|jD]k} | |jz|kr.t-|j|j|j ||d | } |d z }nt#j} |j| l|jrlt#jg|_t#jg|_t#jg|_t+|jD] } | |jz|krst;|j|j|j||d} t;|j|j|j||d} t;|j|j|j||d} n9t#j} t#j} t#j} |j| |j| |j|  |jr)tA|j|j!|j"d|_#dSdS)aInitialize the attention layer. Args: config: The Zamba2 model configuration bare_block_idx: Index of the bare attention block num_hybrid_layers: Total number of hybrid layers cache_config: Configuration for key-value caching quant_config: Configuration for model quantization prefix: Optional prefix for parameter names rgz]attention_hidden_size must be divisible by num_attention_heads (got `attention_hidden_size`: z and `num_heads`: z).Fz .qkv_projr3z.o_projr!z.attn.)rUr/z.linear_q_adapter)r.r/z.linear_k_adapterz.linear_v_adapterT) head_size max_positionrope_parameters is_neox_styleN)$r4r5r rRrTattention_hidden_sizenum_attention_headstotal_num_attention_headsattention_head_dimqkv_sizescale ValueErrorrqkv_projr hidden_sizeo_projr ModuleListdpa_listnum_mem_blocksrangerIdentityappenduse_shared_attention_adapterlinear_q_adapter_listlinear_k_adapter_listlinear_v_adapter_listr) adapter_rank use_mem_ropermax_position_embeddingsr[ rotary_emb)r:rRrSrTrUr.r/tp_sizej block_idxdpalinear_q_adapterlinear_k_adapterlinear_v_adapterr<s r=r5zZamba2Attention.__init__ts#& 688 !2%+%A")/)C&-71<<<<#)#=#H "(";2g= -1d:   #d&D D  ' ( (B262LBB&*%=BBB *  &  #  *%'''     (  &  %%%%      b)) %(==A C$ % t566 & &I600NBB,+J!-$//A// Qkmm M  % % % %  .# D)+r):):D &)+r):):D &)+r):):D &"4#9:: D D v44FF'12+2%1"(;;; ((($(22+2%1"(;;; ((($(22+2%1"(;;; ((($$(*{}}$'){}}$'){}}$*112BCCC*112BCCC*112BCCCC   &1#; & 6" DOOO  r>r?rw position_idsc||\}}||jgdzd\}}}|jjr|j|} t | tjrJ| |} || z}|j |} t | tjrJ| |} || z}|j |} t | tjrJ| |}||z}|jj r| |||\}}|j ||||}||\}}|S)aZForward pass through the attention layer. Args: hidden_states: Input tensor [batch_size, seq_len, hidden_size] position_ids: Position IDs for positional embeddings block_idx: Current shared transformer block index Returns: Output tensor [batch_size, seq_len, hidden_size] dim)rdsplitrarRrmrnr7rrkrorprrrtrhrf)r:r?rwr|qkvrC query_states key_states value_states q_adapter q_lora_output k_adapter k_lora_output v_adapter v_lora_outputys r=rDzZamba2Attention.forwardsf }--Q14DM?Q;NTV1W1W. j, ; 3 829=I!)R[99 9 99%Im44M'-7L29=I!)R[99 9 99%Im44M#m3J29=I!)R[99 9 99%Im44M'-7L ; # '+lJ(( $L* %DM) $\:| L L{{1~~1r>NNr*rFrGrHrIrrJr rrKr5rLrMrDrNrOs@r=rQrQls,026@@@@ @ "D( @ )4/ @@ @@@@@@D+|++l +  ++++++++r>rQc eZdZdZ ddededeeefdedzded df fd Z d e j d ed e j fd Z xZ S) Zamba2MLPaFeed-forward MLP layer for the Zamba2 model. Implements a gated feed-forward network that projects inputs to a larger intermediate size, applies GELU activation with gating, then projects back to the original size. Includes optional adapter layers for model adaptation. Nr*rRrSrTr.r/rVc t||_t|_||_|j|_|j|_t|jd|jgz|jj ||d|_ t|j|j|jj ||d|_ |j dkrtd|j dt|_t#jg|_t)|jD]j}||jz|kr-t-|j|jd|jgz||d| }nt#j}|j|kd S) a!Initialize the MLP layer. Args: config: The Zamba2 model configuration bare_block_idx: Index of the bare block in the model num_hybrid_layers: Total number of hybrid layers quant_config: Configuration for model quantization rXz .gate_up_projr3z .down_projgeluz5Only GELU activation is supported (got `hidden_act`: )z.gate_up_proj_adapter_list.r/N)r4r5rRr rurTreintermediate_sizeradd_bias_linear gate_up_projr down_proj hidden_actrcract_fnrrggate_up_proj_adapter_listrjrir)rqrkrl) r:rRrSrTr.r/rwgate_up_proj_adapterr<s r=r5zZamba2MLP.__init__,s  ;== !2!-!'!97   '( (,%+++    +  "  ,%(((       & &;&,&7;;; !ll *,r):):&t566 H HI600NBB'1&'/00 $LLLL ((($$(*{}}$  * 1 12F G G G G H Hr>r?rwc||\}}|j|}t|tjrJ||}||z}||}||\}}|S)aPForward pass through the MLP layer. Args: hidden_states: Input tensor [batch_size, seq_len, hidden_size] block_idx: Current shared transformer block index Returns: Output tensor [batch_size, seq_len, hidden_size] after applying gated feed-forward transformation )rrr7rrkrr)r:r?rwgate_up_statesrCadapterrBoutputs r=rDzZamba2MLP.forwardks!--m<<0;gr{33333gm,, '+5 N33 NN=11  r>rE)rFrGrHrIrrJdictrrKr5rLrMrDrNrOs@r=rr$s37 =H=H=H=H S> =H )4/ =H  =H =H=H=H=H=H=H~U\celr>rceZdZdZ ddededededzdedzd ed dffd Z d e j d e j dede j d e j f dZ xZ S)Zamba2AttentionDecoderLayeraSingle decoder layer combining attention and feed-forward networks. This layer implements a standard transformer block with: - Input layer normalization - Multi-head self-attention - Pre-feed-forward layer normalization - Feed-forward network (MLP) Nr*rRrSrTrUr.r/rVc:tt|||||||_t |||||d|_t d|jz|j|_ t |j|j|_ dS)aInitialize the decoder layer. Args: config: The Zamba2 model configuration bare_block_idx: Index of the bare block num_hybrid_layers: Total number of hybrid layers cache_config: Configuration for key-value caching quant_config: Configuration for model quantization prefix: Optional prefix for parameter names rSrTrUr.r/z .feed_forward)rSrTr.r/rXepsN) r4r5rQ self_attnr feed_forwardrre rms_norm_epsinput_layernormpre_ff_layernorm)r:rRrSrTrUr.r/r<s r=r5z$Zamba2AttentionDecoderLayer.__init__s& ) )/%%    & )/%+++     'q6+='=6CVWWW '(:@S T T Tr>r?original_hidden_statesrw positionsctj||gd}||}||||}||}|||}|S)aForward pass through the decoder layer. Args: hidden_states: Input tensor from previous layer original_hidden_states: Original input tensor for residual connection block_idx: Current shared transformer block index positions: IDs for positional embeddings Returns: Transformed hidden states after attention and feed-forward rr)r|rw)rw)rL concatenaterrrr)r:r?rrwrs r=rDz#Zamba2AttentionDecoderLayer.forwards0) 2 3   ,,];;  "'  --m<< ))-9)MM r>rrrOs@r=rrs,026,U,U,U,U ,U "D( ,U )4/ ,U,U ,U,U,U,U,U,U\,|,!& , , < ,  ,,,,,,,,r>rc eZdZdZ ddededzdedzdedzded df fd Z dd e j d e j dzd e j dzde j dzd e j f dZ xZ S)Zamba2MambaDecoderLayerzSingle Mamba decoder layer with normalization. This implements a Mamba block. It includes input normalization and can process sequences using either chunked or full computation depending on configuration. Nr*rR model_configrUr.r/rVcRt|j|jz}t |j|j|j||j|j|j |j ||j z|j d||||d|_ t|j|j |_dS)zInitialize the Mamba decoder layer. Args: config: The Zamba2 model configuration quant_config: Configuration for model quantization siluz.mixer)ressm_state_sizeconv_kernel_sizer use_conv_biasuse_biasn_groups num_headshead_dimr activationrrUr.r/rN)r4r5 mamba_expandrer mamba_d_state mamba_d_convrr mamba_ngroups n_mamba_headsrmambarr)r:rRrrUr.r/rr<s r=r5z Zamba2MambaDecoderLayer.__init__s #/&2DD *!/#0/ .+)*&&*>>,%%%$$$    & 'v'9v?RSSSr>r?transformer_hidden_statesrrcv|}|||z}||}||}||z}|S)a Forward pass through the Mamba decoder layer. Args: hidden_states: Input tensor [batch_size, seq_len, hidden_size] transformer_hidden_states: Optional output from transformer path Added to input if provided (used in hybrid architecture) positions: Optional position IDs (unused in Mamba) original_hidden_states: Optional original inputs (unused in Mamba) Returns: Transformed hidden states with residual connection applied )rr)r:r?rrrresidualrs r=rDzZamba2MambaDecoderLayer.forwardsV(! % 0),EEM,,];; M**!6) r>NNNr*)NNN)rFrGrHrIrr r rrKr5rLrMrDrNrOs@r=rrs,0+/26 %T%T%T"D(%T"D( %T )4/ %T  %T %T%T%T%T%T%TT:>)-6: ((|($)<$#6(<$& ( !& t 3 (  ((((((((r>rceZdZdZ ddededededzdedzd e dzd e d dffd Z d e j de j de j d e j fdZxZS)Zamba2HybridLayeraYHybrid layer combining Transformer and Mamba architectures. This layer implements the hybrid architecture described in the Zamba paper, where a shared transformer pathway processes input in parallel with a Mamba pathway. The transformer output is projected and added to the Mamba input for enhanced representation learning. Nr*shared_transformerrRrwrrUr.r/rVct||_||_t |j|jd||d|_t||||||_dS)zInitialize the hybrid layer. Args: shared_transformer: Transformer decoder layer for attention pathway Fz.linearr3rrUr.r/N) r4r5rwrrrelinearr mamba_decoder) r:rrRrwrrUr.r/r<s r=r5zZamba2HybridLayer.__init__Qs ""4&    %%%%     5 %%%    r>r?rrc||||j|}||\}}|||}|S)aForward pass through the hybrid layer. Processes input through parallel transformer and Mamba paths: 1. Transformer path processes input with attention 2. Transformer output is projected to match hidden size 3. Projected output is added to Mamba path input 4. Final output combines both paths' representations Args: hidden_states: Input tensor [batch_size, seq_len, hidden_size] original_hidden_states: Original input for transformer residual connection positions: Position IDs for positional embeddings Returns: Output tensor combining transformer and Mamba representations )rrwr)r)rrwrr)r:r?rrrrC layer_outputss r=rDzZamba2HybridLayer.forwardrsq0%)$;$; #9n %<% % !(,{{3L'M'M$!1** &?+  r>r)rFrGrHrIrrrJr r rrKr5rLrMrDrNrOs@r=rrHs,0+/26  7    "D(  "D(  )4/         B(|(!& (< (  ((((((((r>rc eZdZdZdddededdffdZd ejdejfd Z dd ejd ejd ejdzdeje zfd Z de e eejfdeefdZxZS) Zamba2ModelzCore Zamba2 model combining transformer and Mamba architectures. The model processes input through a sequence of hybrid and Mamba-only layers, using token embeddings and final layer normalization. r*r vllm_configr/rVNc  t|jj |j}|j |j|j}t|}|rJ |_ j |_ t|j j |_ dt jDt fdt! jD}g}t jD]\}} t't)|z| dkrAt+|} |} |t/| | | k|t1 | t3j||_t9 j j|_dS)zInitialize the Zamba2 model. Args: vllm_config: Configuration object containing model, cache, quantization and LoRA settings prefix: Optional prefix for parameter names in state dict ci|]\}}|| Sr).0rw layer_idxs r= z(Zamba2Model.__init__..s.   $ 9 y   r>c Vg|]%}t|t&S)r)rlen)ridxrUrRlayer2block_mapr/r.s r= z(Zamba2Model.__init__..sV   ,#&&)/&:&:!-!-$;    r>hybridrrN)r4r5r hf_configrUr. lora_configboolrR vocab_sizerre embed_tokens enumeratehybrid_layer_idsrrjrilayers_block_typerKrnextrlrrrrglayersrrfinal_layernorm)r:rr/rris_lora_enabledblocksrr layer_typeblockrwrUrRrr.r<s ` @@@@r=r5zZamba2Model.__init__s- )3"/ "/ "/ !- {++""""  +3 O       (1&2I(J(J            !!677      %.v/G%H%H   !Iz_-- 9::FX%%V +I6  %!%1%1%1%     +%1%1%1% mF++  'v'9v?RSSSr> input_idsc,||S)zConvert input token IDs to embeddings. Args: input_ids: Tensor of input token IDs Returns: Embedded representation of the input tokens )rr:rs r=embed_input_idszZamba2Model.embed_input_idss  +++r>r inputs_embedsc|||}|}tj|}t|jD]\}}||||}|}||}|S)aIForward pass through the model. Args: input_ids: Input token IDs positions: Position IDs for embeddings inputs_embeds: Optional pre-computed input embeddings Returns: Either final hidden states or intermediate tensors for pipeline parallelism N)rr)rrLclonerrr) r:rrrr?rrlayerrs r=rDzZamba2Model.forwards$   00;;M% "']!;!; )$+ 6 6 * * Iu!E'=#M *MM,,];; r>weightscgd}t|}t}|D]\}}|D]>\}}} ||vr |||}||} | j} | | || n/||vrK||} t | dt } | | ||||S)N))rdq_projq)rdk_projk)rdv_projv weight_loader)rnamed_parameterssetreplacergetattrradd) r:rstacked_params_mapping params_dict loaded_paramschkpt_weight_name loaded_weight param_name weight_nameshard_idparamrs r= load_weightszZamba2Model.load_weights's" " " 4002233 "%%% 07 1 1 , }5K 4 41 K&777$5$=$=k:$V$V!#$56 % 3  e]H===$K77#$56 '@U V V  e]333   / 0 0 0 0r>rA)rFrGrHrIr rKr5rLrMrr rDrtuplerrrNrOs@r=rrs9 BDRTRTRTzRT3RTRTRTRTRTRTRTh , ,%, , , , ,.2 !!<!<!|d* ! + + !!!!FHU3 3D-E$F3s8r>rc eZdZdZeddddZeddd eej ej ffd Z eddd eee e fee e e fffd Z ed ee e ffd Zd ddeded dffdZdejd ejfdZ ddejdejdejdzded ejf dZdejd ejdzfdZdeeeejfd eefdZxZS)Zamba2ForCausalLMa1Zamba2 model with causal language modeling head. This class wraps the core Zamba2 model and adds: - A language modeling head for next token prediction - Mamba state caching functionality - Support for model parallelism and quantization - Sampling capabilities for text generation r6zA.weightzB.weight)A_logz0.weightz1.weight)orig_to_new_substrrr rVcjtj|jj|jj|jjSrA)rmamba2_state_dtyperdtyperUmamba_cache_dtypemamba_ssm_cache_dtype)clsrs r=!get_mamba_state_dtype_from_configz3Zamba2ForCausalLM.get_mamba_state_dtype_from_configWs4 );  $ *  $ 6  $ :   r>c |j}|jj}|j|jz}t j||j|j|j |j |j |j S)a3Calculate shapes for Mamba's convolutional and state caches. Args: vllm_config: vLLM config Returns: Tuple containing: - conv_state_shape: Shape for convolutional state cache - temporal_state_shape: Shape for state space model cache )r tp_world_sizerrr state_size conv_kernel) parallel_configrrrrermamba2_state_shapetensor_parallel_sizerr mamba_headdimrr)rrr rrs r=!get_mamba_state_shape_from_configz3Zamba2ForCausalLM.get_mamba_state_shape_from_configbsi &5,6 %2Y5JJ(;/)>,-, .!.    r>c(tjSrA)rmamba2_state_copy_func)rs r=get_mamba_state_copy_funcz+Zamba2ForCausalLM.get_mamba_state_copy_funcs+BDDDr>r*rr/Nc|jj}|j}t||_||_||_|j|_t|t|d|_ t|j |j t|d|_ |j |j j|_ t!|j |_dS)a~Initialize the Zamba2 model for causal language modeling. Args: vllm_config: Configuration containing model, cache, quantization, LoRA and scheduler settings prefix: Optional prefix for parameter names Raises: AssertionError: If prefix caching is enabled (not supported by Mamba) model)rr/lm_headrN)rrscheduler_configr4r5rRrrr'r)rrrer* tie_weightsrrlogits_processor)r:rr/rRr+r<s r=r5zZamba2ForCausalLM.__init__s)3&7  & 0'4!#L,I,I    &     22    |// 0GHH !00A B Br>rc6|j|S)zConvert input token IDs to embeddings. Args: input_ids: Tensor of input token IDs Returns: Embedded representation of the input tokens )r)rrs r=rz!Zamba2ForCausalLM.embed_input_idssz)))444r>rrkwargsc 4||||}|S)aIForward pass through the model. Args: input_ids: Input token IDs positions: Position IDs for embeddings inputs_embeds: Optional pre-computed input embeddings **kwargs: Additional arguments passed to cache manager Returns: Output hidden states )r))r:rrrr/r?s r=rDzZamba2ForCausalLM.forwards(&     r>r?c<||j|}|S)zCompute logits for next token prediction. Args: hidden_states: Hidden states from model forward pass Returns: Logits for next token prediction )r-r*)r:r?logitss r=compute_logitsz Zamba2ForCausalLM.compute_logitss &&t|]CC r>rcXt|}|||jS)N)mapper)r%rhf_to_vllm_mapper)r:rloaders r=rzZamba2ForCausalLM.load_weightss+"4((""743I"JJJr>rA)rFrGrHrIr&r6 classmethodrrLrrrJr$rr'r rKr5rMrrrDr3rrrrNrOs@r=rrDs]& ""   !  u{EK' (   [  !  uS#Xc3m 44 5   [ :E%0BDV0V*WEEE[EBD%C%C%Cz%C3%C%C%C%C%C%C%CN55%,5555.2 <<|d*     6 |       KHU3 3D-E$FK3s8KKKKKKKKr>r)DrIcollections.abcr itertoolsrtypingrrLr transformersrvllm.attention.layerrvllm.compilation.decoratorsr vllm.configr r r vllm.distributedr %vllm.model_executor.layers.activationr$vllm.model_executor.layers.layernormr!vllm.model_executor.layers.linearrrrrr+vllm.model_executor.layers.logits_processorr-vllm.model_executor.layers.mamba.mamba_mixer2r,vllm.model_executor.layers.mamba.mamba_utilsrrrr'vllm.model_executor.layers.quantizationr+vllm.model_executor.layers.rotary_embeddingr3vllm.model_executor.layers.vocab_parallel_embeddingrr-vllm.model_executor.model_loader.weight_utilsr vllm.sequencer interfacesr"r#r$utilsr%r&r'Moduler)rQrrrrrrrr>r=rOs%$$$$$ %%%%%%******======<<<<<<<<<<AAAAAA<<<<<<888888HGGGGGEEEEEE GFFFFF@@@@@@POOOOO------KKKKKKKKKKAAAAAAAAAA44444444nuuuuubiuuup````` ```Fddddd")dddNWWWWWbiWWWtRRRRR RRRjccccc")cccL\K\K\K\K\K =(