.`i<`dZddlmZddlmZddlmZddlmZddl Z ddl m Z ddl m Z dd l mZdd lmZdd lmZmZdd lmZmZdd lmZddlmZddlmZddlmZmZm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1m2Z2m3Z3m4Z4m5Z5ee6Z7ede8dzde8dzde j9fdZ:Gdde j9Z;Gd d!e j9Z<Gd"d#e j9Z=eGd$d%e j9Z>Gd&d'e j9e.e/Z?dS)(z?Inference-only Gemma model compatible with HuggingFace weights.)Iterable)cache)islice)AnyN)nn) GemmaConfig) Attention)support_torch_compile) CacheConfig VllmConfig) get_pp_group$get_tensor_model_parallel_world_size) init_logger) GeluAndMul) GemmaRMSNorm)MergedColumnParallelLinearQKVParallelLinearRowParallelLinear)LogitsProcessor)QuantizationConfig)get_rope)VocabParallelEmbedding)default_weight_loader)IntermediateTensors) SupportsLoRA SupportsPP)AutoWeightsLoaderis_pp_missing_parameter'make_empty_intermediate_tensors_factory make_layers maybe_prefix hidden_acthidden_activationreturnc|.|td||tdS|dkrtdS|dkrtdStd|d) Na~Gemma's activation function was incorrectly set to exact GeLU in the config JSON file when it was initially released. Changing the activation function to approximate GeLU (`gelu_pytorch_tanh`). If you want to use the legacy `%s`, edit the config JSON to set `hidden_activation=%s` instead of `hidden_act`. See https://github.com/huggingface/transformers/pull/29402 for more details.tanh) approximategelu_pytorch_tanhgelunonezActivation function z# is not supported for Gemma models.)loggerwarningr ValueError)r#r$s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/gemma.py_get_gemma_act_fnr0<s   ! NN$   f---- 1 1 1f---- f $ $f---- R: R R R   c^eZdZ d dedededzdedzdedzded dffd Zd ZxZS) GemmaMLPN hidden_sizeintermediate_sizer#r$ quant_configprefixr%ctt||gdzd||d|_t ||d||d|_t |||_dS)NFz .gate_up_projbiasr7r8z .down_proj)super__init__r gate_up_projr down_projr0act_fn)selfr5r6r#r$r7r8 __class__s r/r>zGemmaMLP.__init__[s 6   ! #%+++    +  %(((    ( 4EFF r1c||\}}||}||\}}|SN)r?rAr@)rBxgate_up_s r/forwardzGemmaMLP.forwardusD&&q))  KK ~~a  1r1)NNNr4) __name__ __module__ __qualname__intstrrr>rI __classcell__rCs@r/r3r3Zs "&(,26GGGG$J G : G )4/ GG GGGGGG4r1r3ceZdZ ddededededeeefd ed edzd edzd ed dffd Z de j de j d e j fdZ xZ S)GemmaAttention Nr4r5 num_heads num_kv_headshead_dimrope_parametersmax_position_embeddings cache_configr7r8r%c $t||_t} ||_|j| zdksJ|j| z|_||_|j| kr|j| zdksJn| |jzdksJtd|j| z|_||_ |j|j z|_ |j|j z|_ |j dz|_ t||j |j|jd|| d|_t|j|j z|d|| d|_t#|j ||d |_t'|j|j |j |j||| d  |_dS) NrrgFz .qkv_projr;z.o_projT) max_positionrW is_neox_stylez.attn)rUrYr7r8)r=r>r5rtotal_num_headsrTtotal_num_kv_headsmaxrUrVq_sizekv_sizescalingrqkv_projro_projr rotary_embr attn) rBr5rTrUrVrWrXrYr7r8tp_sizerCs r/r>zGemmaAttention.__init__}s &688(#g-2222-8".  "g - -*W499999T4499994#:g#EFF  nt}4 (4=8 }d* )  M   #%'''    (  4= 0 %%%%     # M0+      N M L*%%###    r1 positions hidden_statesc"||\}}||j|j|jgd\}}}||||\}}||||}||\} }| S)N)dim)rcsplitr`rarerfrd) rBrhriqkvrHqkv attn_outputoutputs r/rIzGemmaAttention.forwards }--Q))T[$, E2)NN1ay!Q//1ii1a(( KK ,,  r1)rSNNr4)rJrKrLrMdictrNrr rr>torchTensorrIrOrPs@r/rRrR|s (,+/26@ @ @ @  @  @ c3h @ "%@ "D(@ )4/@ @  @ @ @ @ @ @ D < |          r1rRc eZdZ d dededzdedzdeddf fd Zd ej d ej d ej dzde ej ej ffd Z xZ S)GemmaDecoderLayerNr4configrYr7r8r%c t|j|_t|j|j|j|j|j|j|||d |_ t|j|j |j t|dd||d|_t|j|j|_t|j|j|_dS)Nz .self_attn) r5rTrUrVrXrWrYr7r8r$z.mlp)r5r6r#r$r7r8eps)r=r>r5rRnum_attention_headsnum_key_value_headsrVrXrW self_attnr3r6r#getattrmlpr rms_norm_epsinput_layernormpost_attention_layernorm)rBryrYr7r8rCs r/r>zGemmaDecoderLayer.__init__s !-'(03_$*$B"2%%(((    ($6(%f.A4HH%???     ,F,>FDWXXX(4  F$7) ) ) %%%r1rhriresidualc||}||}n|||\}}|||}|||\}}||}||fS)N)rhri)rrrr)rBrhrirs r/rIzGemmaDecoderLayer.forwards  $H 00??MM&*&:&:=(&S&S #M8''  #'"?"? x"X"X x// h&&r1)NNr4) rJrKrLrr rrNr>rurvtuplerIrOrPs@r/rxrxs,026    "D( )4/           B'<'|',% ' u|U\) * ''''''''r1rxc eZdZdddedeffdZdejdejfdZ ddejd ejd e d zd ejd zdeje zf d Z de e eejfde efdZxZS) GemmaModelr4r8 vllm_configr8ct|jj|j|j|_tjj |_ tj fd|d\|_ |_|_t!j j|_|jj dz}|dt)j|dt-d d gj |_dS) Nc*t|S)Nr)rx)r8rYryr7s r/z%GemmaModel.__init__..s , l6r1z.layersrr{g? normalizerF) persistentrir)r=r> model_config hf_configrYr7ryr vocab_sizer5 embed_tokensr!num_hidden_layers start_layer end_layerlayersrrnormregister_bufferrutensorr make_empty_intermediate_tensors)rBrr8rrYryr7rCs @@@r/r>zGemmaModel.__init__s+ )3"/ "/  2      9D  $      %%% 9 9 9 5$.$+!!39LMMM [,c1  \5< +C+CPUVVV/V j )6+=0 0 ,,,r1 input_idsr%c,||SrE)rrBrs r/embed_input_idszGemmaModel.embed_input_ids's  +++r1Nrhintermediate_tensors inputs_embedsc|tjr'||}n||}||jz}d}n|d}|d}t |j|j|jD]}||||\}}tjst||dS| ||\}}|S)Nrir)rir) r is_first_rankrrrrrr is_last_rankrr) rBrrhrrrirlayerrHs r/rIzGemmaModel.forward*s >> ' 8( - $ 4 4Y ? ? T_ ,MHH0AM+J7HDK)94>JJ  E&+e'' #M88 ~~* &"/XFF  99]H== qr1weightsc(gd}t|}t}|D]\}}|D]i\}}} ||vr |||}|dr||vr;t ||rL||} | j} | | || nU|dr||vrt ||r||} t| dt} | | || ||S)N))rcq_projro)rck_projrp)rcv_projrq)r? gate_projr)r?up_projrz.bias weight_loader) rtnamed_parameterssetreplaceendswithrrrradd) rBrstacked_params_mapping params_dict loaded_paramsname loaded_weight param_name shard_nameshard_idparamrs r/ load_weightszGemmaModel.load_weightsHsg" " " 4002233 "%%% #* $ $ D-4J 4 40 JT))||J ;;==))d+.E.E*466#D) % 3  e]H=====))d+.E.E*466#D) '@U V V  e]333   d # # # #r1rE)rJrKrLr rNr>rurvrrrIrrrrrOrPs@r/rrsAC   z 3      @,,%,,,,,.2 <<2D8  |d*  + + <$HU3 3D-E$F$3s8$$$$$$$$r1rc ,eZdZgdddgdZdddedeffd Zd ejd ejfd Z dd ejdejde d zdejd zd eje zf dZ dejd ejd zfdZ de eeejfd eefdZxZS)GemmaForCausalLM)rrrrr)rcr?r4rrr8c8t|jj}|j}||_|jsJ||_t|t|d|_ t|j |_ |j j |_ dS)Nmodel)rr8)r=r>rrr7rytie_word_embeddingsrr"rrrlogits_processorr)rBrr8ryr7rCs r/r>zGemmaForCausalLM.__init__|s )3"/  ))))(#L,I,I    !00A B B J 6 ,,,r1rr%c6|j|SrE)rrrs r/rz GemmaForCausalLM.embed_input_idssz)))444r1Nrhrrc6|||||}|SrE)r)rBrrhrrris r/rIzGemmaForCausalLM.forwards) y"6   r1ricF||jj|}|SrE)rrr)rBrilogitss r/compute_logitszGemmaForCausalLM.compute_logitss#&&tz'> NN r1rclt||jjrdgnd}||S)Nzlm_head.) skip_prefixes)rryrr)rBrloaders r/rzGemmaForCausalLM.load_weightssC" +/;+JTJ<rurvrrrIrrrrrrOrPs@r/rros{      BD   z 3      $55%,5555<@-1  < < 2D8  |d*  + +    |  ,HU3 3D-E$F,3s8,,,,,,,,r1r)@__doc__collections.abcr functoolsr itertoolsrtypingrrur transformersrvllm.attention.layerr vllm.compilation.decoratorsr vllm.configr r vllm.distributedr r vllm.loggerr%vllm.model_executor.layers.activationr$vllm.model_executor.layers.layernormr!vllm.model_executor.layers.linearrrr+vllm.model_executor.layers.logits_processorr'vllm.model_executor.layers.quantizationr+vllm.model_executor.layers.rotary_embeddingr3vllm.model_executor.layers.vocab_parallel_embeddingr-vllm.model_executor.model_loader.weight_utilsr vllm.sequencer interfacesrrutilsrrr r!r"rJr,rNModuler0r3rRrxrrr1r/rsl$FE$$$$$$ $$$$$$******======////////OOOOOOOO######<<<<<<====== HGGGGGFFFFFF@@@@@@VVVVVVOOOOOO------00000000 X   d  Tz Y    :ryDMMMMMRYMMM`6'6'6'6'6' 6'6'6'rffffffffR:,:,:,:,:,ry, :,:,:,:,:,r1