`i_ ddlZddlmZddlmZmZmZddlZddlm Z ddl m Z m Z ddl mZmZddlmZmZmZddlmZdd lmZmZdd lmZmZdd lmZdd lmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4m5Z5ddl6m7Z7m8Z8m9Z9m:Z:ddl;mZ?Gdde+eZ@GddeZAGdde:ZBGdde7ZCGdde jDZEGdd e/ZFGd!d"e2ZGGd#d$e3ZHGd%d&e-ZIGd'd(eZJdZKGd)d*e1ZLd+eMd,eeMeMeMeMgeNffd-ZOGd.d/e0ZPGd0d1e.ZQGd2d3e jRZSd4eejTd5eejTd6eMd,eefd7ZUGd8d9e9ZVGd:d;e8ZWGd<d=eLZXGd>d?eeLZYgd@ZZdS)AN)Callable)AnyOptionalUnion)Cache DynamicCache)PretrainedConfiglayer_type_validation)create_causal_maskcreate_masks_for_generate!create_sliding_window_causal_mask)FlashAttentionKwargs) GenericForSequenceClassificationGradientCheckpointingLayer)BaseModelOutputWithPast SequenceClassifierOutputWithPast)rope_config_validation)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg) Gemma2Config) Gemma2AttentionGemma2ForCausalLM Gemma2MLP Gemma2ModelGemma2PreTrainedModel Gemma2RMSNormGemma2RotaryEmbeddingapply_rotary_pos_embeager_attention_forward)PaligemmaCausalLMOutputWithPast!PaliGemmaForConditionalGenerationPaliGemmaModelPaligemmaModelOutputWithPast)SiglipVisionConfigcTeZdZdZdZ ddZdS)Gemma3TextConfiga,! This is the configuration class to store the configuration of a [`Gemma3TextModel`]. It is used to instantiate an Gemma3Text model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Gemma3Text-7B. e.g. [google/gemma3_text-7b](https://huggingface.co/google/gemma3_text-7b) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 262208): Vocabulary size of the Gemma3Text model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`Gemma3TextModel`] hidden_size (`int`, *optional*, defaults to 2304): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 9216): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 26): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 4): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 131072): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. tie_word_embeddings (`bool`, *optional*, defaults to `True`): Whether to tie weight embeddings rope_theta (`float`, *optional*, defaults to 1000000.0): The base period of the RoPE embeddings. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. query_pre_attn_scalar (`float`, *optional*, defaults to 256): Scaling factor used on the attention scores sliding_window (`int`, *optional*, defaults to 4096): In Gemma3Text, every other layer uses sliding window attention. This is the size of the sliding window. layer_types (`list`, *optional*): Attention pattern for each layer. final_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the logits. attn_logit_softcapping (`float`, *optional*): Scaling factor when applying tanh softcapping on the attention scores. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings used in global attention. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`list[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`list[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE rope_local_base_freq (float, *optional*, defaults to 10000.0): The base period of the RoPE embeddings for local attention. use_bidirectional_attention (`bool`, *optional*, defaults to `False`): If True, the model will attend to all text tokens instead of using a causal mask. This does not change behavior for vision tokens. ```python >>> from transformers import Gemma3TextModel, Gemma3TextConfig >>> # Initializing a Gemma3Text gemma3_text-7b style configuration >>> configuration = Gemma3TextConfig() >>> # Initializing a model from the gemma3_text-7b style configuration >>> model = Gemma3TextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` gemma3_text@ $gelu_pytorch_tanh{Gz?ư>Trr.AFN@c ~tjd| |||d||_| _|_|_|_|_|_|_ | _ | _ | _ |_ |_|_|_|_|_|_|_|_|_|rjdzdz_|_|_t3|dd_j%fdt9jD_t;jjdS)N) pad_token_id bos_token_id eos_token_idtie_word_embeddingsrr;sliding_window_patterncLg|] }t|dzjzrdnd!S)r;sliding_attentionfull_attention)bool_sliding_window_pattern).0iselfs }/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/gemma3/modular_gemma3.py z-Gemma3TextConfig.__init__..sI   (,QUd6R,R'S'Si##Yi   )r __init__ vocab_sizemax_position_embeddings hidden_sizeintermediate_sizenum_hidden_layersnum_attention_headshead_dimnum_key_value_headsinitializer_range rms_norm_eps use_cache rope_thetaattention_biasattention_dropouthidden_activationquery_pre_attn_scalarsliding_windowfinal_logit_softcappingattn_logit_softcapping layer_typesuse_bidirectional_attentionrope_local_base_freq rope_scalingrgetrKranger )rNrTrVrWrXrYr[rZrbrUr\r]r^rArCrBrDr_r`rarcrdrgrerfrjrirhkwargss` rOrSzGemma3TextConfig.__init__s> ! %%% 3       %'>$&!2!2#6   #6 !2("$,!2!2%:",'>$&<#&+F( & A#'#6!#;q"@D $8!(t$$$(.zz2JA'N'N$   #    t566   D  d.0FGGGGGrQ)r0r1r2r3r4r5r6r7r8r9r:Trr;rTr<Fr=r6r>NNNNr?F)__name__ __module__ __qualname____doc__ model_typerSrRrQrOr.r.:sttlJ- ' ! $#%$)9JHJHJHJHJHJHrQr.ceZdZdZdZddddZeedZ dde e ee e e ffde e ee e e ffdededededeffd ZxZS) Gemma3Configa This is the configuration class to store the configuration of a [`Gemma3ForConditionalGeneration`]. It is used to instantiate an Gemma3ForConditionalGeneration according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the PaliGemma-2B. e.g. [google/gemma-3-4b](https://huggingface.co/google/gemma-3-4b) Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: text_config (`Union[Gemma3TextConfig, dict]`, *optional*): The config object of the text backbone. vision_config (`Union[AutoConfig, dict]`, *optional*): Custom vision config or dict. mm_tokens_per_image (`int`, *optional*, defaults to 256): The number of tokens per image embedding. boi_token_index (`int`, *optional*, defaults to 255999): The begin-of-image token index to wrap the image prompt. eoi_token_index (`int`, *optional*, defaults to 256000): The end-of-image token index to wrap the image prompt. image_token_index (`int`, *optional*, defaults to 262144): The image token index to encode the image prompt. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. Example: ```python >>> from transformers import Gemma3ForConditionalGeneration, Gemma3Config, SiglipVisionConfig, Gemma3TextConfig >>> # Initializing a Siglip-like vision config >>> vision_config = SiglipVisionConfig() >>> # Initializing a Gemma3 Text config >>> text_config = Gemma3TextConfig() >>> # Initializing a Gemma3 gemma-3-4b style configuration >>> configuration = Gemma3Config(vision_config, text_config) >>> # Initializing a model from the gemma-3-4b style configuration >>> model = Gemma3TextConfig(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```gemma3image_token_indexboi_token_indexeoi_token_index)image_token_id boi_token_id eoi_token_id) text_config vision_configNr6r9r|r}mm_tokens_per_imager\c |)t}tdn!t|tr tdi|}t|tr t di|}n*|(t }td||_||_||_||_ ||_ ||_ ||_ tjdi|dS)Nz@text_config is None, using default Gemma3TextConfig text config.zFvision_config is None, using default SiglipVisionConfig vision config.rR)r.loggerinfo isinstancedictr,r|r}rrwrxrvr\superrS) rNr|r}rrwrxrvr\rm __class__s rOrSzGemma3Config.__init__<s  *,,K KKZ [ [ [ [  T * * :*99[99K mT * * b.????MM  ".00M KK` a a a&*#6 ..!2!2""6"""""rQ)NNr6r~rrr9)rnrorprqrr attribute_mapr.r, sub_configsrrrstrrintfloatrS __classcell__rs@rOrtrts . . `J-))M (+KJNMQ#&&&!(#'##e$4d38n$DEF# &8$sCx.&H IJ#! #  #  ##!##########rQrtceZdZdS)Gemma3ModelOutputWithPastNrnrorprRrQrOrr^DrQrceZdZdS)Gemma3CausalLMOutputWithPastNrrRrQrOrrbrrQrc PeZdZdZd dedededeffd Zdejffd Z xZ S) Gemma3TextScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. ?num_embeddings embedding_dim padding_idx embed_scalect||||dtj|ddS)NrF) persistent)rrSregister_buffertorchtensor)rNrrrrrs rOrSz&Gemma3TextScaledWordEmbedding.__init__ksK  DDD ]EL,E,ERWXXXXXrQ input_idsct||j|jjzSN)rforwardrtoweightdtype)rNrrs rOrz%Gemma3TextScaledWordEmbedding.forwardos4wwy))D,<,?,? @Q,R,RRRrQ)r) rnrorprqrrrSrTensorrrrs@rOrrfsYYsY3YSY_dYYYYYYSSSSSSSSSSSrQrc$eZdZdeffd ZxZS) Gemma3MLPconfigcJt|dSrrrSrNrrs rOrSzGemma3MLP.__init__t!      rQrnrorpr.rSrrs@rOrrssE!/!!!!!!!!!!rQrc*eZdZddedeffd ZxZS) Gemma3RMSNormr:dimepscNt||dS)Nrrr)rNrrrs rOrSzGemma3RMSNorm.__init__ys& Sc*****rQ)r:)rnrorprrrSrrs@rOrrxsP++C+e++++++++++rQrc&eZdZddeffd ZxZS)Gemma3RotaryEmbeddingNrcJt|dSrr)rNrdevicers rOrSzGemma3RotaryEmbedding.__init__~rrQrrrs@rOrr}sJ!!/!!!!!!!!!!rQrc&eZdZdedeffd Zeddd dd ejd ejd e ejde e d e ej d e e deeje eje eejffdZxZS)Gemma3Attentionr layer_idxcF|j|dk|_t|||jr|jnd|_|jj |_t|j |j |_ t|j |j |_ dS)NrHr) rg is_slidingrrSrdrrh is_causalrrZr]q_normk_normrNrrrs rOrSzGemma3Attention.__init__s ,Y7;NN +++7;Pf33D![DD#V=PQQQ #V=PQQQ rQpast_key_valuepast_key_values4.58new_nameversionN hidden_statesposition_embeddingsattention_maskcache_positionrmreturnc |jdd}g|d|jR}|||dd} |||dd} |||dd} || } || } |\} } t| | | | \} } |&| | |d}| | | |j |\} } t}|j jdkrt|j j}||| | | |f|jr|jnd|j|jd|\}}|jg|dR}||}||fS)Nr;r)sincosreagerr=)dropoutscalingrd)shaperZq_projview transposek_projv_projrrr&updaterr'r_attn_implementationrtrainingrarrdreshape contiguouso_proj)rNrrrrrrm input_shape hidden_shape query_states key_states value_statesrr cache_kwargsattention_interface attn_output attn_weightss rOrzGemma3Attention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT {{<00 [[,, &S#7 jRUWZ#[#[ j  &#&snUUL'6'='=j,X\Xfht'u'u $J (? ; +w 6 6"9$+:Z"[ $7$7      % /3mDD**L. % %  % % ! \*k);;;;;;FFHH kk+.. L((rQ)NN)rnrorpr.rrSrrrrr LongTensorrrtuplerrrs@rOrrsR/RCRRRRRR_%0A6RRR ,059 -)-)|-)#\-)!. -) "% -) !!12 -)-.-) u|Xel3XeEL>Q5RR S-)-)-)SR-)-)-)-)-)rQrc\eZdZdedeffd Zeddd dd ejd ejd ejd e ejde ej de e de e de e de ej de eje e ejejfffdZxZS)Gemma3DecoderLayerrrct||_|j|_||_|j||_t|||_t||_ t|j|j |_ t|j|j |_t|j|j |_t|j|j |_dS)N)rrr)rrSrrVrrgattention_typer self_attnrmlprr]input_layernormpost_attention_layernormpre_feedforward_layernormpost_feedforward_layernormrs rOrSzGemma3DecoderLayer.__init__s  !-"$0;()LLLV$$,T-=6CVWWW(5d6FFL_(`(`(`%)6t7GVM`)a)a)a&*78HfNa*b*b*b'''rQrrrrNFrposition_embeddings_globalposition_embeddings_localr position_idsoutput_attentionsr^rrc ^|} ||}|jjr|} n|} |jd|| |||||| d| \}} ||}| |z}|} ||}||}||}| |z}|f}|r|| fz }|S)N)rrrrrrr^rrR)rrrrrrr)rNrrrrrrrr^rrmresidualrself_attn_weightsoutputss rOrzGemma3DecoderLayer.forwards!,,];;  > $ =";  "< +94> , ' 3)%+/) , ,  , , ( (55mDD  =0  66}EE // 77 FF  =0  "  , )+ +GrQ)NNNFFN)rnrorpr.rrSrrrrrrrJr FloatTensorrrrs@rOrrsO c/ cC c c c c c c_%0A6RRR 2637+/,1$)5900|0%*L0$)< 0 !. 0 u/0 0"%0$D>0D>0!!120 u (51BEDU1U+V"WW X000SR00000rQrc eZdZdZgdZdZdS)Gemma3PreTrainedModel)rSiglipVisionEmbeddingsSiglipEncoderLayer#SiglipMultiheadAttentionPoolingHeadctj||t|tr |jjdSd|jjvr |j jdSdS)NRMSNorm) r _init_weightsrGemma3MultiModalProjectormm_input_projection_weightdatazero_rrnr)rNmodules rOr z#Gemma3PreTrainedModel._init_weights s|%dF333 f7 8 8 '  - 2 8 8 : : : : : &*3 3 3 M  $ $ & & & & &4 3rQN)rnrorpbase_model_prefix_no_split_modulesr rRrQrOrrs;'''''rQrrdrc Zdtdtdtdtdtf fd }|S)zA Enables a bidirectional mask within the sliding window. batch_idxhead_idxq_idxkv_idxrc0t||z kS)zA token can attend to any other token if their absolute distance is within the (exclusive) sliding window size (distance < sliding_window).)abs)rrrrrds rO inner_maskz1_bidirectional_window_overlay..inner_masks56>""^33rQrrJ)rdrs` rO_bidirectional_window_overlayrsL 4c4S44c4d444444 rQc"eZdZUeed<deffd Z ddeejdeej deejdee deej d ee d ee d ee d eejd e edefdZxZS)Gemma3TextModelrc.t|t|j|j|j|jjdz|_tj |}|j |_ ddi|_ t||_dS)N?)r rope_typedefaultr)rrSrrTrVrr embed_tokenscopydeepcopyrir_rjrrotary_emb_localrs rOrSzGemma3TextModel.__init__&s    :  v143CQUQ\QhjmQm    v&&"7*I6 5V D D DrQNrrrr inputs_embedsr^routput_hidden_statesrrmrc ||n |jj}||n |jj}||n |jj}|du|duz rt d|jr%|jr|rtdd}|| |}|r||jst|j}| B|| nd} tj | | |jdz|j} || d}t#|x} t$si|j||| ||d} | }|jjr"d | d <t+|jj|d <t/d i| t1d i|d } |}|||}|||}|rd nd}|rd nd}|jd|jjD]=}|r||fz }||f||| |j||||| d | }|d}|r ||dfz }>||}|r||fz }t?||||S)N:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Fr"rr;rr input_embedsrrrrcBtjdtjS)NT)r)rrrJ)argss rOz)Gemma3TextModel.forward..ps TY^Yc@d@d@drQor_mask_functionrIrHrR)rrrrrrr^r)last_hidden_staterr attentions) rrr(r^ ValueErrorgradient_checkpointingrr warning_oncer#r get_seq_lengthrarangerr unsqueezerrr$rhrrdr r rotary_embr&layersrXrnormr)rNrrrrr'r^rr(rrmpast_seen_tokenscausal_mask_mapping mask_kwargssliding_mask_kwargsrrrall_hidden_statesall_self_attns decoder_layer layer_outputss rOrzGemma3TextModel.forward5sU2C1N--TXT_Tq$8$D $+Jj "+!6IIDK>>O  !CRC^==???de "\  =#6q#99$+N  )33A66L?-FF + -"0"0#2 , K#."2"2"4"4 {6 t2d2d ./:WX\XcXr:s:s#$67#5"C"C{"C"C%F%]%]I\%]%]##  & &*__]L%Q%Q"$($9$9-$V$V!#7@BBD0:d![)H4;+H)HI 6 6M# 6!m%55!)M +E*C2=3OP) /"3#-    M*!,M  6=#3"55 -00  2 -!1 1 &+++%     rQ NNNNNNNNN)rnrorpr.__annotations__rSrrrrrrrJrrrrrrs@rOrr#sM  E/ E E E E E E"151537+/59$(,0/359o o E,-o !.o u/0 o "% o   12 o D>o $D>o 'tno !!12o +,o  !o o o o o o o o rQrc4eZdZUeed<dZdeffd ZxZS)Gemma3ForCausalLMrlanguage_modelcrt|t||_dSr)rrSrmodelrs rOrSzGemma3ForCausalLM.__init__s.    $V,, rQ)rnrorpr.rGrrSrrs@rOrIrIsY (-/----------rQrIc:eZdZdeffd ZdejfdZxZS)r rcttjt j|jj|jj|_ t|jj|jj |_ t|jj|jjz|_t|jdz|_|j|jz|_tj|j|j|_dS)Nrr) kernel_sizestride)rrSnn Parameterrzerosr}rVr|r rlayer_norm_epsmm_soft_emb_normr image_size patch_sizepatches_per_imagertokens_per_siderO AvgPool2davg_poolrs rOrSz"Gemma3MultiModalProjector.__init__s *,, K,8&:L:X Y Y+ + '!.  ,&2F2U! ! ! "%V%9%DH\Hg%g!h!h"6#=s#BCC1T5II 1A$JZ[[[ rQvision_outputsc|j\}}}|dd}||||j|j}|}||}|d}|dd}||}tj ||j }| |S)Nr;r) rrrrXrr[flattenrUrmatmulr type_as) rNr\ batch_size_ seq_lengthreshaped_vision_outputspooled_vision_outputsnormed_vision_outputsprojected_vision_outputss rOrz!Gemma3MultiModalProjector.forwards$2$8! Az"0":":1a"@"@"9"A"A  D$:D.inner_masks;v(