`iӷ(ddlZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m Z ddl m cmZddlmZddlmZmZddlmZdd lmZmZdd lmZdd lmZdd lmZm Z dd l!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6m7Z7e.j8e9Z:ee,dGddeZ;ee,dGdde*Z<Gdde j=Z>Gdd e j=Z?Gd!d"e j=Z@Gd#d$e j=ZAGd%d&e j=ZBGd'd(e j=ZCGd)d*e j=ZDGd+d,e j=ZEGd-d.e j=ZFGd/d0e j=ZGGd1d2e&ZHGd3d4e jIZJGd5d6e j=ZKGd7d8e j=ZLGd9d:e j=ZMGd;de jPd?eQd@e jPfdAZR dgdCe j=dDe jPdEe jPdFe jPdGee jPdHeSdIeeSdJeeSd@eTe jPe jPffdKZU dhdLe jPdMe jPdNe jPdOee jPdPeQf dQZVGdRdSe j=ZWGdTdUeZXe,GdVdWe&ZYe,dXGdYdZeYZZe,d[Gd\d]eYeZ[Gd^d_e j=Z\e,d`GdadbeYZ]e,dcGdddeeYeZ^gdfZ_dS)iN)CallableSequence) dataclass)OptionalUnion)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg) AutoModel)Gemma3nAudioConfig Gemma3nConfigGemma3nTextConfigGemma3nVisionConfigzL Base class for Gemma3n outputs, with hidden states and attentions. ) custom_introc\eZdZUdZdZeejed<dZ eejed<dS)Gemma3nModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nimage_hidden_statesaudio_hidden_states) __name__ __module__ __qualname____doc__r(rtorch FloatTensor__annotations__r)/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/gemma3n/modeling_gemma3n.pyr'r'3sP  8<%"34;;;7;%"34;;;;;r2r'zS Base class for Gemma3n causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<dZee ejed<dZeejed<dZeejed <dS) Gemma3nCausalLMOutputWithPastaF loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr(r))r*r+r,r-r6rr.r/r0r7r8r r9tupler:r(r)r1r2r3r5r5Ms$)-D(5$ %,,,*.FHU& '...'+OXe_+++8Tdimeps with_scalec t||_||_|jr-t jt j||_dS| dt j dddS)Nweight?F persistent) super__init__r@rAnn Parameterr.onesrCregister_buffertensor)selfr?r@rA __class__s r3rHzGemma3nRMSNorm.__init__qsz $ ? P,uz#77DKKK  5<+<+< O O O O Or2c|tj|ddd|jzz S)NrT)keepdim)r.sqrtpowmeanr@)rNxs r3_normzGemma3nRMSNorm._norm{s85:aeeAhhmmBm==HIIIIr2rVreturnc|||jz}||SN)rWfloatrCtype_as)rNrVoutputs r3forwardzGemma3nRMSNorm.forward~sBAGGII&&):):)<)<<~~a   r2cHt|jjd|jS)Nz, eps=)r;rCshaper@rNs r3 extra_reprzGemma3nRMSNorm.extra_reprs% )**<<$(<<T) r*r+r,intr[boolrHrWr.Tensorr^rb __classcell__rOs@r3r=r=psPPCPePPPPPPPJJJ!!%,!!!! =======r2r=ceZdZdeffd ZdejdejdejfdZdejde d e d e d e d e d e dejfdZ dejdejdejfdZ xZ S)%Gemma3nAudioRelativePositionEmbeddingconfigc$t||_|jj|_|jj|_|j|jz|_td|jj dz |_ |jj |_ tj|j|j|jzd|_d}d}|jdz}t!jt%|t%|z t|dz dz }|t'jt'j|| zz}|d|ddd dS) Nrr FbiasrDg@rinv_timescalesrE)rGrHrjconf_num_attention_heads num_heads hidden_sizechannelshead_dimmaxconf_attention_context_left max_backwardconf_attention_context_right max_forwardrILinearpos_projmathlogr[r.exparangerL unsqueeze)rNrj min_timescale max_timescalenum_timescaleslog_timescale_incrementrnrOs r3rHz.Gemma3nAudioRelativePositionEmbedding.__init__sg  = /  7 4;#JQ#NOO;C $-$-1OV[\\\   !+"&(5+?+?% BVBV+V"W"WZ]^lop^prsZtZt"t&5<3O3OSjRj3j)k)kk    " " , ,Q / / 9 9! < <      r2positiondtyperXcN|d}||j|jt jz}t jt j|t j |gd}| |S)NrQ)devicerr?) r[rrntorr.float32catsincostype)rNrr scaled_time timing_signals r3_get_timing_signal_1d_posz?Gemma3nAudioRelativePositionEmbedding._get_timing_signal_1d_poss>>##--b11!4!7!7xV[Vc!7!d!dd  59[#9#959[;Q;Q"RXZ[[[ !!%(((r2term_bd_before_shift batch_sizerpnum_query_blocksquery_block_sizekey_context_sizemax_span_plus_1c|dz|z }d|f} tj|| } | |||||dzzf} | ddddddd||zf} | |||||f} | S)aZPerforms the relative shift. Args: term_bd_before_shift: Tensor of shape [B, N, U, W, F_span]. batch_size (B), num_heads (N), num_query_blocks (U), query_block_size (W), key_context_size (C = W+L+R), max_span_plus_1 (F_span = L+R+1). Returns: Tensor of shape [B, N, U, W, C]. r rN)rI functionalpadreshape)rNrrrprrrrpad_amount_last_dim padding_tupleterm_bd_paddedterm_bd_reshapedterm_bd_slicedterm_bd_shifteds r3_relative_shiftz5Gemma3nAudioRelativePositionEmbedding._relative_shifts4 0!3F/0 **+?OO *11  $4q$89    *!!!QQQ3X5EHX5X3X*XY)00      r2querieskeysc X|j\}}}}}|j\}}} }}tj|j|j dz d|jd} | jd} || |j} | | } | d| |j |j  d}|ddddd}|ddddd}tj||}|ddddd}|ddd}| ||||z|}tj||}| ||||| }||||||| | }||zS) Nr rQrrrrr)r`r.r~rvrxrrrrrzrrprssqueezepermutematmulr)rNrrrrrrprs_r pos_indicesrsin_emb_timing_signalprojected_sin_embsin_emb queries_pkeys_p_tterm_ac q_permuted s_permuted q_reshapedterm_bd_unshifed_matmulterm_bd_unshifedrs r3r^z-Gemma3nAudioRelativePositionEmbedding.forwards OVmK $&6 8'+z$11l4#4t7G6G!6KRX_Xfgggqq   &+A. $ > > w}!?! !  !MM*?@@#++APTP]^^ff   OOAq!Q22 <<1aA..,y(33__Q1a33 __Q1--  '' I?ORb?bdlmm #(,z:"F"F3::        ..         ((r2) r*r+r,r!rHr.rerrrcrr^rfrgs@r3riris 1      .)%,)u{)W\Wc)))) ;#l;; ;  ;  ;;; ;;;;zL)u|L)5<L)ELL)L)L)L)L)L)L)L)r2riceZdZdeffd ZdejdededejfdZdejdejfd Z dejdejfd Z dejd ej dejfd Z xZ S) Gemma3nAudioAttentionrjct||_|jj|_|jj|_|j|jz|_|jj|_|jj |_ td|jj dz |_ |jj|_|j|j z|j z|_t#||_t'jt+j|jf|_t'j|j|j|jzd|_t'j|j|j|jzd|_t'j|j|j|jzd|_|jdz}dt*jjt+jdz }|d||z !d t+j"t+j#|j|jft*j$ d j%}t+j"t+j#|j|jft*j$ |j |j z }t+j#|j|jft*j$ }||z|z}|d |d |d t+j|j&d dS)Nrr FrlrDq_scalerEr)diagonallocal_causal_valid_masksoftcap)'rGrHrjrorprqrsconf_attention_chunk_size chunk_sizerwmax_future_horizonrtrumax_past_horizonconf_attention_logit_capattention_logits_soft_cap context_sizerirelative_position_embeddingrIrJr.zeros per_dim_scaleryq_projk_projv_projrsoftplusrMrLclonedetachtrilrKrdTr[)rNrjr r_softplus_0lower_causal_maskupper_causal_maskrrOs r3rHzGemma3nAudioAttention.__init__6s  =;2(DN: +?"&+"J #At{'NQR'R S S)-)M& Od.CCdF]]+PQW+X+X(\%+t}6F*G*GHHi 0$.4=2PW\]]] i 0$.4=2PW\]]] i 0$.4=2PW\]]] -%UX099%,s:K:KLLL  Y<)?(F(F(H(H(O(O(Q(Q^cddd!J J)4?;5: N N N     "J J):;5: N N N*T-DD   #(*dot?P-QY^Yc"d"d"d"9 > @ @      r2rVpad_left pad_rightrXc|j^}}}|||g|R}|||g|R}tj|||gd}|S)Nr r)r` new_zerosr.r) rNrVrrbatchr tail_shapeleftrights r3 _pad_dim1zGemma3nAudioAttention._pad_dim1asl !q:{{E89j99:: UI; ;;<< ItQ&A . . .r2r9c$|j}|dd\}}||jzdz |jz}||jz|z x}dkr||d|}|||jf|ddz}||}|S)aETurns a sequence to non overlapping blocks. Args: hidden_states: a tensor of [batch, time, ...]. Returns: A tensor of [batch, num_blocks, block_size, ...], with necessary paddings, where output[:, i, ...] are x[:, i*block_size:(i+1)*block_size, ...]. Nrr r)r`rrr contiguous)rNr9r`bt num_blocks padding_len permute_dimss r3_convert_to_blockz'Gemma3nAudioAttention._convert_to_blockhs#RaRy1$/)A-$/A %7!; ;Kq @ @ NN=![IIM:t7%)C %--l;;FFHH r2c0|j}|j|jzdz }||||}|j}|j}|d||}|jdkr"|jdkrtj|dd}| S)aExtracts temporal context for every block. Args: hidden_states: a tensor of [batch, time, ...]. Returns: A tensor of [batch, num_blocks, context_size, ...], with necessary paddings, where context_size = block_size + left_context + right_context, and output[:, i, ...] are x[:, start-left_context:end+right_context, ...], start = i * block_size, end = (i + 1) * block_size. r ) dimensionsizesteprrrQ)source destination) rrrrrunfoldndimr.movedimr)rNr9rr frame_len frame_step x_unfoldeds r3_extract_block_contextz,Gemma3nAudioAttention._extract_block_context~s(+do=A }h JJ % _ #))AIJ)WW    ! !jo&9&9z"!LLLJ$$&&&r2maskc g|jdd|j|jR}|||}|||}|||}tj j |j }ddd|jf}| |} ||jz| z}|jdd\} } ||} ||} ||}| jd}|}||}|jdkr@|jd|jdz|jkr|| ||j}|j| ||jfkr&t'd|jd| d|d|jd |dd }|jd d d }tj|||j}|| | }|j|j}||z }tj|}||z}tj||tj|jj}tj j |dtj! |j }|j\}}}}}|jd}|"d ddddd||}|"d ddddd||}tj#||} | |||||"d dddd}!|!| ||j$z|j|jf}!|!ddd| f}!|!S)NrQr rrrz%Shape of extracted_valid_mask_blocks z is not (z, z) after potential reshape.rr?rr)%r`rprsrrrrrr.rIrrrviewrrrrr ValueErrorrr logical_andrrrrtanhwherefinforminsoftmaxrrbmmr)"rNr9r qkv_shape query_states key_states value_statesper_dim_scale_spbroadcast_shapeper_dim_scale_sp_broadcastrq_time query_blocks key_blocks value_blocksroriginal_valid_maskextracted_valid_mask_blockscondition_from_input_validitycondition_from_causalityfinal_condition_for_wherer7 softcap_val probabilitiesb_dimn_dimu_dimw_dimc_dimh_dimprob_bunv_bun result_bmmcontext_vectorss" r3r^zGemma3nAudioAttention.forwardsNm)#2#.NN NN {{=1199)DDOOQQ [[//77 BBMMOO {{=1199)DDOOQQ  8.778JKKaDM2%5%:%:?%K%K"#dl25OO )/3 F--l;; 00<< 22<@@ '-a0 $e'+&A&ABU&V&V# ( , 1 1+1!47R7XYZ7[[_c_ppp*E*M*M,d.?++ ' ' ,    1   V/5VV@JVV$VV(,(9VVV )D(M(Ma(P(P(Z(Z[](^(^%$(#?#I#I!#L#L#V#VWX#Y#Y#c#cde#f#f %*$5 ) $ ' '(E(L M M% % ! 11, KKloofm44 +%F##+%6 FL@Y@Y@]^^+33F%-3XX[[bnbt[uu -:,?)ueUE"2& ((Aq!Q77??E5QQ$$Q1a33;;BuMMYx// $,,UE5%OOWWXY[\^_abdeff)11 4?2    *!!!WfW*5r2)r*r+r,r!rHr.rercrrr BoolTensorr^rfrgs@r3rr5s) 1) ) ) ) ) ) V5<335<u| ,.'EL.'U\.'.'.'.'`dU\d9Idelddddddddr2rcdeZdZdZ d dedeedeffd Zdej dej fd Z xZ S) Gemma3nAudioCumulativeGroupNormaApplies Group Normalization cumulatively over the time dimension. This layer normalizes the input by calculating the mean and variance cumulatively over the time dimension (dim 1). The statistics are computed over all feature dimensions (specified by `feature_dims` and `num_channels`) for elements marked as valid by the optional `mask`. If a `mask` is provided (True for valid, False for invalid/padded), invalid time steps do not contribute to the statistics calculation, and their corresponding output values are zeroed out. Scale and bias, if enabled, are applied per-channel (last dimension). This behavior is similar to JAX's `GroupNormalization` with `num_groups=1` and `cumulative=True`. MbP? num_channels feature_dimsr@c Vt||_t||_||_t jtj ||_ ttddt|jzdz|_ dS)Nrr )rGrHr"r;r#r@rIrJr.rKrCrangelenreduction_axes)rNr"r#r@rOs r3rHz(Gemma3nAudioCumulativeGroupNorm.__init__&s (!,//l5:l#;#;<< $E!QT5F1G1G-G!-K$L$LMMr2r9rXc|j|jfz}|jdd|kr"td|jddd||j}t j}||}t j||}t j ||j d}t j |d }t j ||j d} t j | d } t j | d } || z } || z d} t j | |j d}t j |d }|| z }|| z t j||jzz}|j|}dg|dz z|jgz}|||z}||z}||S) zApplies cumulative group norm, optionally using a mask. Args: hidden_states: Input tensor, shape [B, T, *feature_dims, C]. Returns: Normalized tensor with the same shape as x. rNzInput tensor shape suffix z> does not match expected suffix (feature_dims + num_channels) rTr?rRr rrD)r)r#r"r`rrr.rr ones_likesumr'cumsumclamprTrsqrtr@rCr?r)rNr9expected_input_suffix input_dtype calc_dtypex_calc mask_calcsum_values_at_tcum_sum_valueselements_in_group_at_tcum_count_elementssafe_cum_count_elementscum_meansquared_diff_from_meansum_sq_diff_at_tcum_sum_sq_diff cum_variance normalized_xscalescale_view_shape final_outputs r3r^z'Gemma3nAudioCumulativeGroupNorm.forward8s !% 1T5F4H H  qrr "&; ; ;Q]-@-DQQ9NQQ  $) ] !!*-- OF*===  )F0CTRRRo1==="'9$:MW[!\!\!\"\*@aHHH"'+.@c"J"J"J"$;; #)8"3!8!8!;!; 9%;AT^bccc ,'7Q???')@@ )U[9P-Q-QQ  z**3-"3"3"5"5"9:d>O=PP#ejj1A&B&BB $i/ {+++r2)r!) r*r+r,r-rcrr[rHr.rer^rfrgs@r3r r s( NNNsmN NNNNNN$G,U\G,elG,G,G,G,G,G,G,G,r2r c peZdZdZ d dedededeeeeefffd Zdej d ej fd Z xZ S) Gemma3nAudioSSCPConvBlockzA single convolution block for the SubSampleConvProjection. This block consists of a 2D convolution, followed by CumulativeGroupNorm, and a ReLU activation. It handles manual padding for the convolution. rrrrrjidxinput_freq_dimmanual_paddingc"t||_||_|dkrdn|jj|dz }|jj|}|jj|\}}|jj|\} } tj||||f| | fdd|_ ||jdz|jdz} | |z | zdz} t|| f|jj |_ tj |_dS)Nrr )rrF) in_channels out_channels kernel_sizestridepaddingrm)r"r#r@)rGrHrjrGsscp_conv_channel_sizesscp_conv_kernel_sizesscp_conv_stride_sizerIConv2dconvr sscp_conv_group_norm_epsnormReLU activation)rNrjrErFrGrIrJkernel_hkernel_wstride_hstride_w f_in_padded f_out_convrOs r3rHz"Gemma3nAudioSSCPConvBlock.__init__s+  ,!88aa)KCRSG)T {9#> ![>sC(![>sC(I#%h'     %t':1'==@STU@VV !H,9A= 3%$ 4    '))r2audio_encodingsrXctj||jdd|jjj}||}|dddd}| |}|dddd}| |S)Nconstantr)modevaluerrrr ) FrrGrrRrCrrrrTrV)rNr]audio_encodings_paddedaudio_encodings_conv x_for_normx_normedaudio_encodings_normeds r3r^z!Gemma3nAudioSSCPConvBlock.forwards"#8KR\dg!h!h!h!k!k I  "" "  $yy)?@@*11!Q1==HHJJ 99Z((!)!1!1!Q1!=!=!H!H!J!J5666r2)rD) r*r+r,r-r!rcr;rHr.rer^rfrgs@r3rCrCs5A )$)$")$)$ )$ c3S01 )$)$)$)$)$)$V7u|7 77777777r2rCcHeZdZdeffd ZdejdejfdZxZS)#Gemma3nAudioSubSampleConvProjectionrjct||_|j}g}g}t dD]r}|j|\}}|j|\}} d} |dz } d} d} | | | | f}|||| z| z}||z | zdz}|||}std|j||d|_ td|d||d|_ |j d}|d}||z|_ tj|j |jjd|_dS)Nrrr )rErFrjrGrQFrl)rGrHrjinput_feat_sizer%rOrPappendrCconv_0conv_1rNinput_proj_in_featuresrIryrqinput_proj_linear)rNrjcurrent_f_for_block_inputcalculated_block_paddingcalculated_f_out_dimsirWrXrYrZ pad_t_top pad_t_bottom pad_f_left pad_f_rightmanual_padding_tupler[f_out_after_conv final_c_out final_f_outrOs r3rHz,Gemma3nAudioSubSampleConvProjection.__init__s  $*$:!#% "q 9 9A!'!=a!@ Hh!'!=a!@ HhI#a#R#R %=$B!1i#;#C#CAq)V^J^#_#_ ))$<==+o8N7NPTPfgg,t~~o/N/NNNr2 r*r+r,r!rHr.rerr^rfrgs@r3rrsA1AAAAAAOu|OUEUOZ_ZfOOOOOOOOr2rcHeZdZdeffd ZdejdejfdZxZS) Gemma3nAudioConformerFeedForwardrjcHt||_|dt j|jjdt|jj|_ tj |jj|jjdzd|_ tj |jjdz|jjd|_ t|jj|_t j|jj|_dS)NrFrErrl)rGrHrjrLr.rMrr=rqpre_layer_normrIry ffw_layer_1 ffw_layer_2post_layer_normconf_residual_weightpost_layer_scalers r3rHz)Gemma3nAudioConformerFeedForward.__init__-s   0%,t{?\2]2]joppp,T[-DEE9T[%UXY>Y`efff9T[%>(,(8(8(I(I-,,_==(,(8(8(I(I+o8N7NPTPfgg..???T-BBCCr2rrgs@r3rr,s| O1 O O O O O O Du| D D D D D D D D Dr2rcHeZdZdeffd ZdejdejfdZxZS) Gemma3nAudioConformerLightConv1drjc t||_t|jj|jj|_tj|jj|jjdzd|_ tj |jj|jj|jj dd|jjd|_ | dtj|jjd t|jj|jj|_tj|jj|jjd|_|jj dz |_dS) Nr@rFrlr r)rIrJrKrLrMgroupsrmrrE)rGrHrjr=rq rms_norm_epsrrIry linear_startConv1dconf_conv_kernel_sizedepthwise_conv1drLr.rMr conv_norm linear_endcausal_paddingrs r3rHz)Gemma3nAudioConformerLightConv1d.__init__Fs2  ,T[-D$+JbcccIdk&=t{?VYZ?Zafggg "  /0 9;*! ! !  0%,t{?\2]2]joppp' (?T[E]^^^)DK$;T[=T[`aaa"k?!Cr2r]rXcF|}||}||}tjj|d}|ddd}tj||j df}| |}|ddd}tj ||j |j }| |}tj|}||}||z}|S)NrQrrrr )rrr.rIrglurrbrrrr-rrrr)rNr]audio_encodings_residualaudio_encodings_permutedaudio_encodings_permuted_paddedr]s r3r^z(Gemma3nAudioConformerLightConv1d.forward[s#2 --o>>++O<<(-11/r1JJ#2#:#:1a#C#C *+%0H4K^`aJb*c*c'//0OPP)11!Q::+o8N7NPTPfgg..99-,,_==///:: #;; r2rrgs@r3rrEsrD1DDDDDD*u| r2rcVeZdZdeffd ZdejdejdejfdZxZ S)Gemma3nAudioConformerBlockrjct||_t|j|_t |j|_t|j|_t|j|_ | dtj |jj dt|jj|_dS)NrFrE)rGrHrjrffw_layer_startr attentionrlconv1d ffw_layer_endrLr.rMrr=rqrTrs r3rHz#Gemma3nAudioConformerBlock.__init__qs  ? LL7 DD7 DD =dkJJ 0%,t{?\2]2]joppp"4;#:;; r2r]rrXc||}|||}|}||d|jz}||}||}tj||j |j }| |}|S)NrQ) rrrrrrrr.r-rrT)rNr]rvalidity_mask_for_lconvaudio_encodings_for_lconv_inputr]s r3r^z"Gemma3nAudioConformerBlock.forward|s..??...II#1/*9z0Gemma3nAudioEncoder.__init__..s" ^ ^ ^A ' / / ^ ^ ^r2) rGrHrjrisubsample_conv_projectionrI ModuleListr%conf_num_hidden_layers conformerrs `r3rHzGemma3nAudioEncoder.__init__sl     )LV)T)T& ^ ^ ^ ^v?\9]9] ^ ^ ^  r2rrXc||}|jd}d}tt|jjD]}||jj|dz}t j||j|z}t j ||jddz }|j dkr@|j dkr5| d |jdd}nX|j |j krH|jddkr7|jddkr&||jdkr| d}t j |d|}|jD]} | ||}|jjdkr2|dddd|jjf}|dddd|jjf}|| dd}||fS)aEncodes a batch of MELs. Args: audio_mel: a torch.Tensor of shape [batch, num_frames, num_channels, mel_bins]. Returns: audio_encodings: a torch.Tensor of shape `[batch_size, self.config.audio_soft_tokens_per_image, self.config.audio_config.hidden_size]` audio_mel_mask: a torch.BoolTensor of shape [batch, num_frames]. r rr)rtrQNr)rr`r%r&rjrPr.r~rr-rrexpandgatherrconf_reduction_factor masked_fill) rNrrr]t_subtime_stride_productstride_pair_idxindices current_maskblocks r3r^zGemma3nAudioEncoder.forwards88CC %a($S)J%K%KLL Y YO 4;#D_#UVW#X X   ,u^-BCCCFYY+g>+?+BQ+FGGG   " "w|q'8'8''**11.2Fq2I2NNGG  7< / /$Q'1,, a A%%q)))''**G|NAw?? ^ C CE#eO\BBOO ; ,q 0 0-aaa1U1UDK4U1U.UVO'+O+Odk.O+O(OPL)55l6L6LR6P6PRUVV ,,r2)r*r+r,r-r!r0main_input_namerHr.rerr;r^rfrgs@r3rrs !O 1      5-5-7<7G5- u|U-- .5-5-5-5-5-5-5-5-r2rc PeZdZdZd dedededeffd Zdejffd Z xZ S) Gemma3nTextScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. rDnum_embeddings embedding_dim padding_idx embed_scalect||||dtj|ddS)NrFrE)rGrHrLr.rM)rNrrrrrOs r3rHz'Gemma3nTextScaledWordEmbedding.__init__sK  DDD ]EL,E,ERWXXXXXr2 input_idsct||j|jjzSrZ)rGr^rrrCr)rNrrOs r3r^z&Gemma3nTextScaledWordEmbedding.forwards4wwy))D,<,?,? @Q,R,RRRr2)rD) r*r+r,r-rcr[rHr.rer^rfrgs@r3rrsYYsY3YSY_dYYYYYYSSSSSSSSSSSr2rcLeZdZdZdeffd ZdejdejfdZxZ S)Gemma3nTextLaurelBlockz Learned Augmented Residual Layerrjcjt||_tj|jj|jjd|_tj|jj|jjd|_t|jj|jj |_ dS)NFrlr) rGrHrjrIryrq laurel_rank linear_left linear_rightr=rpost_laurel_normrs r3rHzGemma3nTextLaurelBlock.__init__s  9T[%U\abbbIdk&=t{?V]bccc .t{/FDKLd e e er2r9rXc||}||}||}||zSrZ)rrr)rNr9laurel_hidden_statesnormed_laurel_hidden_statess r3r^zGemma3nTextLaurelBlock.forwardsL-1-=-=m-L-L-1->->?S-T-T&*&;&;>> Yv2teLLL n!<<}!!&8"3444r2)r) r*r+r,r#rcrHr.rer^rrfrgs@r3rrs Q Q0 QS Q Q Q Q Q QU\el5U\5el55555555r2rceZdZdZdeffd ZdejdejfdZdejdejfdZ d ejd ejdejfd Z d ejdejfd Z d ejdejfdZ xZ S)Gemma3nTextAltUpaAlternating Updates (AltUp) The AltUp module wraps transformer layers. The `predict` step modifies the input to the transformer layer, and the `correct` step propagates the output of the transformer layer to the sparsely updated dimensions. See more in the research paper: https://proceedings.neurips.cc/paper_files/paper/2023/file/f2059277ac6ce66e7e5543001afa8bb5-Paper-Conference.pdf rjct||_tjt j|jj|_tj |jj |jj d|_ tj |jj |jj dzd|_ tj |jj|jj d|_ t|jj|jj|_|dt j|jjdzddS)NFrlrrrouter_input_scalegrE)rGrHrjrIrJr.rrqcorrect_output_scaleryaltup_num_inputscorrection_coefsprediction_coefsmodality_routerr=r router_normrLrMrs r3rHzGemma3nTextAltUp.__init__&s   $&LT[=T1U1U$V$V! " $+*F Hdkp q q q " $+*F HdfgHgns t t t!y)@$+B^ejkkk)$+*At{G_``` 15< @WY]@]3^3^kpqqqqqr2rVrXc|||jz}||}tj||SrZ)rr rr.rr[r\)rNrV router_inputsrouteds r3compute_router_modalitiesz*Gemma3nTextAltUp.compute_router_modalities0sV((++d.EE %%m44z&,,..))11!444r2r9cz|||jj}|jrF|jj:|jjj|jj |jj||j g|j dd|jj |jj R dddd}tj| dddd|}| dddd}||z }||S)aPredicts the output of a layer using a trainable map. Args: hidden_states: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` containing the predictions. NrQrr rr)rrjaltup_active_idxtrainingaltup_coef_cliprrCdataclamp_rr`r rr.rrr\)rNr9 modalities all_coefs predictionss r3predictzGemma3nTextAltUp.predict5sA33M$+B^4_`` = pT[8D  ! ( - 4 4dk6Q5QSWS^Sn o o o D ! !* - -  i &ss+ i-1[-I iKO;Kg i i i WQ1a  l=#8#8Aq!#D#DiPP !))!Q155 }$ %%''// >>>r2r activatedc,||}|||jjz }||jjddd}|jj:|jjj |jj |jj||dz}| ddd d}tj ||}||z }||S)a_Corrects the predictions relative to the Args: predictions: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. activated: A 3D tensor of shape `[batch_size, num_tokens, hidden_size]` containing the activated inputs. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` correcting the original predictions relative to the activated input embeddings. r NrDrrrQ)rrjrrepeatr rrrCrrrrr.mulrr\)rNrr r innovationr correcteds r3correctzGemma3nTextAltUp.correctQs33I>> T[-I!JJ &&t{'CQ1MM ; & 2  ! ( - 4 4dk6Q5QSWS^Sn o o o #'"7"7 "C"Cc"I %%aA..88<< Ij)44 [ ##%%--i888r2r%cl||j|jz|S)a  This is only defined as the `forward` so that accelerate hooks can move correctly `correct_output_scale` (which is a nn.Parameter, not a Module) between devices when offloading. It is otherwise only used in `scale_corrected_output` )r\r rNr%s r3r^zGemma3nTextAltUp.forwardns2 !!$";< * * 'z&:Mt/T/T '#044[&BUBYBYZ`BaBabbDNN&DN"("@$*$B! /?+/+<+>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk ^ U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;sBE++E/2E/rZ) r*r+r,r.rer0r#rHno_gradrr^rfrgs@r3r,r,{sl//0//////"U]__ < <_ < < < <