§ `ƒiaãó<—ddlmZmZmZddlZddlmZddlmZddl m Z mZddlm Z ddlmZmZddlmZdd lmZdd lmZmZddlmZmZddlmZmZdd lmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*e$j+e,¦«Z-Gd„dej.¦«Z/Gd„dej.¦«Z0d„Z1d4d„Z2dej3de4dej3fd„Z5 d5dej.dej3d ej3d!ej3d"eej3d#e6d$ee6d%ee6de7ej3ej3ffd&„Z8Gd'„d(ej.¦«Z9Gd)„d*e¦«Z:Gd+„d,ej.¦«Z;e"Gd-„d.e¦«¦«Zgd3¢Z?dS)6é)ÚCallableÚOptionalÚUnionNé)ÚACT2FN)ÚCacheÚDynamicCache)ÚGenerationMixin)Úcreate_causal_maskÚ!create_sliding_window_causal_mask)ÚFlashAttentionKwargs)ÚGradientCheckpointingLayer)ÚBaseModelOutputWithPastÚCausalLMOutputWithPast)ÚROPE_INIT_FUNCTIONSÚdynamic_rope_update)ÚALL_ATTENTION_FUNCTIONSÚPreTrainedModel)ÚUnpack)ÚTransformersKwargsÚauto_docstringÚcan_return_tupleÚlogging)Údeprecate_kwarg)Úcheck_model_inputsé)ÚVaultGemmaConfigcó<‡—eZdZddedefˆfd„ Zd„Zd„Zd„ZˆxZ S) ÚVaultGemmaRMSNormçíµ ÷Æ°>ÚdimÚepscó¬•—t¦« ¦«||_tjtj|¦«¦«|_dS©N)ÚsuperÚ__init__r"ÚnnÚ ParameterÚtorchÚzerosÚweight)Úselfr!r"Ú __class__s €ú†/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/vaultgemma/modeling_vaultgemma.pyr&zVaultGemmaRMSNorm.__init__/s?ø€Ý ‰Œ×ÒÑÔÐØˆŒÝ”l¥5¤;¨sÑ#3Ô#3Ñ4Ô4ˆŒˆˆócó—|tj| d¦« dd¬¦«|jz¦«zS)NééÿÿÿÿT)Úkeepdim)r)ÚrsqrtÚpowÚmeanr")r,Úxs r.Ú_normzVaultGemmaRMSNorm._norm4s8€Ø•5”;˜qŸušu Q™xœxŸ}š}¨R¸˜}Ñ>Ô>ÀÄÑIÑJÔJÑJÐJr/có¸—| | ¦«¦«}|d|j ¦«zz}| |¦«S)Ngð?)r8Úfloatr+Útype_as)r,r7Úoutputs r.ÚforwardzVaultGemmaRMSNorm.forward7sL€Ø—’˜AŸGšG™IœIÑ&Ô&ˆð˜3 ¤×!2Ò!2Ñ!4Ô!4Ñ4Ñ5ˆØ~Š~˜aÑ Ô Ð r/cóH—t|jj¦«›d|j›S)Nz, eps=)Útupler+Úshaper")r,s r.Ú extra_reprzVaultGemmaRMSNorm.extra_repr>s%€Ý˜œÔ)Ñ*Ô*Ð<Ð<°$´(Ð<Ð<Ðœ>¨CÑ/Ñ0€GØ3‰w; q™>œ>¨CÑ/Ñ0€GØGÐÐr/Ú hidden_statesÚn_repÚreturncó¸—|j\}}}}|dkr|S|dd…dd…ddd…dd…f |||||¦«}| |||z||¦«S)zÔ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r@ÚexpandÚreshape)rjrkÚbatchÚnum_key_value_headsÚslenÚhead_dims r.Ú repeat_kvrtts„€ð 2?Ô1DÑ.€EÐ hØ‚z€zØÐØ! ! ! ! Q Q Q¨¨a¨a¨a°°°Ð"2Ô3×:Ò:¸5ÐBUÐW\Ð^bÐdlÑmÔm€MØ× Ò Ð(;¸eÑ(CÀTÈ8ÑTÔTÐTr/çÚmoduleÚqueryÚkeyÚvalueÚattention_maskÚdropoutÚscalingÚsoftcapcóª—|€ |jdz}t||j¦«} t||j¦«} tj|| dd¦«¦«|z}|||z}tj|¦«}||z}|$|dd…dd…dd…d| jd…f}||z}tj |dtj¬¦« |j ¦«}tj |||j¬¦«}tj|| ¦«} | dd¦« ¦«} | |fS) Nçà¿r1réþÿÿÿr2)r!Údtype)ÚpÚtrainingr)rsrtÚnum_key_value_groupsr)ÚmatmulÚ transposeÚtanhr@r'Ú functionalÚsoftmaxÚfloat32Útorr{rƒÚ contiguous)rvrwrxryrzr{r|r}ÚkwargsÚ key_statesÚvalue_statesÚattn_weightsÚcausal_maskÚattn_outputs r.Úeager_attention_forwardr“€sR€ð€Ø”/ 4Ñ'ˆå˜3 Ô ;Ñ<Ô<€JÝ˜U FÔ$?Ñ@Ô@€Lå”< z×';Ò';¸A¸qÑ'AÔ'AÑBÔBÀWÑL€LàÐØ# gÑ-ˆÝ”z ,Ñ/Ô/ˆØ# gÑ-ˆØÐ!Ø$ Q Q Q¨¨¨¨1¨1¨1Ð.D° Ô0@ÀÔ0DÐ.DÐ%DÔEˆØ# kÑ1ˆõ”=×(Ò(¨¸2ÅUÄ]Ð(ÑSÔS×VÒVÐW\ÔWbÑcÔc€LÝ”=×(Ò(¨¸È6Ì?Ð(Ñ[Ô[€LÝ”,˜|¨\Ñ:Ô:€KØ×'Ò'¨¨1Ñ-Ô-×8Ò8Ñ:Ô:€KØ˜Ð$Ð$r/cóD‡—eZdZdZdedefˆfd„Zeddd¬¦« dd ej de ej ej fdeej deed eej deede ej eej ee ej ffd„¦«ZˆxZS)ÚVaultGemmaAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrNÚ layer_idxcó•—t¦« ¦«||_||_t |d|j|jz¦«|_|j|jz|_ |j dz|_|jj|_d|_ tj|j|j|jz|j¬¦«|_tj|j|j|jz|j¬¦«|_tj|j|j|jz|j¬¦«|_tj|j|jz|j|j¬¦«|_|jj|_|j|dkr|jnd|_dS)NrsrTrLÚsliding_attention)r%r&rNr–ÚgetattrrOÚnum_attention_headsrsrqr„Úquery_pre_attn_scalarr|Úattention_dropoutÚ is_causalr'rQÚattention_biasÚq_projÚk_projÚv_projÚo_projÚattn_logit_softcappingÚlayer_typesÚsliding_window©r,rNr–r-s €r.r&zVaultGemmaAttention.__init__¦szø€Ý ‰Œ×ÒÑÔÐØˆŒØ"ˆŒÝ ¨ °FÔ4FÈ&ÔJdÑ4dÑeÔeˆŒ Ø$*Ô$>À&ÔB\Ñ$\ˆÔ!ØÔ3°TÑ9ˆŒØ!%¤Ô!>ˆÔØˆŒå”iØÔ Ô :¸T¼]Ñ JÐQWÔQfð ñ ô ˆŒõ”iØÔ Ô :¸T¼]Ñ JÐQWÔQfð ñ ô ˆŒõ”iØÔ Ô :¸T¼]Ñ JÐQWÔQfð ñ ô ˆŒõ”iØÔ&¨¬Ñ6¸Ô8JÐQWÔQfð ñ ô ˆŒð'+¤kÔ&HˆÔ#Ø7=Ô7IÈ)Ô7TÐXkÒ7kÐ7k˜fÔ3Ð3ÐquˆÔÐÐr/Úpast_key_valueÚpast_key_valuesú4.58©Únew_nameÚversionNrjÚposition_embeddingsrzÚcache_positionrrlcó\—|jdd…}g|¢d‘|j‘R}| |¦« |¦« dd¦«} | |¦« |¦« dd¦«} | |¦« |¦« dd¦«}|\}} t| | || ¦«\} } |&| ||dœ}| | ||j |¦«\} }t}|jjdkrt|jj}||| | ||f|jr|jnd|j|j|jdœ|¤Ž\}}|jg|¢d‘RŽ ¦«}| |¦«}||fS)Nr2rr1)rdrcr®Úeagerru)r{r|r¥r})r@rsrŸÚviewr†r r¡riÚupdater–r“rNÚ_attn_implementationrrƒrœr|r¥r£rorŒr¢)r,rjrrzr¨r®rÚinput_shapeÚhidden_shapeÚquery_statesrŽrrcrdÚcache_kwargsÚattention_interfacer’rs r.r=zVaultGemmaAttention.forward¿så€ð$Ô)¨#¨2¨#Ô.ˆØ8˜Ð8 bÐ8¨$¬-Ð8Ð8ˆà—{’{ =Ñ1Ô1×6Ò6°|ÑDÔD×NÒNÈqÐRSÑTÔTˆØ—[’[ Ñ/Ô/×4Ò4°\ÑBÔB×LÒLÈQÐPQÑRÔRˆ Ø—{’{ =Ñ1Ô1×6Ò6°|ÑDÔD×NÒNÈqÐRSÑTÔTˆà&‰ˆˆSÝ#7¸ÀjÐRUÐWZÑ#[Ô#[Ñ ˆjàÐ&à#&¨sÀnÐUÐUˆLØ'6×'=Ò'=¸jÈ,ÐX\ÔXfÐhtÑ'uÔ'uÑ$ˆJ˜å(?ÐØŒ;Ô+¨wÒ6Ð6Ý"9¸$¼+Ô:ZÔ"[Ðà$7Ð$7ØØØØØð% ð/3¬mÐDDÔ*Ð*ÀØ”LØÔ.ØÔ/ð% ð% ðð% ð% Ñ!ˆ\ð*kÔ)Ð;¨;Ð;¸Ð;Ð;Ð;×FÒFÑHÔHˆØ—k’k +Ñ.Ô.ˆØ˜LÐ(Ð(r/)NN)rBrCrDÚ__doc__rrEr&rr)ÚTensorr?rrÚ LongTensorrr r=rFrGs@r.r•r•£s*ø€€€€€ØGÐGðvÐ/ðv¸Cðvðvðvðvðvðvð2€_Ð%Ð0AÈ6ÐRÑRÔRð,0Ø59ð +)ð+)à”|ð+)ð# 5¤<°´Ð#=Ô>ð+)ð! ¤Ô.ð +)ð " %œð+)ð! Ô!1Ô2ð +)ðÐ-Ô.ð+)ð ˆuŒ|˜X e¤lÔ3°X¸eÀEÄLÔ>QÔ5RÐRÔ Sð+)ð+)ð+)ñSÔRð+)ð+)ð+)ð+)ð+)r/r•cóh‡—eZdZdedefˆfd„Zeddd¬¦« dd ejde ejejfde ejd e ejde ede e de e de ejde eje e ejejfffd„¦«ZˆxZS)ÚVaultGemmaDecoderLayerrNr–cóf•—t¦« ¦«|j|_||_|j||_t ||¬¦«|_t|¦«|_ t|j|j¬¦«|_t|j|j¬¦«|_ dS)N)rNr–©r")r%r&rOrNr¤Úattention_typer•Ú self_attnrIÚmlprÚrms_norm_epsÚinput_layernormÚpre_feedforward_layernormr¦s €r.r&zVaultGemmaDecoderLayer.__init__ïs›ø€Ý ‰Œ×ÒÑÔÐØ!Ô-ˆÔØˆŒØ$Ô0°Ô;ˆÔÝ,°FÀiÐPÑPÔPˆŒÝ Ñ(Ô(ˆŒÝ0°Ô1CÈÔI\Ð]Ñ]Ô]ˆÔå):¸6Ô;MÐSYÔSfÐ)gÑ)gÔ)gˆÔ&Ð&Ð&r/r§r¨r©rªNFrjrrzreÚoutput_attentionsÚ use_cacher®rlc óè—|} | |¦«}|jd||||||||dœ| ¤Ž\}}| |z}|} | |¦«}| |¦«}| |z}|f}|r||fz }|S)N)rjrrzrer¨rÆrÇr®©)rÄrÁrÅrÂ) r,rjrrzrer¨rÆrÇr®rÚresidualÚself_attn_weightsÚoutputss r.r=zVaultGemmaDecoderLayer.forwardúsÃ€ð!ˆØ×,Ò,¨]Ñ;Ô;ˆ à+9¨4¬>ð , Ø'Ø 3Ø)Ø%Ø+Ø/ØØ)ð , ð , ðð , ð , Ñ(ˆ Ð(ð! =Ñ0ˆ à ˆØ×6Ò6°}ÑEÔEˆ ØŸš Ñ/Ô/ˆ Ø =Ñ0ˆ à Ð"ˆØð ,ØÐ)Ð+Ñ+ˆGàˆr/)NNNFFN)rBrCrDrrEr&rr)rºr?rr»rÚboolÚFloatTensorr=rFrGs@r.r½r½îsOø€€€€€ð hÐ/ð h¸Cð hð hð hð hð hð hð€_Ð%Ð0AÈ6ÐRÑRÔRð 26Ø37Ø+/Ø,1Ø$)Ø59ð%ð%à”|ð%ð# 5¤<°´Ð#=Ô>ð%ð! ¤Ô.ð %ð ˜uÔ/Ô0ð%ð" %œð %ð$ Dœ>ð%ð˜D”>ð%ð! Ô!1Ô2ð%ð ˆuÔ (¨5°Ô1BÀEÔDUÐ1UÔ+VÔ"WÐWÔ Xð%ð%ð%ñSÔRð%ð%ð%ð%ð%r/r½có|‡—eZdZUejed<ddefˆfd„ Zej¦«e d„¦«¦«Z ˆxZS)ÚVaultGemmaRotaryEmbeddingÚinv_freqNrNcó•—t¦« ¦«t|d¦«rSt|jt ¦«r9|j d|j d¦«¦«|_nd|_|j|_ |j|_ ||_t|j|_ | |j|¦«\}|_| d|d¬¦«|j|_dS)NÚrope_scalingÚ rope_typeÚtypeÚdefaultrÑF)Ú persistent)r%r&ÚhasattrÚ isinstancerÓÚdictÚgetrÔÚmax_position_embeddingsÚmax_seq_len_cachedÚoriginal_max_seq_lenrNrÚrope_init_fnÚattention_scalingÚregister_bufferrÑÚoriginal_inv_freq)r,rNÚdevicerÑr-s €r.r&z"VaultGemmaRotaryEmbedding.__init__&sêø€Ý ‰Œ×ÒÑÔÐå6˜>Ñ*Ô*ð 'z¸&Ô:MÍtÑ/TÔ/Tð 'Ø#Ô0×4Ò4°[À&ÔBU×BYÒBYÐZ`ÑBaÔBaÑbÔbˆDŒNˆNà&ˆDŒNØ"(Ô"@ˆÔØ$*Ô$BˆÔ!àˆŒÝ/°´Ô?ˆÔà+/×+<Ò+<¸T¼[È&Ñ+QÔ+QÑ(ˆ$Ô(Ø×Ò˜Z¨¸eÐÑDÔDÐDØ!%¤ˆÔÐÐr/cóX—|jddd…df ¦« |jddd¦« |j¦«}|dd…ddd…f ¦«}t |jjt¦«r|jjdkr|jjnd}tj |d¬¦«5| ¦«| ¦«z dd¦«}tj||fd¬ ¦«}| ¦«|jz}| ¦«|jz} ddd¦«n#1swxYwY| |j¬ ¦«| |j¬ ¦«fS)Nrr2rÚmpsÚcpuF)Údevice_typeÚenabledr1rZ©r)rÑr:rnr@r‹rãrÙrÕÚstrr)Úautocastr†r[rcràrdr) r,r7reÚinv_freq_expandedÚposition_ids_expandedrçÚfreqsÚembrcrds r.r=z!VaultGemmaRotaryEmbedding.forward7s¹€ð!œM¨$°°°°4¨-Ô8×>Ò>Ñ@Ô@×GÒGÈÔHZÐ[\ÔH]Ð_aÐcdÑeÔe×hÒhÐijÔiqÑrÔrÐØ ,¨Q¨Q¨Q°°a°a°a¨ZÔ 8× >Ò >Ñ @Ô @Ðå'1°!´(´-ÅÑ'EÔ'EÐkÈ!Ì(Ì-Ð[`ÒJ`ÐJ`a”h”mmÐfkˆÝ Œ^¨¸UÐ CÑ CÔ Cð 5ð 5Ø&×,Ò,Ñ.Ô.Ð1F×1LÒ1LÑ1NÔ1NÑN×YÒYÐZ[Ð]^Ñ_Ô_ˆEÝ”)˜U E˜N°Ð3Ñ3Ô3ˆCØ—'’'‘)”)˜dÔ4Ñ4ˆCØ—'’'‘)”)˜dÔ4Ñ4ˆCð 5ð 5ð 5ñ 5ô 5ð 5ð 5ð 5ð 5ð 5ð 5øøøð 5ð 5ð 5ð 5ðvŠv˜AœGˆvÑ$Ô$ c§f¢f°1´7 fÑ&;Ô&;Ð;Ð;sÃBE+Å+E/Å2E/r$)rBrCrDr)rºÚ__annotations__rr&Úno_gradrr=rFrGs@r.rÐrÐ#sˆø€€€€€€ØŒlÐÐÑð/ð/Ð/ð/ð/ð/ð/ð/ð/ð"€U„]_„_Øð<ð<ñÔñ„_ð<ð<ð<ð<ðz,VaultGemmaModel.__init__..js$ø€ÐhÐhÐh¸9Õ # F¨IÑ 6Ô 6ÐhÐhÐhr/r¿©rNF)r%r&Úpad_token_idÚpadding_idxÚ vocab_sizer'Ú EmbeddingrOÚembed_tokensÚ ModuleListÚrangeÚnum_hidden_layersÚlayersrrÃÚnormrÐÚ rotary_embÚgradient_checkpointingÚ post_initrWs `€r.r&zVaultGemmaModel.__init__csÔøø€Ý ‰Œ×Ò˜Ñ Ô Ð Ø!Ô.ˆÔØ Ô+ˆŒåœL¨Ô):¸FÔQÐRÑRÔRˆŒ Ý3¸6ÐBÑBÔBˆŒØ&+ˆÔ#ð ŠÑÔÐÐÐr/NÚ input_idsrzrer¨Ú inputs_embedsrÇrÆÚoutput_hidden_statesr®rrlc ó8—||n|jj}||n|jj}||n|jj}|du|duzrt d¦«‚|jr%|jr|rt d¦«d}|€| |¦«}|r|€|jst|j¬¦«}| €B|| ¦«nd}tj |||jdz|j¬¦«} |€| d¦«}t#|x}t$¦«s'|j||| ||dœ} t'di| ¤Žt)di| ¤Žd œ}|}| ||¦«}tj|jjd z|j¬¦«}||z}|rdnd}|rdnd}|jd|jj…D]<}|r||fz }||f|||j||||| d œ| ¤Ž}|d}|r||dfz }Œ=| |¦«}|r||fz }t;||||¬¦«S)Nz:You must specify exactly one of input_ids or inputs_embedszX`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`.Frrr)rã)rNÚinput_embedsrzr®r¨re)Úfull_attentionr˜gà?rérÉ)rrzrer¨rÆrÇr®)Úlast_hidden_stater¨rjrõ)rNrÆrrÇÚ ValueErrorrrƒÚloggerÚwarning_oncerr Úget_seq_lengthr)Úaranger@rãr`rÙrÚrrrÚtensorrOrrrrÀrr)r,rrzrer¨rrÇrÆrr®rÚpast_seen_tokensÚcausal_mask_mappingÚmask_kwargsrjrÚ normalizerÚall_hidden_statesÚall_self_attnsÚ decoder_layerÚ layer_outputss r.r=zVaultGemmaModel.forwardss€ð2CÐ1NÐ-Ð-ÐTXÔT_ÔTqÐà$8Ð$DÐ Ð È$Ì+ÔJjð ð"+Ð!6II¸D¼KÔÑ>Ô>ˆOàÐ!ØCRÐC^˜×=Ò=Ñ?Ô?Ð?ÐdeÐÝ"œ\Ø Ð"2°]Ô5HÈÔ5KÑ"KÐTaÔThðñôˆNðÐØ)×3Ò3°AÑ6Ô6ˆLõ°Ð?Ð-ÅÑFÔFð ðœ+Ø -Ø"0Ø"0Ø#2Ø ,ð ðˆKõ#5Ð"CÐ"C°{Ð"CÐ"CÝ%FÐ%UÐ%UÈÐ%UÐ%Uð#ð#Ðð&ˆ ð#Ÿošo¨m¸\ÑJÔJÐõ ”\ $¤+Ô"9¸3Ñ">ÀmÔFYÐZÑZÔZˆ Ø%¨ Ñ2ˆ ð#7Ð@˜B˜B¸DÐØ0Ð:˜˜°dˆà!œ[Ð)H¨4¬;Ô+HÐ)HÔIð 6ð 6ˆMØ#ð 6Ø! mÐ%5Ñ5Ð!à)˜MØð à$7Ø2°=Ô3OÔPØ)Ø /Ø"3Ø#Ø-ð ð ðð ð ˆMð*¨!Ô,ˆMà ð 6Ø =°Ô#3Ð"5Ñ5øàŸ š -Ñ0Ô0ˆ àð 2Ø -Ð!1Ñ1Ðå&Ø+Ø+Ø+Ø%ð ñ ô ð r/) NNNNNNNNN)rBrCrDrr&rrrr)r»rºrrÎrÍrrrr=rFrGs@r.rrasOø€€€€€ðÐ/ððððððð Øð15Ø15Ø37Ø+/Ø59Ø$(Ø,0Ø/3Ø59ðk ðk à˜EÔ,Ô-ðk ð! ¤Ô.ðk ð˜uÔ/Ô0ð k ð " %œðk ð Ô 1Ô2ð k ð˜D”>ðk ð$ Dœ>ðk ð' tœnðk ð! Ô!1Ô2ðk ðÐ+Ô,ðk ð !ðk ðk ðk ñ„^ñÔðk ðk ðk ðk ðk r/rcóz‡—eZdZdgZddiZddgdgfiZˆfd„Zee dd e e jd e e jde e jde e d e e jde e jde ede ede ede e jdeee jfdefd„¦«¦«ZˆxZS)ÚVaultGemmaForCausalLMzlm_head.weightÚlm_headÚcolwise_reprjÚlogitscóþ•—t¦« |¦«t|¦«|_|j|_tj|j|jd¬¦«|_| ¦«dSrK) r%r&rrôrr'rQrOr0rrWs €r.r&zVaultGemmaForCausalLM.__init__ésjø€Ý ‰Œ×Ò˜Ñ Ô Ð Ý$ VÑ,Ô,ˆŒ Ø Ô+ˆŒÝ”y Ô!3°VÔ5FÈUÐSÑSÔSˆŒð ŠÑÔÐÐÐr/Nrrrzrer¨rÚlabelsrÇrÆrr®Úlogits_to_keeprlc ó—||n|jj}| | n|jj} |jd|||||||| | dœ |¤Ž} | j}t|t¦«rt|d¦«n|}| |dd…|dd…f¦«}|jj 2||jj z}tj|¦«}||jj z}d}||j|||j fi|¤Ž}t||| j| j| j¬¦«S)aŠ Example: ```python >>> from transformers import AutoTokenizer, VaultGemmaForCausalLM >>> model = VaultGemmaForCausalLM.from_pretrained("google/gemma-2-9b") >>> tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b") >>> prompt = "What is your favorite condiment?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "What is your favorite condiment?" ```N) rrzrer¨rrÇrÆrr®)Úlossr2r¨rjrõrÉ)rNrÆrrôrrÙrEÚslicer0Úfinal_logit_softcappingr)r‡Ú loss_functionrrr¨rjrõ)r,rrzrer¨rr4rÇrÆrr®r5rrÌrjÚ slice_indicesr2r7s r.r=zVaultGemmaForCausalLM.forwardòsb€ðF2CÐ1NÐ-Ð-ÐTXÔT_ÔTqÐà$8Ð$DÐ Ð È$Ì+ÔJjð ð,6¨4¬:ð, ØØ)Ø%Ø+Ø'ØØ/Ø!5Ø)ð, ð, ðð, ð, ˆð Ô1ˆ å8BÀ>ÕSVÑ8WÔ8WÐk˜~˜o¨tÑ4Ô4Ð4Ð]kˆ Ø—’˜m¨A¨A¨A¨}¸a¸a¸aÐ,?Ô@ÑAÔAˆØŒ;Ô.Ð:Ø˜dœkÔAÑAˆFÝ”Z Ñ'Ô'ˆFØ˜dœkÔAÑAˆFàˆØÐØ%4Ô% f¨f°d´oÐPÐPÈÐPÐPˆDå%ØØØ#Ô3Ø!Ô/ØÔ)ð ñ ô ð r/)NNNNNNNNNNr)rBrCrDÚ_tied_weights_keysÚ_tp_planÚ_pp_planr&rrrr)r»rºrrÎrÍrrErr=rFrGs@r.r/r/ãsˆø€€€€€à*Ð+ÐØ˜=Ð)€HØ˜_Ð-°¨zÐ:Ð;€HððððððØð15Ø15Ø37Ø+/Ø59Ø-1Ø$(Ø,0Ø/3Ø59Ø34ðF ðF à˜EÔ,Ô-ðF ð! ¤Ô.ðF ð˜uÔ/Ô0ð F ð " %œðF ð Ô 1Ô2ð F ð˜Ô)Ô*ðF ð˜D”>ðF ð$ Dœ>ðF ð' tœnðF ð! Ô!1Ô2ðF ð˜c 5¤<Ð/Ô0ðF ð ðF ðF ðF ñ„^ñÔðF ðF ðF ðF ðF r/r/)r/rró)Nr)ruNN)@Útypingrrrr)Útorch.nnr'ÚactivationsrÚcache_utilsrr Ú generationr Ú masking_utilsrrÚmodeling_flash_attention_utilsr Úmodeling_layersrÚmodeling_outputsrrÚmodeling_rope_utilsrrÚmodeling_utilsrrÚprocessing_utilsrÚutilsrrrrÚutils.deprecationrÚ utils.genericrÚconfiguration_vaultgemmarÚ get_loggerrBr!ÚModulerrIr^rirºrErtr:r?r“r•r½rÐrórr/Ú__all__rÉr/r.úrRs'ðð,-Ð,Ð,Ð,Ð,Ð,Ð,Ð,Ð,Ð,à€€€ØÐÐÐÐÐà!Ð!Ð!Ð!Ð!Ð!Ø.Ð.Ð.Ð.Ð.Ð.Ð.Ð.Ø)Ð)Ð)Ð)Ð)Ð)ØRÐRÐRÐRÐRÐRÐRÐRØBÐBÐBÐBÐBÐBØ9Ð9Ð9Ð9Ð9Ð9ØOÐOÐOÐOÐOÐOÐOÐOØKÐKÐKÐKÐKÐKÐKÐKØFÐFÐFÐFÐFÐFÐFÐFØ&Ð&Ð&Ð&Ð&Ð&ØRÐRÐRÐRÐRÐRÐRÐRÐRÐRÐRÐRØ0Ð0Ð0Ð0Ð0Ð0Ø/Ð/Ð/Ð/Ð/Ð/Ø6Ð6Ð6Ð6Ð6Ð6ð ˆÔ ˜HÑ %Ô %€ð=ð=ð=ð=ð=˜œ ñ=ô=ð=ð( ð ð ð ð B”Iñ ô ð ð (ð(ð(ððððð6 U˜Uœ\ð U°#ð U¸%¼,ð Uð Uð Uð Uð$Ø#Ø#ð %ð %ØŒIð %àŒ<ð %ð Œð %ðŒ<ð %ð ˜Uœ\Ô*ð %ðð %ðeŒ_ð %ðeŒ_ð %ðˆ5Œ<˜œÐ%Ô&ð %ð %ð %ð %ðFH)ðH)ðH)ðH)ðH)˜"œ)ñH)ôH)ðH)ðV2ð2ð2ð2ð2Ð7ñ2ô2ð2ðj!<ð!<ð!<ð!<ð!< ¤ ñ!<ô!<ð!<ðHð'ð'ð'ð'ð' ñ'ô'ñ„ð'ð2ð~ ð~ ð~ ð~ ð~ Ð/ñ~ ô~ ñ„ð~ ðBðV ðV ðV ðV ðV Ð5°ñV ôV ñ„ðV ðrTÐ SÐ S€€€r/