`ilNdZddlZddlZddlmZddlmZmZmZddl Z ddl Z ddl m Zddl mZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZddlm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1m2Z2m3Z3ddl4m5Z5dZ6dZ7e2rddl8m9Z9e3j:e;Z dnde?e@e@fdeAde@d ee jBd!e@d"e jCf d#ZD dod$e?d%e@d&ee jCfd'ZEGd(d)eZFGd*d+eZGGd,d-eZHGd.d/ejIZJGd0d1ejIZKGd2d3ejIZLGd4d5eLZMGd6d7ejIZN dpd9ejId:e jOd;e jOdeAd?ee jOfd@ZPGdAdBejIZQGdCdDejIZRGdEdFeZSGdGdHeZTGdIdJejIZUGdKdLejIZVGdMdNejIZWGdOdPejIZXGdQdRejIZYGdSdTejIZZe.GdUdVe)Z[e.GdWdXe[Z\e.dYGdZd[e[Z]e.Gd\d]e[Z^e.d^Gd_d`e[Z_e.daGdbdce[Z`e.Gdddee[ZaGdfdgejIZbGdhdiejIZce.djGdkdle[ZdgdmZedS)qzPyTorch Wav2Vec2 model.N) dataclass)CallableOptionalUnion) load_file)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputMaskedLMOutputSequenceClassifierOutputTokenClassifierOutputWav2Vec2BaseModelOutput XVectorOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputauto_docstring cached_filecheck_torch_load_is_safeis_peft_availableis_torch_flex_attn_availablelogging)Wav2Vec2Configzadapter.{}.binzadapter.{}.safetensors)make_flex_block_causal_maskza Output type of [`Wav2Vec2ForPreTraining`], with potential hidden states and attentions. ) custom_introcLeZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dZeejed <dZeejed <dS) Wav2Vec2ForPreTrainingOutputa loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. contrastive_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The contrastive loss (L_m) as stated in the [official paper](https://huggingface.co/papers/2006.11477). diversity_loss (*optional*, returned when `sample_negative_indices` are passed, `torch.FloatTensor` of shape `(1,)`): The diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentionscontrastive_lossdiversity_loss)__name__ __module__ __qualname____doc__r*rtorch FloatTensor__annotations__r+r,r-r.tupler/r0r1/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/wav2vec2/modeling_wav2vec2.pyr)r)Is$)-D(5$ %,,,48hu01888>B): ;BBB9=8E$56===8rAsequence_lengths r<compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spans~i,6DwNOOoy99 [ (? : :-z)_compute_mask_indices..s 9 9 9!o 9 9 9r;dtyperF)replace) ValueErrornprandomranditemdetachsumtolistrangezerosboolchoicearangelen concatenateonesint32appendarray broadcast_toreshaperGput_along_axis)r=r>r?r@rA batch_sizerL input_lengths spec_aug_maskspec_aug_mask_idxsmax_num_masked_spanrHrIspec_aug_mask_idxdummy_mask_idxoffsetsrJrKs `` ` @@r<_compute_mask_indicesrslsP0#(JQABBB_$$ :^i : :'6 : : :   innQ$$&&G         $  % ##B''..000 9 9 9 9uZ'8'8 9 9 9Hj/:$GGGM11/BBa%55 11,??I,, IlkAo6 7 7RW-    ! !Q & &-q0NN.q1NN (;o(MUWU] ^ ^ ^ao o p   !!"34444"455111aaa:&5H+(V,33J@SVa@abbi $$T4]3Gog 4G'UVV^^'+5G,g5/A"555GVYZGZ-!0CCDm%7B??? r;features_shape num_negativesmask_time_indicesc|\}}tj|}tj|||ftj}||t ntj|t }t|D]}||dz }|||} tj tj|dzdddf|dz|f} tj d||dz|f} | | | kxxdz cc<| | ||||<||xx||zz cc<|S)z> Sample `num_negatives` vectors from feature vectors. )r=rSNrRr#r)size) rVrar^reastyper_rdr]r[rhrWrandint) rtrurvrkrKsequence_length_rangesampled_negative_indices batch_idxhighmapped_masked_indicesfeature_indicessampled_indicess r<_sample_negative_indicesrs #1JIo66 "xz?M.Zbdbjkkk+<*G  &&&RWUckoMpMpMp:&& K K  +//11A5 56G 6R S/")D1H*=*=aaag*FPQS`Habb)++At4!8]:S+TT?:;;;q@;;;MbbqLr +,=i,HI !+++y?/JJ++++ ##r;c&eZdZdfd ZdZxZS)Wav2Vec2NoLayerNormConvLayerrcZt|dkr|j|dz nd|_|j||_t j|j|j|j||j||j |_ t|j |_ dS)Nrr# kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s r<rz%Wav2Vec2NoLayerNormConvLayer.__init__s !#C#Caar;cJ|jdkr|ddddd|j f}|SNr)rrs r<rzWav2Vec2SamePadLayer.forwards;   " ")!!!QQQ0F43F2F0F*FGMr;rrs@r<rr|sLKKKKKr;rc.eZdZdZfdZdZdZxZS)Wav2Vec2FeatureEncoderz.Construct the features from raw audio waveformctjdkr7tdgfdt jdz Dz}nDjdkr!fdt jD}nt djd tj||_ d |_ d |_ dS) Ngrouprrc8g|]}t|dzS)r#r)rrOirs r<rQz3Wav2Vec2FeatureEncoder.__init__..s>NNNIJ,Va!eDDDNNNr;r#layerc2g|]}t|S)r)rrs r<rQz3Wav2Vec2FeatureEncoder.__init__..s4CD*6A>>>r;z`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT) rrfeat_extract_normrr]num_feat_extract_layersrUr ModuleList conv_layersgradient_checkpointing_requires_grad)rrrrs ` r<rzWav2Vec2FeatureEncoder.__init__s   #w . .5fqIIIJNNNNNSTZTruvTvNwNwNNNKK % 0 0HMfNlHmHmKKt1Ittt =55&+#"r;cP|D] }d|_ d|_dSNF) parameters requires_gradrrparams r<_freeze_parametersz)Wav2Vec2FeatureEncoder._freeze_parameterss4__&& ( (E"'E  #r;cr|dddf}|jr|jrd|_|jD] }||}|S)NT)rtrainingrr)r input_valuesr. conv_layers r<rzWav2Vec2FeatureEncoder.forwards[$QQQW-    /4= /*.M '* 6 6J&J}55MMr;)r2r3r4r5rrrrrs@r<rrs\88#####&$$$       r;rceZdZfdZxZS)Wav2Vec2FeatureExtractorct|tjd|jjd|jjdjdtdS)Nz The class `zD` has been depreciated and will be removed in Transformers v5. Use `rz ` instead.)rrwarningswarnrr2 __bases__ FutureWarningrrrs r<rz!Wav2Vec2FeatureExtractor.__init__sy      E$.1 E EN,Q/8 E E E       r;)r2r3r4rrrs@r<rrs8         r;rc$eZdZfdZdZxZS)Wav2Vec2FeatureProjectionc.ttj|jd|j|_tj|jd|j|_ tj |j |_ dS)NrMeps) rrrrrlayer_norm_epsrLinearr projectionDropoutfeat_proj_dropoutdropoutrs r<rz"Wav2Vec2FeatureProjection.__init__sn ,vr':@UVVV)FOB$79KLLz&":;; r;c||}||}||}||fSr)rrr)rr.norm_hidden_statess r<rz!Wav2Vec2FeatureProjection.forwardsC!__];;(:;;  ]33 000r;rrs@r<rrsG<<<<< 1111111r;rmodulequerykeyvaluescalingr head_maskc ||ddz}tj||dd|z} || |z} tj| d} || |ddddz} tj| ||j } tj| |} | dd } | | fS)NrMr&r rr#)pr) rxr6matmulrr functionalsoftmaxviewrr contiguous) rrrrr@r rr kwargs attn_weights attn_outputs r<eager_attention_forwardrs**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L#innQAq&A&AA =((6?([[L,|U33K''1--88::K  $$r;c>eZdZdZ ddededed ed ed ed eeffd Z dde j dee j dee j dee j deede e dee j ee j eee j ffdZxZS)Wav2Vec2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderr is_causalrc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r )r)rrrrrhead_dimrrUr rrrrk_projv_projq_projout_proj) rrrrrrrrrs r<rzWav2Vec2Attention.__init__s "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r;r.key_value_statesr@layer_head_maskoutput_attentionsrrBc |du}|jdd\}} |r |jdn| } || d|jf} || d|jf} ||j| dd} |r|n|}||j| dd}||j| dd}t}|jj dkrt|jj }||| |||f|j sdn|j |j ||d|\}}||| d}||}||dfS)z#Input shape: Batch x Time x ChannelNrMr#r&eagerr)rr r&r )r=rr"rrr r!rr_attn_implementationrrrr rirr#)rr.r$r@r%r&ris_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrs r<rzWav2Vec2Attention.forward s.T9%*3B3/ W/AN"(++wgr4=9 wDM:7t{{=116 FPPQRTUVV -?R))]5T[[005~FPPQRTUVV 7t{{>227HRRSTVWXX (? ; +w 6 6"9$+:Z"[ $7$7      % $}>CC$,L/% % %  % % ! \"))#w;;FFHH mmK00 L$..r;)rFTFN)NNNF)r2r3r4r5rFfloatr_rr$rr6Tensorrrr9rrrs@r<rrs^GG  +/CCCC C  C  CC(CCCCCCD481526,1 3/3/|3/#5<03/!. 3/ "%,/ 3/ $D> 3/-.3/ u|Xel3XeEL>Q5RR S3/3/3/3/3/3/3/3/r;rc$eZdZfdZdZxZS)Wav2Vec2FeedForwardcttj|j|_tj|j|j|_ t|j trt|j |_n |j |_tj|j|j|_tj|j|_dSr)rrrractivation_dropoutintermediate_dropoutrrintermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrs r<rzWav2Vec2FeedForward.__init__As $&Jv/H$I$I!"$)F,>@X"Y"Y f' - - 9'-f.?'@D $ $'-'8D $If&>@RSS j)>??r;c||}||}||}||}||}|Sr)r=rAr;rBrDrs r<rzWav2Vec2FeedForward.forwardNsg// >> 00?? 11-@@ ))-88 ++M:: r;rrs@r<r8r8@sL @ @ @ @ @r;r8c&eZdZfdZddZxZS)Wav2Vec2EncoderLayerctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_dS)NFrrrrrr)rrrrnum_attention_headsattention_dropout attentionrrrCrrrrr8 feed_forwardfinal_layer_normrs r<rzWav2Vec2EncoderLayer.__init__Ys *(0,    z&"788 ,v'9v?TUUU/77 " V-?VEZ [ [ [r;NFc|}||||\}}}||}||z}||}|||z}||}|f}|r||fz }|SNr@r&)rLrrrMrNrr.r@r& attn_residualrrPoutputss r<rzWav2Vec2EncoderLayer.forwardhs% )- .L]*8* * & |Q ]33 % 5  66 %(9(9-(H(HH --m<<  "  '   &Gr;rrrs@r<rGrGXsQ \ \ \ \ \r;rGcXeZdZfdZ ddejdeejdefdZxZ S) #Wav2Vec2EncoderLayerStableLayerNormctt|j|j|jd||_tj|j |_ tj |j|j |_ t||_tj |j|j |_t#|ddt%||_dSd|_dS)NFrIradapter_attn_dim)rrrrrJrKrLrrrCrrrrr8rMrNgetattrWav2Vec2AttnAdapterLayer adapter_layerrs r<rz,Wav2Vec2EncoderLayerStableLayerNorm.__init__}s *(0,    z&"788 ,v'9v?TUUU/77 " V-?VEZ [ [ [ 6-t 4 4 @!9&!A!AD   !%D   r;NFr.r@r&cJ|}||}||||\}}}||}||z}||||z}|j|||z}|f}|r||fz }|SrP)rrLrrMrNr[rRs r<rz+Wav2Vec2EncoderLayerStableLayerNorm.forwards &  66 )- .L]*8* * & |Q ]33 % 5 %(9(9$:O:OP]:^:^(_(__   ))D,>,>},M,MMM "  '   &Gr;r) r2r3r4rr6r6rr_rrrs@r<rVrV|s~&&&&&,26"' |!. r;rVc eZdZfdZ d dejdeejdeded ef d Z de ejdfd ejfd Z xZ S)Wav2Vec2Encoderct|_t|_t jjj|_ t j j |_ t j fdtjD|_d|_dS)Nrc.g|]}tSr:)rGrOrPrs r<rQz,Wav2Vec2Encoder.__init__..s"$k$k$ka%9&%A%A$k$k$kr;Frrrrpos_conv_embedrrrrrrrCrrr]num_hidden_layerslayersrrs `r<rzWav2Vec2Encoder.__init__s  =fEE,v'9v?TUUUz&"788 m$k$k$k$k5QWQiKjKj$k$k$kll &+###r;NFTr.r@r&output_hidden_states return_dictc|rdnd}|rdnd}|;|ddd|jd}d||<|||}||} || z}||}||}tpt|} |j D]a} |r||fz}tj g} |j o| |j jk} | r| r| |||}|d}| rd}|r ||dfz}b|r||fz}|std|||fDSt!||| S) Nr:rMr#r&rrQNNc3K|]}||V dSrr:rOvs r< z*Wav2Vec2Encoder.forward..(mmq_`_l_l_l_l_lmmr;last_hidden_stater.r/) unsqueezerepeatr=_update_full_maskrcrrr r rer6rXrr layerdropr9rrr.r@r&rfrgall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusrdropout_probabilityskip_the_layer layer_outputss r<rzWav2Vec2Encoder.forwards#7@BBD$5?bb4  %$2$<$r6r6r%rrr@r~s r<rsz!Wav2Vec2Encoder._update_full_mask  %{/3FFF343F3FD1V;;"E^UbUh!i!i15EEEnel;;b%@[`%a%a%aN".s" b b bQ 0 8 8 b b br;Frbrs `r<rz'Wav2Vec2EncoderStableLayerNorm.__init__ s  =fEE,v'9v?TUUUz&"788 m b b b b%H`BaBa b b b   ',###r;NFTc|rdnd}|rdnd}|;|ddd|jd}d||<|||}||} || z}||}t pt|} |jD]a} |r||fz}tj g} |j o| |j j k} | r| r| |||}|d}| rd}|r ||dfz}b||}|r||fz}|std|||fDSt!||| S) Nr:rMr#r&rrQric3K|]}||V dSrr:rks r<rmz9Wav2Vec2EncoderStableLayerNorm.forward..Krnr;ro)rqrrr=rsrcrr r rer6rXrrrtrr9rrus r<rz&Wav2Vec2EncoderStableLayerNorm.forwards#7@BBD$5?bb4  %$2$<$rM) flattenexpandr=r6where zeros_liker[meanexplog)probsmask mask_extendedmarginal_probs perplexitys r<_compute_perplexityz1Wav2Vec2GumbelVectorQuantizer._compute_perplexitys   LLNN111dD=9@@MMMK ue6Fu6M6MNNE"YY1Y-- :NN"ZZAZ..NY .59^VZEZ;[;[*[ac d d ddeeiikk r;c|j\}}}||}|||z|jzd}|jrt j||j d |}tj |||z|jdd}| ||}n|d} ||jd| ddd}|||z|jd}| ||}|||zd}|d|jz} | ||z|j|jd} | d||d} | |fS)NrMT)tauhardr r#?r)r=rrrrrrgumbel_softmaxr5rtype_asr6rrargmax new_zerosscatter_rqrrr[) rr.rvrkrKrcodevector_probscodevector_soft_distrcodevector_idxcodevectors_per_grouprs r<rz%Wav2Vec2GumbelVectorQuantizer.forwards3@3F0 O[((77 %**:+G$/+Y[]^^ = W!};;##%%4+;$ <  gm$$  $)="":#?RTUU[[]]ce$$$ 112FHYZZJJ+11b199N,66}7JKKTTN''A..    044Z/5QSWSbdfgg 112BDUVVJ+00o1MrRR 0 : :2 > >AQ Q+00o1Mt`d`moqrr !oob))..z?BOO J&&r;r) r2r3r4r5r staticmethodrrrrs@r<rrjsv (   \ #'#'#'#'#'#'#'#'r;rc$eZdZfdZdZxZS)Wav2Vec2AdapterctjjkrCt jjj|_t jj|_ndx|_|_t j fdtj D|_ j |_ dS)Nc36K|]}tVdSr)Wav2Vec2AdapterLayerras r<rmz+Wav2Vec2Adapter.__init__..s,#k#kQ$8$@$@#k#k#k#k#k#kr;)rroutput_hidden_sizerrrprojrproj_layer_normrr]num_adapter_layersrertrs `r<rzWav2Vec2Adapter.__init__s   $(: : : &"4f6OPPDI#%<0I#J#JD /3 3DI,m#k#k#k#k%PVPiJjJj#k#k#kkk )r;cX|j1|j*||}||}|dd}|jD]=}tj}|jr ||jkr ||}>|dd}|Sr)rrrrerVrWrrt)rr.rlayerdrop_probs r<rzWav2Vec2Adapter.forwards 9 T%9%E IIm44M 00??M%//155 [ 5 5EY--//N= 5^dn%D%D %m 4 4 %//155 r;rrs@r<rrsG * * * * *r;rc$eZdZfdZdZxZS)rcttj|jd|jz|j|jd|_dS)Nr&r#)rr)rrrrradapter_kernel_sizeadapter_striderrs r<rzWav2Vec2AdapterLayer.__init__sU I  % ) )  &(     r;cr||}tj|d}|S)Nr#r )rrrglurs r<rzWav2Vec2AdapterLayer.forwards3 -00  ))-Q)?? r;rrs@r<rrsG     r;rc4eZdZfdZdejfdZxZS)rZctt|j|_|j|_t j|j|_t j |j|j|_ t j |_ t j |j|j|_ dS)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)rrrX input_dimr hidden_dimrrnormrlinear_1ReLUact_fnlinear_2rs r<rz!Wav2Vec2AttnAdapterLayer.__init__s 0 ,L11  $/4>BB gii  $.$/BB r;r.c||}||}||}||}|Sr)rrrrrs r<rz Wav2Vec2AttnAdapterLayer.forwardsL -00  m44  M22  m44 r;)r2r3r4rr6r7rrrs@r<rZrZs[ C C C C CU%6r;rZceZdZUeed<dZdZdZdZdZ dZ dZ dde e jefdeefd Z dd ed e jfd Zd ZdZddefdZdS)Wav2Vec2PreTrainedModelrwav2vec2rTc t|trL|j|jd|j_d|j_dSt|t ro|jjj dd|jj j tj|jdSt|t"rtj |jjddt'jd|jjd|jjzz ztj|jj ddSt|t0r}t'jd|jjz }tj|jj| |tj|jj | |dSt|tjrT|jj d|jj|j |j j dSdSt|tjtjfr?|j j |jj d dSt|tj!rtj"|j|j [t'j|j#|j|jdzz }tj|j | |dSdSdS) zInitialize the weightsTrr#)rstdrr&)abNr)$r>Wav2Vec2ForPreTraining project_hidreset_parameters project_q_is_hf_initializedrrrdatanormal_rzero_rinituniform_rrrmathsqrtr in_channels constant_rr in_featuresrrinitializer_rangerrfill_rkaiming_normal_r)rrks r< _init_weightsz%Wav2Vec2PreTrainedModel._init_weights s f4 5 5" 9   / / 1 1 1   - - / / /48F  126F  / / /  = > > 9   % * 2 2 2 C C C   # ( . . 0 0 0 G  V/ 0 0 0 0 0  ? @ @ 9 GOO " !v{'>q'AFKD['["\]]]     G  fk. 2 2 2 2 2  9 : : 9 !f/;;<._conv_out_length:s&9\K7wWWWZ[[ [r;r#)rrziprrr]rr)rrlrrrrrPs r< _get_feat_extract_output_lengthsz8Wav2Vec2PreTrainedModel._get_feat_extract_output_lengths1s2=1Ddk--+  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMM  _4;9:: _ _ 0 04;C] ^ ^ r;feature_vector_lengthr@c |ddddf}|||}|tj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dgd dg }|S)NrMr rr)rSdevicer#)r) cumsumrtor6longr=r^rSrraflipr_)rrr@rnon_padded_lengthsoutput_lengthsrks r<"_get_feature_vector_attention_maskz:Wav2Vec2PreTrainedModel._get_feature_vector_attention_maskHs ,22r2::111b5A>>?Q_j>kk'**5:66#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOr;c|jjt|jdi}|D]N\}}t |t r4|D]\}}||d||g< Ot |tr9|j D]\}}||dd|g< |S)NzF has no adapter layers. Make sure to define `config.adapter_attn_dim`..lm_head) rrXrUr named_modulesr>rZnamed_parametersjoinWav2Vec2ForCTCr)radapter_weightsrr param_namers r< _get_adaptersz%Wav2Vec2PreTrainedModel._get_adapters\s ; ' /vvvww w ..00 J JLD&&":;; J)/)@)@)B)BJJ%JDIOCHHdJ-?$@$@AA dN + + E#|<<>> E E e?D)T): ; ;<<r;c|D],}t|tr||-t|tr||jdSdS)zc (Re-)initialize attention adapter layers and lm head for adapter-only fine-tuning N)modulesr>rZrrr)rrs r<init_adapter_layersz+Wav2Vec2PreTrainedModel.init_adapter_layersls} llnn + +F&":;; +""6*** dN + + -   t| , , , , , - -r; target_langc 2|jjtd|d||jkr"|s td|ddS|dd}|dd}|d d}|d d}|d d}|d d} |d d} |dd} |dd} | -tjdt| td| } |jj } d}| dur|t |} t| |||||| | | }t|}n9#t$r| rYn*t $r| rtd| d| d|dYnwxYw|t" |} t| |||||| | | }t%t'j|dd}n;#t$rt$rt $rtd| d| d|dwxYw|t-|t-z }t-t-|z }t1|dkr)td|dd|dt1|dkr)td|dd|d|djd}||jjkr=t9j|jj||j|j |_!||j_fd |"D}|#|d!||_dS)"a Load a language adapter model from a pre-trained adapter model. Parameters: target_lang (`str`): Has to be a language id of an existing adapter weight. Adapter weights are stored in the format adapter..safetensors or adapter..bin force_load (`bool`, defaults to `True`): Whether the weights shall be loaded even if `target_lang` matches `self.target_lang`. cache_dir (`Union[str, os.PathLike]`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download of the model weights and configuration files, overriding the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. local_files_only(`bool`, *optional*, defaults to `False`): Whether or not to only look at local files (i.e., do not try to download the model). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, or not specified, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. mirror (`str`, *optional*): Mirror source to accelerate downloads in China. If you are from China and have an accessibility problem, you can set this option to resolve it. Note that we do not guarantee the timeliness or safety. Please refer to the mirror site for more information. Activate the special ["offline-mode"](https://huggingface.co/transformers/installation.html#offline-mode) to use this method in a firewalled environment. Examples: ```python >>> from transformers import Wav2Vec2ForCTC, AutoProcessor >>> ckpt = "facebook/mms-1b-all" >>> processor = AutoProcessor.from_pretrained(ckpt) >>> model = Wav2Vec2ForCTC.from_pretrained(ckpt, target_lang="eng") >>> # set specific language >>> processor.tokenizer.set_target_lang("spa") >>> model.load_adapter("spa") ``` NzCannot load_adapter for - if `config.adapter_attn_dim` is not defined.z#Adapter weights are already set to r cache_dirforce_downloadFresume_downloadproxieslocal_files_onlytokenuse_auth_tokenrevisionuse_safetensorszrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.zV`token` and `use_auth_token` are both specified. Please set only the argument `token`.)filenamerrrrrrrzCan't load the model for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z=' is the correct path to a directory containing a file named cpuT) map_location weights_onlyrzThe adapter weights z has unexpected keys: z, z has missing keys: zlm_head.weightrrScNi|]!\}}|||"Sr:)r)rOrrlr s r< z8Wav2Vec2PreTrainedModel.load_adapter..5s0QQQdaaoa011QQQr;)strict)$rrXrUrloggerwarningpoprrr _name_or_pathWAV2VEC2_ADAPTER_SAFE_FILEformatrsafe_load_fileOSError ExceptionWAV2VEC2_ADAPTER_PT_FILErr6loadr setkeysrbrr= vocab_sizerrrrrSritemsload_state_dict)rr force_loadrrrrrrrrrrmodel_path_or_id state_dictfilepath weight_pathunexpected_keys missing_keystarget_vocab_sizer s @r< load_adapterz$Wav2Vec2PreTrainedModel.load_adapterys6~ ; ' /r rrrss s $* * *: * NNOOOO P P P FJJ{D11 $4e<< **%6==**Y--!::&8%@@ 7D))$4d;;::j$// **%6==  % ME     l#E;4  % ' '188EEH )$%#1$3#%5%'    ,K88    "    "!J5EJJ=MJJ?GJJJ   /66{CCH# )$%#1$3#%5%'    )***"Z!&!%          F1AFF9IFF;CFFF ,,..joo//0037K7K7M7M3N3NN?//1122S9J9J5K5KK   ! # #tKttW[W`W`apWqWqtttuu u    " "nKnnTXT]T]^jTkTknnnoo o''78>qA  6 6 69 .0A$+]a]gDL&7DK "RQQQj>N>N>P>PQQQ  Z666's$0'F G'$G G.=H,,8I$r)T)r2r3r4r$r8base_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrrr6 LongTensorrFrr_rrr rr@r;r:r;r<rrs "$O&*#N%9%9%9PZ^"5#3S#89HPQU0Y]%(:?:J( - - -@'@'@'@'@'@'@'@'r;rc6eZdZdeffd ZdZdZ ddejde ejde ej fd Z e dd e ej de ej de ejd e ed e ed e edeeeffdZxZS) Wav2Vec2Modelrct|||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|jrt+|nd|_|dS)Nr)rrrrfeature_extractorrfeature_projectionmask_time_probmask_feature_probrrr6r6rrmasked_spec_embeddo_stable_layer_normrencoderr^rradapter post_initrs r<rzWav2Vec2Model.__init__>s     !7!?!?";F"C"C  3 & &&*BS*H*H%'\%,v?Q2R2R2[2[2]2]%^%^D "  & 39&AADLL*622DL282DNv...$  r;cbtjdt|dSz Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. The method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.Nrrrfreeze_feature_encoderrs r<freeze_feature_extractorz&Wav2Vec2Model.freeze_feature_extractorR;   Q    ##%%%%%r;c8|jdS Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rFrrTs r<rSz$Wav2Vec2Model.freeze_feature_encoder^s 1133333r;Nr.rvr@ct|jdds|S|\}}}|#|j|j||<n|jjdkr|jr|t||f|jj|jj ||jj }tj ||j tj}|j|j||<|jjdkr|jrt||f|jj|jj|jj}tj ||j tj}|dddfd|d}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTNr)r>r?r@rAr)r>r?rArM)rYrrxrJrrSrHrrsmask_time_lengthmask_time_min_masksr6rrr_rImask_feature_lengthmask_feature_min_masksr)rr.rvr@rkrKrmask_feature_indicess r<_mask_hidden_statesz!Wav2Vec2Model._mask_hidden_statesest{$8$?? ! 4A3E3E3G3G0 O[  (/3/E/H/HI\/]/]M+ , , [ '! + + + 5_-+4 K8-+9 !!! !& ->}G[chcm n n n /3/E/H/HI\/]/]M+ , ; (1 , , ,#8[)+7 K;+< $$$ $)<0D]Mainis#t#t#t #74#@#G#GO]_#`#` 23M. /r;rr&rfrgrBc:||n |jj}||n |jj}||n |jj}||}|dd}|#||jd|d}||\}}| |||}| |||||} | d}|j | |}|s||f| ddzSt||| j | j S) a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Nr#r&Fr)rvr@r@r&rfrgr)rpextract_featuresr.r/)rr&rfuse_return_dictrFrrr=rGrarLrMrr.r/) rrr@rvr&rfrgrdr.encoder_outputss r<rzWav2Vec2Model.forwards|2C1N--TXT_Tq$8$D $+Jj &1%uEN+/*A*ABR*S*S' '00 ->~1  ,, )/!5# '  (* < # LL77M K!#34qrr7JJ J&+-)7&1     r;riNNNNN)r2r3r4r$rrUrSr6r7rrBrarr6r_rr9rrrrs@r<rDrD<sZ~( & & &444:>59 ,,(,$E$56,!!12 ,,,,\269=,0/3&*7 7 u|,7 !.7 $E$56 7 $D> 7 'tn 7 d^7  u-- .7 7 7 ^7 7 7 7 7 r;rDz? Wav2Vec2 Model with a quantizer and `VQ` head on top. cXeZdZdeffd ZdefdZdZdZe dde j d e j d e j defd Z e dd ee jdee jdee jdee jdeedeedeedeeeffdZxZS)rrct|t||_t j|j|_t||_ t j |j |j |_ t j |j|j |_|dSr)rrrDrrrfeat_quantizer_dropoutdropout_featuresr quantizerrrproj_codevector_dimrrrrNrs r<rzWav2Vec2ForPreTraining.__init__s    %f-- " 6+H I I6v>>9V%79STT6#8&:TUU r;rc||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rlr)rrs r<set_gumbel_temperaturez-Wav2Vec2ForPreTraining.set_gumbel_temperatures&1"""r;cbtjdt|dSrPrRrTs r<rUz/Wav2Vec2ForPreTraining.freeze_feature_extractorrVr;cB|jjdSrXrrFrrTs r<rSz-Wav2Vec2ForPreTraining.freeze_feature_encoder! '::<<<<>> import torch >>> from transformers import AutoFeatureExtractor, Wav2Vec2ForPreTraining >>> from transformers.models.wav2vec2.modeling_wav2vec2 import _compute_mask_indices, _sample_negative_indices >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("facebook/wav2vec2-base") >>> model = Wav2Vec2ForPreTraining.from_pretrained("facebook/wav2vec2-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> input_values = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt").input_values # Batch size 1 >>> # compute masked indices >>> batch_size, raw_sequence_length = input_values.shape >>> sequence_length = model._get_feat_extract_output_lengths(raw_sequence_length).item() >>> mask_time_indices = _compute_mask_indices( ... shape=(batch_size, sequence_length), mask_prob=0.2, mask_length=2 ... ) >>> sampled_negative_indices = _sample_negative_indices( ... features_shape=(batch_size, sequence_length), ... num_negatives=model.config.num_negatives, ... mask_time_indices=mask_time_indices, ... ) >>> mask_time_indices = torch.tensor(data=mask_time_indices, device=input_values.device, dtype=torch.long) >>> sampled_negative_indices = torch.tensor( ... data=sampled_negative_indices, device=input_values.device, dtype=torch.long ... ) >>> with torch.no_grad(): ... outputs = model(input_values, mask_time_indices=mask_time_indices) >>> # compute cosine similarity between predicted (=projected_states) and target (=projected_quantized_states) >>> cosine_sim = torch.cosine_similarity(outputs.projected_states, outputs.projected_quantized_states, dim=-1) >>> # show that cosine similarity is much higher than random >>> cosine_sim[mask_time_indices.to(torch.bool)].mean() > 0.5 tensor(True) >>> # for contrastive loss training model should be put into train mode >>> model = model.train() >>> loss = model( ... input_values, mask_time_indices=mask_time_indices, sampled_negative_indices=sampled_negative_indices ... ).loss ```N)r@r&rfrvrgrr#Fr)rvrMr&r z-infir[) reduction)r*r+r,r-r.r/r0r1)$rrerr6r_rrrkrr=rlrrrSrrpermuter|contrastive_logits_temperatureallanyr5rrirxrrr cross_entropyrrr[diversity_loss_weightr)r.r/)rrr@rvr|r&rfrgrTtransformer_featuresrdquantized_featuresr-r*r0r1rkrKrnegative_quantized_featuresr{ neg_is_postargetnum_codevectorss r<rzWav2Vec2ForPreTraining.forwardsD&1%uEN59NN 0A5C5 5 110224>3H3NOO!^^,>??3777. # /7I7O 4J +=*A*A"k*R*R(--//44R88+ '+F*J*JOR++gaAq!! ( 44"47++$ : F-0KKPPQSTTJ~~ 7).vqrr :&%%a++33B AGGF,11333t;FFq!LLTTVVF!}::6<<>>6]b:cc "kCdkFggO.1FF/Y]n]r]r]t]ttN$dk&G.&XXD c24FH]^ahijikikalll(*<>STW^_`_a_aWbb b+1'9"7!/)-)    r;)rt)NNNNNN)r2r3r4r$rrFrorUrSrr6r7r|rrr6 BoolTensorr_rr9r)rrrs@r<rrs ~      1#1111 & & &===  * ,"- \(268 \ 'tn\ d^\  u22 3\ \ \ ^\ \ \ \ \ r;rceZdZfdZe d dejdeejdee dee dee deej d e e e ffd ZxZS) Wav2Vec2ForMaskedLMcRt|tjdtt ||_tj|j |_ tj |j |j |_|dS)NzSThe class `Wav2Vec2ForMaskedLM` is deprecated. Please use `Wav2Vec2ForCTC` instead.)rrrrrrDrrr final_dropoutrrrr0rrNrs r<rzWav2Vec2ForMaskedLM.__init__s     acp   &f-- z&"677 y!3V5FGG  r;Nrr@r&rfrglabelsrBc||n |jj}|||||}|d}||}||} |s| f|ddz} | St | |j|jS)N)r&rfrgrr&)r{r.r/)rrerrrrr.r/) rrr@r&rfrgrrTr.r{outputs r<rzWav2Vec2ForMaskedLM.forwards&1% q 'tn q d^ q&q un$ %qqq^qqqqqr;rzp Wav2Vec2 Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ceZdZddeeffd ZdZdZdZdZ e ddee j d ee j d ee d ee d ee d ee j deeeffdZxZS)rNrct|t||_t j|j|_||_|j td|j dt|dr|j r|jn|j}t j||j |_|dS)a2 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`Wav2Vec2ForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `Wav2Vec2ForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.r)rrrDrrrrrrr0rUrrrrrrrrN)rrrrrs r<rzWav2Vec2ForCTC.__init__s    %f-- z&"677 &   $HHHH *1)G)G vFL^ vF % %djdv y!3V5FGG  r;c|j}|)t|jddtd|d|2t|jddtddS|||ddSdS)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. NrXzCannot pass `target_lang`: rz)By default `target_lang` is set to 'eng'.T)r3)rrYrrUr#infor;)rrs r< tie_weightszWav2Vec2ForCTC.tie_weightss&  "wt{||jjkrt d|jj||||||}|d}||}||} d} |Z||ntj |tj }| | d tj } |dk} | d} || }tj| dtjdd}tjjd 5tj||| | |jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rcrrRrM)rrSr#F)enabled)blankr~ zero_infinityr*r{r.r/)rrerGr0rUrrrr6 ones_likerrr[r masked_selectrr log_softmaxfloat32rbackendscudnnflagsctc_loss pad_token_idctc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrr.r/)rrr@r&rfrgrrTr.r{r*rl labels_masktarget_lengthsflattened_targets log_probsrs r<rzWav2Vec2ForCTC.forward0s"&1% HSM:<<<* & & &===(((26,0/3&*)-D D u|,D !.D $D> D 'tn D d^ D &D  un$ %D D D ^D D D D D r;rz Wav2Vec2 Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZdZe ddee j dee j dee d ee d ee d ee j d e e effd ZxZS)!Wav2Vec2ForSequenceClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_tj|j|j|_|dS)Nrz_Sequence classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r#)rrrrrUrDrrduse_weighted_layer_sumrrr6rd layer_weightsrrclassifier_proj_size projector num_labels classifierrNrr num_layersrs r<rz*Wav2Vec2ForSequenceClassification.__init__s    6= ) ) f.@ q &f-- -1  ( S!#ej.D.Dz.Q!R!RD 6#5v7RSS)F$?ARSS r;cbtjdt|dSrPrRrTs r<rUz:Wav2Vec2ForSequenceClassification.freeze_feature_extractorrVr;cB|jjdSrXrrrTs r<rSz8Wav2Vec2ForSequenceClassification.freeze_feature_encoderrsr;cL|jD] }d|_ dSrrrs r<rz3Wav2Vec2ForSequenceClassification.freeze_base_modelrr;Nrr@r&rfrgrrBcd||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |}||d} n||jd|} | ddd|jd} d || <| d| d ddz } || } d}|Kt)}|| d|jj| d}|s| f|tdz}||f|zn|St-|| |j|j S)  input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`Wav2Vec2Processor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrcr#r rMrr&rr)rrerrrr6stackrrrrrr[rrrr=rqrrrr rrr.r/)rrr@r&rfrgrrTr. norm_weights pooled_output padding_maskexpand_padding_maskr{r*loss_fctrs r<rz)Wav2Vec2ForSequenceClassification.forwardsW.&1%->r1a-H-HHMMRSMTTMM#AJM}55  !)..1.55MMBB=CVWXCY[ijjL"."8"8"<"<"C"CAq-J]^_J`"a"a 25M.. /)--!-44|7G7GA7G7N7N7S7STVXY7Z7ZZM//  '))H8FKKDK,BCCV[[QS__UUD FY)F)G)G!HHF)-)9TGf$$v E'!/)     r;rg)r2r3r4rrUrSrrrr6r6r_rr9rrrrs@r<rrxs" & & &===(((26,0/3&*)-B B u|,B !.B $D> B 'tn B d^ B &B  u.. /B B B ^B B B B B r;rceZdZfdZdZdZdZe ddee j dee j dee j d ee d ee d ee d e e effd ZxZS)#Wav2Vec2ForAudioFrameClassificationct|t|dr|jrt dt ||_|jdz}|jr.tj tj ||z |_ tj|j|j|_|j|_|dS)NrzbAudio frame classification does not support the use of Wav2Vec2 adapters (config.add_adapter=True)r#)rrrrrUrDrrdrrrr6rdrrrrr init_weightsrs r<rz,Wav2Vec2ForAudioFrameClassification.__init__s    6= ) ) f.@ t &f-- -1  ( S!#ej.D.Dz.Q!R!RD )F$68IJJ + r;cbtjdt|dSrrRrTs r<rUz->r1a-H-HHMMRSMTTMM#AJM//  '))H8FKKDO<t>t>tuuD Y)F)G)G!HHFM$!/)     r;rg)r2r3r4rrUrSrrrr6r6r_rr9rrrrs@r<rrs  & & &===(((26)-,0/3&*9 9 u|,9 !.9 & 9 $D> 9 'tn 9 d^9  u++ ,9 9 9 ^9 9 9 9 9 r;rc&eZdZdfd ZdZxZS) AMSoftmaxLoss>@皙?ct||_||_||_t jtj||d|_ t j |_ dS)NT)r) rrscalemarginrrrr6randnrr r*)rrrrrrs r<rzAMSoftmaxLoss.__init__\se   $l5;y*#E#EUYZZZ ')) r;c|}tj|jd}tj|d}t j||}||jz }tj||j }|j t j | ||z}| ||}|S)Nrr r#)rrr normalizerr6mmrone_hotrrrr_r*) rr.rr cos_thetapsionehotr{r*s r<rzAMSoftmaxLoss.forwardds!!((!(<< // 1/EE H]F33 $+%&&vt??ek&++--iHHHyy(( r;)rrrrs@r<rr[sL******       r;rcDeZdZdfd ZdejdejfdZxZS) TDNNLayerrct|dkr|j|dz n |j||_|j||_|j||_|j||_tj |j|jz|j|_ tj |_ dS)Nrr#)rrtdnn_dimrr tdnn_kernelr tdnn_dilationdilationrrkernelrrrs r<rzTDNNLayer.__init__ss rrrrrrrrrrrconv1drrr)rr.rrs r<rzTDNNLayer.forward}s    3 2 2 2 2 2 2    $+y11  O &//155 #(():D.s#QQQy++QQQr;rMr&)rrrDrrdrrrr6rdrrrrrr]rbrtdnnxvector_output_dimrFrrr objectiver)rrr tdnn_layersrs ` r<rzWav2Vec2ForXVector.__init__s)    %f-- -1  ( S!#ej.D.Dz.Q!R!RD 6#5vq7IJJQQQQU3v;O;O5P5PQQQ M+.. !#6?2+>+BFD]!^!^)F$=v?XYY&v'@&BSTT r;cbtjdt|dSrrRrTs r<rUz+Wav2Vec2ForXVector.freeze_feature_extractorrVr;cB|jjdSrXrrrTs r<rSz)Wav2Vec2ForXVector.freeze_feature_encoderrsr;cL|jD] }d|_ dSrrrs r<rz$Wav2Vec2ForXVector.freeze_base_modelrr;rlcDd}|jjD]}|||d}|S)z? Computes the output length of the TDNN layers c||z |zdzS)Nr#r:rs r<rzEWav2Vec2ForXVector._get_tdnn_output_lengths.._conv_out_lengths!;.69A= =r;r#)rr)rrlrrs r<_get_tdnn_output_lengthsz+Wav2Vec2ForXVector._get_tdnn_output_lengthssE  > > >  ;2 L LK,,]KKKMMr;Nrr@r&rfrgrrBc>||n |jj}|jjrdn|}||||||}|jjrx|t}t j|d}tj |j d} || dddz d}n|d}| |}|jD] } | |}|-|d} |d} n|| d} || }g} g} t'|D]k\}}| ||d|fd| ||d|fdlt j| } t j| } t j| | gd}||}||}d}||||}|s||f|tdz}||f|zn|St3||||j|jS) rNTrcr#r rMr)r*r{ embeddingsr.r/)rrerrrr6rrrrrrr[rrrrrr enumeraterfryrFrrrr.r/)rrr@r&rfrgrrTr.r tdnn_layer mean_features std_featuresfeat_extract_output_lengthstdnn_output_lengthsrlengthstatistic_poolingoutput_embeddingsr{r*rs r<rzWav2Vec2ForXVector.forwards.&1%->r1a-H-HHMMRSMTTMM#AJM}55 ) 6 6J&J}55MM  !)..1.55M(,,,33LL*.*O*OP^PbPbghPbPiPi*j*j '"&"?"?@["\"\ ML&':;; J J 6$$]1gvg:%>%C%C%C%J%JKKK##M!WfW*$=$A$Aa$A$H$HIIII!K 66M ;|44L!I}l&CLLL 223DEE!233  >>&&11D F/07;X;Y;Y3ZZF)-)9TGf$$v E(!/)     r;rg)r2r3r4rrUrSrrr6rBrFrrrr6r_r9rrrrs@r<rrsB & & & &===((( eE O 'tn O d^ O &O  um# $O O O ^O O O O O r;r)rrrrrrrDrrr)NrN)fr5rr dataclassesrtypingrrrnumpyrVr6safetensors.torchrr)rtorch.nnr activationsr integrations.deepspeedr integrations.fsdpr modeling_attn_mask_utilsrrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrrprocessing_utilsrrrrrrr r!r"configuration_wav2vec2r$r,r'integrations.flex_attentionr% get_loggerr2r#rr)r9rFr5rBndarrayrsrrrrModulerrrrrr6rrr8rGrVr^rrrrrZrrDrrrrrrrr__all__r:r;r<rs  !!!!!!,,,,,,,,,, 999999%%%%%%!!!!!!@@@@@@777777CBBBBB999999GFFFFFFF&&&&&&322222,5!!KJJJJJJ  H % %!"  77777;77   7B26 tt c?tttU-. t  t Z ttttpZ^!$!$!$*-!$BJ2:BV!$!$!$!$H#=*!;6!;0*****bi***Z29%%%%%RY%%%P     5    1 1 1 1 1 1 1 1, $(,%% I% <% % < % U\* % e_ %% %%%%%29$ry2x'x'x'x'x'ox'x'x'v N N N N N +N N N b Y Y Y Y Y 4Y Y  Y x*q*q*q*q*q1*q*q*qZ S S S S S ,S S  S l p p p p p (?p p  p ff f f f f *Af f f RBI. @ N N N N N 0N N  N b   r;