`iGdZddlZddlZddlmZddlmZmZddlZddl m Z ddl m Z m Z ddlmZddlmZmZd d lmZmZmZmZmZmZmZmZmZd d lmZeje Z!eed Gdde Z"GddeZ#GddeZ$GddeZ%GddeZ&GddeZ'GddeZ(eGddeZ)e Z*Gdd e)eZ+ed!Gd"d#e)Z,Gd$d%eZ-Gd&d'eZ.gd(Z/dS))zPyTorch UniSpeech model.N) dataclass)OptionalUnion) ModelOutputWav2Vec2BaseModelOutput)PreTrainedModel)auto_docstringlogging) Wav2Vec2EncoderWav2Vec2EncoderStableLayerNormWav2Vec2FeatureEncoderWav2Vec2FeatureProjectionWav2Vec2ForCTC!Wav2Vec2ForSequenceClassificationWav2Vec2GumbelVectorQuantizer Wav2Vec2ModelWav2Vec2PositionalConvEmbedding)UniSpeechConfigzh Output type of [`UniSpeechForPreTrainingOutput`], with potential hidden states and attentions. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ ee ejed<dZee ejed<dS) UniSpeechForPreTrainingOutputa loss (*optional*, returned when model is in train mode, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the contrastive loss (L_m) and the diversity loss (L_d) as stated in the [official paper](https://huggingface.co/papers/2006.11477). projected_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Hidden-states of the model projected to *config.proj_codevector_dim* that can be used to predict the masked projected quantized states. projected_quantized_states (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.proj_codevector_dim)`): Quantized extracted feature vectors projected to *config.proj_codevector_dim* representing the positive target vectors for contrastive loss. codevector_perplexity (`torch.FloatTensor` of shape `(1,)`): The perplexity of the codevector distribution, used to measure the diversity of the codebook. Nlossprojected_statesprojected_quantized_statescodevector_perplexity hidden_states attentions)__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rrrrtupler /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/unispeech/modular_unispeech.pyrr-s  )-D(5$ %,,,48hu01888>B): ;BBB9=8E$56===8)meanr%expsumlog)probsmarginal_probs perplexitys r+_compute_perplexityz2UniSpeechGumbelVectorQuantizer._compute_perplexity_s^**Y .59^VZEZ;[;[*[ac d d ddeeiikk r*c|j\}}}||}|||z|jzd}|jrt j||j d |}tj |||z|jdd}| |}n|d}|j|jd|ddd}|||z|jd}| |}|||zd}|d|jz} | ||z|j|jd} | d||d} | |fS)Nr=T)tauhardr;r?)shape weight_projview num_groupstrainingnn functionalgumbel_softmaxfloat temperaturetype_asr%softmaxrEargmax new_zerosscatter_ unsqueeze codevectorsnum_varsr@) selfr batch_sizesequence_length hidden_sizecodevector_probscodevector_soft_distrDcodevector_idxcodevectors_per_groupr[s r+forwardz&UniSpeechGumbelVectorQuantizer.forwardes3@3F0 O[((77 %**:+G$/+Y[]^^ = D!};;##%%4+;$ <  gm$$  $)="":#?RTUU[[]]ce$$$ 112FGGJJ+11b199N6}6 8KLUUN''A..    044Z/5QSWSbdfgg 112BCCJ+00o1MrRR 0 : :2 > >AQ Q+00o1Mt`d`moqrr !oob))..z?BOO J&&r*N)r!r"r# staticmethodrErer)r*r+r9r9^s<\ #'#'#'#'#'r*r9cxeZdZUeed<dZdZdZdZdZ dZ dZ de e jeffdZded e jfd Zd S) UniSpeechPreTrainedModelconfig unispeech input_valuesTc \t|tro|jjjdd|jjjtj |j dSt|trtj |j jddtjd|j jd|j jzz ztj |j jddSt|t&r}tjd|jjz }tj |jj| |tj |jj| |dSt|tjrT|jjd|jj|j |jjdSdSt|tjtjfr?|jj|jjddSt|tjrtj |j|j[tj|j|j|jdzz }tj |j| |dSdSdS) zInitialize the weightsr)r>stdrr )abNrI) isinstancer9rLweightdatanormal_biaszero_rPinituniform_r[r-convmathsqrt kernel_size in_channels constant_r3 projection in_featuresLinearriinitializer_range LayerNorm GroupNormfill_Conv1dkaiming_normal_groups)r]moduleks r+ _init_weightsz&UniSpeechPreTrainedModel._init_weightss f< = = 9   % * 2 2 2 C C C   # ( . . 0 0 0 G  V/ 0 0 0 0 0  @ A A 9 GOO " !v{'>q'AFKD['["\]]]     G  fk. 2 2 2 2 2  : ; ; 9 !f/;;<._conv_out_lengths&9\K7wWWWZ[[ [r*)zipri conv_kernel conv_stride)r]rrr|rs r+ _get_feat_extract_output_lengthsz9UniSpeechPreTrainedModel._get_feat_extract_output_lengthss\  \ \ \ $'t{'> @W#X#X Q Q K,,]KPPMMr*feature_vector_lengthattention_maskc|ddddf}||tj}|jd}tj||f|j|j}d|tj |jd|j|dz f<| dgd dg }|S)Nr=r;r)dtypedevicer)r) cumsumrtor%longrKzerosrrarangeflipbool)r]rrnon_padded_lengthsoutput_lengthsr^s r+"_get_feature_vector_attention_maskz;UniSpeechPreTrainedModel._get_feature_vector_attention_masks,22r2::111b5A>>?QRRUUV[V`aa#)!,  . /~7KTbTi   uv ^%9!%<^EZ[[[]kno]opq',,bT2299"==BBB4HHMMOOr*N)r!r"r#rr'base_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrrr% LongTensorintrrr)r*r+rhrhs #$O&*#N999B eEDTVYDY>Z      ]b]m      r*rhceZdZdefdZdZdZ ddeej deej deej d ee d ee d ee d e e effd ZdS)UniSpeechModelrict||||_t||_t ||_|jdks |jdkrBtj tj |j |_|jrt#||_nt'||_|dS)Nrm)rh__init__rir1feature_extractorr3feature_projectionmask_time_probmask_feature_probrP Parameterr%Tensorr`rxmasked_spec_embeddo_stable_layer_normr7encoderr5 post_init)r]ris r+rzUniSpeechModel.__init__s ))$777 !8!@!@"~1  ,, )/!5# '  (*  K!#34qrr7JJ J'+-)7&1     r*)NNNNN)r!r"r#rrrrrr%rr&rrr(rrer)r*r+rrs"999999 269=,0/3&*2 2 u|,2 !.2 $E$56 2 $D> 2 'tn 2 d^2  u.. /2 2 2 2 2 2 r*rzZ UniSpeech Model with a vector-quantization module and ctc loss for pre-training. c eZdZdeffd ZdefdZdZdZe dde j d e j d e j defd Z e dd ee jdee jdeedeedeedeeeff dZxZS)UniSpeechForPreTrainingrict|t||_t j|j|_t||_ t j |j |j |_ t j |j |j|_t j |j|j|_t j|j|_|dS)N)superrrrjrPDropoutfeat_quantizer_dropoutdropout_featuresr9 quantizerrcodevector_dimproj_codevector_dim project_qr` project_hidnum_ctc_classesctc_proj final_dropoutdropoutr)r]ri __class__s r+rz UniSpeechForPreTraining.__init__+s    '// " 6+H I I7??6#8&:TUU9V%?ASTT &"4f6LMM z&"677  r*rTc||j_dS)zb Set the Gumbel softmax temperature to a given value. Only necessary for training N)rrT)r]rTs r+set_gumbel_temperaturez.UniSpeechForPreTraining.set_gumbel_temperature:s&1"""r*cbtjdt|dS)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. zThe method `freeze_feature_extractor` is deprecated and will be removed in Transformers v5. Please use the equivalent `freeze_feature_encoder` method instead.N)warningswarn FutureWarningrrs r+rz0UniSpeechForPreTraining.freeze_feature_extractor@s;   Q    ##%%%%%r*cB|jjdS)z Calling this function will disable the gradient computation for the feature encoder so that its parameter will not be updated during training. N)rjr_freeze_parametersrs r+rz.UniSpeechForPreTraining.freeze_feature_encoderLs! (;;=====r*rtarget_featuresnegative_featurespredicted_featuresctj||gd}tj||d}||}||z }|S)z Compute logits for contrastive loss based using cosine similarity as the distance measure between `[positive_feature, negative_features]` and `[predicted_features]`. Additionally, temperature can be applied. rr;r=)r%catcosine_similarityrSrU)rrrrTlogitss r+compute_contrastive_logitsz2UniSpeechForPreTraining.compute_contrastive_logitsSsq )_6G$HaPPP();)A)A)C)C_EZEZE\E\bdeee00+% r*NrkrrrrrcN||n |jj}||||||}|d}||d}||\} } || |jjj} | | } tj | d| d |jj} | dd} tj| |j} | dd} | d} || d| | dz} || } || } d}|s#|||| | f|ddzS|| | f|ddzSt/||| | |j|jS) a Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, UniSpeechForPreTraining >>> feature_extractor = AutoFeatureExtractor.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> model = UniSpeechForPreTraining.from_pretrained("microsoft/unispeech-large-1500h-cv") >>> # TODO: Add full pretraining example ```Nrrrr=rmr )rrrrrr )rirrjrrrrrrrrr%emptysizer replace_probr bernoullirrrZ masked_fillrrrrr )r]rkrrrroutputstransformer_featuresrquantized_featuresrprob_replace_matrixsampled_replace_matrixrrs r+rezUniSpeechForPreTraining.forwardgsW*&1%,A,A$.BWB],^,^__!--.@AA#k*>*C*CA*F*FH\HaHabcHdHdeekk K $  2;;AqAA!&1D!E!E!J!J!L!L!O!OPdPk!l!l!7!A!A!Q!G!G!7!A!A"!E!E%112H#NN  * *,B+BC H H  f%%v&& c24FH]^ahijikikalll(*<>STW^_`_a_aWbb b,1'9"7!/)     r*)r)NNNN)r!r"r#rrrrrrrfr%r&rr rrrrr(rre __classcell__)rs@r+rr%su       1#1111 & & &>>>  * ,"- \&26,0/3&* D D u|,D !.D $D> D 'tn D d^ D  u33 4D D D ^D D D D D r*rceZdZdS)UniSpeechForCTCNr.r)r*r+rrr/r*rceZdZdS)"UniSpeechForSequenceClassificationNr.r)r*r+rrr/r*r)rrrrrh)0r$rzr dataclassesrtypingrrr%torch.nnrPmodeling_outputsrrmodeling_utilsr utilsr r wav2vec2.modeling_wav2vec2r rrrrrrrrconfiguration_unispeechr get_loggerr!loggerrr-r1r3r5r7r9rhrrrrr__all__r)r*r+rs !!!!!!"""""""" DDDDDDDD------,,,,,,,,                      544444  H % %  :::::K::   :.     'F        4        !:                &D   *'*'*'*'*'%B*'*'*'ZFFFFFFFFR3J J J J J -}J J J Z B B B B B 6B B  B J     n        )J      r*