`idZddlZddlmZmZddlZddlmZddlmZddl m Z m Z m Z ddl mZdd lmZdd lmZdd lmZdd lmZmZdd lmZddlmZmZmZddlmZddl m!Z!ej"e#Z$dZ%dZ&dZ'gdZ(dZ)dZ*Gddej+Z,Gddej+Z-Gddej+Z.Gddej+Z/Gdd ej+Z0Gd!d"ej+Z1Gd#d$ej+Z2Gd%d&ej+Z3Gd'd(eZ4Gd)d*eZ5d+Z6d,Z7Gd-d.e5Z8e d/e6Gd0d1e5Z9e d2e6Gd3d4e5Z:gd5Z;dS)6zPyTorch M-CTC-T model.N)OptionalUnion)nn)ACT2FN)add_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forward)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputCausalLMOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)logging) MCTCTConfigrzspeechbrain/m-ctc-t-large)rizY"Mr. Quilter is the apostle of the middle classes, and we're glad to welcome his gospel."gv@c(eZdZdZfdZdZxZS)MCTCTConv1dSubsamplerz Convolutional subsampler: a stack of 1D convolution (along temporal dimension) followed by non-linear activation via gated linear units (https://huggingface.co/papers/1911.08460) ct|_|j_t j|j_|j _ |j |j z_ j dkr#|jtd|j_nd_|jdz_|j_|j_t jfdt1jD_dS)NrzbNeed to specify `conv_channels` configuration in `MCTCTConfig` to use multiple convolution layers.c3K|]c\}}tj|dkrjn j||jdz kr j|nj|j|dVddS)rrvalid) kernel_sizestridepaddingN)rConv1d in_channels mid_channels num_layers out_channelsr ).0ikselfs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/deprecated/mctct/modeling_mctct.py z1MCTCTConv1dSubsampler.__init__..Us ) ) 1 I$%FF  0A!0D()DOa,?(?(?!!$$TEV{1~     ) ) ) ) ) ) )super__init__config conv_glu_dimglu_dimrDropout conv_dropoutdropoutnum_conv_layersr%input_feat_per_channelinput_channelsr# conv_channels ValueErrorr$ hidden_sizer& conv_kernelr conv_strider ModuleList enumerate conv_layersr*r0 __class__s` r+r/zMCTCTConv1dSubsampler.__init__9s  * z&"566  0!86;PP ?Q  #+  !' 4D   $D ".2!-( = ) ) ) ) "$"233 ) ) )   r-ctd|jD}tjj|dd||fdd}|dd}|jD]H}||}tj ||j }| |}I|dd}|S)Nc3 K|] }|dzV dS)rN)r'sizes r+r,z0MCTCTConv1dSubsampler.forward..cs&==Ddai======r-rconstantrrdim) sumrtorchr functionalpad transpose contiguousr@glur2r5)r*input_featuresr! hidden_statesconvs r+forwardzMCTCTConv1dSubsampler.forward`s==D,<=====,00!QQXAY[eghii&00A66AACC $ 8 8D D//MM--m-NNM LL77MM%//155@@BB r-__name__ __module__ __qualname____doc__r/rT __classcell__rBs@r+rr3sR % % % % % N       r-rc,eZdZdZfdZ ddZxZS)MCTCTEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ t|_ tj|j|_|dt%j|jdd|dt%j|jt$j|jjddS)N) padding_idx position_ids)rF) persistenttoken_type_idsdtypedevice)r.r/r Embedding vocab_sizer; pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddingsMCTCTLayerNorm LayerNormr3hidden_dropout_probr5register_bufferrKarangeexpandzerosr`rFlongrfrAs r+r/zMCTCTEmbeddings.__init__ss1 !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]" ())z&"<==   EL)GHHOOPWXXej       K)..00 4K\Kc d d d      r-Nrc>||n|dd}|d}||jdd|||zf}|mt|dr2|jddd|f}||d|} | }n+t j|t j|jj}|| |}| |} || z} | | } | | } | S)Nrarrcrrd) rFr`hasattrrcrtrKrurvrfrjrnrpr5) r*rQrcr` inputs_embedspast_key_values_length input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedrn embeddingss r+rTzMCTCTEmbeddings.forwardsB0>/In))+++}OaOaOcOcdgegdgOh  ^  ,QQQ0FVlIl0l-lmL  !t-.. m*.*=aaa*n*M'3J3Q3QR]^_R`bl3m3m0!A!&[ SWSdSk!l!l!l   00@@M $ : :> J J"%:: ^^J// \\*-- r-)NNNNrrUr[s@r+r]r]psYQQ     .wxr-r]c>eZdZfdZdZdZdZ ddZxZS) MCTCTSelfAttentionct|j|jzdkr0t |ds t d|jd|jd|j|_|j|_|j|jz|_tj |j|jd|_ tj |j|jd|_ tj |j|jd|_ tj|j|_|j|_tjd|jzd z |j|_|j|_dS) Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()Fbiasrr)r.r/r;num_attention_headsrxr:attention_head_dimattention_head_size all_head_sizerLinearquerykeyvaluer3attention_probs_dropout_probr5rkrgdistance_embedding is_decoderrAs r+r/zMCTCTSelfAttention.__init__sS    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #)#< !58PPYv143EERRR 9V/1C%PPPYv143EERRR z&"EFF '-'E$"$,q63Q/QTU/UW[Wo"p"p +r-c|dd|j|jfz}|j|}|ddddS)Nrarrr)rFrrviewpermute)r*x new_x_shapes r+transpose_for_scoresz'MCTCTSelfAttention.transpose_for_scoressMffhhssmt'?AY&ZZ AFK yyAq!$$$r-c *t|jdkr6|jtt t|j}|jt|jtt t|S)Nr)lenshaperreversedrangereshape)r*rrs r+reshape_fortranz"MCTCTSelfAttention.reshape_fortransp qw< bclerrrHrr) start_dim)rmathsqrtrrrrrKmatmulrNrweighteinsumrrrLsoftmaxr5rflatten)r*rRattention_mask head_maskoutput_attentionsmixed_query_layer key_layer value_layer query_layerattention_scorespositional_embeddingrelative_position_scoresattention_probs context_layeroutputss r+rTzMCTCTSelfAttention.forwards!JJ}55- $:R0S0SS--dhh}.E.EFF // =0I0IJJ //0ABB !< Y5H5HR5P5PQQ $6=#(<0BDXZeZoZopqstZuZu#v#v #'#J#JKc#d#d +.FF  %/.@ -//0@b/II,,77  - 9O _kBB %--aAq99AABAOO 6G]=/22mM]r-NNF) rVrWrXr/rrrrTrZr[s@r+rrs,,,,,.%%% QQQ ***8 ........r-rc$eZdZfdZdZxZS)rocttjt jd|_tjt jd|_dSNr) r.r/r ParameterrKonessingleton_weightrusingleton_bias)r*rBs r+r/zMCTCTLayerNorm.__init__sS  " UZ]] ; ; l5;q>>::r-c&||jz|jzSN)rrr*rRs r+rTzMCTCTLayerNorm.forwards 559LLLr-rVrWrXr/rTrZr[s@r+rorosN;;;;; MMMMMMMr-roc$eZdZfdZdZxZS)MCTCTSelfOutputc(t||_tj|j|jd|_tj|j|j|_tj |j |_ dSNFr)eps) r.r/r0rrr;denserplayer_norm_epsr3rqr5rAs r+r/zMCTCTSelfOutput.__init__!sr  Yv163EERRR f&8f>STTTz&"<== r-c||}||}|||z}|Srrr5rpr*rR input_tensors r+rTzMCTCTSelfOutput.forward(@ =11  ]33 }|'CDD r-rr[s@r+rr sG>>>>>r-rc2eZdZfdZdZ ddZxZS)MCTCTAttentionctt||_t ||_t |_dSr)r.r/rr*routputset pruned_headsrAs r+r/zMCTCTAttention.__init__0sI &v.. %f-- EEr-ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)NrrrH)rrr*rrrrrrrrrrunion)r*rindexs r+ prune_headszMCTCTAttention.prune_heads6s u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r-NFc|||||}||d|}|f|ddz}|S)Nrr)r*r)r*rRrrr self_outputsattention_outputrs r+rTzMCTCTAttention.forwardHsXyy        ;;|A FF#% QRR(88r-r)rVrWrXr/rrTrZr[s@r+rr/sf""""" ;;;* r-rc$eZdZfdZdZxZS)MCTCTIntermediatecttj|j|jd|_t|jtrt|j|_ dS|j|_ dS)NFr) r.r/rrr;intermediate_sizer isinstance hidden_actstrrintermediate_act_fnrAs r+r/zMCTCTIntermediate.__init__\st Yv163KRWXXX f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r-cZ||}||}|Sr)rrrs r+rTzMCTCTIntermediate.forwardds, =11 00?? r-rr[s@r+rr[sG99999r-rc$eZdZfdZdZxZS) MCTCTOutputcttj|j|jd|_tj|j|j|_tj |j |_ dSr) r.r/rrrr;rrprr3rqr5rAs r+r/zMCTCTOutput.__init__ksl Yv79KRWXXX f&8f>STTTz&"<== r-c||}||}|||z}|Srrrs r+rTzMCTCTOutput.forwardqrr-rr[s@r+rrjsG>>>>> r-rc8eZdZdeffd Z ddZdZxZS) MCTCTLayerr0ctd|_|j|_t ||_t ||_|j|_t||_ dSr) r.r/ seq_len_dimchunk_size_feed_forwardr intermediater attentionrrrrAs r+r/zMCTCTLayer.__init__ysh '-'E$-f55'// +!&)) r-NFc|||||}|d}|dd}t|j|j|j|}|f|z}|S)N)rrr)rrfeed_forward_chunkrr) r*rRrrrself_attention_outputsrr layer_outputs r+rTzMCTCTLayer.forwardsw"& >9HY"0" " 2!4(,0  #T%A4CSUe   /G+r-c\||}|||}|Sr)rr)r*rintermediate_outputrs r+rzMCTCTLayer.feed_forward_chunks2"//0@AA{{#68HII r-r)rVrWrXrr/rTrrZr[s@r+rrxsr *{ * * * * * * *r-rcLeZdZUdZeed<dZdZdZdZ de j fdZ d Z d S) MCTCTPreTrainedModelz An abstract class to handle weights initialization and a simple interface for downloading and loading pretrained models. r0mctctrQTc|jj}t|tjrH|jjd||j|jj nt|tj rR|jjd||j )|jj|j  nt|tj r>|jj |jj dnRt|tr=|jj d|jj t|tjtjfrH|jjd||j"|jj dSdSdS)zInitialize the weightsg)meanstdNg?)r0initializer_rangerrrrdatanormal_rzero_rgr_rpfill_rorrr")r*modulers r+ _init_weightsz"MCTCTPreTrainedModel._init_weightssk+ fbi ( ( / M  & &CS & 9 9 9{&  &&(((  - - / M  & &CS & 9 9 9!- "6#56<<>>>  - - / K  " " $ $ $ M  $ $S ) ) ) )  / / /  # ( . .s 3 3 3  ! & , , . . . fry")4 5 5 ) M  & &CS & 9 9 9{&  &&((((( ) )&&r- input_lengthscd}tt|jj|jj|jjD]9\}}}|dz}|d|zz||dz zz dz }t j||ddz}:|S)zH Computes the output length of the convolutional layers rrtrunc) rounding_mode)ziprr0r6r<r=rKdiv)r*r dilation_ kernel_szr r!s r+ _get_feat_extract_output_lengthsz5MCTCTPreTrainedModel._get_feat_extract_output_lengthss$' $+- . . 0GI`% %  X X Ay& 1nG)AK7(iRSm:TTWXXM!ImV7SSSVWWMMr-ct|jdkr|dddddf}||d}|d}t j||f|j|j}d|t j ||j|dz f<| dg d dg }|S)Nrrarrdrr) rrrrJrFrKrurerfrsflipcumsumrv)r*feature_vector_lengthrsubsampled_lengthsbszs r+"_get_feature_vector_attention_maskz7MCTCTPreTrainedModel._get_feature_vector_attention_masks ~# $ $q ( (+AAAqqq"H5N"BB>CUCUVXCYCYZZ!!##A& ' (0D^Mb    ef S1FGGGI[^_I_`a',,bT2299"==BBB4HHMMOOr-N)rVrWrXrYr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointingr rK LongTensorrrrEr-r+rrs{ &O&*#)))0 e>N    r-raH This model is a PyTorch [torch.nn.Module](https://pytorch.org/docs/stable/nn.html#torch.nn.Module) sub-class. Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage and behavior. Parameters: config ([`MCTCTConfig`]): Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the [`~PreTrainedModel.from_pretrained`] method to load the model weights. a Args: input_features (`torch.LongTensor` of shape `({0})`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`Wav2Vec2CTCTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.FloatTensor` of shape `({0})`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) head_mask (`torch.FloatTensor` of shape `(num_heads,)` or `(num_layers, num_heads)`, *optional*): Mask to nullify selected heads of the self-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~file_utils.ModelOutput`] instead of a plain tuple. c~eZdZdeffd Z d dejdejdejded ed ed ee e ffd Z xZ S) MCTCTEncoderr0c.tj|_t|_t |_tjfdtj D|_ d|_ dS)Nc.g|]}tSrE)r)r'rr0s r+ z)MCTCTEncoder.__init__..s!$a$a$aAZ%7%7$a$a$ar-F) r.r/rqro layer_normrrSrr>rnum_hidden_layerslayersgradient_checkpointingrAs `r+r/zMCTCTEncoder.__init__s    #)#= (**)&11 m$a$a$a$avG_A`A`$a$a$abb &+###r-FTrQrrroutput_hidden_states return_dictreturnc||n |jj}||n |jj}||n |jj}||}||}|!||jd|}tj ||j |j }|t||j}|rdnd} |rdnd} |p|dt!|jkr@t%dt!|jd|ddt'pt)|} t+|jD]d\} } |r| |fz} t-jg}|j o||jjk}|r| r| |||}|d}|rd }|r | |dfz} e|r| |fz} |st3d || | fDSt5|| | S) Nr)ptrainingrErz&The head_mask should be specified for z layers, but it is for .)rRrr)NNc3K|]}||V dSrrE)r'vs r+r,z'MCTCTEncoder.forward..`s(eeqWXWdWdWdWdWdeer-last_hidden_staterR attentions)r0rr,use_return_dictr(rSrrrrLr5rqr1r rerFrr*r:r r r?rKrand layerdroptupler)r*rQrrrr,r-ryrRencoder_statesall_attentions synced_gpusidx encoder_layerdropout_probabilityskip_the_layer layer_outputss r+rTzMCTCTEncoder.forwards2C1N--TXT_Tq$8$D $+Jj &1%>N fee]NN$Seeeee e+>Vd    r-)FFT) rVrWrXrr/rKTensorboolrr;rrTrZr[s@r+r$r$s,{,,,,,,#(%* I I  I  I < I  I # I I  uo% &I I I I I I I I r-r$zaThe bare M-CTC-T Model transformer outputting raw hidden-states without any specific head on top.c,eZdZfdZeedeee e de  dde j dee j dee j d eed eed eed eee ffd ZxZS) MCTCTModelct|||_t||_|dSr)r.r/r0r$encoder post_initrAs r+r/zMCTCTModel.__init__ksI     #F++  r-zbatch_size, sequence_lengthaudio) checkpoint output_type config_classmodalityexpected_outputNrQrrrr,r-r.c"||n |jj}||n |jj}||n |jj}|t d|||||||}|d}|s|f|ddzSt ||j|jS)Nz#You have to specify input_features.rrrr,r-rrr5) r0rr,r8r:rIrrRr7) r*rQrrrr,r-encoder_outputssequence_outputs r+rTzMCTCTModel.forwardts"2C1N--TXT_Tq$8$D $+Jj &1% # 'tn # d^#  uo% &# # # ih# # # # # r-rGzcMCTCT Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC).c"eZdZfdZeeeeee e e  d de j dee j dee j deedeed eed ee jd eeeffd ZxZS) MCTCTForCTCc$t|t||_|jt d|jd|j}tj ||j|_ | dS)NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `MCTCTForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration.) r.r/rGrrhr:rBr;rrctc_headrJ)r*r0output_hidden_sizerBs r+r/zMCTCTForCTC.__init__s    ''   $HHHH  $/ "4f6GHH  r-)rLrMrNrP expected_lossNrQrrrr,r-labelsr.c b|>||jjkrtd|jj||n |jj}|||||||}|d} || } d} |g||n,tj|j ddtj }| | d tj } |dk} | d}|| }tj| dtjdd}tjjd 5tj||| ||jj|jj|jj } dddn #1swxYwY|s| f|t6dz}| | f|zn|St9| | |j|j S) a labels (`torch.LongTensor` of shape `(batch_size, target_length)`, *optional*): Labels for connectionist temporal classification. Note that `target_length` has to be smaller or equal to the sequence length of the output logits. Indices are selected in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size - 1]`. Nz$Label values must be <= vocab_size: rRrra)re)rIrerF)enabled)blank reduction zero_infinity)losslogitsrRr7)maxr0rhr:r8rr]rKrrrvrrJto masked_selectrrL log_softmaxfloat32rNbackendscudnnflagsctc_lossrictc_loss_reductionctc_zero_infinity_HIDDEN_STATES_START_POSITIONrrRr7)r*rQrrrr,r-r`rrRrgrfr  labels_masktarget_lengthsflattened_targets log_probsrs r+rTzMCTCTForCTC.forwards2  &**,,$+2H"H"H\DKDZ\\]] ]%0% E 'tn E d^E )*E  un$ %E E E CBE E E E E r-r[)r[rGr)rs; """""""" """"""rrrrrrrrrrAAAAAA888888CCCCCC::::::@@@@@@@@......mmmmmmmmmm,,,,,,  H % % !2't:::::BI:::z77777bi777tiiiiiiiiXMMMMMRYMMM     bi   )))))RY)))X             ")   $$$$$+$$$NBBBBB?BBBJ @T T T T T 'T T T ng5 5 5 5 5 %5 5 5 pma a a a a &a a a H @ ? ?r-