`i`YdZddlmZmZmZddlZddlmZm Z m Z m Z ddl m Z ddlmZddlmZmZmZejeZGd d e Zd gZdS) z)Feature extractor class for UnivNetModel.)AnyOptionalUnionN)mel_filter_bankoptimal_fft_length spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature)PaddingStrategy TensorTypeloggingc)eZdZdZgdZ d>dedededededededede ededede ed ed!ed"ed#ed$ed%ed&ed'ef(fd( Z d)Z d*Z d+e jd,e jfd-Z d?d.ed/e e jjd,e jfd0Zd?d,ee jfd1Z d@d2ee jeeee jeeefde ed3eeeefd4e ed5ed6e ed7ed/e e jjd8ed9e ede ed:e ed;e eeefd,efd<Zd,eeefffd= ZxZS)AUnivNetFeatureExtractora Constructs a UnivNet feature extractor. This class extracts log-mel-filter bank features from raw speech using the short time Fourier Transform (STFT). The STFT implementation follows that of TacoTron 2 and Hifi-GAN. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: feature_size (`int`, *optional*, defaults to 1): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 24000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). padding_value (`float`, *optional*, defaults to 0.0): The value to pad with when applying the padding strategy defined by the `padding` argument to [`UnivNetFeatureExtractor.__call__`]. Should correspond to audio silence. The `pad_end` argument to `__call__` will also use this padding value. do_normalize (`bool`, *optional*, defaults to `False`): Whether to perform Tacotron 2 normalization on the input. Normalizing can help to significantly improve the performance for some models. num_mel_bins (`int`, *optional*, defaults to 100): The number of mel-frequency bins in the extracted spectrogram features. This should match `UnivNetModel.config.num_mel_bins`. hop_length (`int`, *optional*, defaults to 256): The direct number of samples between sliding windows. Otherwise referred to as "shift" in many papers. Note that this is different from other audio feature extractors such as [`SpeechT5FeatureExtractor`] which take the `hop_length` in ms. win_length (`int`, *optional*, defaults to 1024): The direct number of samples for each sliding window. Note that this is different from other audio feature extractors such as [`SpeechT5FeatureExtractor`] which take the `win_length` in ms. win_function (`str`, *optional*, defaults to `"hann_window"`): Name for the window function used for windowing, must be accessible via `torch.{win_function}` filter_length (`int`, *optional*, defaults to 1024): The number of FFT components to use. If `None`, this is determined using `transformers.audio_utils.optimal_fft_length`. max_length_s (`int`, *optional*, defaults to 10): The maximum input length of the model in seconds. This is used to pad the audio. fmin (`float`, *optional*, defaults to 0.0): Minimum mel frequency in Hz. fmax (`float`, *optional*): Maximum mel frequency in Hz. If not set, defaults to `sampling_rate / 2`. mel_floor (`float`, *optional*, defaults to 1e-09): Minimum value of mel frequency banks. Note that the way [`UnivNetFeatureExtractor`] uses `mel_floor` is different than in [`transformers.audio_utils.spectrogram`]. center (`bool`, *optional*, defaults to `False`): Whether to pad the waveform so that frame `t` is centered around time `t * hop_length`. If `False`, frame `t` will start at time `t * hop_length`. compression_factor (`float`, *optional*, defaults to 1.0): The multiplicative compression factor for dynamic range compression during spectral normalization. compression_clip_val (`float`, *optional*, defaults to 1e-05): The clip value applied to the waveform before applying dynamic range compression during spectral normalization. normalize_min (`float`, *optional*, defaults to -11.512925148010254): The min value used for Tacotron 2-style linear normalization. The default is the original value from the Tacotron 2 implementation. normalize_max (`float`, *optional*, defaults to 2.3143386840820312): The max value used for Tacotron 2-style linear normalization. The default is the original value from the Tacotron 2 implementation. model_in_channels (`int`, *optional*, defaults to 64): The number of input channels to the [`UnivNetModel`] model. This should match `UnivNetModel.config.model_in_channels`. pad_end_length (`int`, *optional*, defaults to 10): If padding the end of each waveform, the number of spectrogram frames worth of samples to append. The number of appended samples will be `pad_end_length * hop_length`. return_attention_mask (`bool`, *optional*, defaults to `True`): Whether or not [`~UnivNetFeatureExtractor.__call__`] should return `attention_mask`. )input_featuresnoise_sequence padding_mask]Fd hann_window N& .>?h㈵> 'ă@@T feature_size sampling_rate padding_value do_normalize num_mel_bins hop_length win_length win_function filter_length max_length_sfminfmax mel_floorcentercompression_factorcompression_clip_val normalize_min normalize_maxmodel_in_channelspad_end_lengthc tjd||||d|||_||_||_||_||_| |_| |_| t|dz } | |_ | |_ | |_ | |z|_ |jt|j|_n |j|_|jdzdz|_t#|j|jd|_t'|j|j|j|j |jdd|_||_||_||_||_||_||_||_dS) N)r#r$r%return_attention_maskrT) window_lengthnameperiodicslaney)num_frequency_binsnum_mel_filters min_frequency max_frequencyr$norm mel_scale)super__init__r&r'r(r)r*r+r-floatr.r/r,num_max_samplesrn_fftn_freqsr windowrr$ mel_filtersr0r1r2r3r4r5r6)selfr#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r8kwargs __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/univnet/feature_extraction_univnet.pyrFz UnivNetFeatureExtractor.__init__es|2  %''"7       )($$(* <''!+D "(+m;   %+DO<S)cpuT)devicecopy)detachtonumpy).0rXs rP z8UnivNetFeatureExtractor.batch_decode..sAeeexX__&&))T)BBHHJJeeerQNc6g|]\}}|d|SNrD)rirXwaveform_lengthss rPrz8UnivNetFeatureExtractor.batch_decode..s.aaa[Q"7$4Q$7"78aaarQ) enumerate)rM waveformsrs `rP batch_decodez$UnivNetFeatureExtractor.batch_decodesH fe[deee  'aaaaIV_L`L`aaaIrQ raw_speechpadding max_length truncationpad_to_multiple_of return_noisepad_end pad_lengthr8return_tensorsc  | | nj} |<|jkr0tdjjdjdjd|d n(t djjdt|tj ot|j d k}|r*t|j d krtd |pHt|ttfo,t|d tj ttf}|r d |D}n|s;t|tj s!tj|tj}n^t|tj rD|jtjtjur|tj}|s!tj|tjg}| r  nj fd|D}t)d|i}||||nj||| }|d}fd|D}t|d trd|D|d<nd|D|d<|d}|d|D|d<|rfd|dD}||d<| rfd|dD|d<| || }|S)a Main method to featurize and prepare for the model one or several sequence(s). Args: raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition pipeline. padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`): Select a strategy to pad the input `raw_speech` waveforms (according to the model's padding side and padding index) among: - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single sequence if provided). - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum acceptable input length for the model if that argument is not provided. - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different lengths). If `pad_end = True`, that padding will occur before the `padding` strategy is applied. max_length (`int`, *optional*): Maximum length of the returned list and optionally padding length (see above). truncation (`bool`, *optional*, defaults to `True`): Activates truncation to cut input sequences longer than `max_length` to `max_length`. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_noise (`bool`, *optional*, defaults to `True`): Whether to generate and return a noise waveform for use in [`UnivNetModel.forward`]. generator (`numpy.random.Generator`, *optional*, defaults to `None`): An optional `numpy.random.Generator` random number generator to use when generating noise. pad_end (`bool`, *optional*, defaults to `False`): Whether to pad the end of each waveform with silence. This can help reduce artifacts at the end of the generated audio sample; see https://github.com/seungwonpark/melgan/issues/8 for more details. This padding will be done before the padding strategy specified in `padding` is performed. pad_length (`int`, *optional*, defaults to `None`): If padding the end of each waveform, the length of the padding in spectrogram frames. If not set, this will default to `self.config.pad_end_length`. do_normalize (`bool`, *optional*): Whether to perform Tacotron 2 normalization on the input. Normalizing can help to significantly improve the performance for some models. If not set, this will default to `self.config.do_normalize`. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. [What are attention masks?](../glossary#attention-mask) return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.np.array` objects. - `'np'`: Return Numpy `np.ndarray` objects. Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.rr9z2Only mono-channel audio is supported for input to rcNg|]"}tj|tj#Srsrbasarrayrx)rspeechs rPrz4UnivNetFeatureExtractor.__call__..s)XXX6"*V2:>>>XXXrQrsc\g|](}tj|djzfj)S)r)constant_values)rbrcr(r%)rrXrrMs rPrz4UnivNetFeatureExtractor.__call__..sJx!Z$/%A!BTXTfgggrQr)rrrrr8c:g|]}|SrD)rn)rrXrMs rPrz4UnivNetFeatureExtractor.__call__..s'ZZZxD00::ZZZrQcNg|]"}tj|tj#Srrrmels rPrz4UnivNetFeatureExtractor.__call__..s*/n/n/nVY 3bj0Q0Q0Q/n/n/nrQcLg|]!}|tj"SrD)astyperbrxrs rPrz4UnivNetFeatureExtractor.__call__..s&/c/c/c3 2:0F0F/c/c/crQattention_maskcNg|]"}tj|tj#Sr)rbrint32)rarrays rPrz4UnivNetFeatureExtractor.__call__..s*-l-l-lTYbjbh.O.O.O-l-l-lrQrcRg|]#}|jd$S)r)r{shape)rr rqrMs rPrz4UnivNetFeatureExtractor.__call__..sA##K$5a$8)DDrQrc:g|]}|SrD)rU)rr rMs rPrz4UnivNetFeatureExtractor.__call__..s20000;{++000rQ)r&r$ ValueErrorrO__name__loggerwarning isinstancerbndarraylenrlisttuplerrxrtfloat64rr6r rcrHgetconvert_to_tensors)rMrr$rrrrrrqrrr&r8ris_batched_numpy is_batchedbatched_speech padded_inputsrmel_spectrogramsrrzs` ` ` rP__call__z UnivNetFeatureExtractor.__call__sX(4'?||TEV  $ 222 W$.JaWW)-);WW)-);WWFSWWW3 NN\W[WeWn\\\    &j"*==[#jFVBWBWZ[B[  ZJ$4 5 5 9 9XRVXXYY Y% zD%= 1 1 lz*Q-RTR\^ceiQj7k7k   7XXZXXXJJ 7Jz2:$F$F 7JbjAAAJJ  BJ / / 7J4DQSQ[H\H\4\4\#**2:66J D*ZrzBBBCJ  '1'=4CVJ *J &'7&DEE %/%;zzAU!1"7 !  '**+;<<ZZZZ>ZZZ nQ' . . d/n/n]m/n/n/nN+ , ,/c/cRb/c/c/cN+ ,'**+;<<  %-l-l]k-l-l-lN> *  5#12B#CE05N+ ,  0000?MN^?_000N+ ,  %+>>~NNNrQcht}gd}|D] }||vr||= |S)N)rKrLrIrJrH)rEto_dict)rMoutputnamesr;rOs rPrzUnivNetFeatureExtractor.to_dictsI""QPP ! !Dv~~4L rQ)rrrFrrrrrrrNrFrrr r!r"rTr) NTNTNTNFNNNN)r __module__ __qualname____doc__model_input_namesrdrGboolstrrrFrUrWrbrrnru Generatorr{rrrr rr rdictrr __classcell__)rOs@rPrrsCCJMLL""")'+ $$'&*21!# "-J-J-J-J- J-  J-  J-J-J-J- }J-J-J-uoJ-J-J- "!J-"$#J-$%J-&'J-()J-*+J-J-J-J-J-J-Xhhhhhh.% .%rz.%.%.%.%f48BI/0  @RZ@P4(,59$(,0!37$(&*04;?__"*d5k4 3CT$u+EVVW_ }_tS/12 _ SM _  _%SM__BI/0__SM_sm_ (~_!sJ!78_ ____B c3h          rQr)rtypingrrrrrb audio_utilsrrr r !feature_extraction_sequence_utilsr feature_extraction_utilsr utilsr rr get_loggerrrr__all__rDrQrPrs0/''''''''''\\\\\\\\\\\\IIIIII4444449999999999  H % %kkkkk6kkk\ % %rQ