`i\AdZddlmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZmZerddlZejeZGd d e Zd gZdS) z% Feature extractor class for Whisper )OptionalUnionN)is_torch_available)mel_filter_bank spectrogramwindow_function)SequenceFeatureExtractor) BatchFeature) TensorTypeloggingceZdZdZdgZ d#fd Zd ejd ed ejfdZ d$dejd ed ejfdZ e d%de ejde ejde d e ejfdZ d&deeje e e eje e e fdedeedeeeefdeedeedeedeed eed eed!eed efd"ZxZS)'WhisperFeatureExtractora Constructs a Whisper feature extractor. This feature extractor inherits from [`~feature_extraction_sequence_utils.SequenceFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. This class extracts mel-filter bank features from raw speech using a custom numpy implementation of the `Short Time Fourier Transform` which should match pytorch's `torch.stft` equivalent. Args: feature_size (`int`, *optional*, defaults to 80): The feature dimension of the extracted features. sampling_rate (`int`, *optional*, defaults to 16000): The sampling rate at which the audio files should be digitalized expressed in hertz (Hz). hop_length (`int`, *optional*, defaults to 160): Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients. chunk_length (`int`, *optional*, defaults to 30): The maximum number of chunks of `sampling_rate` samples used to trim and pad longer or shorter audio sequences. n_fft (`int`, *optional*, defaults to 400): Size of the Fourier transform. padding_value (`float`, *optional*, defaults to 0.0): Padding value used to pad the audio. Should correspond to silences. dither (`float`, *optional*, defaults to 0.0): Adds dithering. In other words, adds a small Gaussian noise to each frame. E.g. use 0.0001 to add dithering with a normal distribution centered around 0.0 with standard deviation 0.0001 (assuming [-1,+1] range of raw_speech). The value 0.0 means no dithering. Dithering has similar effect as `spectrogram(mel_floor=...)`. It reduces the high log_mel_fbank values for signals with hard-zero sections, when VAD cutoff is present in the signal. input_featuresP>Fc tjd||||d| ||_||_||_||z|_|j|z|_||_||_td|dzz|dd|dd|_ dS) N) feature_size sampling_rate padding_valuereturn_attention_maskrg@@slaney)num_frequency_binsnum_mel_filters min_frequency max_frequencyrnorm mel_scale) super__init__n_fft hop_length chunk_length n_samples nb_max_framesrditherr mel_filters) selfrrr)r*r(rr-rkwargs __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/whisper/feature_extraction_whisper.pyr'z WhisperFeatureExtractor.__init__Hs  %''"7        $(% 5!^z9* * 5A:~( '   waveform_batchdevicereturnc |dkrtd|dg}|D]}t|t|jd|j|jd|j|jd}|dddd f}tj|| d z }|d zd z }| |tj |}|S) z Compute the log-mel spectrogram of the provided audio, gives similar results to Whisper's original torch implementation with 1e-5 tolerance. cpuz Got device `z` for feature extraction, but feature extraction on CUDA accelerator devices requires torch, which is not installed. Either set `device='cpu'`, or install torch according to the official instructions: https://pytorch.org/get-started/locally/hanng@log10) frame_lengthr)powerr-r.log_melN @@) ValueErrorrr r(r)r-r.npmaximummaxappendarray)r/r4r5log_spec_batchwaveformlog_specs r2_np_extract_fbank_featuresz2WhisperFeatureExtractor._np_extract_fbank_featuresls U??qvqqq  & , ,H" F33!Z?{ ,   H 3B3'Hz(HLLNNS,@AAH 3#-H  ! !( + + + +.11r3r8rHc$tj||tj}tj|j|}|jdkr1||jtj|j|j |j zz }tj ||j|j |d}|dddf d z}tj|j|tj}|j|z}tj|d }|d krQ|d d d dd d } tj|| dz }n*tj||dz }|dzdz }|dkr&|}|S)z Compute the log-mel spectrogram of the audio using PyTorch's GPU-accelerated STFT implementation with batching, yielding results similar to cpu computing with 1e-5 tolerance. )r5r)dtyper5T)windowreturn_complex.Nr>rg|=)min)dimkeepdimrrr?r@r8)torch from_numpytofloat32 hann_windowr(r-randnshaperLr5stftr)absr.Tclampr:rPrDrCdetachr8numpy) r/rHr5rMrY magnitudesr.mel_specrImax_vals r2_torch_extract_fbank_featuresz5WhisperFeatureExtractor._torch_extract_fbank_featuress #H--00GG"4:f=== ;#    ek(._g_n&o&o&oo oHz(DJ_cddd#ss(^''))Q. &t'788;;FEMRR =:-;xU33399;; <<>>Q  llq$l77:>>1d>SSTUVG}Xw}==HH}Xx||~~/CDDHsNc) U??((,,..H~~r3 input_valuesattention_maskrc|tj|tj}g}t||dD]\}}||d|z tj|d|dzz }||jdkr|||d<| |n d|D}|S)z[ Every array in the list is normalized to have zero mean and unit variance Nr>Hz>rcg|]C}||z tj|dzz DS)rf)meanrBsqrtvar).0xs r2 zCWhisperFeatureExtractor.zero_mean_unit_var_norm..s@"b"b"bPQALBGAEEGGdN4K4K#K"b"b"br3) rBrFint32zipsumrhrirjrXrE)rcrdrnormed_input_valuesvectorlength normed_slices r2zero_mean_unit_var_normz/WhisperFeatureExtractor.zero_mean_unit_var_norms  %Xnbh??N"$ "%lN4F4Fr4J4J"K"K 9 9 &)=)=)?)? ?276RYSYRY?K^K^K`K`cgKgChChh L.q111,9L)#**<8888  9#c"bUa"b"b"b ""r3TN max_length raw_speech truncationpad_to_multiple_ofreturn_tensorsrpaddingr do_normalizereturn_token_timestampsc |<|jkr0tdjjdjdjd|d n(tdjjdt |tjot|j d k} | r*t|j d krtd | pHt |ttfo,t |d tjttf}|r d |D}n|s;t |tjs!tj |tj}n^t |tjrD|jtjtjur|tj}|stj |gjg}t'd|i}|||r|nj|||p| }| rK|d|dj|d<tj|dd |d<|dd d d }t7rjnj}||d | }t |d trd|D|d<n||d<|rL|dddddjf}|dj d jzd kr|ddddf}||d<| 9tdjjdfd|D|d<|| |}|S)a Main method to featurize and prepare for the model one or several sequence(s). Implementation uses PyTorch for the STFT computation if available, otherwise a slower NumPy based one. Args: raw_speech (`np.ndarray`, `list[float]`, `list[np.ndarray]`, `list[list[float]]`): The sequence or batch of sequences to be padded. Each sequence can be a numpy array, a list of float values, a list of numpy arrays or a list of list of float values. Must be mono channel audio, not stereo, i.e. single float per timestep. truncation (`bool`, *optional*, default to `True`): Activates truncation to cut input sequences longer than *max_length* to *max_length*. pad_to_multiple_of (`int`, *optional*, defaults to None): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta), or on TPUs which benefit from having sequence lengths be a multiple of 128. return_attention_mask (`bool`, *optional*): Whether to return the attention mask. If left to the default, will return the attention mask according to the specific feature_extractor's default. [What are attention masks?](../glossary#attention-mask) For Whisper models, `attention_mask` should always be passed for batched inference, to avoid subtle bugs. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors instead of list of python integers. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return Numpy `np.ndarray` objects. sampling_rate (`int`, *optional*): The sampling rate at which the `raw_speech` input was sampled. It is strongly recommended to pass `sampling_rate` at the forward call to prevent silent errors and allow automatic speech recognition pipeline. padding_value (`float`, *optional*, defaults to 0.0): The value that is used to fill the padding values / vectors. do_normalize (`bool`, *optional*, defaults to `False`): Whether or not to zero-mean unit-variance normalize the input. Normalizing can help to significantly improve the performance of the model. device (`str`, *optional*, defaults to `'cpu'`): Specifies the device for computation of the log-mel spectrogram of audio signals in the `_torch_extract_fbank_features` method. (e.g., "cpu", "cuda") return_token_timestamps (`bool`, *optional*, defaults to `None`): Deprecated. Use `return_attention_mask` instead from which the number of frames can be inferred. Whether or not to return the number of frames of the input raw_speech. These num_frames can be used by the model to compute word level timestamps. Nz3The model corresponding to this feature extractor: z& was trained using a sampling rate of zI. Please make sure that the provided `raw_speech` input was sampled with z and not .zDIt is strongly recommended to pass the `sampling_rate` argument to `zN()`. Failing to do so can result in silent errors that might be hard to debug.rrz2Only mono-channel audio is supported for input to rcZg|](}tj|gtjj)SrL)rBasarrayrUr[)rkspeechs r2rmz4WhisperFeatureExtractor.__call__..s.\\\v"*fXRZ@@@B\\\r3rr)r{rvrxryrrd)rdr)axiscNg|]"}tj|tj#Sr)rBrrU)rkfeatures r2rmz4WhisperFeatureExtractor.__call__..Bs*.s.s.sY`rz'/T/T/T.s.s.sr3r>z,`return_token_timestamps` is deprecated for z~ and will be removed in Transformers v5. Use `return_attention_mask` instead, as the number of frames can be inferred from it.c>g|]}t|jzSr%)lenr))rk raw_speech_ir/s r2rmz4WhisperFeatureExtractor.__call__..Vs)*o*o*oT`3|+<+<+O*o*o*or3 num_frames)!rrAr1__name__loggerwarning isinstancerBndarrayrrXlisttuplerrUrLfloat64astyper[r padr+rurstackget transposerrbrJr) warning_onceconvert_to_tensors)r/rwrxryrzrr{rvrr|r5r}r0is_batched_numpy is_batchedbatched_speech padded_inputsrextract_fbank_featuresrescaled_attention_masks` r2__call__z WhisperFeatureExtractor.__call__s[H  $ 222 W$.JaWW)-);WW)-);WWFSWWW3 NN\W[WeWn\\\    &j"*==[#jFVBWBWZ[B[  ZJ$4 5 5 9 9XRVXXYY Y% zD%= 1 1 lz*Q-RTR\^ceiQj7k7k   7\\Q[\\\JJ 7Jz2:$F$F 7JbjAAAJJ  BJ / / 7J4DQSQ[H\H\4\4\#**2:66J 6*j\2245J%'7&DEE %/CzzT^!1"7"G< !    `.2.J.J./,-=>"0/K//M* + /1h}EU7V]^._._._M* +'**+;<<FFq!QOO3E2F2F kD . .DLk 0/q0A6JJ nQ' . . =.s.sdr.s.s.sM* + +/=M* + F&34D&EaaaI[I[DOI[F[&\ # -.4Q7$/IQNN*A!!!SbS&*I'.EM* + " .   Wt~?VWWW   +p*o*o*odn*o*o*oM, '  %)<<^LLMr3)rrrrrrrF)r8)r) TNNNrvNNNr8N)r __module__ __qualname____doc__model_input_namesr'rBrstrrJrb staticmethodrfloatrurboolrintr r r __classcell__)r1s@r2rr$sHB**#" " " " " " HSUWU_<  bj # Z\Zd    >be##2:&#8rs#"""""""""""""HHHHHHHHHHIIIIII444444((((((((LLL  H % %wwwww6wwwt % %r3