*`iN@nddlZddlZddlmZddlmZddlZddlm Z ddl m Z m Z m Z ejeZdZdeded dfd ZGd d eeZeGd dZeGddZeGddZeGddZGddZdS)N) dataclass)Enum)Audio) AudioChunk AudioURLChunk AudioURLType num_samplesmult_ofreturnc8||zdksJd|d|dS)Nrz num_samples= must be a multiple of mult_of=)r r s z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/audio.py_check_mult_ofrs7  A % % %'Y+'Y'Yw'Y'Y % % % % %ceZdZdZdZdZdS)TranscriptionFormatzTranscription format. Should be set by the tokenizer for correct encoding. Attributes: - INSTRUCT: The instruct format. - STREAMING: The streaming format. instruct streamingN)__name__ __module__ __qualname____doc__INSTRUCT STREAMINGrrrrrs$HIIIrrc:eZdZUdZeed<eed<eed<ddZdS) AudioSpectrogramConfiga^Configuration for generating an audio spectrogram. Attributes: num_mel_bins: Number of mel bins, typically 80 or 128. hop_length: Length of the overlapping windows for the STFT used to obtain the Mel Frequency coefficients, typically 160. window_size: Window size of the Fourier transform, typically 400. num_mel_bins hop_length window_sizer Nc|jdksJ|j|jdksJ|j|jdksJ|jdSNr)rr r!selfs r __post_init__z$AudioSpectrogramConfig.__post_init__9sc 1$$$d&7$$$"""DO"""!###T%5#####rr N)rrrrint__annotations__r&rrrrr'sWOOO666666rrceZdZUdZeed<eed<eed<dZedzed<e j Z e ed<dZ edzed<dZ edzed <dZedzed <dZedzed <dd Zed efdZded efdZed efdZed efdZed efdZed efdZed efdZed efdZdS) AudioConfiga`Configuration for audio processing. Attributes: sampling_rate: Sampling rate of the audio. frame_rate: Number of frames per second accepted by the tokenizer model. encoding_config: Configuration for audio spectrogram. chunk_length_s: Whether to pad an audio into multiples of chunk_length_s seconds (optional). sampling_rate frame_rateencoding_configNchunk_length_stranscription_formattranscription_delay_msstreaming_look_ahead_msstreaming_look_back_msstreaming_n_left_pad_tokensr cr|jdksJ|j|jdksJ|j|j?|jdksJ|j|jdksJd|jd|j|jsh|jJd|jd|jJd|jd|jJd|jd|jJd|jd|jr|jJd|jd |jJd|jd |jJd|jd |jJd|jd d |jz }|jdks Jd |j|zdksJd|jd ||jJd |jddSdS)Nrz7chunk_length_s and sampling_rate must both be > 0, got z and zself.transcription_delay_ms=z must be None.zself.streaming_look_ahead_ms=zself.streaming_look_back_ms=!self.streaming_n_left_pad_tokens= must be set.@@z*{self.transcription_delay_ms=} must be > 0z) must be a multiple of frame_duration_ms=zself.chunk_length_s=z cannot be set in streaming.) r-r,r/ chunk_frames is_streamingr1r2r3r4)r%frame_duration_mss rr&zAudioConfig.__post_init__]s"""DO"""!A%%%t'9%%%   *&***D,?***$q(((x$J]xxdhdvxx)((  r.668g4;V8g8g8g666/779iDs rnum_audio_tokenszAudioConfig.num_audio_tokenssb t+6 6! ; ; )d.B.M"MPQ"QRRII!T%9%DDIyT%>>???rc|jsJd|jd|jJd|jdt|jdz |jz}||S)Nz1Can't call num_delay_tokens if self.is_streaming=.z;Can't call num_delay_tokens if self.transcription_delay_ms=r8)r:r1r(r,rE)r% delay_lens rnum_delay_tokenszAudioConfig.num_delay_tokenss YY"YDDU"Y"Y"YYY *66 Md.I M M M7663f [["[dFW"["["[[[ %)-LLLrcx|jsJd|jd|jJd|jd|jS)Nz2Can't call n_left_pad_tokens if self.is_streaming=rGr6r7)r:r4r$s rn_left_pad_tokenszAudioConfig.n_left_pad_tokenssY ZZ"ZTEV"Z"Z"ZZZ /;;=p@`=p=p=p;;;//rr')rrrrr(r)rNrr/rrr0r1r2r3r4r&propertyboolr:rErIr9rLrDrRrTrrrr+r+?s%++++#'NEDL'''1D0L-LLL,0EDL///-1UT\000+/EDL///.2t222ffff>JdJJJXJ@#@#@@@@0#000X0=c===X= :#:::X:&c&&&X& MCMMMXM03000X000rr+c4eZdZUdZeeed<eed<dS) AudioEncodingzEncapsulates the tokens and audio data for an audio chunk. Attributes: tokens: Text tokens corresponding to this audio chunk. audio: Original audio waveform data. tokensaudioN)rrrrlistr(r)rrrrrXrXs5 I LLLLLrrXcDeZdZUdZedzed<edzed<edzed<dS)SpecialAudioIDsaSpecial text tokens corresponding to audio token sequence. Attributes: audio: Token representing audio. begin_audio: Token representing the beginning of audio. streaming_pad: Token representing streaming pad of audio. Only relevant for steaming models. NrZ begin_audio streaming_pad)rrrrr(r)rrrr]r]sN :t:rr]c\eZdZdZdededdfdZdejde d e dejfd Z d e d e de e e ffd Z d e de de fdZdee fdZde dee fdZded e defdZdedefdZdedefdZdeezdefdZede fdZede fdZede fdZdS) AudioEncodera Encodes audio chunks into a format suitable for further processing. Attributes: audio_config: Configuration for audio processing. encoding_config: Configuration for audio spectrogram. special_ids: Special tokens for audio encoding. audio_config special_idsr Nc:||_|j|_||_dSr=)rbr.rc)r%rbrcs r__init__zAudioEncoder.__init__s#(+;&r audio_arrayr,is_online_streamingc|jjrG||jd|}t j|d||jdz f}n|jjr<||jd|\}}t j|||f}ndt|j trJ|jd|j j kr/t j|d|j j |jdz f}|S)zPad the audio array to the desired length. Args: audio_array: Audio data as a numpy array. sampling_rate: Sampling rate of the audio. Returns: Padded audio array. r) rbr/next_multiple_of_chunk_framesshapenppadr:_get_streaming_pad isinstancer.rr!)r%rfr,rgrjleft_pad right_pads rrmzAudioEncoder.pads   + m,0,N,N{O`acOdfs,t,t )&q2OR]RcdfRg2g.hiiKK   + m"&"9"9+:KB:OQd"e"e Hi&x.CDDKK t+-C D D m!"%(<(HHH&q$2F2RU`UfgiUj2j.kllKrr c|jj}|rd}nKt|||zz |z}|jj}t||z}t ||||z }|jj}t||z}t ||||fSr#)rbrLr(rRrrT) r%r rgr rq_extra_right_pad_tokens_extra_right_pad_samples_extra_left_pad_tokensrps rrnzAudioEncoder._get_streaming_pads#<  2IIW g(=>'IJJI&*&7&J #'*75L+L'M'M $ 3W = = = 1 1I "&!2!Dw!7788x)))""raudio_array_lenc||jjksJd|d|jj|jjJd|jjdtj||jjz |jjzS)zCalculate the next multiple of chunk frames. Args: audio_array_len: Length of the audio array. sampling_rate: Sampling rate of the audio. Returns: The next multiple of chunk frames. zExpected sampling_rate=z' to be self.audio_config.sampling_rate=NzMCan't call next_multiple_of_chunk_frames if self.audio_config.chunk_length_s=rG)rbr,r/rBrCr9)r%rvr,s rrjz*AudioEncoder.next_multiple_of_chunk_framess 1 ???? Q Q Qt/@/N Q Q@?? /;; _4;L;[ _ _ _<;;y4+<+IIJJTM^Mkkkrct|jjtsJd|jj|jjJ|jj|jjz}|jg|z}|S)NzQAudio encoder must be spectrogram encoder, got self.audio_config.encoding_config=)rorbr.rr1rTrIr_)r%stream_pad_prefix_lenrYs r_encode_streaming_tokensz%AudioEncoder._encode_streaming_tokens,s$+;=STT  bd>O>_ b b  T 7CCC!% 1 CdFWFh h$%(== r signal_lengthc||jjzdkr%tj||jjz dz }n||jjz}tj||jjz }|jg|jg|zz}|Sr@)r.r rBrCrbrDbegin_audio_token audio_token)r%r{rErYs r_encode_audio_tokensz!AudioEncoder._encode_audio_tokens8s 4/: :a ? ? Imd6J6U&UXY&YZZMM)T-A-LLM9]T5F5[%[\\()T-=,>AQ,QQ rrZc<||jj|rlt|jjjdz |jjjz }|jjd|zdks Jd|jjdd|| |j|jj||_|jj tj kr| }n%||jjd}t||S)Nrirzaudio.audio_array.shape[-1]=r)rYrZ)resamplerbr,absr.r!r rfrkrmr0rrrzrrX)r%rZrgr rYs r encode_audiozAudioEncoder.encode_audioDs" t(6777  !1=ADDUDeDppG$*2.8A===R5$*2.RRRR>==!HHU%68I8WYlmm   15H5R R R2244FF..u/@/Fq/IJJF    rcontentcbtj|j}||dSNF)rg)rfrom_raw_audio input_audior)r%rrZs r_encode_audio_chunkz AudioEncoder._encode_audio_chunk^s.$W%899  E BBBrcF|}|tjtjhvrt j|j}nC|tjkrt j|j}nt j|j}| |dSr) get_url_typerfilefile_urir from_fileurlfrom_url from_base64r)r%rurl_typerZs r_encode_audio_url_chunkz$AudioEncoder._encode_audio_url_chunkbs''))  )<+@A A AOGK00EE ) ) )N7;//EE%gk22E  E BBBrct|tr||St|tr||St dt |)zCall the encoder on an audio chunk or URL chunk. Args: content: Audio or URL chunk to encode. Returns: Encoded audio data and tokens. zUnsupported content type: )rorrrr ValueErrortype)r%rs r__call__zAudioEncoder.__call__nso g} - - K//88 8  , , K++G44 4I$w--IIJJ Jrcb|jjJd|jjd|jjS)zGet the audio token.Nzself.special_ids.audio=r7)rcrZr$s rr~zAudioEncoder.audio_token~s<%113\d6F6L3\3\3\111%%rcb|jjJd|jjd|jjS)zGet the begin audio token.Nzself.special_ids.begin_audio=r7)rcr^r$s rr}zAudioEncoder.begin_audio_tokens<+779hDN>\;l;l;l999--r)rrrrr+r]rerlndarrayr(rVrmtuplernrjr[rzrrrXrrrrrrrUr~r}r_rrrrarasG'[''SW'''' rz#TX]_]g4#c##QVWZ\_W_Q`####4lSlQTlY\llll& $s)     # $s)     % d }    4C:C-CCCC C} C C C C CK ] :K}KKKK &S&&&X& ,3,,,X, .s...X...rra)loggingrB dataclassesrenumrnumpyrlmistral_common.audior&mistral_common.protocol.instruct.chunkrrr getLoggerrloggerrQr(rstrrrr+rXr]rarrrrs !!!!!!&&&&&&ZZZZZZZZZZ  8 $ $#%ZZcZdZZZZ     #t    6666666 6. v0v0v0v0v0v0v0 v0r                    w.w.w.w.w.w.w.w.w.w.r