`i@$ddlZddlmZddlmZmZmZddlZddl m Z m Z e rddl Z e rddl ZddlmZmZddlmZddlmZmZmZmZdd lmZmZGd d ed ZGdded ZGddeZdgZdS)N)Path)AnyOptionalUnion)is_soundfile_availableis_torch_available) AudioInputmake_list_of_audio) BatchFeature) AudioKwargsProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInputc6eZdZUeeeefed<dS)CsmAudioKwargsencoded_length_kwargsN)__name__ __module__ __qualname__rdictstrr__annotations__z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/csm/processing_csm.pyrr%s+#DcN333333rrF)totalcNeZdZUeed<ddddgdgdgddd d d d d idZdS)CsmProcessorKwargs audio_kwargsTleftF)padding padding_sideadd_special_tokens)rrr( rr( rr(r)r(r(r(r-r(r(r(r(r(r(r)r()r(r(r(r(r(r(r(r(r(r(r(r(r(r(r() kernel_sizesstrides dilationsuse_causal_convi])r sampling_ratereturn_tensorspt) text_kwargsr" common_kwargsN)rrrrr _defaultsrrrr!r!)s    ""'  !Q P PHHHJJJ#' && #  +D1IIIrr!c >eZdZdZddgZdZdZ dfd ZeddZ d e d e e e ee e e ffd eefd Z ddee eeeeeefd ee deedeed eef dZedZxZS) CsmProcessora  Constructs a Csm processor which wraps [`EncodecFeatureExtractor`] and [`PretrainedTokenizerFast`] into a single processor that inherits both the audio feature extraction and tokenizer functionalities. See the [`~CsmProcessor.__call__`] for more information. The preferred way of passing kwargs is as a dictionary per modality, see usage example below. ```python from transformers import CsmProcessor from datasets import load_dataset ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") audio = ds[0]["audio"]["array"] processor = CsmProcessor.from_pretrained("sesame/csm-1b") processor( text=["<|begin_of_text|>[0]What are you working on?<|end_of_text|><|AUDIO|><|audio_eos|><|begin_of_text|>[1]I'm figuring out my budget.<|end_of_text|>"], audio=audio, text_kwargs = {"padding": False}, audio_kwargs = {"sampling_rate": 16000}, common_kwargs = {"return_tensors": "pt"}, ) # this should error out because EncodecFeatureExtractor expects a 24kHz audio :) ``` Args: feature_extractor ([`EncodecFeatureExtractor`]): The feature extractor is a required input. tokenizer ([`PreTrainedTokenizer`, `PreTrainedTokenizerFast`]): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. feature_extractor tokenizerEncodecFeatureExtractorPreTrainedTokenizerFastNct|ds'd|_||j|_n|j|_|j|_t|ds'd|_||j|_n|j|_|j|_t |||dS)N audio_tokenz <|AUDIO|>audio_eos_tokenz <|audio_eos|>) chat_template)hasattrrBconvert_tokens_to_idsaudio_token_idrCaudio_eos_token_idsuper__init__)selfr=r>rD __class__s rrJzCsmProcessor.__init__fs y-00 ;*D "+"A"A$BR"S"SD  (4D "+":D y"344 C#2D &/&E&EdFZ&[&[D # ##,#, - - = #l2]BJ$x;?'CCaGFRUVVJJraudio saving_pathkwargsc tstdt|}t|tt fr|g}nDt|t tfrtd|Dstdt|t|krtd|j tfi|}|d}|d}t||D]m\}}t|tjr8|}t'j|||ndS)Nz/Please install `soundfile` to save audio files.c3NK|] }t|ttfV!dSN) isinstancerr).0ps r z*CsmProcessor.save_audio..s3@q@q`aAPSUY{A[A[@q@q@q@q@q@qrzAInvalid input path. Please provide a string, or a list of stringsz5The number of audio and saving paths must be the samer"r5)r ImportErrorr rdrrlisttupleall ValueErrorlen _merge_kwargsr!rNtorchTensorcpufloatnumpysfwrite) rKr^r_r` output_kwargsr"r5 audio_valuerfs r save_audiozCsmProcessor.save_audiosu&'' QOPP P#5)) kC; / / b&-KK[4-88 bS@q@qep@q@q@q=q=q b`aa a u::[)) ) )TUU U**     %^4 $_5 !%55 4 4NK+u|44 @)oo//5577==?? HQ ] 3 3 3 3 4 4rF?text output_labelsdepth_decoder_labels_ratioc  jtfdjji|}|d}|d}|d} | dd} | dkrt jjdt|tr|g}nDt|ttfrtd |Dst d fd |D} d } |t|}t|} t| d kr:| t| kr'|t d t d| d| d||difd|D} | }g}|D]}g}j|vrY|d }j|z}|||jdd}j|vYd|vr.|d|d d}d|v.|||}j|fi|}i}||||ddgg}}d }| D]} | d krP|t+jd |t/jdgX|t+jd|||| zDd|t/jd|||| zDd|| z }j|fi|}|dd||t9d|D fd|D}t/j|d |d<|r|djk}|j d }|d kr8t/j!|dtE|d|z z}||}n|}t/j#|djk|dj$kz|dd!}d"||ddd f|dddff<||d#<tK|| $S)%a Main method to prepare text(s) and audio to be fed as input to the model. This method forwards the `text` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] to encode the text. To prepare the audio, this method forwards the `audio` arguments to EncodecFeatureExtractor's [`~EncodecFeatureExtractor.__call__`]. Please refer to the docstring of the above two methods for more information. Args: audio (`np.ndarray`, `torch.Tensor`, `list[np.ndarray]`, `list[torch.Tensor]`): The audio or batch of audio to be prepared. Each audio can be a NumPy array or PyTorch tensor. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). output_labels (bool, *optional*, default=False): Whether to return labels for training. Indices will be in `[config.audio_token_id, -100, -101]`. - `config.audio_token_id` indicates an audio frame (considering sequence length elements as frames) - `-100` will be ignored in the loss computation - `-101` indicates the audio frame will be used only for the backbone model (using the first codebook token as labels) depth_decoder_labels_ratio (float, *optional*, default=1.0): The ratio of audio frames to keep for the depth decoder labels. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **input_values** -- List of audio values to be fed to a model. Returned when `audio` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **labels** -- List of labels for the audio frames. Returned when `output_labels=True`. tokenizer_init_kwargsr8r"r9r6Nr7z% only supports `return_tensors='pt'`.c3@K|]}t|tVdSrc)rdr)rets rrgz(CsmProcessor.__call__..s-9[9[QR*Q:L:L9[9[9[9[9[9[rzAInvalid input text. Please provide a string, or a list of stringscDg|]}|jSr)countrB)rerrKs r z)CsmProcessor.__call__..s(CCC1774#344CCCrrz@No audio were provided, but there are audio tokens in the promptz)The number of audio tokens in each text (z7) should be the same as the number of provided audios (z).rcBg|]}j|jdfiS)r]shape)re audio_arrayrrKs rrz)CsmProcessor.__call__..sE%%%]h(():2)>XXBWXX%%%rz r(return_attention_maskrcg|]D}t|tjr&|n|ESr)rdrorprqrsreels rrz)CsmProcessor.__call__..=sQ$&5?r5<4P4P X 0 0 0VXr)axisc(g|]}|jdSrrrs rrz)CsmProcessor.__call__..Es%^%^%^rbhrl%^%^%^r)dim padding_maskc30K|]}|jdVdS)rNr)recut_idxss rrgz(CsmProcessor.__call__..Ns)RR(.,RRRRRRrc ~g|]9}tjj|d|jdz fd:S)rr)value)ronn functionalpadr)rermax_lens rrz)CsmProcessor.__call__..OsU$$$#''1gr@R6R2S[]'^^$$$rinput_values_cutoffs input_idsryiilabels)data tensor_type)&rnr!r> init_kwargspoprlrLrrdrrirjrkr rmsumcopyrBappendreplaceupdatenpzerosrotensor concatenatecumsumr=maxstackrGnonzerorrandpermintwhererHr )!rKrzr^r{r|r`rvr8r"r9r6n_audio_in_textn_audionum_audio_tokens_listnum_audio_tokens_list_copy expanded_textsample replace_strnum_audio_tokensexpanded_audio_tokenencodingrconcatenated_audioroffset audio_inputsaudio_frame_idxsn_audio_frames rand_idxsskip_frames_idxsrrrs!` @@r__call__zCsmProcessor.__call__s^+*   "&."<   $M2 $^4 %o6 &**+;TBB T ! ! 7^^^__ _ dC  b6DDTD%=11 bc9[9[VZ9[9[9[6[6[ b`aa aCCCCdCCC  &u--E%jjG   ! # #33G3G(G(G} !cddd >>>29>>>  $0$4$45Lb$Q$Q !%%%%%lq%%% !*?)C)C)E)E &M - - &&00'A'E'Ea'H'H$+/+;>N+N(&&';<<<#^^D,$66+66 H     4d ; ; ;792 4 F* & &a<<&--bhqkk:::(// bT0B0BCCCC&--*/'9I0I*J"$ )// %^%^U6FU\L\C\=]%^%^%^__ffkmfnng%FF1412DUU UUL   ^T 2 2 2 KK % % %RR=QRRRRRG$$$$ 4$$$ ,1;7KQR+S+S+SD' (  $ $[ 1T5H HQQSS -3A6N)S00!N>::;sSSTWqSqAr=s=s;st #3I#>  #3 [k"d&99d;>OSWSj>jk[!F FJF#AAAqD)+;AAAqD+AA B#DN>BBBBrcv|jj}|jj}d|D}t||zdgzS)Ncg|] }|dk| S)rr)renames rrz2CsmProcessor.model_input_names..qs$(r(r(r$[_cq[q[q[q[q[qrr)r>model_input_namesr=ri)rKtokenizer_input_namesfeature_extractor_input_namess rrzCsmProcessor.model_input_namesjsO $ @(,(>(P%)s(r:W(r(r(r%),IIMcLddeeerrc)NNNN)NFry)rrr__doc__ attributesfeature_extractor_classtokenizer_classrJ staticmethodr]r rrrrirr!rxrrrboolrrrpropertyr __classcell__)rLs@rr<r<>s!!F&{3J7/O  TTTTTT,###\#J 4 43d5d+;&<<= 4+, 4 4 4 4J'+(-69 dCdCuY(94 ?DQbLccdedC #dC ~ dC %-UO dC +, dCdCdCdCLffXfffffrr<) rOpathlibrtypingrrrrsrutilsrr ro soundfilert audio_utilsr r feature_extraction_utilsr processing_utilsr rrrtokenization_utils_baserrrr!r<__all__rrrrs  ''''''''''????????LLL99999999444444UUUUUUUUUUUUCCCCCCCC44444[4444)*tftftftftf>tftftfn  r