`i|xddlmZddlmZmZddlZddlmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZd dlmZd dlm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*ddl+m,Z,ej-e.Z/eedGddeZ0Gdde%Z1Gdde&Z2Gdde#Z3Gdde Z4Gdd e!Z5ed!eGd"d#eZ6eGd$d%e$e6Z7Gd&d'ej8Z9ed(Gd)d*e"eZ:Gd+d,ej8Z;eGd-d.e$Z<ed/Gd0d1e6e,Z=gd2Z>dS)3) dataclass)OptionalUnionN)check_model_inputs)Cache DynamicCache)GenerationMixin)create_causal_mask)BaseModelOutputWithPastCausalLMOutputWithPast)PreTrainedModel)Unpack) ModelOutputauto_docstringcan_return_tuplelogging) AutoModel)LlamaAttentionLlamaDecoderLayerLlamaForCausalLMLlamaMLP LlamaModel LlamaRMSNormLlamaRotaryEmbeddingTransformersKwargs) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejdfed<dZee ejdfed<dZeejed <dZeejed <dZee ed <dZee ejdfed <dZee ejdfed <dZeejed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r%rtorch FloatTensor__annotations__r&r'rr(tupler)r*r+r,r-r.r/w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/csm/modular_csm.pyr$r$1sK8)-D(5$ %,,,*.FHU& '...'+OXe_+++=AM8E%"3S"89:AAA:>Ju0#567>>>6:!23:::8<(5#45<<<59!8E?999KO%0A30F*G!HOOOHLhuU->-C'DELLL15M8E-.55555r9r$ceZdZdS) CsmRMSNormNr0r1r2r8r9r:r<r<bDr9r<ceZdZdS)CsmRotaryEmbeddingNr=r8r9r:r@r@fr>r9r@ceZdZdS)CsmMLPNr=r8r9r:rBrBjr>r9rBceZdZdS) CsmAttentionNr=r8r9r:rDrDnr>r9rDceZdZdS)CsmDecoderLayerNr=r8r9r:rFrFrr>r9rFz[ The bare Csm Model outputting raw hidden-states without any specific head on top. cXeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ e edZfdZxZS)CsmPreTrainedModelconfigmodelTrFr')r(r)ct|t|trM|j}t |dz D]5}|jj|d|j j 4dSdS)Nrg)meanstd) super _init_weights isinstanceCsmCodebooksHead num_codebooksrangeweightdatanormal_rIinitializer_range)selfmodulerRi __class__s r:rOz CsmPreTrainedModel._init_weightss f%%% f. / / ["0M=1,-- [ [ "1%--3DKAR*RU[Upqq')y1LfN`gl'm'm'm$$$r9N input_idsbackbone_last_hidden_stateattention_mask position_idsr' inputs_embeds use_cachecache_positionkwargsreturnc |:tjstdd}|du|duz rt d|r|t |j}|^||nd} | |j dn |j d} ||j n|j } tj | | | z| }|tj |dz d} | |j z}|||z}|ddk}| ||dddf<n:tjs|rtd ||}t#|j||||| }|}|d}|||}|jd|jjD]}||f||||||d | }||}t/||r|nd S) aJ backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. NzCustom `position_ids` were provided but will be ignored. CSM depth decoder automatically determines position_ids from `cache_position` and as it requires them to be identical across the batch, the provided position_ids will be ignored.z;You must specify exactly one of input_ids or inputs_embeds.)rIrrdevice)minzvWhen the first codebook token is provided, `backbone_last_hidden_state` should also be provided for correct inference.)rI input_embedsrxr|r'ry)rxryr'r{r|position_embeddings)last_hidden_stater')r4compiler is_compilinglogger warning_once ValueErrorr rIget_seq_lengthshaperarangeclamprorqwarningrtr unsqueeze rotary_emblayersnum_hidden_layersnormr )rXrvrwrxryr'rzr{r|r}past_seen_tokensinputs_seq_lengthr codebook_idxsoffsetinput_ids_are_first_codebook causal_maskr(r decoder_layers r:forwardzCsmDepthDecoderModel.forwards&  #EN,G,G,I,I #   M    L  -t"; < \Z[[ [  ?0*$+>>>O  !CRC^==???de :G:S 3A 6 6YbYhijYk -:-F]))IL\F"\*:$?R !. R u/0 R "% R   12R D>R !!12R +,R  u-- .R R R ^R R R R R r9rhc&eZdZfdZddZxZS)rQct||_tjt j|jdz |||_dS)Nr)rNrlrRrm Parameterr4emptyrT)rXrsrRror[s r:rlzCsmCodebooksHead.__init__sM *l5;t/AA/E{T^#_#_`` r9Nc|-jd}|jtj|n|dz }|j|fdt jdDtjdS)Nrc g|]:}tjdd|ddf|j;SN)rm functionallinearT).0 codebook_idxcodebook_weightr(s r: z,CsmCodebooksHead.forward..sX    M qqq,/A!BOT`DaDc d d   r9rdim)rrTr4rrSstack)rXr(r| seq_lengthrrs ` @r:rzCsmCodebooksHead.forwards  !&,Q/J"k%,z*B*BCOO*Q.M"k-8O     %o&;A&> ? ?    Mq999 r9rr0r1r2rlrrerfs@r:rQrQsQaaaaa r9rQa$ The CsmDepthDecoder Model transformer, with a [`CsmCodebooksHead`] on top, which can be seen a position-specific language modeling head, allowing to use a different linear layer for each codebook (e.g. position 0 is the first codebook and uses the first codebook head, etc.) c(eZdZdZdZdZfdZ ddejde e de ejde ej de ejf fd Z e e dde ejd e ej de ejd e ejde ee eej fde ej d e ejd e ede ejdeeejfdeedeeeffdZxZS)CsmDepthDecoderForCausalLMNct||`t|j|j|j|_t||_ dSr) rNrllm_headrQrsrRrocodebooks_headrhrJrus r:rlz#CsmDepthDecoderForCausalLM.__init__sQ    L.v/A6CWY_Yjkk)&11 r9rvr'rxrzr|c tj|||||fi|}|dddk}|s|d|d|S)Nr|rrwry)rNprepare_inputs_for_generationpop) rXrvr'rxrzr|r} model_inputsis_first_generation_stepr[s r:rz8CsmDepthDecoderForCausalLM.prepare_inputs_for_generations=uww<  ~  Y_  $00@#A!#D#I ' ;   9 : : : (((r9rrwrylabelsr{logits_to_keepr}r~c |jd|||||||| d| } | d} t| tr)| dkrtdd}nt| d}n| }|| dd|ddf| | |nd}|}d}|:|dddf}|jd|d|jj|d| }t||| j | j | j S) a backbone_last_hidden_state (`torch.FloatTensor` of shape `(batch_size, backbone_hidden_size)`, *optional*): The last hidden state of the backbone model. Such input is required when the first codebook token (the one generated by the backbone model) is provided in the `input_ids` argument. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. )rvrwrxryr'rzr{r|rrN.)r&rro shift_labels)r%r&r'r(r)r8) rJrPintslicer contiguous loss_functionrIror r'r(r))rXrvrwrxryr'rzrr{r|rr}outputsr( slice_indicesr&r%rs r:rz"CsmDepthDecoderForCausalLM.forward5sr2$*  'A)%+')        nc * * +"" %a %~ot < < *M$$ !!!]AAA- .Q_Qk}0M0Mqu  ""$$  !#qrr'?5577L%4%dt{7M\hlrD&#3!/)     r9NNNN) NNNNNNNNNr)r0r1r2_tied_weights_keys_tp_plan_pp_planrlr4rrrr5rrrrrlistrrrrr7r rrerfs@r:rr sHH22222,0595959 #"%!!12    12  !!12 ,15BF1537KO59-1$(5934@ @ E,-@ %-U->$?@ !. @ u/0 @ "%tE4E/F(F"GH @   12@ )*@ D>@ !!12@ c5</0@ +,@  u,, -@ @ @ ^@ @ @ @ @ r9rc$eZdZfdZdZxZS)CsmBackboneModelEmbeddingscttj|j|jz|j|_|dtj |j|jzddS)Naudio_tokens_offsetsF) persistent) rNrlrmrnrRrorsembed_audio_tokensregister_bufferr4rrus r:rlz#CsmBackboneModelEmbeddings.__init__{s "$,0DvGX0X[a[m"n"n  "EL1E$F$FIZ$Zgl      r9cl|||jz}|d}|S)Nrr)rrsum)rXrvrs r:rz"CsmBackboneModelEmbeddings.forwards9..y4;T/TUU #''A'.. r9rrfs@r:rrzsG     r9rcHeZdZfdZeefdZxZS)CsmBackboneModelcrt|t||_dSr)rNrlrrqrus r:rlzCsmBackboneModel.__init__s1    6v>>r9c 6tjdi|S)a& input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length, num_codebooks) or (batch_size, sequence_length)`): 1. (batch_size, sequence_length): corresponds to the input sequence prepared with the processor from the text prompt. Such input requires `input_values` to be provided so that audio can be encoded in codebook tokens and then merged with the text tokens. 2. (batch_size, sequence_length, num_codebooks): codebook tokens generated during the autoregressive decoding. Such input is not meant to be used by end users. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) r8)rNr)rX super_kwargsr[s r:rzCsmBackboneModel.forwards!uww.....r9)r0r1r2rlrrrrerfs@r:rrsh????? / / / /^ / / / / /r9rz The Csm model consists of two llama-like auto-regressive transformer models: a backbone model that predicts the first codebook token and a depth decoder that predicts the other codebook tokens. c eZdZddgZfdZdZdZdZefdZ fdZ dd e e j d e e j d e e j d e e j de e j f dZ dd e jde ede e jde e jde e jf fd Zee dd e e jd e e j de e j d e e j de e jde eeee jfde e jd e e jde ede e jdeee j fdeedeeeffdZxZS)CsmForConditionalGenerationz5backbone_model.embed_tokens.embed_audio_tokens.weightz'depth_decoder.model.embed_tokens.weightct||j|_tj|j|jd|_tj|j|j|_ t ||_ t |j|_t!j|j|_|dSrj)rNrlrormrrrsrrntext_vocab_sizeembed_text_tokensr _from_configbackbone_modelrdepth_decoder_config depth_decoderr from_config codec_config codec_model post_initrus r:rlz$CsmForConditionalGeneration.__init__s     +y!3V5FUSSS !#f.DfFX!Y!Y.;;FCC7DDVE`aa$01DEE r9c|jjSrrrqrXs r:get_input_embeddingsz0CsmForConditionalGeneration.get_input_embeddingss"//r9c||j_dSrr)rXvalues r:set_input_embeddingsz0CsmForConditionalGeneration.set_input_embeddingss+0(((r9c|jjr6||jjj|jjjdSdSr)rItie_codebooks_embeddings_tie_or_clone_weightsrrqrrrJrs r: _tie_weightsz(CsmForConditionalGeneration._tie_weightssS ; /   & &#0C"(5       r9c|ddrtj|i|\}}ntj|i|}dtfdt |jD}t |jjddi||D]}t|j|zd|vr||fS|S)Noutput_loading_infoFdepth_decoder_cVi|]%\}}||d|&Sr) startswith)rattrrprefix prefix_lens r: z?CsmForConditionalGeneration.from_pretrained..sJ   ev&&  u   r9_from_model_config) getrNfrom_pretrainedlenvarsgeneration_configitemsrupdatedelattr) clsargsr}rJ loading_infodepth_decoder_attrsrrrr[s @@r:rz+CsmForConditionalGeneration.from_pretraineds' ::+U 3 3 ="9%''"94"J6"J"J E<<+EGG+TzQCsmForConditionalGeneration._merge_input_ids_with_input_values..s(&O&Orrx{&O&O&O&O&O&Or9c tg|]4}tj|ddd|jdz f5S)r)rmrpadr)rrmax_audio_framess r:rzRCsmForConditionalGeneration._merge_input_ids_with_input_values..sBrrrZ\R]&&rAq!5EQR 5S+TUUrrrr9)rdtypeiTas_tuple)rzr)$rrmrrdiffr4rmaxrexpandrrno_gradziprSrrencode audio_codes transposeappendrget_audio_codes_maskrIaudio_token_idrrqonesrRlongcodebook_eos_token_idsqueezeaudio_eos_token_idrepeatrnonzero)rXrvr rrrz audio_lengthsinput_values_maskaudio_tokens_listbatch_input_valuesbatch_input_values_cutoffsrZ start_idxend_idx audio_batch codec_outputs codebook_idsbatched_audio_token_idsaudio_codes_maskr&audio_token_mask audio_embedsaudio_eos_frame_idsaudio_eos_embedsaudio_eos_token_masklabels_expanded depth_decoder_ignore_frames_idxsrs @r:"_merge_input_ids_with_input_valuesz>CsmForConditionalGeneration._merge_input_ids_with_input_valuess*..y99  ##%=#4#45I6#R#R 01E1JKPPRRM)-!*;q$A "O9O&P (,(8(?(? @U@UVW@X@X(Y(Y '4'@'J'J1b'Q'Q )00aAAAA B$'&O&O=N&O&O&O#O#O */+rrrr`qrrr++'$(#3#H#HIZ#[#[ ! \ \ \ \ \ \ \ \ \ \ \ \ \ \ \$"[7N(N: .;;>> import torch >>> from transformers import CsmForConditionalGeneration, AutoProcessor >>> from datasets import load_dataset, Audio >>> model_id = "sesame/csm-1b" >>> torch_device = "cuda" if torch.cuda.is_available() else "cpu" >>> processor = AutoProcessor.from_pretrained(model_id) >>> ds = load_dataset("hf-internal-testing/dailytalk-dummy", split="train") >>> # ensure the audio is 24kHz >>> ds = ds.cast_column("audio", Audio(sampling_rate=24000)) >>> conversation = [] >>> # prepare a conversation with text and corresponding audio >>> for text, audio, speaker_id in zip(ds[:4]["text"], ds[:4]["audio"], ds[:4]["speaker_id"]): ... conversation.append( ... { ... "role": f"{speaker_id}", ... "content": [{"type": "text", "text": text}, {"type": "audio", "path": audio["array"]}], ... } ... ) >>> inputs = processor.apply_chat_template( ... conversation, ... tokenize=True, ... return_dict=True, ... output_labels=True, ... ).to(torch_device) >>> model = CsmForConditionalGeneration.from_pretrained(model_id, device_map=torch_device) >>> output = model(**inputs) >>> output.loss.backward() ```Nrrzr)rvrxryr'rzr{r|r)r&rrorrrr.r)rTr)rvrwr{ return_dictr) r%r/r*r&r'r(r)r+r,r-r.r8)rCrArrPrrrrrIroallrRrmrrr-rr%r$r'r(r)r&)rXrvr rxrryr'rzrr{r|rr}rDbackbone_outputsbackbone_hidden_statesrbackbone_logitsr%r/r*depth_decoder_outputsbackbone_labels train_maskdepth_decoder_input_ids train_idxsbackbone_last_hidden_statesdepth_decoder_labelss r:rz#CsmForConditionalGeneration.forwardXsf  Y^q%8%8 CC<)=vM*/:M"8,FI.4.  )%+')      "2!!48B>SV8W8Wk~ot444]k ,,'=aaaPQPQPQ>Q'RSS ! $  $QQQ1WoO.D.&4;KaekM "!!!QQQ(+t388R8@@@J&,Z&8>] @Y\]@]>]9]&^ #&(m&7&78OQW_`&7&a&a ##++T+::J*@APZ[\P]`aPacdcdcdAd*e '#)*#5 $6D$6%1+F# + %%  %% !"7!;  #55D '1",<*8'2AVAb!6!=!=hl$0+@*O*O$0)>(K(KI^Ij%:%E%Ept    r9r) NNNNNNNNNNr)r0r1r2rrlrrr classmethodrr rr4rrArrr5rrrrrrrrrr7r$rrerfs@r:rrs! @1 000111[411111-1/37;)- PBPBEL)PBu|,PB'u|4 PB & PB %,  PBPBPBPBj,0595959 #"%!!12    12  !!12 >15/3157;37KO59-1$(5934[ [ E,-[ u|,[ !. [ 'u|4 [ u/0 [ "%tE4E/F(F"GH[   12[ )*[ D>[ !!12[ c5</0[ +,[  u'' ([ [ [ ^[ [ [ [ [ r9r)rHrrhrr)? dataclassesrtypingrrr4torch.nnrmtransformers.utils.genericr cache_utilsrr generationr masking_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrautorllama.modeling_llamarrrrrrrrconfiguration_csmrr generation_csmr! get_loggerr0rr$r<r@rBrDrFrHrhModulerQrrrr__all__r8r9r:res "!!!!!"""""""" 999999........))))))//////OOOOOOOO------&&&&&&KKKKKKKKKKKK                    @???????......  H % %  '6'6'6'6'6 '6'6   '6V             -        X        >        '    [[[[[[[  [4\ \ \ \ \ :'9\ \ \ ~ry.c c c c c !1?c c c L        /////z///. P P P P P "46HP P  P f    r9