`iddlmZddlmZmZmZddlZddlmZddl m Z ddl m Z ddl mZmZddlmZdd lmZdd lmZdd lmZdd lmZmZdd lmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.m/Z/ddl0m1Z1e(j2e3Z4ee&dGdde$Z5edGddej6Z7Gddej6Z8Gd d!ej6Z9d"Z:dHd#Z;d$ej<d%e=d&ej<fd'Z> dId)ej6d*ej<d+ej<d,ej<d-eej<d.e?d/e?d0e"e%fd1Z@Gd2d3ej6ZAGd4d5eZBe&d6e&Gd7d8e ZCe&Gd9d:eCZDGd;dd?eCeZFGd@dAej6ZGe&GdBdCeCZHe&dDGdEdFeCe1ZIgdGZJdS)J) dataclass)CallableOptionalUnionN)check_model_inputs)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging)deprecate_kwarg) AutoModel) CsmConfigCsmDepthDecoderConfig)CsmGenerationMixinz: Base class for the model autoregressive outputs. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejdfed<dZee ejdfed<dZeejed <dZeejed <dZee ed <dZee ejdfed <dZee ejdfed <dZeejed<dS)CsmOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. depth_decoder_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the depth decoder model. depth_decoder_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the depth decoder (scores for each vocabulary token before SoftMax). depth_decoder_past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). depth_decoder_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the optional initial embedding outputs. depth_decoder_attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each layer) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. backbone_loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction) of the backbone model. Nlosslogitspast_key_values. hidden_states attentionsdepth_decoder_lossdepth_decoder_logitsdepth_decoder_past_key_valuesdepth_decoder_hidden_statesdepth_decoder_attentions backbone_loss)__name__ __module__ __qualname____doc__r&rtorch FloatTensor__annotations__r'r(r r)tupler*r+r,r-r.r/r0x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/csm/modeling_csm.pyr%r%2sK8)-D(5$ %,,,*.FHU& '...'+OXe_+++=AM8E%"3S"89:AAA:>Ju0#567>>>6:!23:::8<(5#45<<<59!8E?999KO%0A30F*G!HOOOHLhuU->-C'DELLL15M8E-.55555r:r%RMSNormc,eZdZdfd ZdZdZxZS) CsmRMSNormư>cttjt j||_||_dS)z9 CsmRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parameterr5onesweightvariance_epsilon)self hidden_sizeeps __class__s r;rBzCsmRMSNorm.__init__dsD l5:k#:#:;; #r:c|j}|tj}|ddd}|tj||jzz}|j||zS)NrT)keepdim) dtypetor5float32powmeanrsqrtrGrF)rHr) input_dtypevariances r;forwardzCsmRMSNorm.forwardls|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r:cHt|jjd|jS)Nz, eps=)r8rFshaperGrHs r; extra_reprzCsmRMSNorm.extra_reprss& )**II$2GIIIr:)r?)r1r2r3rBrWr[ __classcell__rKs@r;r>r>bsb$$$$$$;;;JJJJJJJr:r>c|eZdZUejed<ddeffd Zeje dZ xZ S)CsmRotaryEmbeddinginv_freqNconfigctt|drSt|jt r9|jd|jd|_nd|_|j|_ |j|_ ||_ t|j|_ | |j |\}|_|d|d|j|_dS)N rope_scaling rope_typetypedefaultr`F persistent)rArBhasattr isinstancercdictgetrdmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenrar rope_init_fnattention_scalingregister_bufferr`original_inv_freq)rHradevicer`rKs r;rBzCsmRotaryEmbedding.__init__zs  6> * * 'z&:Mt/T/T '#044[&BUBYBYZ`BaBabbDNN&DN"("@$*$B! /?+/+<+>@@GG HZ[\H]_acdeehhijiqrr ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk ^ U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;sBE++E/2E/N) r1r2r3r5Tensorr7r rBno_gradrrWr\r]s@r;r_r_wsl//y//////"U]__ < <_ < < < <