`i/2:dZddlmZddlmZddlmZmZerddlm Z m Z m Z m Z m Z mZmZmZdZnBddlmZed Z ed Z ed Z ed Z ed Z edZdededefdZdZejeZGddeZdgZdS)zxLSTM configuration.)Optional)PretrainedConfig)is_xlstm_availablelogging)BackendModeTypeChunkwiseKernelType DtypeTypeSequenceKernelTypeStepKernelTypeWeightModeTyperound_up_to_next_multiple_ofxLSTMLargeConfigT)Literal)traintrain_with_padding inference)chunkwise--native_autogradzparallel--native_autograd)float32bfloat16float16native_sequence__nativenative)singlefusedx multiple_ofreturnc8t||zdz |z|zS)z0Rounds up x to the next multiple of multiple_of.)int)rrs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/xlstm/configuration_xlstm.pyrr2s#Q_q([8KGHHHFcAleZdZdZdZ dBdededeedeed eed!ed"ed#ed$ed%ed&ed'ed(ed)e d*e d+e d,e d-ed.ed/e d0ed1e d2ed3ed4ed5ed6ed7ed8ed9ed:ed;ef@fd< Zed=Zed>Zed?Zed@ZdAZxZS)C xLSTMConfiga This is the configuration class to store the configuration of a [`xLSTM`]. It is used to instantiate a xLSTM model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the xLSTM-7b [NX-AI/xLSTM-7b](https://huggingface.co/NX-AI/xLSTM-7b) model. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (int, optional, *optional*, defaults to 50304): Vocabulary size of the xLSTM model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`xLSTMModel`]. Defaults to the GPT2-NeoX tokenizer size. hidden_size (int, optional, *optional*, defaults to 4096): Dimensionality of the embeddings or hidden states. embedding_dim (int, optional, *optional*, defaults to 4096): Dimensionality of the embeddings or hidden states, use hidde_size if None. num_hidden_layers (int, optional, *optional*, defaults to 32): Number of blocks of the xLSTM model. num_blocks (int, optional, *optional*, defaults to 32): Number of blocks of the xLSTM model, use num_hidden_layers if None. num_heads (int, optional, *optional*, defaults to 8): Number of heads for the xLSTM Layer/Cell. use_bias (bool, optional, *optional*, defaults to `False`): Whether to use biases in the xLSTM model. norm_reduction_force_float32 (bool, optional, *optional*, defaults to `True`): Whether to force the float32 norm reduction op to be done in fp32 precision. tie_word_embeddings (bool, optional, *optional*, defaults to `False`): Whether to tie word embeddings to the lm head weights. add_out_norm (bool, optional, *optional*, defaults to `True`): Whether to add an output norm after the blocks before the LMHead. norm_eps (float, optional, *optional*, defaults to 1e-06): Norm eps for RMSNorm and Layer Norm. qk_dim_factor (float, optional, *optional*, defaults to 0.5): Scale factor for the query and key dimension. v_dim_factor (float, optional, *optional*, defaults to 1.0): Scale factor for the value dimension. chunkwise_kernel (ChunkwiseKernelType, optional, *optional*, defaults to `"chunkwise--native_autograd"`): Kernel type for chunkwise processing mode. sequence_kernel (SequenceKernelType, optional, *optional*, defaults to `"native_sequence__native"`): Kernel type for sequence processing mode. step_kernel (StepKernelType, optional, *optional*, defaults to `"native"`): Kernel type for step processing mode. mode (BackendModeType, optional, *optional*, defaults to `"inference"`): Operation mode (inference is needed for generation). chunk_size (int, optional, *optional*, defaults to 64): Internal chunk size. return_last_states (bool, optional, *optional*, defaults to `True`): If to return the last states / cache internally. Needed as True for generation. autocast_kernel_dtype (DtypeType, optional, *optional*, defaults to `"bfloat16"`): Kernel dtype for the states. eps (float, optional, *optional*, defaults to 1e-06): Epsilon for the mLSTM cell post norm. inference_state_dtype (DtypeType, optional, *optional*, defaults to `"float32"`): Kernel dtype for states in inference. ffn_proj_factor (float, optional, *optional*, defaults to 2.667): Size factor of the post-up projection gated Feed Forward network. ffn_round_up_to_multiple_of (int, optional, *optional*, defaults to 64): Size factor round value of the post-up projection gated Feed Forward network. gate_soft_cap (float, optional, *optional*, defaults to 15.0): Gate soft cap scale. output_logit_soft_cap (float, optional, *optional*, defaults to 30.0): Output logit soft cap scale. weight_mode (`Literal`, *optional*, defaults to `"single"`): Whether parallel linear layers are separated or fused (single). use_cache (bool, optional, *optional*, defaults to `True`): Whether to use the cache (xLSTMCache). pad_token_id (int, optional, *optional*, defaults to 1): Pad token id needed for generation. bos_token_id (int, optional, *optional*, defaults to 0): BOS token id needed for generation. eos_token_id (int, optional, *optional*, defaults to 2): EOS token id needed for generation. max_inference_chunksize (int, optional, *optional*, defaults to 16384): Limit the chunk size for inference to save memory. Example: ```python >>> from transformers import xLSTMConfig, xLSTMModel >>> # Initializing a xLSTM configuration >>> configuration = xLSTMConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = xLSTMModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```xlstmN FTư>??rrrr@rrtV@.@>@rr r@ vocab_size hidden_size embedding_dimnum_hidden_layers num_blocks num_headsuse_biasnorm_reduction_force_float32tie_word_embeddings add_out_normnorm_eps qk_dim_factor v_dim_factorchunkwise_kernelsequence_kernel step_kernelmode chunk_sizereturn_last_statesautocast_kernel_dtypeepsinference_state_dtypeffn_proj_factorffn_round_up_to_multiple_of gate_soft_capoutput_logit_soft_cap weight_mode use_cache pad_token_id bos_token_id eos_token_idmax_inference_chunksizec! $||_||n||_||n||_||n||_||n||_||_||_| |_| |_| |_ ||_ | |_ | |_ ||_ ||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_||_| |_tAj!d|||| d|!dS)N)rQrRrPr<)"r4r5r6r7r8r9r:r<r=r>r;r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSsuper__init__)#selfr4r5r6r7r8r9r:r;r<r=r>r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrOrPrQrRrSkwargs __class__s# r"rWzxLSTMConfig.__init__sjX%*5*A;;}.;.G]][6G6S!2!2Yc(2(>**DU"  #6 (  ,H)*( 0.& $"4%:"%:".+F(*%:"&"((('>$ %%% 3         r#c>t|j|jzdSNr.)r)rr5r?rXs r"qk_dimzxLSTMConfig.qk_dims*+  t1 1    r#c>t|j|jzdSr\)rr5r@r]s r"v_dimzxLSTMConfig.v_dims*+  t0 0    r#c |j|jzSN)r^r9r]s r" qk_head_dimzxLSTMConfig.qk_head_dims{dn,,r#c |j|jzSrb)r`r9r]s r" v_head_dimzxLSTMConfig.v_head_dimszT^++r#ctrtdid|jd|jd|jd|jd|jd|jd|jd|j d |j d |j d |j d |j d |jd|jd|jd|jd|jd|jd|jd|jd|jd|jd|jd|jS|S)Nr4r6r8r9r:r=r>r;r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNrU)external_xlstmrr4r5r7r9r:r=r>r;r?r@rArBrCrDrErFrGrHrIrJrKrLrMrNr]s r"to_xlstm_block_configz!xLSTMConfig.to_xlstm_block_config s  #??".. 11..    ".. .2-N-N#00".."&!6!6!% 4 4!,, YY!" ??#$$(#:#:%&'+&@&@'(HH)*'+&@&@+.!% 4 4/0-1,L,L14#0056'+&@&@78!,,9 >Kr#) r'r(Nr)Nr*FTFTr+r,r-rrrrr.Trr+rr/r.r0r1rTr rr2r3)__name__ __module__ __qualname____doc__ model_typer!rboolfloatr r r rr r rWpropertyr^r`rcrerh __classcell__)rZs@r"r%r%<sY Y vJ '++-$(-1$)!"!0L.G&. +#'+5+4!&+-#'+&.',SX X X X  } X $C= X SM X X X '+X "X X X X X ".#X $,%X &$'X *+X ,-X 0!1X 2 )3X 45X 6 )7X :;X <&)=X @AX B %CX F$GX JKX LMX NOX PQX R"%SX X X X X X t  X   X --X-,,X,!!!!!!!r#r%N)rltypingrconfiguration_utilsrutilsrrxlstm.xlstm_large.modelrr r r r r rrrgrr! get_loggerriloggerr%__all__rUr#r"rys 33333300000000                    NNHIO! %89I !:;X&N./NII#I#IIIIN  H % %ooooo"oood /r#