`iPHdZddlmZddlmZejeZGddeZGddeZ Gdd eZ Gd d eZ Gd d eZ gdZ dS)zBlt model configuration)PretrainedConfig)loggingcHeZdZdZdZ dfd ZxZS)BltLocalEncoderConfigzB Configuration class for the Blt Local Encoder component. blt_local_encoderFNh㈵>`Asilu {Gz?c ||_||_||_||_||_||_|p||_||z|_|ptd|zdz |_ ||_ | |_ | |_ | |_ | |_| |_||_||_|ddt'jdi|ddidSNrtie_word_embeddingsF vocab_sizecross_attn_all_layers cross_attn_khidden_size_global hidden_sizenum_attention_headsnum_key_value_headshead_dimintintermediate_sizenum_hidden_layers rms_norm_epsdropoutmax_position_embeddings rope_theta rope_scaling hidden_actinitializer_rangepopsuper__init__selfrrrrrr r!r%r&r'r(r)r*r+r$r,kwargs __class__s }/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/blt/configuration_blt.pyr/zBltLocalEncoderConfig.__init__(%%:"("4&#6 #6#M:M #':: !2!Nc!k/A:M6N6N!2( '>$$($!2  ($///==6==u======)rFr r r r Nr rrrrNrrr__name__ __module__ __qualname____doc__ model_typer/ __classcell__r3s@r4rrs%J#  %#(>(>(>(>(>(>(>(>(>(>r6rcHeZdZdZdZ dfd ZxZS)BltLocalDecoderConfigzB Configuration class for the Blt Local Decoder component. blt_local_decoderrTr r r r N rrrrrrrc ||_||_||_||_||_||_|p||_||z|_|ptd|zdz |_ ||_ | |_ | |_ | |_ | |_| |_||_||_|ddt'jdi|ddidSrrr0s r4r/zBltLocalDecoderConfig.__init__Qr5r6)rTr r r r NrBrrrrNrrrr7r>s@r4r@r@Js%J"  %#(>(>(>(>(>(>(>(>(>(>r6r@c@eZdZdZdZ dfd ZxZS)BltGlobalTransformerConfigzG Configuration class for the Blt Global Transformer component. blt_global_transformerr r Nrrrrrc  X||_||_|p||_||z|_| pt d|zdz |_||_||_||_||_ ||_ | |_ | |_ | |_ | ddtjdi| ddidSr)rr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/)r1rr r!r%r&r'r(r)r*r+r$r,r2r3s r4r/z#BltGlobalTransformerConfig.__init__s '#6 #6#M:M #':: !2!Nc!k/A:M6N6N!2( '>$$($!2  ($///==6==u======r6) r r NrGrrrHrNrrIrr7r>s@r4rErE|st*J  $ > > > > > > > > > >r6rEc@eZdZdZdZ dfd ZxZS)BltPatcherConfiga Configuration class for the Blt Patcher/Entropy model component. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt patcher model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling the patcher model. hidden_size (`int`, *optional*, defaults to 768): Dimension of the hidden representations. num_hidden_layers (`int`, *optional*, defaults to 14): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. max_position_embeddings (`int`, *optional*, defaults to 8192): The maximum sequence length that this model might ever be used with. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. intermediate_size (`int`, *optional*, defaults to 2048): Dimension of the MLP representations. rope_scaling (`dict`, *optional*): Dictionary containing the RoPE scaling configuration. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. blt_patcherr N rr@r rc  t||_||_||_||_||z|_||n||_||_||_||_| |_ d|_ | ptd|jzdz |_ | |_ | |_| ddt!jdi| ddidS)NrrrrFr)rrr%r r"r!r(r&r'r)r+r#r$r*r,r-r.r/)r1rrr%r r!r(r&r'r)r$r*r,r2r3s r4r/zBltPatcherConfig.__init__s %&!2#6 #':: :M:Y#6#6_r '>$( $ !2!Sc!d>N:NQR:R6S6S(!2  ($///==6==u======r6) rrNrOrPNrQrrrRr Nrr7r>s@r4rLrLsu##JJ $!>!>!>!>!>!>!>!>!>!>r6rLcdeZdZdZdZdgZeeee dZ dfd Z xZ S) BltConfigas This is the configuration class to store the configuration of a [`BltModel`]. It is used to instantiate a Blt model according to the specified arguments, defining the model architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 260): Vocabulary size of the Blt model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`BltModel`]. max_position_embeddings (`int`, *optional*, defaults to 4096): The maximum sequence length that this model might ever be used with. patch_in_forward (`bool`, *optional*, defaults to `True`): Whether to perform patching during the forward pass. patch_size (`int`, *optional*, defaults to 4): Size of the patches used in the patching mechanism. patching_mode (`str`, *optional*, defaults to `"entropy"`): The mode used for patching, such as entropy-based patching. patching_threshold (`float`, *optional*, defaults to 1.34): Threshold value used for determining when to apply patches. patching_batch_size (`int`, *optional*, defaults to 1): Batch size used during the patching process. max_patch_length (`int`, *optional*): Maximum length of patches that can be generated. cross_attn_k (`int`, *optional*, defaults to 2): Number of cross-attention heads used in the model. encoder_hash_byte_group_size (`list`, *optional*): List of byte group sizes used in the encoder hash function. encoder_hash_byte_group_vocab (`int`, *optional*, defaults to 500002): Vocabulary size for the encoder hash byte groups. encoder_hash_byte_group_nb_functions (`int`, *optional*, defaults to 1): Number of hash functions used in the encoder byte grouping. patcher_config (`BltPatcherConfig`, *optional*): Configuration for the patcher component of the model. encoder_config (`BltLocalEncoderConfig`, *optional*): Configuration for the local encoder component of the model. decoder_config (`BltLocalDecoderConfig`, *optional*): Configuration for the local decoder component of the model. global_config (`BltGlobalTransformerConfig`, *optional*): Configuration for the global transformer component of the model. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether to tie weight embeddings. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rope_theta (`float`, *optional*, defaults to 500000.0): The base period of the RoPE embeddings. rope_scaling (`dict`, *optional*): Dictionary containing the RoPE scaling configuration. ```python >>> from transformers import BltModel, BltConfig >>> # Initializing a Blt configuration >>> configuration = BltConfig() >>> # Initializing a model from the configuration >>> model = BltModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` Checkpoint: [facebook/blt](https://huggingface.co/facebook/blt) bltpast_key_values)patcher_configencoder_configdecoder_config global_configrrHTentropy]?r Nr "Frrc ||_||_||_||_||_||_||_||_||_||_ ||_ | dd|_ | dd|_ | d|_| dd|_| |_| pgd|_| |_| |_| 0t)| |_t,d nYt1| t2r(| d |t)di| |_nt1| t(r| |_|0t7| |_t,d nYt1|t2r(|d |t7di||_nt1|t6r||_|0t;| |_t,d nYt1|t2r(|d |t;di||_nt1|t:r||_|0t?| |_ t,dnYt1|t2r(|d |t?di||_ nt1|t>r||_ |jj!|jz}||j j!kr|nd|j _"|#ddtIj%dd|i|dS)Npatching_devicecudarealtime_patchingTpatching_threshold_add monotonicityF)rr\r)r,z8patcher_config is None, using default Blt patcher configr,z8encoder_config is None, using default Blt encoder configz8decoder_config is None, using default Blt decoder configz6global_config is None, using default Blt global configrr)&rr(r,r)r*patch_in_forward patch_size patching_modepatching_thresholdpatching_batch_sizemax_patch_lengthgetrarcrdrerencoder_hash_byte_group_sizeencoder_hash_byte_group_vocab$encoder_hash_byte_group_nb_functionsrLrXloggerinfo isinstancedict setdefaultrrYr@rZrEr[rencoder_cross_output_sizer-r.r/)r1rr(rirjrkrlrmrnrrprqrrrXrYrZr[rr,r)r*r2rxr3s r4r/zBltConfig.__init__>s2%'>$!2$(!1$*"4#6 0%zz*;VDD!',?!F!F&,jj1I&J&J#"JJ~u==)-I,^L^L^L^)-J*4X1  !"2EV"W"W"WD  KKR S S S S  - - 1  % %&9;L M M M"2"D"D^"D"DD   (8 9 9 1"0D   !"7J["\"\"\D  KKR S S S S  - - 1  % %&9;L M M M"7"I"I."I"ID   (= > > 1"0D   !"7J["\"\"\D  KKR S S S S  - - 1  % %&9;L M M M"7"I"I."I"ID   (= > > 1"0D   !;N_!`!`!`D  KKP Q Q Q Q  t , , /  $ $%8:K L L L!;!L!Lm!L!LD    'A B B /!.D %)$7$CdFW$W!)BdFXFd)d)d % %jn 4  ($///KK-@KFKKKKKr6)rrHTr\r]r^r Nr Nr_r NNNNFrrN) r8r9r:r;r<keys_to_ignore_at_inferencerLrr@rE sub_configsr/r=r>s@r4rUrUs@@DJ#4"5*//3 K $,%)&,-.!+`L`L`L`L`L`L`L`L`L`Lr6rU)rUrLrr@rEN)r;configuration_utilsrutilsr get_loggerr8rsrr@rErLrU__all__rr6r4rsN333333  H % %/>/>/>/>/>,/>/>/>d/>/>/>/>/>,/>/>/>d'>'>'>'>'>!1'>'>'>TI>I>I>I>I>'I>I>I>XlLlLlLlLlL lLlLlL^   r6