`idZddlmZmZddlmZddlmZeje Z GddeZ Gdd eZ Gd d eZ Gd d eZGddeZgdZdS)zFLAVA model configurations)AnyOptional)PretrainedConfig)loggingceZdZdZdZdZ d dededededededededededededededeffd Z xZ S)!FlavaImageConfiga This is the configuration class to store the configuration of a [`FlavaImageModel`]. It is used to instantiate an FLAVA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. image_size (`int`, *optional*, defaults to 224): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 16): The size (resolution) of each patch. num_channels (`int`, *optional*, defaults to 3): The number of input channels. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. mask_token (`bool`, *optional*, defaults to `True`): Whether to use a mask token or not. Used in MIM (Masked Image Modeling) loss for FLAVA. vocab_size (`int`, *optional*, defaults to 8192): Vocabulary size of the [`FlavaImageCodebook`] used in conjunction with [`FlavaImageModel`] for MIM (Masked Image Modeling) loss for FLAVA. Example: ```python >>> from transformers import FlavaImageConfig, FlavaImageModel >>> # Initializing a FlavaImageModel with style configuration >>> configuration = FlavaImageConfig() >>> # Initializing a FlavaImageModel model (with random weights) from the style configuration >>> model = FlavaImageModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```flava_image_model image_config gelu{Gz?-q=rT hidden_sizenum_hidden_layersnum_attention_headsintermediate_size hidden_acthidden_dropout_probattention_probs_dropout_probinitializer_rangelayer_norm_eps image_size patch_size num_channelsqkv_bias mask_token vocab_sizec tjdi|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_||_dSN)super__init__rrrrrrrrrrr r!r"r#r$)selfrrrrrrrrrrr r!r"r#r$kwargs __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/flava/configuration_flava.pyr)zFlavaImageConfig.__init__Zs& ""6"""&!2#6 !2$#6 ,H)!2,$$(  $$)r r r rrrrrrrrrTTr __name__ __module__ __qualname____doc__ model_typebase_config_keyintfloatboolr) __classcell__r,s@r-r r s: : x%J$O!##%!% %(.1#' %!#%#%#%#%! #%  #%  #%##%',#%!#%#%#%#%#%#%#% !#%#%#%#%#%#%#%#%#%#%r.r ceZdZdZdZdZ d!dedededededededededededededede ffd Z xZ S)"FlavaTextConfigaS This is the configuration class to store the configuration of a [`FlavaTextModel`]. It is used to instantiate an FLAVA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 30522): Vocabulary size of the BERT model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`FlavaTextModel`]. type_vocab_size (`int`, *optional*, defaults to 2): The vocabulary size of the `token_type_ids` passed when calling [`FlavaTextModel`]. Note that even though text encoder allows `token_type_ids`'s value as 2, for text-only pretraining and fine-tuning, only 1 is used similar to RoBERTa. max_position_embeddings (`int`, *optional*, defaults to 512): The maximum sequence length that this model might ever be used with. Typically set this to something large just in case (e.g., 512 or 1024 or 2048). For VL, max_length passed to model is 77. position_embedding_type (`str`, *optional*, defaults to `"absolute"`): Type of position embedding. Choose one of `"absolute"`, `"relative_key"`, `"relative_key_query"`. For positional embeddings use `"absolute"`. For more information on `"relative_key"`, please refer to [Self-Attention with Relative Position Representations (Shaw et al.)](https://huggingface.co/papers/1803.02155). For more information on `"relative_key_query"`, please refer to *Method 4* in [Improve Transformer Models with Better Relative Position Embeddings (Huang et al.)](https://huggingface.co/papers/2009.13658). hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 12): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.1): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. image_size (`int`, *optional*, defaults to 224): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to 16): The size (resolution) of each patch. num_channels (`int`, *optional*, defaults to 3): The number of input channels. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. Example: ```python >>> from transformers import FlavaTextConfig, FlavaTextModel >>> # Initializing a FlavaTextModel with style configuration >>> configuration = FlavaTextConfig() >>> # Initializing a FlavaTextModel model (with random weights) from the style configuration >>> model = FlavaTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```flava_text_model text_config:wabsoluter r rrrrrrTr$type_vocab_sizemax_position_embeddingsposition_embedding_typerrrrrrrrr pad_token_idr"c tjdi|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_||_dSr&)r(r)r$rCrDrErrrrrrrrrr"rF)r*r$rCrDrErrrrrrrrrrFr"r+r,s r-r)zFlavaTextConfig.__init__s& ""6"""$.'>$'>$&!2#6 !2$#6 ,H)!2,  (r.)r?r@rArBr r r rrrrrrrT) r0r1r2r3r4r5r6strr7r8r)r9r:s@r-r<r<sE E N$J#O  '*'1!##%!% %(.1#' %!#)#)#)#)"% #) "% #)  #)#)!#)#)#)##)',#)!#)#)#) !#)#)#)#)#)#)#)#)#)#)r.r<cpeZdZdZdZdZ dd ededededededededededeffd Z xZ S)FlavaMultimodalConfiga This is the configuration class to store the configuration of a [`FlavaMultimodalModel`]. It is used to instantiate an FLAVA model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. num_hidden_layers (`int`, *optional*, defaults to 6): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 12): Number of attention heads for each attention layer in the Transformer encoder. intermediate_size (`int`, *optional*, defaults to 3072): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_probs_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. qkv_bias (`bool`, *optional*, defaults to `True`): Whether to add a bias to the queries, keys and values. use_cls_token (`bool`, *optional*, defaults to `True`): Whether to use an extra CLS token for multimodal settings. Usually needed by the FLAVA model. Example: ```python >>> from transformers import FlavaMultimodalConfig, FlavaMultimodalModel >>> # Initializing a FlavaMultimodalModel with style configuration >>> configuration = FlavaMultimodalConfig() >>> # Initializing a FlavaMultimodalModel model (with random weights) from the style configuration >>> model = FlavaMultimodalModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```flava_multimodal_modelmultimodal_configr r rrrrrTrrrrrrrrrr" use_cls_tokenc  tjdi| ||_||_||_||_||_||_||_||_ | |_ | |_ | |_ dSr&) r(r)rrrrrrrrrr"rN)r*rrrrrrrrrr"rNr+r,s r-r)zFlavaMultimodalConfig.__init__)s~ ""6"""&!2#6 !2$#6 ,H)!2,  *r.) r rMr rrrrrrTTr/r:s@r-rJrJs2 2 h*J)O!"#%!% #&,/#' %"++++! +  +  +!+'*+!+++++++++++++r.rJcVeZdZdZdZ dd ed ed ed edededeffd ZxZS)FlavaImageCodebookConfigflava_image_codebookimage_codebook_configrr@rTr num_groupsinput_channelsnum_blocks_per_grouprr$freezerc tjdi|||_||_||_||_||_||_||_dSr&) r(r)rVrWrXrr$rYr) r*rVrWrXrr$rYrr+r,s r-r)z!FlavaImageCodebookConfig.__init__vs] ""6"""$,$8!&$ !2r.)rTrr@rUrTr) r0r1r2r4r5r6r7r)r9r:s@r-rQrQGs'J-O)Z$%#'3333" 3  3  33!3333333333r.rQc)HeZdZdZdZeeeedZ d"d e e e e fd e e e e fde e e e fde e e e fdedededededededededededededededef(fd Zed ed ededefd!ZxZS)# FlavaConfiga [`FlavaConfig`] is the configuration class to store the configuration of a [`FlavaModel`]. It is used to instantiate FLAVA model according to the specified arguments, defining the text model, image model, image codebook and multimodal model configs. Instantiating a configuration with the defaults will yield a similar configuration to that of the FLAVA [facebook/flava-full](https://huggingface.co/facebook/flava-full) architecture. Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: text_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`FlavaTextConfig`]. image_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`FlavaImageConfig`]. multimodal_config (`dict`, *optional*): Dictionary of configuration options used to initialize [`FlavaMultimodalConfig`]. hidden_size (`int`, *optional*, defaults to 768): Dimensionality of the encoder layers and the pooler layer. layer_norm_eps (`float`, *optional*, defaults to 1e-12): The epsilon used by the layer normalization layers. projection_dim (`int`, *optional*, defaults to 512): Dimensionality of text and image projection layers. logit_scale_init_value (`float`, *optional*, defaults to 2.6592): The initial value of the *logit_scale* parameter. Default is used as per the original FLAVA/CLIP implementation. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. ce_ignore_index (`int`, *optional*, defaults to -100): Cross entropy index to ignore. mim_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MIM (Masked Image Modeling) unimodal loss mlm_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MLM (Masked Language Modeling) unimodal loss global_contrastive_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to global contrastive cross-alignment loss. itm_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to image-text matching multimodal loss. mmm_image_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MMM loss's image part. mmm_text_weight (`float`, *optional*, defaults to 1.0): Weight to be assigned to MMM loss's text part. global_backprop_contrastive (`bool`, *optional*, defaults to `True`): Whether to use global backpropgation through all workers in contrastive loss. skip_unmasked_multimodal_encoder (`bool`, *optional*, defaults to `True`): Whether to skip running unmasked multimodal encoder whose outputs are not used by FLAVA losses. return_loss (`bool`, *optional*, defaults to `True`): Whether to return loss or not kwargs (*optional*): Dictionary of keyword arguments. Example: ```python >>> from transformers import FlavaConfig, FlavaModel, FlavaForPreTraining >>> # Initializing a FlavaConfig with style configuration >>> configuration = FlavaConfig() >>> # Initializing a FlavaModel and FlavaForPreTraining model (with random weights) from the style configuration >>> model = FlavaModel(configuration) >>> model_pre = FlavaForPreTraining(configuration) >>> # Accessing the model configuration >>> configuration = model.config >>> configuration_pre = model_pre.config ``` flava)r>r rLrSNr rT/L F@r?r r>rLrSrrprojection_dim init_codebooklogit_scale_init_valuerce_ignore_index mim_weight mlm_weightglobal_contrastive_weight itm_weightmmm_image_weightmmm_text_weightglobal_backprop_contrastive skip_unmasked_multimodal_encoder return_lossc ^|dd}|dd}|dd}|dd}tjdi|||i}tdi|}|D]I\}}||vr@|||kr4|dkr.||vr d|d|d}nd |d }t |J||||i}tdi|}d |vr'd |d D|d <|D]I\}}||vr@|||kr4|dkr.||vr d|d |d}nd|d }t |J||||i}tdi|}|D]I\}}||vr@|||kr4|dkr.||vr d|d|d}nd|d }t |J||||i}tdi|} | D]I\}}||vr@|||kr4|dkr.||vr d|d|d}nd|d }t |J|| |i}t d|i}t d|i}t d|i}t dtdi||_ tdi||_ tdi||_tdi||_||_||_||_||_| |_| |_d|_| |_| |_| |_||_||_||_||_||_||_||_ dS)Ntext_config_dictimage_config_dictmultimodal_config_dictimage_codebook_config_dicttransformers_version`zp` is found in both `text_config_dict` and `text_config` but with different values. The value `text_config_dict["z"]` will be used instead.zk`text_config_dict` is provided which will be used to initialize `FlavaTextConfig`. The value `text_config["z"]` will be overridden.id2labelc4i|]\}}t||Sr')rH).0keyvalues r- z(FlavaConfig.__init__..!s1222(2UCHHe222r.zs` is found in both `image_config_dict` and `image_config` but with different values. The value `image_config_dict["zn`image_config_dict` is provided which will be used to initialize `FlavaImageConfig`. The value `image_config["z` is found in both `multimodal_config_dict` and `multimodal_config` but with different values. The value `multimodal_config_dict["z}`multimodal_config_dict` is provided which will be used to initialize `FlavaMultimodalConfig`. The value `multimodal_config["z` is found in both `image_codebook_config_dict` and `image_codebook_config` but with different values. The value `image_codebook_config_dict["z`image_codebook_config_dict` is provided which will be used to initialize `FlavaImageCodebookConfig`. The value `image_codebook_config["zR`image_config` is `None`. initializing the `FlavaImageConfig` with default values.zP`text_config` is `None`. Initializing the `FlavaTextConfig` with default values.z\`multimodal_config` is `None`. initializing the `FlavaMultimodalConfig` with default values.zc`image_codebook_config` is `None`. initializing the `FlavaImageCodebookConfig` with default values.r`r')!popr(r)r<to_dictitemsloggerinfoupdater rJrQr r>rLrSrarbrrrrcinitializer_factorrdrerfrgrhrirjrkrlrm)"r*r r>rLrSrrrarbrcrrdrerfrgrhrirjrkrlrmr+rorprqrr_text_config_dictrxrymessage_image_config_dict_multimodal_config_dict_image_codebook_config_dictr,s" r-r)zFlavaConfig.__init__s.6"::&8$??"JJ':DAA!',Dd!K!K%+ZZ0Ld%S%S"""6"""  '" !0 C C2B C C K K M M 05577 ) ) U+%%%;s3C*C*COeHeHe...[[[T&U&U&]&]&_&_ #6;;== ) ) U+++9J39O0O0OTW[qTqTq444sssTWsss sVYsss KK(((  $ $%< = = = % 1$,(*%+C*`*`E_*`*`*h*h*j*j ':??AA ) ) U000!6s!;;;555888''']`''' z]`zzz KK((( " ( ()D E E E  L KKl m m m  K KKj k k k  $ "  KKv w w w ($& ! KKu   -<<|<<*99[99!6!K!K9J!K!K%=%V%V@U%V%V",*&,!2&<#"%.$$)B&$ 0.+F(0P-&r.c |d||||d|S)a& Instantiate a [`FlavaConfig`] (or a derived class) from flava text model configuration, flava image model configuration, flava multimodal model and flava codebook model configuration. Returns: [`FlavaConfig`]: An instance of a configuration object )r r>rLrSr')r|)clsr r>rLrSr+s r- from_configszFlavaConfig.from_configssh"s %--//#++--/7799"7"?"?"A"A       r.)NNNNr rr Tr^rr_r`r`r`r`r`r`TTT)r0r1r2r3r4r<r rJrQ sub_configsrdictrHrr6r7r8r) classmethodrr9r:s@r-r\r\sCCJJ&(2!9 K26046::> %!"(.#'#+."%!$,015 +D'D'tCH~.D'd38n-D'$DcN3 D' (S#X7 D'  D'D'D'D'!&D'!D'D'D'D'$)D' !D'" #D'$%D'&&*'D'(+/)D'*+D'D'D'D'D'D'L & % 1  8    [     r.r\)r\rQr rJr<N)r3typingrrconfiguration_utilsrutilsr get_loggerr0r~r r<rJrQr\__all__r'r.r-rss! 333333  H % %c%c%c%c%c%'c%c%c%Ln)n)n)n)n)&n)n)n)bS+S+S+S+S+,S+S+S+lA3A3A3A3A3/A3A3A3Hk k k k k "k k k \ v u ur.