ePidZddlmZmZmZddlZddlmZm Z m Z Gdde j Z Gdde j Z Gd d e j ZGd d e j ZGd de j ZGdde j ZGdde jZGdde jZGdde jZGdde jZGdde jZGdde jZdS)z0Declares specification of the Transformer model.)OptionalTupleUnionN)attention_spec common_spec model_specc/,eZdZddejjdejjddddddddddddddddfdedede d e d ejd ed ejd e de de de de de de ede ede ede de e j de de de ede e de f.dZdS)TransformerEncoderSpecTFN' num_layers num_headspre_norm no_final_norm activationnum_source_embeddingsembeddings_mergelayernorm_embeddingrelative_positionrelative_attention_biasffn_glurms_normmulti_query_attention num_heads_kvhead_dim rotary_dimrotary_interleaverotary_scaling_typerotary_scaling_factor rotary_basesliding_windowqk_normpre_post_layer_normc  | rdkrtdd| |_tjd||_||_tjd||_tjd||_dt|D|_ d|_ s st|_ |r|stj |_|rtj |_|,tjd ||_  f d t|D|_dS) a'Initializes a Transformer encoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of dimensions per attention head. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. sliding_window: Max sequence length to retain in KV Cache. qk_norm: Apply layer normalization to the query and key projections. pre_post_layer_norm: Add post layer norm for each pre norm layer. Nr 5Enabling multi_query_attention implies num_heads_kv=1int16int8c4g|]}tjS)rEmbeddingsSpec).0_s v/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ctranslate2/specs/transformer_spec.py z3TransformerEncoderSpec.__init__..Ss.   -.K & ( (   Trint32cH g|]}t    S)) rrrrrrrrrrr r"r#)TransformerEncoderLayerSpec)r+r,rrrr#r"rrrr rrrrs r-r.z3TransformerEncoderSpec.__init__..`sa    ("3(?!)!%"3$7&;'$7      r/) ValueErrorrnpdtypetyperrrrrange embeddingsscale_embeddingsPositionEncoderSpecposition_encodingsr LayerNormSpec layer_normrr!layer)selfr rrrrrrrrrrrrrrrrrrr r!r"r#s ```` ``````` ``r-__init__zTransformerEncoderSpec.__init__ sx ! 'LA,=,= KL%:"'**// ::  (6**// ;; " 0 0 5 56F G G  278M2N2N   !%  <)@ <&9&;&;D #  KM K)7JJJDO  T'2'@('S'S'SD $  %"$(7"3"3"8"8"H"HD                 :&&!    r/)__name__ __module__ __qualname__r ActivationRELUEmbeddingsMergeCONCATintboolrrRotaryScalingTypefloatrAr)r/r-r r s #-8-C-H%&8C8S8Z$)"'(-&+&*"&$("&JN'("(,"'$)1f f f f  f  f  * f  #f &5f "f  f "&f f f  $f smf 3-!f "SM#f $ %f &&n&FG'f ( %)f *+f ,! -f .$/f 0"1f f f f f f r/r cJeZdZdejjddddddddddddddddddddddddddddddddf"ded ed ed ejd ed ededededededededededededeededee j de de deded ed!ed"ed#ed$eed%eed&eed'eej d(eed)eed*ed+eefHd,Z ed-ZdS).TransformerDecoderSpecTFr Nr rr rrrrwith_encoder_attentionrproject_in_outrralignment_layeralignment_headsrralibialibi_use_positive_positions scale_alibirrrrr original_max_position_embeddingsmax_position_embeddingsparallel_residualshared_layer_normr#rrrr! quant_typequant_group_size quant_bitsr" external_pre_post_encoder_layersc%< #$t|_r"|stdrtd|rdkrtddtjd||_||_tjd||_tjd| |_ tjd| |_ tj |_ d|_tj|_||_||_||_,tjd |_ s s|st-|_|r|stj |_|rtj |_tj|_$ # fd t;|D|_d |_|p|k|jd <|r0tj|_ tj|_!| r | |jd<|"|jd<|!|jd<dSdS)a. Initializes a Transformer decoder specification. Args: num_layers: Number of layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. with_encoder_attention: Enable the encoder attention sublayers. no_final_norm: Disable the final layer norm in the pre-norm architecture. project_in_out: Add linear transformations after the embedding layer and before the final layer. relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: Add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. sliding_window: Max sequence length to retain in KV Cache. quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) external_pre_post_encoder_layers: if the encoder attention pre and processing is done outside the attention. z/The GPT-J block expects a pre-norm architecturez-The GPT-J block does not have cross attentionNr r%r&r'Tr1r0cg|]H}tdidd d dd ddddd d d d d d dddd dIS)rPrrrrrrrrr rWrXrYrZr#rrr!r"r^r))TransformerDecoderLayerSpec)r+r,r^rrrXrrWrYr#r"rrrr rrrrrZr!rPs r-r.z3TransformerDecoderSpec.__init__..s6   .- (   '='= "3"3 )@(?     "   &:  #4"3 %8$7 '<&; (K 2R1Q )@(? #4"3 #4"3 %8$7  *\! ""# $ .~% & ' (2R1Q)    r/Frquantization_typequantization_bitsquantization_group_size)"dict_configr4r5r6r7rrrrRrSrr*r9r:rOPTIONAL scale_outputsrTrUrVr!r;r<r=r>r LinearSpec projectionr8r?start_from_zero_embedding project_in project_out)%r@r rrrrrPrrQrrrRrSrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"r^s% ` `` `` `````````` ``` ``r-rAzTransformerDecoderSpec.__init__us"pvv  R T !RSSS% R !PQQQ  'LA,=,= KL'**// ::  (6**// ;;!x0055oFF!x0055oFF%466 $'0 ,H)&  %"$(7"3"3"8"8"H"HD ! <+ < <"&9&;&;D #  KM K)7JJJDO  T'2'@('S'S'SD $%022                       .:&&/    2*/&0E1 I % ,-  8)466DO*577D   G0:DL, -0:DL, -6FDL2 3 3 3 G Gr/c|jSN)rfr@s r-configzTransformerDecoderSpec.configs |r/)rBrCrDrrErFrIrJrrrKrL QuantizationrApropertyrqr)r/r-rNrNts -8-C-H$)'+#$"'(-! -2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(;@KdGdGdGdG dG  * dG " dG!%dGdGdG dG"&dGdGdGdGdG !dG"'+#dG$%dG&SM'dG( )dG*&n&FG+dG, %-dG./dG0+.1dG2"%3dG4 5dG6 7dG8"9dG: $;dG<sm=dG>3-?dG@! AdGB[56CdGD#3-EdGFSMGdGHIdGJ+34.KdGdGdGdGLXr/rNc neZdZ d deededeejd ed ed ef d Z dS)r3FNTr r rrrrr r#ctjd|||||||| | | | |  |_t|||_|rt j||_t j||_t j||_ t j||_ t|jdt|jddSdS)NT) self_attentionrrrrrr!rrrrr r"glurr0r>) rMultiHeadAttentionSpecrvFeedForwardSpecffnrr=input_layer_normpost_attention_layer_normpre_feedforward_layer_normpost_feedforward_layer_normdelattr)r@rrrrrrr!rrrrr r"r#s r-rAz$TransformerEncoderLayerSpec.__init__!s "-C/$;%)!/ 3"7#   #wBBB  ,$/$=x$P$P$PD !-8-F!...D */:.G!///D +0;/H!000D , D' 6 6 6 DHl + + + + + , ,r/)FFFFNNNNTNr r FF) rBrCrDrrIrJrrKrLrAr)r/r-r3r3 s %$("&JN'("$)/,/,SM/, /,&n&FG/, %/,/,"/,/,/,/,/,/,r/r3c>eZdZ ddZdS) raTFNr r rcRtjd||||||| | | | |||||_|r!tj||||||du|_t |||_| ru|rt j|_n0t j|_ t j|_ t|jdt|jd|rt j||_ t j||_ |r6|r4t j||_ t j||_ t j||_t j||_t|jdt|jddSdS)NT)rvrrrrrrrr rWrXrrr!r"F)rrrr!r"has_normrwr>r0)rryrv attentionrzr{rr=rZr|r}r*external_post_encoder_attention_layer_norm)external_pre_encoder_attention_layer_normr~r)r@rPrrrrrrrrr rWrXrYrZr#rrr!r"r^s r-rAz$TransformerDecoderLayerSpec.__init__Ts.-C/$;!/ 3"7#-M$;%)   $ " +B!)!-9UB DN#wBBB  ,  M)4)B)D)D&&(3(A(C(C%1<1J1L1L. D' 6 6 6 DHl + + +  ,$/$=x$P$P$PD !-8-F!...D *& *J -x@@@? -x@@@> /:.G!///D +0;/H!000D , D' 6 6 6 DHl + + + + +1 , ,r/)TFFFFNTNr r rrFFFNNNFFrBrCrDrAr)r/r-raraSsl $ % )* !!).+W,W,W,W,W,W,r/raceZdZddZdS)rzFctj||_tj|_tj|_|rtj|_dSdS)Nr0)rr=r>rilinear_0linear_1linear_0_noact)r@rxrs r-rAzFeedForwardSpec.__init__s`%3XFFF#.00 #.00  ;"-"8":":D    ; ;r/N)FFrr)r/r-rzrzs(;;;;;;r/rzceZdZdZdS)r;c(tj|_dSro)rrg encodingsrps r-rAzPositionEncoderSpec.__init__s#,r/Nrr)r/r-r;r;s#-----r/r;c6eZdZdZddeeffd ZxZS)TransformerConfigz%Configuration for Transformer models.Nlayer_norm_epsilonc >tjdd|i|dS)zInitializes the configuration for Transformer models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr)superrAr@rkwargs __class__s r-rAzTransformerConfig.__init__. II,>I&IIIIIr/rorBrCrD__doc__rrLrA __classcell__rs@r-rrs_//JJ8E?JJJJJJJJJJr/rc 6eZdZdZdedeffd Zedddej j dddej j dddddf d e eeeeffd ed ed ed edej dedededej dededededefdZedZedZdZdZdZxZS)TransformerSpeczDescribes a Transformer model. The specification is invariant to hidden dimensions but requires to explicitly set the number of layers and attention heads. encoderdecoderc>t|tstdt|tstdt ||_||_|j d|jj dS)zInitializes a Transformer model specification. Args: encoder: The encoder specification. decoder: The decoder specification. 1encoder argument must be a TransformerEncoderSpec1decoder argument must be a TransformerDecoderSpecrN) isinstancer TypeErrorrNrrArrrf add_attributer)r@rrrs r-rAzTransformerSpec.__init__s'#9:: QOPP P'#9:: QOPP P    "" #T\%G     r/FTrOr r rwith_relative_positionrrrrRrSrrrrrrrct|ttfr|\}}n||}}t|||||| | | || | || }t |||||| || ||| || }|||S)aCreates a Transformer model specification. Args: num_layers: Number of encoder and decoder layers, or a 2-tuple if the number is different. num_heads: Number of attention heads. with_relative_position: Use relative position representations in the self-attention layers as described in https://arxiv.org/abs/1803.02155. pre_norm: Enable the pre-norm Transformer architecture. no_final_norm: Disable the final layer norm in the pre-norm architecture. activation: Activation to apply in the feed-forward network. alignment_layer: Layer index selected for alignment. alignment_heads: Number of attention heads selected for alignment. num_source_embeddings: Number of source embeddings. embeddings_merge: When :obj:`num_source_embeddings` > 1, specify how the embeddings are merged. layernorm_embedding: Apply layer normalization after the embedding layer. relative_attention_bias: Use relative attention bias in the self-attention layers as described in the T5 paper https://arxiv.org/abs/1910.10683. ffn_glu: Use gated linear units in the FFN layer as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. multi_query_attention: Use multi-query attention. ) rrrrrrrrrrr) rrrrrrrRrSrrr)rlisttupler rN)clsr rrrrrrRrSrrrrrrrnum_encoder_layersnum_decoder_layersrrs r- from_configzTransformerSpec.from_configsV j4- 0 0 L5? 2  2 25? 2 (  '!"7- 34$;"7    )  '! 34$;++"7    s7G$$$r/cdS)Nrr)rps r-namezTransformerSpec.name7s  r/cdS)Nr)rps r-revisionzTransformerSpec.revision;qr/ctSro)rrps r-get_default_configz"TransformerSpec.get_default_config?s """r/c.d|jjDS)Nc2g|]}|jjdS)r)weightshape)r+specs r-r.z>TransformerSpec.get_source_vocabulary_size..Cs"III !!$IIIr/)rr9rps r-get_source_vocabulary_sizez*TransformerSpec.get_source_vocabulary_sizeBsII1HIIIIr/c:|jjjjdSNrrr9rrrps r-get_target_vocabulary_sizez*TransformerSpec.get_target_vocabulary_sizeE|&-3A66r/)rBrCrDrr rNrA classmethodrrErFrGrHrrIrrJrrsrrrrrrrs@r-rrs  - 8N      * (-#-8-C-H! %&8C8S8Z$)(-&+!O%O%#uS#X./O%O%!% O%  O%  O% *O%O%O% #O%&5O%"O%"&O%O%O% $!O%O%O%[O%b!!X!X###JJJ7777777r/rc6eZdZdZddeeffd ZxZS)TransformerDecoderModelConfigz-Configuration for Transformer decoder models.Nrc >tjdd|i|dS)zInitializes the configuration for Transformer decoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr)rrs r-rAz&TransformerDecoderModelConfig.__init__Lrr/rorrs@r-rrI_77JJ8E?JJJJJJJJJJr/rc@eZdZdZdeffd Zedejj ddddddddddddddd d dddddddddddfd e d e d e d ejde de de de de de de de de de e de de e jdedede de de de d e d!e d"e e d#e e d$e e d%e ejd&e e d'e e d(e f>d)Zed*Zed+Zd,Zd-ZxZS).TransformerDecoderModelSpecz3Describes a Transformer decoder model (e.g. GPT-2).rct|tstdt||_|jjD] \}}|j ||!dS)z|Initializes a Transformer decoder model specification. Args: decoder: The decoder specification. rN) rrNrrrArrqitemsrfr)r@rkeyvaluers r-rAz$TransformerDecoderModelSpec.__init__Ys '#9:: QOPP P  ,-3355 3 3JC L & &sE 2 2 2 2 3 3r/TFNr r rr rrrrrrQrrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"c t||fid|d|d|ddd|d|d|d | d | d | d | d | d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|d|} || S) a! Creates a Transformer decoder model specification. Args: num_layers: Number of decoder layers. num_heads: Number of attention heads. pre_norm: Enable the pre-norm Transformer architecture. activation: Activation to apply in the feed-forward network. layernorm_embedding: Apply layer normalization after the embedding layer. no_final_norm: Do not apply layer normalization after the last decoder block. project_in_out: Add a linear layer after the embedding layer and another one before the final output projection. with_relative_position: Enable relative position representations modules. ffn_glu: Use gated linear units in the FFN layers as described in https://arxiv.org/abs/2002.05202. rms_norm: Use the root mean square layer normalization. alibi: Use attention with linear biases. alibi_use_positive_positions: Use positive positions in the ALiBi definition. scale_alibi: Apply the dot product scale factor to ALiBi. rotary_dim: Apply rotary embeddings to these first N dimensions. If 0, rotary embeddings are applied to all dimensions. rotary_interleave: Interleave the head dimensions when rotary embeddings are applied. Otherwise the head dimensions are sliced in half. rotary_scaling_type: Type of RoPE scaling. rotary_scaling_factor: Factor used in the RoPE scaling. rotary_base: The base period of the rotary embeddings. original_max_position_embeddings: The original max position embeddings for Su rope embeddings max_position_embeddings: The max position embeddings for Su rope embeddings parallel_residual: Use parallel residual connections in each layer block, as used by the GPT-J and GPT-NeoX models. shared_layer_norm: When using parallel residual, share the input and post attention layer norms. pre_post_layer_norm: add post layer norm for each pre norm layer multi_query_attention: Use multi-query attention (alias for num_heads_kv=1). num_heads_kv: Number of attention heads for the key and value. head_dim: Number of head sliding_window: max sequence length to retain KV cache quant_type: quantization type used (like awq... for lower bit quantization) quant_group_size: group size of the lower bit quantization quant_bits: number of bit of the quantization (ex: 4bit) rrrrPFrrQrrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r")rN)!rr rrrrrrQrrrrTrUrVrrrrr rWrXrYrZr#rrrr!r[r\r]r"rs! r-rz'TransformerDecoderModelSpec.from_configgsX)  ! ! ! X! "z ! !4 3 ! $)5 ! (-! *>! 54! G! X! %! *F)E! $ ! "z! 0/!! "!4 3#! $#8"7%! &$ '! (.N-M)! *%<$;+! ,0/-! .0//! 0!4 31! 2#8"73! 4&5! 6X7! 8*>9! :"z;! <.-=! >"z?! @GA! Fs7||r/cdS)NrNr)rps r-rz TransformerDecoderModelSpec.name''r/cdS)Nr)rps r-rz$TransformerDecoderModelSpec.revisionrr/ctSro)rrps r-rz.TransformerDecoderModelSpec.get_default_config,...r/c:|jjjjdSrrrps r-get_vocabulary_sizez/TransformerDecoderModelSpec.get_vocabulary_sizerr/)rBrCrDrrNrArrrErFrIrJrrrKrLrrrrsrrrrrrs@r-rrVs== 3 6 3 3 3 3 3 3 -8-C-H$)#$',-2!$("&JN'("01'("'"'$)&+&*"&(,9=*.$(Annnn n  * n " nnn!%nnnn'+nnSMn  !n"&n&FG#n$ %%n&'n(+.)n*"%+n, -n. /n0"1n2 $3n4sm5n63-7n8! 9n:[56;n<#3-=n>SM?n@Annn[n`((X(X///7777777r/rc6eZdZdZddeeffd ZxZS)TransformerEncoderModelConfigz-Configuration for Transformer encoder models.Nrc >tjdd|i|dS)zInitializes the configuration for Transformer encoder models. Args: layer_norm_epsilon: The layer norm epsilon value. **kwargs: Additional configuration. rNr)rrs r-rAz&TransformerEncoderModelConfig.__init__rr/rorrs@r-rrrr/rceZdZdZdejjfdededejffd Z e dZ e dZ d Z d ZxZS) TransformerEncoderModelSpecz2Describes a Transformer encoder model (e.g. BERT).Fr pooling_layerpooling_activationcxt|tstdt||_|jd|jj|rFtj |_ tj d||_dSdS)zInitializes a Transformer encoder model specification. Args: encoder: The encoder specification. pooling_layer: Add the pooling layer. pooling_activation: The activation to apply after the pooling layer. rrr'N)rr rrrArrfrrrri pooler_denser5r6r7pooler_activation)r@rrrrs r-rAz$TransformerEncoderModelSpec.__init__s'#9:: QOPP P   "" #T\%G     O + 6 8 8D %'Xf%5%5%:%:;M%N%ND " " " O Or/cdS)Nr r)rps r-rz TransformerEncoderModelSpec.namerr/cdS)Nr r)rps r-rz$TransformerEncoderModelSpec.revisionrr/ctSro)rrps r-rz.TransformerEncoderModelSpec.get_default_configrr/cF|jjdjjdSr)rr9rrrps r-rz/TransformerEncoderModelSpec.get_vocabulary_sizes|&q)06q99r/)rBrCrDrrrETanhr rJrArsrrrrrrs@r-rrs<< $5@5K5P OO'OO(2 OOOOOO4((X(X///:::::::r/r)rtypingrrrnumpyr5ctranslate2.specsrrr LayerSpecr rNr3rarzr;SequenceToSequenceModelConfigrSequenceToSequenceModelSpecrLanguageModelConfigrLanguageModelSpecrrrr)r/r-rs66))))))))))EEEEEEEEEEg g g g g Z1g g g TiiiiiZ1iiiX0,0,0,0,0,*"60,0,0,fX,X,X,X,X,*"6X,X,X,v;;;;;j*;;;-----*.--- J J J J J @ J J J}7}7}7}7}7j<}7}7}7@ J J J J JJ$B J J JN7N7N7N7N7*">N7N7N7b J J J J JJ$B J J J):):):):):*">):):):):):r/