Pi38ddlZddlmZddlZddlmcmZddlmZddlm Z ddl m Z ej e ZGddejZdS)N)Optional)nn) _MaskType)KVCachec'eZdZdZddddddddddd ded ed ed ed ejd ejdejdejdeejdeejdeejdeedede de deedee deeddf&fdZ dede j deddfdZdZ d&ddd d!e jd"ee jd#eed$ee jde jf d%ZxZS)'Gemma2Attentionaz Adapated from official Google Pytorch Implementation: https://github.com/google/gemma_pytorch/blob/80881c2e6e797ef1913a4a705d4b40394791cc58/gemma/model.py#L213 to match torchtune style. A new attention had to be added since nn.functional.scaled_dot_product_attention does allow soft capping Args: embed_dim (int): embedding dimension for the model num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure ``num_heads % num_kv_heads == 0``. For standard MHA set ``num_kv_heads == num_heads``, for GQA ``num_kv_heads < num_heads``, and for MQA set ``num_kv_heads == 1``. head_dim (int): dimension of each head, calculated by ``embed_dim // num_heads``. q_proj (nn.Module): projection layer for query. k_proj (nn.Module): projection layer for key. v_proj (nn.Module): projection layer for value. output_proj (nn.Module): projection layer for output. pos_embeddings (Optional[nn.Module]): positional embeddings layer, e.g. RotaryPositionalEmbeddings. q_norm (Optional[nn.Module]): normalization layer for query, e.g. RMSNorm. For decoding, this is applied before updating from kv_cache. This means it will only support token wide normalization and not batch or sequence wide normalization. k_norm (Optional[nn.Module]): normalization layer for key, must be set if q_norm is. kv_cache (Optional[KVCache]): KVCache object used to cache key and value max_seq_len (int): maximum sequence length supported by the model. This is needed to compute the RoPE Cache. Default: 4096. is_causal (bool): sets the default mask to causal when no mask is provided attn_dropout (float): dropout value passed onto the scaled_dot_product_attention function. This argument is ignored if the self.training is False. Default value is 0.0. sliding_window_size (Optional[int]): size of the sliding window if None no sliding window is applied softcapping (Optional[float]): capping value used for soft caping, if None no capping is performed query_pre_attn_scalar (Optional[int]): value used for pre attention normalisation, if None head_dim is used instead Raises: ValueError: If ``num_heads % num_kv_heads != 0``, **or** if ``embed_dim % num_heads != 0``, **or** if ``attn_dropout < 0`` or ``attn_dropout > 1``, **or** if ``q_norm`` is defined without k_norm or vice versa NiTggI@) pos_embeddingsq_normk_normkv_cache max_seq_len is_causal attn_dropoutsliding_window_size softcappingquery_pre_attn_scalar embed_dim num_heads num_kv_headshead_dimq_projk_projv_proj output_projr r r r r rrrrrreturnct||zdkrtd|d|d||zdkrtd|d|d|dks|dkrtd|d t| t| z rtd ||_||_||_||_||_| |_ ||_ | |_ ||_ ||_ ||_||_| |_| |_| |_||_||_| |d z|_n|jd z|_d |_dS) Nrz num_heads (z%) must be divisible by num_kv_heads ()z embed_dim (z") must be divisible by num_heads (zattn_dropout (z) must be between 0.0 and 1.0z!q and k norm must be set togethergF)super__init__ ValueErrorboolrrrrrr rr rrrrr r r rrscaling cache_enabled)selfrrrrrrrrr r r r r rrrrr __class__s v/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/gemma2/_attention.pyr zGemma2Attention.__init__<s,  | #q ( (1i11!-111  y A % %+i++'+++  !  |a//ViVVVWW W <<$v,, & B@AA A#("(  &"!    &  ,$7 & ,0$6DLL=$.DL # batch_sizedtypec|jtddSt|||j|j||_d|_dS)aQSetup key value caches for attention calculation. If called after kv_cache is already setup, this will be skipped. Args: batch_size (int): batch size for the caches. dtype (torch.dtype): dtype for the caches. max_seq_len (int): maximum sequence length model will be run with. NzWKey value caches are already setup. You cannot call ``setup_caches()`` twice. Skipping.)r)r rrr*T)r loggerwarningrrrr$)r%r)r*r s r' setup_cachezGemma2Attention.setup_cachesj = $ NNi     $%'!^ DM"&D   r(cd|jtd|jdS)zzReset the key value caches. Raises: RuntimeError: if key value caches are not already setup. Nz>Key value caches are not setup. Call ``setup_caches()`` first.)r RuntimeErrorreset)r%s r' reset_cachezGemma2Attention.reset_caches= = P  r()mask input_posxyr3r4c |)t|tjstd|j\}}}| |jdnd}||} |j|jz} | |||j| z|j } |j | | |} | dd} |j | | } |7|j |jstd|j j} |j j} n||} ||} | ||d|j } |j | | |} | |||jd|j } | |||jd|j } |j|jkrF| |||j| |j } | |||j| |j } | ||d|j } | ||d|j } | dd} | dd} |j|| } |j %|jr|j | | \} } | |jtj| | dd } |Ltjtj||ftj |j}|j tjkr(tj!|"d d}|j#ltj$|}tj%|d|j#zdztj||j#dz z}tj!|dk|d }|&d kr|'d}|j((| |j(z } tj)| } | |j(z} | |z} tUj+| ,d -| } tj| | } | dd.||d} |/| S) a Args: x (torch.Tensor): input tensor with shape [b x s_x x d] for the query y (Optional[torch.Tensor]): second input tensor with shape [b x s_y x d], is the input for k and v. For self attention, x=y. Optional only with kv_cache enabled. mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Either: A boolean tensor with shape ``[b x s x s]``, ``[b x s x self.encoder_max_cache_seq_len]``, or ``[b x s x self.encoder_max_cache_seq_len]`` if using KV-cacheing with encoder/decoder layers. A value of True in row ``i`` and column ``j`` means token ``i`` attends to token ``j``. A value of False means token ``i`` does not attend to token ``j``. If no mask is specified, a causal mask is used by default. A :class:`~torch.nn.attention.flex_attention.BlockMask` for document masking in a packed sequence created via `create_block_mask `_. We use :func:`~torch.nn.attention.flex_attention.flex_attention` when computing attention with block masks. Default is None. input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b x s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Raises: NotImplementedError: If ``mask`` is provided, but mask is not an instance of ``torch.Tensor``. ValueError: If no ``y`` input and ``kv_cache`` is not enabled. Returns: torch.Tensor: output tensor with attention applied Notation used for tensor shapes: - b: batch size - s_x: sequence length for x - s_y: sequence length for y - n_h: num heads - n_kv: num kv heads - d: embed dim - h_d: head dim Nz5Block masks are not implemeted yet, use packed=False.rr)r4zAMust provide y input or use kv_cache to enable streaming decoding)sizer*g T%66 FF1c4,x7 G G   *##A#;;A KK1   ; " AA 9}$D,>$ W %A %AA  AA AAq#r4=11A".''Y'??q#t0!T]CCAq#t0!T]CCA~!222HHQT%6$-PPHHQT%6$-PP !S"dm44A !S"dm44A Aq!!A Aq!!A{&KKNN}(T-?(}++Aq111 t| q{{1a     <: s*"QX,, D : # #;t//11=!DDD  # /t,,H :"t77!; 8T%=%ABBCL;|q0$ FFD 88::??>>!$$D   'd..FZ''Fd..F$6<<>>r222::1==fa((!!!Q''224499!S"EE'''r()N)__name__ __module__ __qualname____doc__intrModulerrr"rWr r>r*r.r2r?rre __classcell__)r&s@r'rrsF&&f/3&*&*&*!-1'+/3)G#G#G#G# G#  G#  G# G# G# G#YG#!+G##G##G#7#G#G# !G#"#G#$&c]%G#&e_'G#( (})G#* +G#G#G#G#G#G#R&&&+k&@C& &&&&4   %)n( %),0 n(n(n( <n( EL !n( y! n( EL) n( n(n(n(n(n(n(n(n(r(r)loggingtypingrr>torch.nn.functionalr functionalrU!torchtune.modules.attention_utilsrtorchtune.modules.kv_cacher getLoggerrfr,rkrr(r'rus 777777......  8 $ $F(F(F(F(F(biF(F(F(F(F(r(