Pi4ddlZddlmZddlZddlmZddlmZmZddlm Z ej e Z Gddej ZdS)N)Optional)nn) _MaskType_sdpa_or_flex_attention)KVCachec!eZdZdZdddddddddeded ed ed ejd ejd ejdejdeejdeejdeejdeedede de ddf fdZ dede j deddfdZdZ d"dddde jdee jdeed ee jde jf d!ZxZS)#MultiHeadAttentionu/Multi-headed attention layer with support for grouped query attention (GQA) introduced in https://arxiv.org/abs/2305.13245v1. GQA is a version of multiheaded attention (MHA) which uses fewer key/value heads than query heads by grouping n query heads for each key and value head. Multi-Query Attention is an extreme version where we have a single key and value head shared by all query heads. Following is an example of MHA, GQA and MQA with num_heads = 4 (credit for the documentation: `litgpt.Config `_). :: ┌───┐┌───┐┌───┐┌───┐ ┌───┐ ┌───┐ ┌───┐ │ v ││ v ││ v ││ v │ │ v │ │ v │ │ v │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ │ │ │ ┌───┐┌───┐┌───┐┌───┐ ┌───┐ ┌───┐ ┌───┐ │ k ││ k ││ k ││ k │ │ k │ │ k │ │ k │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ ┌──┴──┐ ┌──┴──┐ ┌────┬──┴─┬────┐ ┌───┐┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐┌───┐ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ ◀──────────────────▶ ◀──────────────────▶ ◀──────────────────▶ MHA GQA MQA n_kv_heads =4 n_kv_heads=2 n_kv_heads=1 Args: embed_dim (int): embedding dimension for the model num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure ``num_heads % num_kv_heads == 0``. For standard MHA set ``num_kv_heads == num_heads``, for GQA ``num_kv_heads < num_heads``, and for MQA set ``num_kv_heads == 1``. head_dim (int): dimension of each head, calculated by ``embed_dim // num_heads``. q_proj (nn.Module): projection layer for query. k_proj (nn.Module): projection layer for key. v_proj (nn.Module): projection layer for value. output_proj (nn.Module): projection layer for output. pos_embeddings (Optional[nn.Module]): positional embeddings layer, e.g. RotaryPositionalEmbeddings. q_norm (Optional[nn.Module]): normalization layer for query, e.g. RMSNorm. For decoding, this is applied before updating from kv_cache. This means it will only support token wide normalization and not batch or sequence wide normalization. k_norm (Optional[nn.Module]): normalization layer for key, must be set if q_norm is. kv_cache (Optional[KVCache]): KVCache object used to cache key and value max_seq_len (int): maximum sequence length supported by the model. This is needed to compute the RoPE Cache. Default: 4096. is_causal (bool): sets the default mask to causal when no mask is provided attn_dropout (float): dropout value passed onto the scaled_dot_product_attention function. Default value is 0.0. Raises: ValueError: If ``num_heads % num_kv_heads != 0``, **or** if ``embed_dim % num_heads != 0``, **or** if ``attn_dropout < 0`` or ``attn_dropout > 1``, **or** if q_norm is defined without k_norm or vice versa NiT)pos_embeddingsq_normk_normkv_cache max_seq_len is_causal attn_dropout embed_dim num_heads num_kv_headshead_dimq_projk_projv_proj output_projr r r rrrrreturncdt||zdkrtd|d|d||zdkrtd|d|d|dks|dkrtd|d t| t| z rtd ||_||_||_||_||_| |_ ||_ | |_ ||_ ||_ ||_||_| |_| |_| |_t'|_d |_dS) Nrz num_heads (z%) must be divisible by num_kv_heads ()z embed_dim (z") must be divisible by num_heads (zattn_dropout (z) must be between 0.0 and 1.0z!q and k norm must be set togetherF)super__init__ ValueErrorboolrrrrrrrrrrrrr r r r_attention_call cache_enabled)selfrrrrrrrrr r r rrrr __class__s o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/attention.pyrzMultiHeadAttention.__init__Ss&  | #q ( (1i11!-111  y A % %+i++'+++  !  |a//ViVVVWW W <<$v,, & B@AA A#("(  &"!    &  , 788 # batch_sizedtypec|jtddSt|||j|j||_d|_dS)aQSetup key value caches for attention calculation. If called after kv_cache is already setup, this will be skipped. Args: batch_size (int): batch size for the caches. dtype (torch.dtype): dtype for the caches. max_seq_len (int): maximum sequence length model will be run with. NzWKey value caches are already setup. You cannot call ``setup_caches()`` twice. Skipping.)r(rrrr)T)rloggerwarningrrrr#)r$r(r)rs r& setup_cachezMultiHeadAttention.setup_cachesk = $ NNi     $%'!. DM"&D   r'cd|jtd|jdS)zReset the key value caches.Nz>Key value caches are not setup. Call ``setup_caches()`` first.)r RuntimeErrorreset)r$s r& reset_cachezMultiHeadAttention.reset_caches; = P  r')mask input_posxyr2r3c |j\}}}| |jdnd}||} |j|jz} | |||j| z|j} |j|| |} | dd} |j|| } |6|j |j std|j j } |j j } n||} ||} | ||d|j} | ||d|j} |j|| |} | dd} | dd} |j|| } |j %|j r|j | | \} } |j|jkr||j| d|jf} | d| dd} | d| dd} || | | ||jr|jnd|j duo |duo|j }|dd||d}||S) a Args: x (torch.Tensor): input tensor with shape [b x s_x x d] for the query y (Optional[torch.Tensor]): second input tensor with shape [b x s_y x d], is the input for k and v. For self attention, x=y. Optional only with kv_cache enabled. mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Either: A boolean tensor with shape ``[b x s x s]``, ``[b x s x self.encoder_max_cache_seq_len]``, or ``[b x s x self.decoder_max_cache_seq_len]`` if using KV-cacheing with encoder/decoder layers. A value of True in row ``i`` and column ``j`` means token ``i`` attends to token ``j``. A value of False means token ``i`` does not attend to token ``j``. If no mask is specified, a causal mask is used by default. A :class:`~torch.nn.attention.flex_attention.BlockMask` for document masking in a packed sequence created via `create_block_mask `_. We use :func:`~torch.nn.attention.flex_attention.flex_attention` when computing attention with block masks. Default is None. input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b x s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Raises: ValueError: If no ``y`` input and ``kv_cache`` is not enabled. Returns: torch.Tensor: output tensor with attention applied Notation used for tensor shapes: - b: batch size - s_x: sequence length for x - s_y: sequence length for y - n_h: num heads - n_kv: num kv heads - d: embed dim - h_d: head dim Nrr)r3zAMust provide y input or use kv_cache to enable streaming decodingr )r2 dropout_pr)shaperrrviewrr transposer rr#r k_cachev_cacherrr update unsqueezeexpandflattenr"trainingrr contiguousr)r$r4r5r2r3bs_x_s_yqq_per_kvkv expand_shapeoutputs r&forwardzMultiHeadAttention.forwardsbG 3Magajjq KKNN>T%66 FF1c4,x7 G G   *##A#;;A KK1   ; " AA 9}$D,>$ W %A %AA  AA AAq#r4=11Aq#r4=11A".''Y'?? Aq!!A Aq!!A{&KKNN}(T-?(}++Aq111 >T. . .t0(B NL A%%l33;;AqAAA A%%l33;;AqAAA%% +/=Ad''cmt+O O &  !!!Q''224499!S"EE'''r')N)__name__ __module__ __qualname____doc__intrModulerrr!floatrtorchr)r-r1TensorrrO __classcell__)r%s@r&r r s>>V/3&*&*&*!#?#?#?#?# ?#  ?#  ?# ?# ?# ?#Y?#!+?##?##?#7#?#?# !?#"#?#$ %?#?#?#?#?#?#B&&&+k&@C& &&&&4%)z( %),0 z(z(z( <z( EL !z( y! z( EL) z( z(z(z(z(z(z(z(z(r'r )loggingtypingrrWr!torchtune.modules.attention_utilsrrtorchtune.modules.kv_cacher getLoggerrPr+rUr r'r&r`s PPPPPPPP......  8 $ $^(^(^(^(^(^(^(^(^(^(r'