PiE0ddlZddlmZddlZddlmcmZddlmZddl m Z ddl m Z mZddlm Z ejeZGddejZGdd ejZd ejjd dfd Zd ejjd ejjfd ZdS)N)Optional)nn)KVCache) _MaskType_sdpa_or_flex_attentionc!xeZdZdZdddddddddeded ed ed ejd ejd ejdejdeejdeejdeejdeedede de ddf fdZ dede j deddfdZdZdddde jde jdeed ee jde jf d!ZxZS)"MultiHeadAttentionuN NOTE: torch.export.export() friendly MultiHeadAttention, modified from torchtune.modules.attention.MultiHeadAttention Major differences: - Rewrite `if y is None` to torch.cond(). - Logic becomes `if all values of y are NaN`, to make torch.compile() happy. - No input mutations in both false and true branches, so we need to copy kv values back into kv cache after torch.cond(). - Added a SDPA module - SDPA module includes transpose and expanding kv dimensions. - Makes it easy to swap with custom SDPAs that are needed by the users of exported program. - Uses new kv cache - This potentially can be merged with torchtune.modules.kv_cache. - Changed += to .add_ to avoid mutating module attributes. - Added clone() method. Multi-headed attention layer with support for grouped query attention (GQA) introduced in https://arxiv.org/abs/2305.13245v1. GQA is a version of multiheaded attention (MHA) which uses fewer key/value heads than query heads by grouping n query heads for each key and value head. Multi-Query Attention is an extreme version where we have a single key and value head shared by all query heads. Following is an example of MHA, GQA and MQA with num_heads = 4 (credit for the documentation: `litgpt.Config `_). :: ┌───┐┌───┐┌───┐┌───┐ ┌───┐ ┌───┐ ┌───┐ │ v ││ v ││ v ││ v │ │ v │ │ v │ │ v │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ │ │ │ ┌───┐┌───┐┌───┐┌───┐ ┌───┐ ┌───┐ ┌───┐ │ k ││ k ││ k ││ k │ │ k │ │ k │ │ k │ └───┘└───┘└───┘└───┘ └───┘ └───┘ └───┘ │ │ │ │ ┌──┴──┐ ┌──┴──┐ ┌────┬──┴─┬────┐ ┌───┐┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐┌───┐ ┌───┐┌───┐┌───┐┌───┐ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ │ q ││ q ││ q ││ q │ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ └───┘└───┘└───┘└───┘ ◀──────────────────▶ ◀──────────────────▶ ◀──────────────────▶ MHA GQA MQA n_kv_heads =4 n_kv_heads=2 n_kv_heads=1 Args: embed_dim (int): embedding dimension for the model num_heads (int): number of query heads. For MHA this is also the number of heads for key and value num_kv_heads (int): number of key and value heads. User should ensure ``num_heads % num_kv_heads == 0``. For standard MHA set ``num_kv_heads == num_heads``, for GQA ``num_kv_heads < num_heads``, and for MQA set ``num_kv_heads == 1``. head_dim (int): dimension of each head, calculated by ``embed_dim // num_heads``. q_proj (nn.Module): projection layer for query. k_proj (nn.Module): projection layer for key. v_proj (nn.Module): projection layer for value. output_proj (nn.Module): projection layer for output. pos_embeddings (Optional[nn.Module]): positional embeddings layer, e.g. RotaryPositionalEmbeddings. q_norm (Optional[nn.Module]): normalization layer for query, e.g. RMSNorm. For decoding, this is applied before updating from kv_cache. This means it will only support token wide normalization and not batch or sequence wide normalization. k_norm (Optional[nn.Module]): normalization layer for key, must be set if q_norm is. kv_cache (Optional[KVCache]): KVCache object used to cache key and value max_seq_len (int): maximum sequence length supported by the model. This is needed to compute the RoPE Cache. Default: 4096. is_causal (bool): sets the default mask to causal when no mask is provided attn_dropout (float): dropout value passed onto the scaled_dot_product_attention function. Default value is 0.0. Raises: ValueError: If ``num_heads % num_kv_heads != 0`` ValueError: If ``embed_dim % num_heads != 0`` ValueError: If ``attn_dropout < 0`` or ``attn_dropout > 1`` ValueError: if q_norm is defined without k_norm or vice versa NiT)pos_embeddingsq_normk_normkv_cache max_seq_len is_causal attn_dropout embed_dim num_heads num_kv_headshead_dimq_projk_projv_proj output_projr r r rrrrreturnc t||zdkrtd|d|d||zdkrtd|d|d|dks|dkrtd|d t| t| z rtd ||_||_||_||_||_| |_ ||_ | |_ ||_ ||_ ||_||_| |_| |_| |_t'|_t+|j|j|j|jr|jnd |j |j|j |_d |_dS)Nrz num_heads (z%) must be divisible by num_kv_heads ()z embed_dim (z") must be divisible by num_heads (zattn_dropout (z) must be between 0.0 and 1.0z!q and k norm must be set togetherr )rrrrr attention_fnrF)super__init__ ValueErrorboolrrrrrrrrrrrrr r r r_attention_callSDPAtraining_sdpa cache_enabled)selfrrrrrrrrr r r rrrr __class__s w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/_export/attention.pyr zMultiHeadAttention.__init__fs&  | #q ( (1i11!-111  y A % %+i++'+++  !  |a//ViVVVWW W <<$v,, & B@AA A#("(  &"!    &  , 788*n].2mD**n-]    # batch_sizedtypec|jtddSt|||j|j|d|_|j|j_d|_dS)aQSetup key value caches for attention calculation. If called after kv_cache is already setup, this will be skipped. Args: batch_size (int): batch size for the caches. dtype (torch.dtype): dtype for the caches. max_seq_len (int): maximum sequence length model will be run with. NzWKey value caches are already setup. You cannot call ``setup_caches()`` twice. Skipping.F)r,rrrr-transpose_cacheT)rloggerwarningInferenceKVCacherrr&r')r(r,r-rs r* setup_cachezMultiHeadAttention.setup_cachesz = $ NNi     -%'!. % DM#'-DJ !%D   r+cd|jtd|jdS)zReset the key value caches.Nz>Key value caches are not setup. Call ``setup_caches()`` first.)r RuntimeErrorreset)r(s r* reset_cachezMultiHeadAttention.reset_caches; = P  r+)mask input_posxyr8r9cL|j\}}|}jjz}||j|zj}j|}j|}fdfd} fd} j| Jd|\} } ntj tj | | | |f\} } } jj| jj| jj| || | ||}|S)a* Args: x (torch.Tensor): input tensor with shape [b x s_x x d] for the query y (torch.Tensor): second input tensor with shape [b x s_y x d], is the input for k and v. For self attention, x=y. If all values are NaN, we read from kv cache. mask (Optional[_MaskType]): Used to mask the scores after the query-key multiplication and before the softmax. Either: A boolean tensor with shape ``[b x s x s]``, ``[b x s x self.encoder_max_cache_seq_len]``, or ``[b x s x self.encoder_max_cache_seq_len]`` if using KV-cacheing with encoder/decoder layers. A value of True in row ``i`` and column ``j`` means token ``i`` attends to token ``j``. A value of False means token ``i`` does not attend to token ``j``. If no mask is specified, a causal mask is used by default. A :class:`~torch.nn.attention.flex_attention.BlockMask` for document masking in a packed sequence created via `create_block_mask `_. We use :func:`~torch.nn.attention.flex_attention.flex_attention` when computing attention with block masks. Default is None. input_pos (Optional[torch.Tensor]): Optional tensor which contains the position ids of each token. During training, this is used to indicate the positions of each token relative to its sample when packed, shape [b x s]. During inference, this indicates the position of the current token. If none, assume the index of the token is its position id. Default is None. Returns: torch.Tensor: output tensor with attention applied Notation used for tensor shapes: - b: batch size - s_x: sequence length for x - s_y: sequence length for y - n_h: num heads - n_kv: num kv heads - d: embed dim - h_d: head dim Nr9cb|jd}|}|}||dj}||dj}j|}j|}||fS)Nrr=)shaperrviewrr r )r;s_ykvbr9r(s r* calculate_kvz0MultiHeadAttention.forward..calculate_kvs'!*C AA AAq#r4=11Aq#r4=11A".''Y'??{&KKNNa4Kr+c^j}|j|j|jfSN)rclonek_cachev_cache cache_pos)r;rr(s r*true_fnz+MultiHeadAttention.forward..true_fn's,}**,,H#X%5x7II Ir+c|\}}j}||||j|j|jfSrH)rrIupdaterJrKrL)r;rCrDrrFr(s r*false_fnz,MultiHeadAttention.forward..false_fn+sQ<??DAq}**,,H OOAq ! ! !#X%5x7II Ir+zAMust provide y input or use kv_cache to enable streaming decoding)r8)r@rrrrArr r rtorchcondisnanallitemrJcopy_rKrLr&r)r(r:r;r8r9s_x_qq_per_kvrMrPrCrDrLoutputrErFs` ` @@r*forwardzMultiHeadAttention.forwards\G 3 KKNN>T%66 FF1c4,x7 G G   *##A#;;A ; " AA       ( J J J J J J J J J J J =  R <??DAqq $j A""$$))++WhOAq) M ! ' ' * * * M ! ' ' * * * M # ) )) 4 4 4Aq!Q$77'''r+)__name__ __module__ __qualname____doc__intrModulerrr"floatr rQr-r3r7Tensorrr\ __classcell__r)s@r*r r sNNv/3&*&*&*!#H#H#H#H# H#  H#  H# H# H# H#YH#!+H##H##H#7#H#H# !H#"#H#$ %H#H#H#H#H#H#T&&&+k&@C& &&&&8%),0 p(p(p( <p( <p( y! p( EL) p( p(p(p(p(p(p(p(p(r+r ceZdZdZdedededededdf fd Z dd ej d ej d ej d edede e dej fdZ xZ S)r$zr TorchTune's SDPA which can be optimized and can be swapped out for a more efficient implementations. rrrrrrNct||_||_||_|j|jz|_||_||_||_||_ dSrH) rr rrrrZrr _attention_fnr) r(rrrrrrrr)s r*r z SDPA.__init__Msd ("  $*;; (")  r+rYrCrDbszseq_lenr8c |dd}|dd}|dd}|j|jkrdd|jddf}|d|dd}|d|dd}||||||j|j duo |duo|j }|dd ||dS)Nrr?)r8 dropout_pr) transposerrrZ unsqueezeexpandflattenrirrr contiguousrA) r(rYrCrDrjrkr8 expand_shaper[s r*r\z SDPA.forwardas7 KK1   KK1   KK1   >T. . .DM2r:L A%%l33;;AqAAA A%%l33;;AqAAA## 'mt+O O $  1%%002277WbIIIr+rH)r]r^r_r`rarcr"r rQrdrrr\rerfs@r*r$r$Gs !!! !  !  ! !!!!!!6%)!J!J <!J <!J < !J  !J  !Jy!!J !J!J!J!J!J!J!J!Jr+r$modulerc~|D]\}}t|tjryt ||t|j|j|j|j|j |j |j |j |j |j|j|j|j|j|jt)|dS)N)rrrrrrrrr r r rrrr)named_children isinstanceTorchTuneAttentionr setattrrrrrrrrrr r r rrrrreplace_mha_with_inference_mha)runamechilds r*_replace_mha_with_inference_mhar~s,,..22 e e/B C C 2 "#o#o!&!3"^ < < < % 1#(#7 < <"^ % 1#o!&!3    , +5 1 1 1 1122r+c$t||S)z Replace TorchTune's MHA with an inference friendly version of MHA that separates out the inference-related parts for further optimization. )r~)rus r*r{r{s $F+++ Mr+)loggingtypingrrQtorchtune.modules.attentionmodules attentionryr"torchtune.modules._export.kv_cacherr2!torchtune.modules.attention_utilsrrtorchtune.modules.kv_cache getLoggerr]r0rbr r$r~r{r+r*rsj 888888888JJJJJJPPPPPPPP......  8 $ $o(o(o(o(o(o(o(o(d ;J;J;J;J;J29;J;J;J|2EHO22222858?uxr+