Pis' UddlZddlmZmZmZmZddlZddlmZddlm Z ddl m Z m Z e Z ejed<e rjddlmZmZmZdZeZejd d ejd ejd ejdedejf dZeejefZnejZdeejdejfdZdeejdejfdZdeejdefdZdefdZdS)N)CallableListOptionalUnion)nn)_SUPPORTS_FLEX_ATTENTION) get_loggerlog_once_log) BlockMaskcreate_block_maskflex_attentionc@ tjtdS#t$rv}td|d tjtddcYd}~S#t$r$}td|dd}~wwxYwd}~wwxYw) NF)dynamicz,Compiling flex_attention failed with error 'z%'. Retrying with mode='max-autotune'.z max-autotune)rmodez-Compiling flex_attention failed with error: 'z', Updating your pytorch version to nightlies may solve it, or you can setin your config dataset.packed=False to avoid using flex attention.)torchcompiler Exceptionr info)es u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/attention_utils.pycompile_flex_attentionrs =??? ?    IIgqggg    }^UXXXXXXXXX    YAYYY    s8 BBA'!B' B1BBBBF) recursiveqkv block_maskreturnc(t||||S)Nr)flex_attention_compiled)rrrrs rcompile_friendly_flex_attentionr"2s'q!Q:FFFFseq_lensct|}g}t|D]H}tjdt ||D}||Itj|}|S)a Convert a batch tensor of seq lens into integer IDs denoting sample ownership. For example, seq_lens = [2, 3, 1] would return [0, 0, 1, 1, 1, 2]. Args: seq_lens (List[torch.Tensor]): Sequence lengths of samples in each pack in the batch, shape (batch_size, n), where n is the max number of sequences in a pack and can vary across packs. Returns: Tensor: Document IDs of shape (batch_size, max_seq_len). cdg|]-\}}tj|f|tj|j.S)dtypedevice)rfulllongr).0iseq_lens r z3_get_document_ids_from_seq_lens..UsE   Aw G:q 7>RRR   r#)lenrangercat enumerateappendstack)r$ batch_sizebatch_document_ids sample_idx document_idss r_get_document_ids_from_seq_lensr;@sXJJ'' 0 0 y  "+HZ,@"A"A     !!,////%788 r#cg}t|}t|D]B}dt||D}|t j|Ct j|S)a  Given a batch tensor of seq lens defining the lengths of samples in each pack, Construct a 2D block causal mask for each pack in the batch. For example, if a single sample's seq_lens is [3, 2, 1], the mask would be:: mask = [ [1, 0, 0, 0, 0, 0], [1, 1, 0, 0, 0, 0], [1, 1, 1, 0, 0, 0], [0, 0, 0, 1, 0, 0], [0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 1], ] Args: seq_lens (List[torch.Tensor]): Sequence lengths of samples in each pack in the batch, shape (batch_size, n), where n is the max number of sequences in a pack and can vary across packs. Returns: Tensor: Block causal mask of shape (batch_size, max_seq_len, max_seq_len). c g|]>\}}tjtj||tj|j?Sr')rtrilonesboolr)r,s rr0z,create_block_causal_mask..zsT   7 J 7G5:gnUUU     r#)r1r2r4r5r block_diagr6)r$batch_block_attn_masksr7r9block_attn_maskss rcreate_block_causal_maskrD_s0 XJJ''KK   ((<==     %%e&68H&IJJJJ ;- . ..r#ctrHt|j\}}dfd}t ||d||dSt |S)a Create a block causal document mask for a batch of packed sequences. If flex attention is supported by the current hardware, block causal logic and passing this into :func:`torch.nn.attention.flex_attention.create_block_mask`. The resultant BlockMask is a compressed representation of the full block causal mask. If on an older version, a standard 2D block causal mask is created and returned. Args: seq_lens (List[torch.Tensor]): Sequence lengths of samples in each pack in the batch, shape (batch_size, n), where n is the max number of sequences in a pack and can vary across packs. Returns: _MaskType: BlockMask or Tensor if torch version < 2.5.0. cudacF||k}||f||fk}||zS)a Defines the logic of a block causal mask by combining both a standard causal mask and a block diagonal document mask. See :func:`~torchtune.modules.attention_utils.create_block_causal_mask` for an illustration. )bhq_idxkv_idx causal_mask document_maskr:s rmask_modz*packed_block_causal_mask..mask_mods5 6/K(E2l1f96MMM. .r#N)r))r$)rr;shapetocreate_block_causal_mask_flexrD)r$r7 max_seq_lenrOr:s @rpacked_block_causal_maskrTs$ ;6x@@ "."4 K#v..  / / / / /-          (::::r#cftrUdtjdtjdtjdttdt dt dtjfd}nTdtjdtjdtjdttdt dt dtjfd }|S) aE Helper function to decide when to call flex attention or SDPA. It will use flex attention if ALL of the following conditions are met, otherwise it will default to SDPA: - torch version >= 2.5.0 - we are sample packing, therefore mask is a BlockMask - torch.cuda.get_device_capability() >= (7, 5) rrrmask dropout_p is_causalrc0t|trIttdtj|dkrt dt||||S||dddddddf}tj ||||||S)NzOUsing flex attention for attention computation since a BlockMask was passed in.)levelgzCFlex attention does not support dropout. Please set dropout to 0.0.r  attn_maskrWrX) isinstancer r r loggingDEBUG ValueErrorr"r functionalscaled_dot_product_attentionrrrrVrWrXs r_attention_callz0_sdpa_or_flex_attention.._attention_calls$ ** e!- s??$]7# #4AAA .D}AA"'' Br#ct||dddddddf}tj||||||S)Nr[)rrarbrcs rrdz0_sdpa_or_flex_attention.._attention_calls[AAAtQQQM*===## > r#)rrTensorr _MaskTypefloatr@)rds r_sdpa_or_flex_attentionris G- |- |- |- 9% -   -   - \- - - - - b | | | 9%        \    , r#) r^typingrrrrrrtorchtune.utils._import_guardrtorchtune.utils._loggingr r r Logger__annotations__!torch.nn.attention.flex_attentionr r rRrrr!compilerdisablerfr"rgr;rDrTrirHr#rrrs-222222222222 BBBBBB99999999!z||gn###+ &5466  ^e,,G <G <G <G G  GGG-,GelI-.II I5<  \>#/tEL'9#/el#/#/#/#/L1;5< 1;1;1;1;1;hSSSSSSSr#