Pid/ddlZddlmZmZmZddlZddlmcmZ ddlmZm Z ddl m Z ddl mZGddejZGdd ejZGd d ejZGd d ejZdededede fdZdS)N)ListOptionalUnion)nnTensor)MultiHeadAttention) _get_clonesceZdZdZdddejdeejeejej fdejde de d e d e d e e d dffd Z de d e fdZxZS) T5Encodera The T5 encoder module. T5 paper: https://arxiv.org/abs/1910.10683 Args: token_embedding (nn.Embedding): PyTorch embedding layer to place tokens in an embedding space. layers (Union[nn.Module, List[nn.Module], nn.ModuleList]): A single encoder layer. final_norm (nn.Module): Module that applies normalization to the output of the encoder num_heads (int): The number of attention heads. rel_pos_num_buckets (int): Number of discrete buckets to divide the relative positions into. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` rel_pos_max_dist (int): Maximum distance for relative positions. Distances beyond this are grouped into the last bucket. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` max_seq_len (int): The maximum sequence length (context length) of the model. num_layers (Optional[int]): Number of encoder layers, only define when layers is not a list. Raises: AssertionError: If ``num_layers`` is set and layer is a list, **or** ``num_layers`` is not set and layer is an ``nn.Module``. N) num_layerstoken_embeddinglayers final_norm num_headsrel_pos_num_bucketsrel_pos_max_dist max_seq_lenr returnct||_||_||_t |||||_d|_t|tj r ||_dSt|trtj ||_dSt|tj std|tdt|||_dS)N) num_bucketsmax_distrrz.num_layers is defined, layers must be a modulez0num_layers is not defined, layers must be a list)super__init__r rrT5EncoderRelativePositionBiasrelative_position_biasr isinstancer ModuleListlistModuleAssertionErrorr ) selfr rrrrrrr __class__s p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/t5/_encoder.pyrzT5Encoder.__init__+s .$&&C+%# ' ' ' # fbm , , : DKKK  % % :-//DKKKfbi00 W$%UVVV!$%WXXX%fj99DKKKtokensc |j\}}||jkrtd|d|jd||}|}|jD]}|||}||S)a Args: tokens (Tensor): input tensor with shape ``[bsz, max_seq_len]`` Returns: Tensor: output tensor with shape [bsz, max_seq_len, embed_dim] Raises: ValueError: if seq_len of tokens is bigger than max_seq_len z seq_len (z6) of input tensor should be smaller than max_seq_len ())shaper ValueErrorr rrr)r!r%bszseq_lenx rel_pos_biaslayers r#forwardzT5Encoder.forwardNs| W T% % %9G99%)%5999    ( (2244 [ ' 'Ea&&AAq!!!r$)__name__ __module__ __qualname____doc__r Embeddingrrrrintrrrr/ __classcell__r"s@r#r r sF%)!:!:!:!:bibi"-?@ !: I !:  !:!!:!:!:SM!: !:!:!:!:!:!:F"f"""""""""r$r c jeZdZdZdedejdejdejddf fd Zd ed edefd Z xZ S) T5EncoderLayeray Single layer of the T5 encoder (standard transformer layer with relative position bias). Args: attn (MultiHeadAttention): Attention module. mlp (nn.Module): Feed-forward module. sa_norm (nn.Module): Normalization to be applied before self-attention. mlp_norm (nn.Module): Normalization to be applied before the feed-forward layer. attnmlpsa_normmlp_normrNct||_||_||_||_dS)N)rrr:r;r<r=)r!r:r;r<r=r"s r#rzT5EncoderLayer.__init__ys;     r$r,r-c|||||z}||||z}|S) Args: x (Tensor): input tensor with shape [bsz, seq_len, embed_dim] rel_pos_bias (Tensor): relative position bias with shape [1, num_heads, max_seq_len, max_seq_len] See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` Returns: Tensor: output tensor with shape [bsz, seq_len, embed_dim] )r:r<r;r=)r!r,r-s r#r/zT5EncoderLayer.forwardsM  $,,q//<88 8 q))** *r$) r0r1r2r3rrrrrr/r6r7s@r#r9r9ns !  !Y ! ! ) !  ! ! ! ! ! !  v &        r$r9c|eZdZdZdedededejdejdejdejffd Zd ed ed efd Z xZ S)T5EncoderSelfAttentionaR Self-attention for the T5 encoder. Standard self-attention with two differences: - No scaling factor - Add "relative position bias" to the attention scores. (See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias`) Args: embed_dim (int): The model dimension. num_heads (int): Number of attention heads. head_dim (int): Dimension of the attention heads (should equal `embed_dim // num_heads`) q_proj (nn.Module): Projection layer for query. k_proj (nn.Module): Projection layer for key. v_proj (nn.Module): Projection layer for value. output_proj (nn.Module): Projection layer for output. Raises: ValueError: If ``embed_dim % num_heads != 0``, **or** if ``embed_dim // num_heads != head_dim`` embed_dimrhead_dimq_projk_projv_proj output_projct||zdkrtd|d|d||z|krtd|d||_||_||_||_||_||_dS)Nrz embed_dim (z") must be divisible by num_heads (r'z head_dim (z)) must be equal to embed_dim // num_heads) rrr)rrDrErFrGrH) r!rCrrDrErFrGrHr"s r#rzT5EncoderSelfAttention.__init__s  y A % %+i++'+++   !X - -PXPPP #     &r$r,r-rc`|j\}}}||}||}||}||||j|jdd}||||j|jdd}||||j|jdd}tj ||dd} | |z } tj | d | j} tj | |} | dd|||} || S)r@)dim)r(rErFrGviewrrD transposetorchmatmulFsoftmaxfloattodtypereshaperH) r!r,r-r*r+rCqkv attn_score attn_weightattn_outs r#r/zT5EncoderSelfAttention.forwardsn#$'Wi KKNN KKNN KKNN FF3 ? ? I I!Q O O FF3 ? ? I I!Q O O FF3 ? ? I I!Q O O\!Q[[R%8%899 l" i 0 0 2 2;;;>>z?OPP < Q//%%a++33C)LL)))r$) r0r1r2r3r5rrrrr/r6r7s@r#rBrBs.''' '  '  ' 'Y'''''':**v*&********r$rBc@eZdZdZdedededeffd ZdefdZxZS) ra- Computes binned birectional relative position bias for the T5 encoder. It places relative positions into buckets and for each bucket, learns bias values for each attention head. Args: num_buckets (int): Number of discrete buckets to divide the relative positions into. max_dist (int): Maximum distance for relative positions (distances beyond this are grouped into the last bucket) num_heads (int): Number of attention heads in the transformer. max_seq_len (int): Maximum sequence length (context length). rrrrct||_tj|||_|dt|||ddS)Nrelative_position_to_bucketF) persistent)rrrrr4 embeddingregister_buffer#_calc_birectional_rel_pos_to_bucket)r!rrrrr"s r#rz&T5EncoderRelativePositionBias.__init__sv &k9==  ) / X{ S S      r$rc||j}|gddS)z Returns: torch.Tensor: relative position bias tensor with shape [1, num_heads, max_seq_len, max_seq_len] )rLrrKr)rdrbpermute unsqueeze)r!r,s r#r/z%T5EncoderRelativePositionBias.forward s> NN4; < <yy##--a000r$) r0r1r2r3r5rrr/r6r7s@r#rrs    *- := LO      1 1 1 1 1 1 1 1 1r$rrrrrctj|tjdddf}tj|tjdddf}||z }tj|}|dz}|dz}||k} |tj||z t j||z z ||z ztjz} tj| tj | |dz } |dktj|ztj | || z} | S)a Calculate the mapping from relative positions to bucket indices. NOTE: This is for the T5 encoder (birectional), not the decoder (unidirectional). Args: num_buckets (int): Number of discrete buckets to divide the relative positions into. max_dist (int): Maximum distance for relative positions (distances beyond this are grouped into the last bucket) max_seq_len (int): Maximum sequence length (context length). Returns: Tensor: shape=[max_seq_len, max_seq_len], range=[0, num_buckets] )rXNrLrKr) rRarangelongabslogrVmathrWmin full_likewhere) rrrquery_positions key_positionsrelative_positionsabs_relative_positionshalf_num_buckets max_exactis_exactrelative_position_if_not_exactrbs r#rfrfsb l;ejAAA!!!T'JOLEJ???aaaHM&8"Y'9:: #a' A%I% 1H&/ (..009<== (8i' ( ( ) i ' ) bnn &" &+Y& 68H18LMM&&" $6#9"="= ### ;(*H# '&r$)rotypingrrrrRtorch.nn.functionalr functionalrTrtorchtune.modulesrtorchtune.modules.transformerr rr r9rBrr5rfr$r#rs (((((((((( 000000555555Z"Z"Z"Z"Z" Z"Z"Z"z$$$$$RY$$$NT*T*T*T*T*RYT*T*T*n&1&1&1&1&1BI&1&1&1R.'.' #.'25.' .'.'.'.'.'.'r$