PiB nddlmZddlmZmZmZddlmZddlm Z de de de de d e d e d e d e d e de fdZ dS))nn) T5EncoderT5EncoderLayerT5EncoderSelfAttention) FeedForward)RMSNorm embed_dimmlp_dim num_headshead_dim num_layersrel_pos_num_bucketsrel_pos_max_dist vocab_sizenorm_eps max_seq_lenc tj||} tj} t|D]} t |||tj||dtj||dtj||dtj||d} t tj||dtj||dtj||dtj}t| |t||t||}| |t||}t| | ||||| S)aM Builder for the T5 encoder. T5 paper: https://arxiv.org/abs/1910.10683 Args: embed_dim (int): The model dimension. mlp_dim (int): The inner dimension of the feed forward layers. num_heads (int): The number of attention heads. head_dim (int): The dimension of the attention heads (should equal `embed_dim // num_heads`) num_layers (int): Number of encoder layers. rel_pos_num_buckets (int): Number of discrete buckets to divide the relative positions into. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` rel_pos_max_dist (int): Maximum distance for relative positions. Distances beyond this are grouped into the last bucket. See: :class:`~torchtune.models.t5._encoder.T5EncoderRelativePositionBias` vocab_size (int): Vocab size of the model's tokenizer. norm_eps (float): Small value added to denominator for numerical stability. max_seq_len (int): The maximum sequence length (context length) of the model. Returns: T5Encoder F)bias)r r r q_projk_projv_proj output_proj) gate_proj down_projup_proj activation)eps)attnmlpsa_normmlp_norm)token_embeddinglayers final_normr rrr) r Embedding ModuleListrangerLinearrGELUrrappendr)r r r r r rrrrrr"r#_rrlayerr$s {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/t5/_component_builders.py t5_encoderr.sFl:y99O ]__F :  %9Y >>>9Y >>>9Y >>> )YUCCC   i 7???i???Iiu===wyy    I8444YH555      e111J '/)   N) torchrtorchtune.models.t5._encoderrrrtorchtune.modules.feed_forwardrtorchtune.modules.rms_normrintfloatr.r/r-r7s  766666......JJ JJ J  J  JJJJJJJJJJr/