.`ihddlZddlZddlmcmZddlmZmZGddejZdGde dejjfdZ dHd e d e e d e d e dejf d Z GddejZGddejZGddejZGddejZGddejZGddejZdede dedede e de e d e e ddfd!ZGd"d#ejZGd$d%ejZGd&d'ejZGd(d)ejZGd*d+ejZGd,d-ejjZ dId/ed0e d1e d2e d3ed4e defd5ZGd6d7ejZ Gd8d9ee Z!d:ed;edzdefd<Z"Gd=d>ejZ#Gd?d@ejj$Z%dAe dBe de fdCZ&dDedEe defdFZ'dS)JN)Tensornnc0eZdZdZdededdffd ZxZS) BlockBasezBlock abstract module input_size output_sizereturnNcdt||_||_dSN)super__init__rr)selfrr __class__s {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/phi4mm_utils.pyr zBlockBase.__init__s. $&)__name__ __module__ __qualname____doc__intr __classcell__rs@rrrsX'3'S'T''''''''''rrrelunamer cF|}|dkrtjdS|dkrtjS|dkrtjS|dkrtjS|dkrtjSt|)zSelect an activation function by name Args: name: str activation function name, one of ["relu", "gelu", "swish", "sigmoid"], default "relu". rT)inplacegeluswishsigmoididentity)lowerrReLUGELUSiLUSigmoidIdentityNotImplementedError)rs rget_activationr(s ::<>ABTWL qqq!do-qqq01DOdo.AA111DEG  qqq!do-qqq01DG;t||aaa4?Q+>>ABTWL@@qqq!do-qqq01T\\aaa4?Q+>>AB66 IIiii r)rTF rrrrrrTboolr rrSrrs@rrVrV{s<" #B#B#B#B #B  #B  #B#B #B#B#B#B#B#BJFrrVc PeZdZdZ d dedededededd f fd Zd edefd ZxZS)DepthWiseSeperableConv1da+DepthWiseSeperableConv1d module used in Convnet module for the conformer, for more details see: https://arxiv.org/pdf/2005.08100v1.pdf Args: input_dim: int input channel size. depthwise_seperable_out_channel: int if set different to 0, the number of depthwise_seperable_out_channel will be used as a channel_out of the second conv1d layer. otherwise, it equals to 0, the second conv1d layer is skipped. kernel_size: int kernel_size depthwise_multiplier: int number of input_dim channels duplication. this value will be used to compute the hidden channels of the Conv1D. padding: int, optional padding for the conv1d, default: 0. rrWdepthwise_seperable_out_channelrYdepthwise_multiplierr_r Ncttj|||z|d|||_|dkr!tj||z|ddd|_ntj|_||_dS)Nr.)r_groupsr)r r rr`dw_convpw_convr&rp)rrWrprYrqr_rs rr z!DepthWiseSeperableConv1d.__init__s y  , ,        +a / /900/ DLL;==DL/N,,,rrLcp||}|jdkr||}|S)z4 Args: x: input tensor r)rtrprurks rrSz DepthWiseSeperableConv1d.forwards5 LLOO  /1 4 4 QAr)r rrrrrr rrSrrs@rroros: OOO*-O O " O  O OOOOOO>  F        rroc#eZdZdZ dded ed ed ed ed edededededededededededdf"fd ZddZ de de fdZ xZ S) ConvModuleay ConvModule Module for the conformer block. for more details see: https://arxiv.org/pdf/2005.08100v1.pdf Args: input_dim: int input channel size. ext_pw_out_channel: int if > 0, ext_pw_out_channel is a dim channel size for the last pointwise conv after swish activation. depthwise_seperable_out_channel: int if set different to 0, the number of depthwise_seperable_out_channel will be used as a channel_out of the second conv1d layer. otherwise, it equal to 0, the second conv1d layer is skipped. ext_pw_kernel_size: int kernel size of the conv pointwise of the conformer. kernel_size: int kernel size. depthwise_multiplier: int number of input_dim channels duplication. this value will be used to compute the hidden channels of the Conv1D. dropout_rate: float dropout rate. causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation. default False chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. chunk_size: int, optional chunk size for cnn. default 18 activation: str, optional activation function used in ConvModule, default: "relu". glu_type: str, optional activation function used for the glu, default: "sigmoid". bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. export: bool, optional, if set to True, padding is equal to 0. This is for inference, or onnx export. Typically this is set by the export program or the decoder program, and it isn't present in your config file. default False FrrrTrWext_pw_out_channelrpext_pw_kernel_sizerYrq dropout_rater\ batch_normchunk_se chunk_size activationrZr[linear_glu_in_convmexportr Ncttj||_||_||_||_||_| |_ ||_ ||_ ||_ | | |_||_| rtj||_t%| |_tj||_||_|r |rdn|dz }n|dz dz}t/||||||_|dkr$||krtj|||_dSdS|dkrtj||z||_dSdS)Nrr.rNr^)r r r LayerNorm layer_normrWr{r|rprZr[rr\_add_ext_pw_layerr~rY BatchNorm1dbn_layerr(actDropoutdropoutrrodw_sep_conv_1dLinearln2)rrWr{rpr|rYrqr}r\r~rrrrZr[rrr_rs rr zConvModule.__init__Zs& ,y11""4"4/N,  &#6      $&  6N955DM!*--z,//   -!6aa{QGG"Q1,G6  +       +a / /;;;9%DiPP<;$q((9Y1E%EyQQ)(rc(tjx|_x|_x|_|_tj|_dx|_|_|j dkrF|j rPtj |j |j |j d|j dz |_|j dkrd|_nGd|_n?tj |j |j |j d|j dz dz|_d|_|jr,t|j |j |j|j|_n7t%|j |j |j |j|j|j |_|j |j kr-d|_tj|j |j |_dSd|_dSt(jt)jd|_t(jt)jd|_dS) z This function is an extension of __init__ function and dedicated to the convolution module creation of the conformer. Frr.r^TrNN)rr&ln1glurrasqueeze_excitation apply_ln1fix_len1r{r\r`rWr|r GLULinearrZr[rVrr1rconespw_conv_simplify_wrdpw_conv_simplify_brs rrzConvModule._add_ext_pw_layers KMM  48 dmd.A#%+--)..  "a ' '{ &&(iN++!4q8 '''#*Q..$(DMM$)DMM&(iN++!4q8Q> '''#!& ' $N+M$ ,N++M$K ~!888!%9T%t6::aaa9T4q899111<=~ HHQKKd-a0043J13MMCd-a0043J13MMCc A IIiii     " " ; 34+a//!!!QQQ1D,q01112A 4   % )))$$A A )))$$A ? ! a  A HHQKK  "a ' '##A&&A} >aaa< 7! ;<<<=~ )IIiii((HHQKKIIiii(( )))$$AA A&&|||44AD+A..1H1KKA ! A LLOOr) FFrrzrrTFFr N) rrrrrfloatrmrTr rrrSrrs@rryrysf88F  ! $)#<R<R<R <R*- <R  <R  <R"<R<R<R<R<R<R<R<R<R "!<R"#<R$ %<R<R<R<R<R<R|;I;I;I;Iz11F11111111rryc NeZdZdZ d dededededd f fd Zd edefd Z xZ S)ra`Linear + GLU module Args: input_dim: int input size output_dim: int output size. glu_type: activation function name used in glu module. default "sigmoid" (swish function). bias_in_glu: bool, optional If True, the addtive bias is added. Default False. rTrWrXrZr[r Ncttj||dz||_t d||_dS)NrNr0)r r rrlinearrGrb)rrWrXrZr[rs rr zGLULinear.__init__sI i :>;GG 2x(( rrLcV||}||S)zFGLULinear forward Args: x: input tensor. )rrbrks rrSzGLULinear.forward#s# KKNN||ArrTrlrs@rrr s  $" ) ) ) ) )  )  ) ) ) ) ) )Frrc ReZdZdZ ddededededed d f fd Zd e d e fd Z xZ S) FeedForwardaFeedForward Module. For more details see Conformer paper: https://arxiv.org/pdf/2005.08100.pdf Args: d_model: int input size. d_inner: int output size. dropout_rate: float, dropout rate. activation: str, activation function name, one of ["relu", "swish", "sigmoid"], sigmoid activation is only used with "glu_in_fnn=True", default "sigmoid". bias_in_glu: bool, optional rTd_modeld_innerr}rr[r Nc`t||_||_t j||_t||||}t j|t j |t j ||t j ||_ dSr ) r r rrrrrr Sequentialrrnet)rrrr}rr[modulers rr zFeedForward.__init__As   ,w//7GZEE=  J| $ $ Igw ' ' J| $ $   rrLcV|||}|S)zRFeedForward forward function. Args: x: input tensor. )rr)rrLouts rrSzFeedForward.forwardVs& hhtq))** rr) rrrrrrrTrmr rrSrrs@rrr-s0$                  *Frr state_dictprefixlocal_metadatastrict missing_keysunexpected_keys error_msgscF|dz}||vr||dSdS)zPerform pre-hook in load_state_dict for backward compatibility. Note: We saved self.pe until v.0.5.2 but we have omitted it later. Therefore, we remove the item "pe" from `state_dict` for backward compatibility. peN)pop)rrrrrrrks r _pre_hookrbs5"  AJqrc `eZdZdZ ddedededed d f fd Zd ed efd Zded efdZ xZ S)T5RelativeAttentionLogitBiasaS This module implements the relative position bias described in Section 2.1 of the T5 paper: https://arxiv.org/pdf/1910.10683.pdf The Huggingface implementation is used as a reference https://github.com/huggingface/transformers/blob/v4.30.0/src/ transformers/models/t5/modeling_t5.py#L435 Modifies attention as Q*K^T + B, where B is a learned scalar bias based on relative position of the query and key. It is HxNxN, where H is the number of heads, N is the sequence length. I've made these modifications to the original T5 bias: - Skipping of the bucketing step. Original T5 bias converted rel position distances into logarithmically increasing buckets. This is supposed to help with length generalization. - I just directly use rel position index as bias values, as we don't need length generalization (40s max is good enough for ASR encoder), and it keeps ONNX export simple. - I've also extended it so that biases can be asymmetric, the default implementation treats L->R and R->L the same. Asymmetric was found to yield better results in my experiments. Args: num_heads: int Number of attention heads num_buckets: int Number of buckets to use for relative attention bias. This is the size of the learnable bias parameter. Bucketing is not yet supported, so this defaults to -1 which means no bucketing is used (max_distance determines size of bias param). max_distance: int Maximum distance to use for relative attention bias. With num_buckets=-1, this directly controls the max size of the bias parameter. When num_buckets > 0 is supported, this will control the maximum distance for logarithmic bucketing after which all positions are in the same bucket. symmetric: bool Whether to use symmetric or asymmetric biases. symmetric=False uses 2x number of bias params to distinguish L->R from R->L. This was found to be better for the encoder. r0F num_heads num_buckets max_distance symmetricr NcRt||_||_||_||_|jdk|_|jr||_ntd|js|xjdzc_tj |j|j|_ dS)Nrz;T5 attention bias with bucketed positions is not yet testedrN) r r rrrr_skip_bucketingr'r Embedding bias_values)rrrrrrs rr z%T5RelativeAttentionLogitBias.__init__s "&("#/!3   +D  %M ~ "    !  <(8$.IIrrLc|d}tj||jtjdddf}tj||jtjdddf}||z }|||j k|j }|||jdz k|jdz }|jr|}n||}|j r| }n ||j dzz }| |}| dddd}|S)Nr.devicedtyperNr)sizer1r5rr2 masked_fillrr_bucket_relative_positionrabsrrrjr6)rrLmaxposcontext_positionmemory_positionrelative_positionbias_idxt5_rel_att_biass rrSz$T5RelativeAttentionLogitBias.forwardsh <qxuzRRR AAtG  ,vahejQQQ !!!G ,.>>.99 !2 2 2T5F4F  .99  1A 5 5t7H17L     I(HH556GHHH > .||~~HH (A- -H**844)11!Q::DDQGGrrcd}|jsS|xjdzc_||dktj|jzz }tj|}n(tj|tj| }|jdz}||k}|tj| |z tj|j |z z |j|z ztjz}tj|tj ||jdz }|tj |||z }|S)NrrNr.)r\rtor1r2rmin zeros_likelogrmathr full_likewhere)rrrelative_buckets max_exactis_smallrelative_position_if_larges rrz6T5RelativeAttentionLogitBias._bucket_relative_positionsr{     "   !2Q!6 : : !! !! ! !& *; < <  !&!5#34E#F#F""!  $) $y0&/ I'--//); < <ht(9455 6)+ - "UZ.. &" &+Y & O68H18L M M& & " EK ')C    r)r0rF) rrrrrrmr rrSrrrs@rrrxs))\  JJJJ J  J  JJJJJJ.  F    D$ 6$ f$ $ $ $ $ $ $ $ rrc teZdZdZd dedededdffd Zd ejddfd Z d ejdejfd Z xZ S) AbsolutePositionalEncodingaiAbsolute Positional encoding module. This module implement Absolute sinusoidal positional encoding from: https://arxiv.org/pdf/1706.03762.pdf Args: d_model: int Input embedding size. dropout_rate: float dropout rate max_len: int, optional Maximum input length sequence, Default 5000 rr}max_lenr Nct||_tj|j|_t j||_ d|_ | t j d d||tdS)z'Construct an PositionalEncoding object.pNr.)r r rrsqrtxscaler1rrrr extend_petensorr8"_register_load_state_dict_pre_hookr)rrr}rrs rr z#AbsolutePositionalEncoding.__init__s  i -- x'','77  u|C((//7;;<<< // :::::rrLc|j|jd|dkrW|jj|jks|jj|jkr+|j|j|j|_dSt j|d|j}t jd|dt j  d}t j t jd|jdt j tj d|jz z}t j||z|dddddf<t j||z|dddddf<| d}||j|j|_dS) zSReset the positional encodings. Args: x: input tensor Nr.)rrrrrNg@r)rrrrrr1rdrr5float32r6exprrsincos)rrLrpositiondiv_terms rrz$AbsolutePositionalEncoding.extend_pes 7 47<<??affQii#?#?w}''47>QX+E+E'**1718*DD F [DL 1 1<166!99EMBBBLLQOO9 LDL!5= A A A!!DL01 2  i8 344111add7 i8 344111add7 \\!__%%qxqw%77rc||||jz|jddd|dfz}||S)zAdd positional encoding. Args: x: Input tensor. shape is (batch, time, ...) Returns: Encoded tensor. Its shape is (batch, time, ...) Nr.)rrrrrrks rrSz"AbsolutePositionalEncoding.forward2sS q  Odgaaa166!99n5 5||Ar)r) rrrrrrr r1rrrSrrs@rrrs  ;;;5;3;RV;;;;;;85<8D8888*  %,        rrc<eZdZdZdeddffd ZdedefdZxZS)MeanVarianceNormLayerzMean/variance normalization layer. Will subtract mean and multiply input by inverted standard deviation. Typically used as a very first layer in a model. Args: input_size: int layer input size. rr Nct||_tjt j||_tjt j||_ dSr ) r r rrrcr1rd global_meanr global_invstd)rrrs rr zMeanVarianceNormLayer.__init__Ms^ $< J(?(?@@\%*Z*@*@AArinput_c&||jz |jzS)zWMeanVarianceNormLayer Forward Args: input_: input tensor. )rr)rrs rrSzMeanVarianceNormLayer.forwardSs ))T-???rrwrs@rrrBsB3B4BBBBBB @f@@@@@@@@@rrceZdZdZ ddeded ed ed eezd ed edededdffd Z ddededzde eedzffdZ ddededzdee eedzfzffd Z xZ S) CausalConv1Da A causal version of nn.Conv1d where each step would have limited access to locations on its right or left All arguments are the same as nn.Conv1d except padding. If padding is set None, then paddings are set automatically to make it a causal convolution where each location would not see any steps on its right. If padding is set as a list (size of 2), then padding[0] would be used as left padding and padding[1] as right padding. It would make it possible to control the number of steps to be accessible on the right and left. This mode is not supported when stride > 1. padding[0]+padding[1] should be equal to (kernel_size - 1). r.rTrdN in_channels out_channelsrYstrider_dilationrsbias padding_moder c d|_||dz |_|dz |_n|dkr||dz krtdt |t r||_||_nnt |t rFt|dkr3|d|dz|dz kr|d|_|d|_ntd|d|j|_t ||||d|||| | |  dS)Nr.z3No striding allowed for non-symmetric convolutions!rNrzInvalid padding param: !) rrrYrr_rrsrrrr) cache_drop_size _left_padding_right_padding ValueError isinstancerlistr9_max_cache_lenr r rrrrYrr_rrsrrrrrs rr zCausalConv1D.__init__msE $ ?!,qD "(1*D  {{w+/99 !VWWW'3'' G%,"&-##7D)) GLLA%%AJ+{Q>>%,QZ"&-aj## !E7!E!E!EFFF"0 #%#%  rrLcachecV|%tj||j|jf}|}ntj|d|jf}t j||gd}|jdkr|ddddd|j f}n|}|dddd|d df}||fS)Nr4rr0rO)Fr4r r r1catr r)rrLrnew_x next_caches r update_cachezCausalConv1D.update_caches =E!$"4d6I!JKKKEJJE!!T%8!9:::EIuen"555E#a''"111aaa)@D,@+@)@#@A " #AAAqqq5::b>>/*;*;$;MM$(3)4(,(9#'<$(#.     MM(3)4(,(9#'<$($6#. (    HOO$/%2$% ! $    j)))+ G# ,JJ & &DL !D #DO~ (%)%6%:"&*lQ&6#&81&<##&*&7!&;%A"'+'81'<&B#&'#4-.. , ,>MM$(3)6(,(9#'<$( MM(3)6(,(9#'<$($6 ( j)))+ - ,0- - -!KDL !D #DO~ (%)%6%:"&*lQ&6#&81&<##&*&7!&;%A"'+'81'<&B#&'#4-.. , ,>MM$(3$(#5Q#>#>!)%2(,(9#'<$(       MM(3$(#5Q#>#>!)%2(,(9#'<$($6(      j)))+ = ,@0 0 0!KDL !D #DO"&"3a"7A!=D #'#4q#8Q">D  MMHOO$/%0$($5#| $ 2* $HOO$/(,(:a(?(?HH]$% ! $     ,(K MM* % % %4-122 , , (3)4(,(9#'<$($6#. ((3$(#5Q#>#>!)%2()#$$%#$(  0 j)))+ 5 ,:F FFFGG G 5 5 5 WEK@@@I$!!/$2EE -|/- Jx}s:'FQQDH&*D # # E E EDH&+D # #F FFFGG GH'0 rcd|jgS)Nr.r(rs rget_sampling_framesz'NemoConvSubsampling.get_sampling_frames3s4*++rcd|jdzgS)Nrr.rLrs rget_streaming_cache_sizez,NemoConvSubsampling.get_streaming_cache_size6s4*Q.//rrLmaskc|jr|dn|dd}|jdkr|jr|jdkr3d|jz |jz|jz}t j||k}nd}|rQ||\}}|s6|j dkr| |}nA| |}n+| |}n| |}|jrX| \}}}} | |dd||d}n|dd}||dfS|jd} |d} t j| |jz } |jr%|jr| |jz} | | dkxxdz cc<t jd| |j | dd| dk}||dfS) al Forward method for NeMo subsampling. Args: x: input tensor mask: input mask Returns: x: Resulting tensor from subsampling (B, T // time_reduction_factor, feat_out) pad_mask: tensor of padded hidden state sequences (B, 1, T // time_reduction_factor) r.rNr0Tr$Nr)r)rEr6 transposer+r8r=r1numelconv_split_by_batchr7conv_split_by_channelrFrrreshapeshapesumceilr(r,r<r5rr8)rrLrPx_ceil need_to_splitsuccessbctfmax_audio_length feature_lenspadding_lengthfeature_lens_remainderpad_masks rrSzNemoConvSubsampling.forward9s`#5 LAKKNNN1;;q!;L;L  0B 6 64;R 6499 !44t|CdlR % A 7 !%  !!55a88 7)(M99 66q99 IIaLLIIaLL ! A  " "JAq!QQ**221a<<==AA Aq!!A <d7N71:xx{{ L43J$JKK > =d: =%1D4K%K " 1Q6 7 7 71 < 7 7 7<#3AHEEELL    " "B   $ $Q ' '(($$Q''''rc|jdkrAtj5d|jz }|jdzdz}|jdz}tjj|jdj | |tjj|jdj | |tdt|jdD]}tjj|j|j | |tjj|j|j | |tjj|j|dzj | |tjj|j|dzj | ||j |jz|jz dz}tjj|jj | |tjj|jj | |ddddS#1swxYwYdSdS)Nr$?rNgrrr.)r7r1no_gradr>r8rinituniform_rFweightrrBr9r:r9r;r)rscaledw_maxpw_maxr=fc_scales rreset_parametersz$NemoConvSubsampling.reset_parametersysF   - - K Kd//+Q.47,d2 &&ty|':UFEJJJ &&ty|'85&%HHH C NNA66UUCHM**49S>+@6'6RRRHM**49S>+>PPPHM**49S1W+=+DvgvVVVHM**49S1W+=+BVGVTTTT !NT]:T=OOTXX &&tx 8LLL &&tx}xiJJJ) K K K K K K K K K K K K K K K K K K . -sHIIIc|\}}}}|dkr|dfSjdkrj}n[djz jzjz}t jt jtj||z d}d|z}||z}|dkr|dfStj fdtj ||dDdfS)z:Tries to split input by batch, run conv and concat resultsr.FrRrNrc:g|]}|SrF).0rPrs r z;NemoConvSubsampling.conv_split_by_batch..s%QQQe5!!QQQrT) rr+r8r=rrZrr1rTrsplit)rrLr^_cfr[rnew_batch_sizes` rrUz'NemoConvSubsampling.conv_split_by_batchsVVXX 1a 66e8O  01 4 46BBT004<?$,NF $(5;q>>F#:A>>??AABb Q  e8O IQQQQu{1na/P/PQQQ      rc  jd|}jd|}tjdz D] |\}}}}jdkrj}nAt jt jtj |dz d}d|z}t||z}|dkrd}t||z}|dkrd} j dzdz||}tj fdtj ||dDd}j dzdz|} |S)zOFor dw convs, tries to split input by time, run conv and concat resultsrr.rRrNrcHg|]}jdzdz|S)rru)rvrPrHrs rrwz=NemoConvSubsampling.conv_split_by_channel..s3SSS%1q519%e,,SSSrr#)rFrBr;rr+rrZrr1rTrchannel_chunked_convrrx) rrLryr_r`rzrnew_cnew_trHs ` @rrVz)NemoConvSubsampling.conv_split_by_channelsv DIaLOO DIaLOOt)A-.. ( (AJAq!Q4q88: Idhu{1~~'=qAABBTRLLEzzRLLEzz)) !a%!)$eQA  SSSSS%+aPQ:R:RSSSA% !a%!)$Q''AArrFrc d}g}tj||dD]1}|d}|jrtj||jdz |jdz |jdz |jdz f}tj ||j |||zddddddf|j |||z|jd|}nYtj ||j |||zddddddf|j |||z|j|j |}| |||z }3tj|dS)z$Performs channel chunked convolutionrr.rN)rrr_rs)r1rxrr,rr3r4r>r=conv2drlrr r@r) rrFrrLind out_chunksrPstepch_outs rr~z(NemoConvSubsampling.channel_chunked_convs  [J22  E::<<?D~  )))A- q()A- q( *--KcDj 0!!!QQQ 9:3t#34< .--KcDj 0!!!QQQ 9:3t#34< . .   f % % % 4KCCyQ'''rc\|dkr|dkr|dzdkrtd||_dS)Nr0r.rNrr0)r r+)rr+s r'change_subsampling_conv_chunking_factorz;NemoConvSubsampling.change_subsampling_conv_chunking_factorsP - 2 20A5501499S 1Q---rr)rrrrrr"rrTr1Modulermr rrMrOrrrSrqrUrVr~rrrs@rr"r"sJ#$( 01&-bgii_1_1_1_1 _1  _1  _1+._1HO_1_1 _1_1_1_1_1_1B ,T#Y,,,,0$s)0000>(>(v}>(vvPT}?T9U>(>(>(>(@KKKK2 V fdl0C    4$v$&$$$$L((HO((14((9?(( ((((((((T Q03 Q  Q Q Q Q Q Q Q Qrr"r.r3r4rYrr5r6c:||z }d}t|D]e}tj|tj|z||z}|rtj|ntj|}f|tjS)z^Calculates the output length of a Tensor passed through a convolution or max pooling layerrhr)rBr1divrrrZfloorr) r3r4rYrr5r6add_padonerHs rrDrD s"K/GC :  MM)GJJU[J99GCVLLsR)2L%*W%%% G8L8L ::EI: & &&rceZdZdZd fd ZddeddfdZ dded edzd edzd edzdeeeedzedzff d Z xZ S) AttModulezAttention abstraction moduler NcVtd|_dS)NF)r r export_mode)rrs rr zAttModule.__init__ s'  rTmodec||_dS)zset the export modeN)r)rrs r set_exportzAttModule.set_export$srrLmemorypos_embatt_maskc||||fS)zAttModule forward Args: x: input tensor. memory: memory tensor. pos_emb: positional encoder embedding. att_mask: attention mask tensor. rt)rrLrrrs rrSzAttModule.forward(s&'8++rr)T)NNN) rrrrr rmrrrrSrrs@rrrs&&!!!!!!  t t    !%!%"& ,, , ,$ , 4- , vvv}ftm; < ,,,,,,,,rrc4eZdZdZddedeeeffdZdS)AttBlockzBAttention Block module to support both Attention and Block module.Frr cd|jfS)zmemory dimensionsr.)r)rrs r memory_dimszAttBlock.memory_dims=s4?##rN)F)rrrrrmrrrrtrrrr:sILL$$4$E#s(O$$$$$$rrscoresrPc|t|dd}||tj }tj|d|d}ntj|d}|S)Nr.rr0rOr)r6eqrr1infsoftmax)rrPattns rmasked_softmaxrBs ~~a  ##A&&##D59*55}V,,,88 #  }V,,, KrcHeZdZUdZejjeed<ejje ed<ejje ed<ejje ed< dd e d e d ede de de de de de ddffd Z d de de de de dzde dzde dzde dzde fdZxZS)!MultiHeadedAttentionaMulti-Head Attention layer with optional relative position embedding and GLU. Args: n_head: int the number of heads. n_feat: int input size features. dropout_rate: float dropout rate. attention_inner_dim: int, optional the attention dimension used in the class, it can be different from the input dimension n_feat. default: -1 (equal to n_feat). use_pt_scaled_dot_product_attention: bool, optional if set True, use pytorch scaled dot product attention in training. NOTE: this will NOT be used in ONNX decoding due to a lack of support. In that case, we use the original attention implementation, which shows no regression. default: False. n_value: int, optional if set to values other than -1, use a different dimension for value. With the default value (i.e. -1), it is backward compatible. group_size: int, optional. must divide `n_head` if group_size > 1: GQA if group_size = 1: MHA if group_size = n_head: MQA inv_sqrt_d_khh_kgr0rTFr.n_headn_featr}attention_inner_dimrZr[#use_pt_scaled_dot_product_attentionn_value group_sizer Nc .t|dkr|}|dkr|}||zdksJ||z|_dtj|jz |_||_|| zdks Jd| |_|| z|_tj |||_ tj ||| z|_ tj ||| z|_ tj || z||_tjdt$dz|_tj||_||_||_|r| dkrt1dtjj|_tjj|_tjj|_tjj j !|_"dS)Nr0rrhzgroup_size must divide n_headrr.z'Cannot use PT Scaled Attention with GQA)#r r d_krrrrrrrrlinear_qlinear_klinear_v linear_outr1jit Attributerrrrr}rr ao quantization QuantStubquant_qquant_x DeQuantStubdequant quantizedFloatFunctionalffunc) rrrr}rrZr[rrrrs rr zMultiHeadedAttention.__init__ts  b==G " $ $"( "V+q0000'&0$)DH"5"55 "a''')H'''Z' &*=>>  &*=*KLL  '+>*+LMM )$7:$EwOOI''ftm<< zL111 (3V0 . H:>>FGG G x,6688 x,6688 x,88:: X[*::<< rquerykeyr/pos_kpos_vrPrelative_attention_biasc( |d}|||d|j|j} |||d|j|j} |||d|j|j} |jr4tj s| ddn| dd|j z} | dd} | dd} |jr@tj s!d} |I|d}|||z} n|} |j| jkr| | j} tjjtjjjjtjjjjtjjjjtjjjjg5tjj| | | | |j} dddn #1swxYwYn|j|jkr?| ||j|jd|j} tjd| | }n)tj| | dd}||j|jkrtjd | |}n| ||jzd|j dd}tj|| dd}| dd||j|d|d}||z}n|}|||z}tC||}||_"|#|}tj|| j| } || ||jz|d|d dd}tj|| dd ||j|d|j}| |z} | dd |d|j|jz} |$| S) a4Compute 'Scaled Dot Product Attention'. Args: query: query tensor (batch, time1, size) key: key tensor (batch, time2, size) value: value tensor (batch, time1, size) pos_k: key tensor used for relative positional embedding. pos_v: value tensor used for relative positional embedding. mask: mask tensor (batch, time1, time2) relative_attention_bias: bias added to attention logits w.r.t. relative positions (1, n_head, time1, time2) rr0r.rNN) attn_mask dropout_pzb g h t d, b h s d -> b h t szb g h t d, t s d -> b h t s)%rrviewrrrrrrr1r is_scriptingrSrr6rrr attention sdpa_kernel SDPBackendFLASH_ATTENTIONEFFICIENT_ATTENTIONMATHCUDNN_ATTENTIONr3scaled_dot_product_attentionr}rWreinsummatmul contiguousrrrr)rrrr/rrrPrn_batchqrvrrLAB reshape_qrrp_attn reshape_attnattn_vs rrSzMultiHeadedAttention.forwards.**Q-- MM% % %gr4648 D D MM#   # #GR48 D D MM% % %gr48TX F F7 7@E @V@V@X@X 7AKK1   Q""T%66 KK1   KK1    3I EI >II $I:(( ) QW 5 5I#//H&1AH&1EH&16H&1A   H'DD'"/ E                v!!IIgtvtxTXFFL!@!QGGLAKKB$7$788 6TX%% %BAuMMAA g.DH=="1a  !5??2r#:#:A Aq))..A 1 AQ&2"99!&$//DDI\\$''F VYYqw//33A %%''T'DF*EJJqMM5::a==IIYq!__Lu55Yq!__Z\\T'465::a==$(CC  J KK1   ( ( * * / /TX=P Q Q q!!!s:/I55I9<I9)r0rTFr0r.r )rrrrr1rFinalr__annotations__rrTrmr rrSrrs@rrrQs:)/%(((( ys   ys$& 49.=.=.=.= .= ! .=  .=.=.2.=.=.= .=.=.=.=.=.=p26r"r"r"r" r" } r" } r"tmr""($r" r"r"r"r"r"r"r"r"rrcBeZdZdZejjdefdZdS)MultiSequentialz,Multi-input multi-output torch.nn.Sequentialr c|D]}||}|S)zForward method implementation.rt)rargsms rrSzMultiSequential.forwards%  A1d8DD rN) rrrrr1rignorerrSrtrrrrsE66 Yrr input_layertime_reductioncN|dvr|dkrdS|dvr|dkrdS|dvr|dkrdSd S) aGet an offset. We will use the offset for determining #frames of a subsampled feature. Args: input_layer: Type of an input layer time_reduction: time reduction factor for downsampling a feature Returns: int: offset )r nemo_convr#r)rr.rrt)rrs r get_offsetr$sY---.A2E2Eqk!!n&9&9q---.A2E2Eq 1rxs_pad max_seq_lenc\|j\}}}|dd}tj|ddddfd|fd|f}|j\}}}||d||}|ddd d}|d||}|S) a( For a given tensor with shape of (N, T, D), if sequence length T is longer than max_seq_len, this function unfold it to a (NT', max_seq_len, D) where T' is T // max_seq_len. Args: xs_pad: input tensor with shape (N, T, D) max_seq_len: maximum sequence length r0r.Nr.)rYrrrrN)rXrSrunfoldrrjr)rrryDnew_bszslens r unfold_tensorr7slGAq!   b" % %F XsD!!!| $;F |GQ [["k4 8 8F ^^Aq!Q ' ' 2 2 4 4F [[[! , ,F Mr)r)rr)r.)(rr1torch.nn.functionalrr3rrrrrTr(rrrErGrVroryrrdictrmrrrrr`rrArr"rDrrrrrrrrrtrrrsd ''''' '''$$$%(/$$$$2WX0"0" 0"!%c0"9<0"PS0" \0"0"0"0"f*****")***0^^^^^ry^^^B@@@@@ry@@@FggggggggT!!!!! !!!H11111")111j    s)  #Y S  ,I I I I I 29I I I X::::::::|@@@@@BI@@@4[[[[[29[[[|3333329333lWQWQWQWQWQ%(/WQWQWQ@ '' '''  '  '  ' ''''&,,,,, ,,,:$$$$$y)$$$    4-      E"E"E"E"E"29E"E"E"Peh) C      &&svr