.`ipddlZddlZddlmZmZddlZddlZddlm cm Z ddlm Z m Z ddl mZddlmZddlmZddlmZmZmZmZmZmZmZmZmZmZmZGdd e jZ Gd d ej!e jZ"Gd d e"Z#Gdde jZ$Gdde jZ%dS)N)AnyLiteral)Tensornn)CheckpointWrapper)FullyShardedDataParallel)PretrainedConfig) AbsolutePositionalEncoding ConvModule FeedForwardMeanVarianceNormLayerMultiHeadedAttentionMultiSequentialNemoConvSubsamplingT5RelativeAttentionLogitBiasadaptive_enc_mask get_offset unfold_tensorc3HeZdZdZ d3dedededededededededededededed ed!ed"ed#ed$ed%ed&ed'ed(ed)ed*d+f2fd, Z d4d-e j d.e j d/e j d0e j d1e d+zd*e e j e j e j e j ff d2Z xZ S)5ConformerEncoderLayeraConformerEncoder Layer module. for more details see conformer paper: https://arxiv.org/abs/2005.08100 This module implement the Conformer block layer. Args: d_model: int attention dim. ext_pw_out_channel: int if > 0, ext_pw_out_channel is a dim channel size for the last pointwise conv after swish activation. depthwise_seperable_out_channel: int if set different to 0, the number of depthwise_seperable_out_channel will be used as a channel_out of the second conv1d layer. otherwise, it equals to 0, the second conv1d layer is skipped. depthwise_multiplier: int number of input_dim channels duplication. this value will be used to compute the hidden channels of the Conv1D. n_head: int the number of heads for multihead attention module. d_ffn: int output size of the feed_forward blocks. ext_pw_kernel_size: int kernel size of the conv pointwise of the conformer. kernel_size: int kernel size. dropout_rate: float dropout rate. causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation in ConvModule layer of the conformer. default False activation: str, optional activation function name, one of ["relu", "swish", "sigmoid"], sigmoid activation is only used with "glu_in_fnn=True", default "relu". chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. default 0. chunk_size: int, optional chunk_size for cnn. default 18 conv_activation: str, optional activation function used in ConvModule part of the conformer, default "relu". conv_glu_type: str, optional activation function used for the glu inside the ConvModule part of the conformer. default: "sigmoid". bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. attention_inner_dim: int, optional if equal to -1, attention dim for linears k/q/v is equal to d_model. otherwise attention_inner_dim is used. default -1. attention_glu_type: str, optional activation function for glu used in the multihead attention, default "swish". activation_checkpointing: str, optional a dictionary of {"module","interval","offload"}, where "module": str accept ["transformer", "attention"] to select which module should do activation checkpointing. "interval": int, default 1, interval of applying activation checkpointing, interval = 1 means that we apply checkpointing on every layer (if activation), otherwise, we apply it every x interval. "offload": bool, default False, if set to True, we offload activation to cpu and reload it during backward, otherwise, we recalculate activation in backward. default "". export: bool, optional if set to True, it removes the padding from convolutional layers and allow the onnx conversion for inference. default False. use_pt_scaled_dot_product_attention: bool, optional if set to True, use pytorch's scaled dot product attention implementation in training. attn_group_sizes: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attn_group_sizes < attention_heads = Grouped-Query Attention attn_group_sizes = attention_heads = Multi-Query Attention r皙?FrelusigmoidTswishd_modelext_pw_out_channeldepthwise_seperable_out_channeldepthwise_multipliern_headd_ffnext_pw_kernel_size kernel_size dropout_ratecausal batch_norm activationchunk_se chunk_sizeconv_activation conv_glu_type bias_in_glulinear_glu_in_convmattention_inner_dimattention_glu_typeactivation_checkpointingexport#use_pt_scaled_dot_product_attentionattn_group_sizesreturnNctt||| | ||_t ||| ||||||_t ||||||| | | | |||||||_t||| | ||_tj ||_ tj ||_ dS)N)r$d_innerr,r/r4)r: group_size)r9) super__init__r feed_forward_inr self_attnr convfeed_forward_outr LayerNormlayer_norm_att layer_norm)selfr$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r; __class__s {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/phi4mm_audio.pyrAzConformerEncoderLayer.__init__s 6 *%!#    .      0S'       +           !    &!,%!# ! ! ! !l733,w//xpos_kpos_vmaskrelative_attention_biasc <|d||zz}||}|||||||||z}|||z}|d||zz}||}||||fS)aConformerEncoder forward. Args: x: input feature of shape (batch, max_time_in, size) pos_k: positional key embedding. pos_v: positional value embedding. mask: mask for x (batch, max_time_in) relative_attention_bias: bias added to attention logits w.r.t. relative positions (1, n_head, time1, time2) g?rQ)rBrGrCrDrErH)rIrMrNrOrPrQnorm_xouts rKforwardzConformerEncoderLayer.forwards$ d**1--- -$$Q''       $;     !  d++A... .ooa  E5$&&rL)rrrrrrrrrFFrrrrr TFr!r"r#FFrN)__name__ __module__ __qualname____doc__intfloatboolstrrAtorchrtuplerV __classcell__rJs@rKrr%s,ccN"#/2$%"#!  %& $)#%")(*49 !3K0K0K0 K0*- K0 " K0  K0K0 K0K0K0K0K0K0K0K0 !K0"#K0$%K0&"'K0(!)K0* +K0,#&-K0./K00.21K023K04 5K0K0K0K0K0K0f26 #'#' <#'|#'| #' l #' "($ #' u|U\5<E F#'#'#'#'#'#'#'#'rLrc%eZdZdZ d1d ed eeezd eeezdedededededededede ee fdzdede ee fdzde ddede ee fdzddf$fd Z dee jzdee jzfd Zejde fd!Z d2d eeezdzd eeezdzdeeeffd"Zd#ejdejfd$Zd%e jd&e jdee je jffd'Zd%e jdee jdze jdzffd(Zd)ed*ed eeezd eeezde jf d+Z d2d,e jd&e jd-eeezdzd.eeezdzdee je jdze jdze je jfee je jdze jdze je je jfzf d/Zdefd0ZxZS)3TransformerEncoderBaseaj The Base class for Transformer based encoders Please set causal = True in streaming model Args: input_size: int input feature dimension. chunk_size: int, list(int) Number of frames for each chunk This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training Some examples for the 2 cases: chunk_size = 12 chunk_size = [6, 8, 12, 24] left_chunk: int, list(int) Number of chunks used for masking in streaming mode. This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training. When chunk_size is a list, left_chunk must be a list with same length. Some examples for the 2 cases: left_chunk = 6 left_chunk = [12, 9, 6, 3] attention_dim: int, optional attention dimension. default 256. attention_heads: int, optional the number of heads. default 4 input_layer: str, optional input layer type before Conformer, one of ["linear", "conv2d", "custom", "vgg2l", "embed"], default "conv2d" cnn_out: int, optional the number of CNN channels before Conformer. default -1. cnn_layer_norm: bool, optional layer norm between Conformer and the first CNN. default False. time_reduction: int, optional time reduction factor default 4 dropout_rate: float, optional dropout rate. default 0.1 padding_idx: int, optional padding index for input_layer=embed default -1 relative_attention_bias_args: dict, optional use more efficient scalar bias-based relative multihead attention (Q*K^T + B) implemented in cmb.basics.embedding. [T5/ALiBi]RelativeAttentionLogitBias usage: relative_attention_bias_args={"type": t5/alibi} additional method-specific arguments can be provided (see transformer_base.py) positional_dropout_rate: float, optional dropout rate after positional encoding. default 0.0 nemo_conv_settings: dict, optional A dictionary of settings for NeMo Subsampling. default None conv2d_extra_padding: str, optional Add extra padding in conv2d subsampling layers. Choices are (feat, feat_time, none, True). if True or feat_time, the extra padding is added into non full supraframe utts in batch. Default: none attention_group_size: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attention_group_size < attention_heads = Grouped-Query Attention attention_group_size = attention_heads = Multi-Query Attention rr nemo_convr!FNnoner input_sizer1 left_chunk attention_dimattention_heads input_layercnn_outcnn_layer_normtime_reductionr, padding_idxrelative_attention_bias_argspositional_dropout_ratenemo_conv_settingsconv2d_extra_paddingfeat feat_timerhTattention_group_sizeencoder_embedding_configr<c dt||_||_||_||_||_||_||_| |_ ||_ ||_ |jdkr]d|j ||ddtj dd}|r(||dD]}||vs Jdtdi||_nt#d |zt%|| |_| r| d nd|_|jd krj|j|jzd ks Jd t-|j|jz| dd| dd|_nt0t3|j d|_dS)Nrf dw_stridingrrF subsamplingsubsampling_factorfeat_infeat_out conv_channels subsampling_conv_chunking_factorr/ is_causalrrr6{i} should be specified outside of the NeMo dictionaryzunknown input_layer: typet5r'attention_group_size must divide n_headt5_bias_max_distanceit5_bias_symmetric) max_distance symmetricri)r@rArirmr1rjrk num_headsryrprtrzrReLUupdaterembed ValueErrorr pos_embgetrelative_attention_bias_typerrelative_attention_bias_layerNotImplementedErrorr encoder_embedding)rIrir1rjrkrlrmrnrorpr,rqrrrsrtruryrzdefault_nemo_conv_settingsirJs rKrAzTransformerEncoderBase.__init__Gs$( $&$$*($8!,"4(@%  { * *,&*&9%)!$45 gii" * * &" *112DEEEFA$6666P7666-,DJJ4{BCC C1 2   ,  ( , ,V 4 4 4 )  , 4 4>D$==BBB9CBB2N$";;9==*D7::;NPUVV 222D . .& %!6  ), 7" " rL feature_lenscV|jdkr|jdddv}|jdd}|r|rt|tr.t j||jz ntj||jz }||jz}t|tr||dkxxdz cc<n |dkr|dz }|St|tr tjn t j}|||jz SdS) ajfeature_lens: int return updated feature lens. This used to return a different lambda function for each case that computed the right thing. That does not work within Torchscript. If you really need this to be faster, create nn.Module()-s for all the cases and return one of them. Torchscript does support that. rfr~r|)r|stridingstriding_conv1drFrN) rmrtr isinstancerr`ceilrplongmathr\)rIrsubsampling_causal_condr lens_changefeature_lens_remainder ceil_funcs rKcompute_lens_changez*TransformerEncoderBase.compute_lens_changesT  { * *&*&=&A&A}''' #/33KGGI #4 #",77GEJ|d.AABBGGIII<$2E#EFF *68K)K&lF33% 6! ;<<<A<<<<+q001$K""%/ c%B%BR  I9\D,??@@ @/ + *rLcdS)z'Abstract forward method implementation.NrrIs rKrVzTransformerEncoderBase.forwardsrLc||j}||j}t|trt t jdt|d}||}t|tstdt|t|krtd||}n|}|}||fS)z>If chunk size is a list, we will randomly select a chunk size.Nr)r)lowhighsizez5Since chunk_size is a list, left_chunk must be a listzBThe length of left_chunk must be the same as length of chunk_size.) r1rjrlistr\r`randintlenr)rIr1rjchunk_size_indexchunk_size_train_effleft_chunk_train_effs rK_chunk_size_selectionz,TransformerEncoderBase._chunk_size_selections  J  J j$ ' ' ." !#j//EEE   $..>#? j$//  K:#j//11 X$..>#? #- #- #%999rLrct|t}t|t}|}|r|j}|r|j}|SrW)rrr_checkpoint_wrapped_modulemodule)rIris_embed_using_act_chkptis_embed_fsdp_wrapped embed_classs rK_get_embed_classz'TransformerEncoderBase._get_embed_classsN#-e5F#G#G *52J K K # ;:K  ',KrL input_tensormasksc||j}t|tsJ|||\}}||fSrW)rrrr)rIrrrs rK_forward_embeddings_corez/TransformerEncoderBase._forward_embeddings_coresS++DJ77 +':;;;;;"jju== eU""rLcJd}d}|j||}||fSrW)rr)rIrrNrOs rK_position_embeddingz*TransformerEncoderBase._position_embeddings9  - 5<<Le|rLseq_len batch_sizec|||\}}tjd||}t|||d|ddg}|S)Nr) left_windowr!)rnparanger unsqueezeexpand) rIrrr1rjrrchunk_start_idxenc_streaming_masks rK_streaming_maskz&TransformerEncoderBase._streaming_masks6:5O5O  6 6 22 )Aw0DEE 6J   Yq\\ VZR( ) ) "!rLxs_pad chunk_size_nc left_chunk_ncc|t||jd}|dkrtd|d|jd}||||j|j}|jr(|}|}|}| ||\}}|} | ||| z} n||} n| } |>|||||} |jr| } ||| z} n| } nd} | |\} }||| || |fS|| || || fS)aForwarding the inputs through the top embedding layers Args: xs_pad: torch.Tensor input tensor masks: torch.Tensor input mask chunk_size_nc: (optional, default is None) chunk size for non-causal layers left_chunk_nc: (optional, default is None) # of left chunks for non-causal layers rrzFThe sequence length after time reduction is invalid: z. Your input feature is too short. Consider filtering out the very short sentence from data loaderN) r\rshaperrr1rjis_cudacudarr)rIrrrrrrrrstreaming_maskhs_maskenc_streaming_mask_nc hs_mask_ncrNrOs rKforward_embeddingsz)TransformerEncoderBase.forward_embeddingssJd..v|A??@@ a<< \!_ !11 Z$/   > #!3!8!8!:!: [[]]F ";;L%PP e+  %%*;n,GG  GG$G  $$($8$8]M%% !~ E(=(B(B(D(D% "%:: 2 J// == u  w= =UE7E:EErLc6t|j|jS)a!Returns offset used when retaining inputs for decoding. This is essentially, how many additional frames have to be added to the front-end CNN input to ensure it can produce a single output. So if the "padding" parameter is 0, typically offset will be > 0. )rrmrprs rKrz!TransformerEncoderBase.get_offsetos$*D,?@@@rL)rrrfr!Frrgr!NrgNrhrN)NN)rXrYrZr[r\rr_r^r]dictrrrAr`rrabcabstractmethodrVrarrModulerrrrrrrbrcs@rKreresQFFZ! &$!>B),48KQ$%:>%R R R $s)OR $s)O R  R  R R R R R R R '+38nt&;R "'R !cNT1R &&GH!R ""#R $#'sCx.4"7%R & 'R R R R R R h"A%,."A u| "A"A"A"AH 66666 .2-1::$s)Od*:$s)Od*: sCx ::::@ bi BI    #!L#16# u|U\) *#### !L  u|d"EL4$77 8    """$s)O " $s)O "  """"81504 TFTF TF|TFT#Y- TF T#Y- TF  L L4  L4  L L     L L4  L4  L L L    TFTFTFTFlACAAAAAAAArLrecOeZdZUdZeeed<ddddddd d d d d d dddd ddddd d dd d gdddd ddd ddf#dedeeezdeeezdedzdedededededede d e d!ed"e d#ed$ed%ed&ed'ed(ed)ed*ed+ed,e d-e d.ed/e d0edeed1ed2e ee fdzd3ed4e d5e ee fdzd6e d7d8e d9ed:e ee fdzd;dfNfd< Z d=ejd;ejdzfd>Zd?ejd@ejdAejdzd;ejfdBZejjd?ejdCejd;eejejffdDZxZS)EConformerEncoderarConformerEncoder module. see original paper for more details: https://arxiv.org/abs/2005.08100 Please set causal = True in streaming model Args: input_size: int input feature dimension. chunk_size: int, list(int) Number of frames for each chunk This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training Some examples for the 2 cases: chunk_size = 12 chunk_size = [6, 8, 12, 24] left_chunk: int, list(int) Number of chunks used for masking in streaming mode. This variable can take 2 forms: int: Used for inference, or single chunk size training list(int) : Used only for variable chunk size training. When chunk_size is a list, left_chunk must be a list with same length. Some examples for the 2 cases: left_chunk = 6 left_chunk = [12, 9, 6, 3] num_lang: int This parameter is used to store the number of languages in the lang_dict, only used for multiseed/multilingual models. default None. attention_dim: int, optional attention dimension. default 256. attention_heads: int, optional the number of heads. default 4 linear_units: the number of units of position-wise feed forward. default 2048 num_block: number of Transformer layer. default 6 dropout_rate: float, optional dropout rate. default 0.1 input_layer: str, optional input layer type before Conformer, one of ["linear", "conv2d", "custom", "vgg2l", "embed"], default "conv2d" causal: bool, optional if set to True, convolution have no access to future frames. default False. batch_norm: bool, optional if set to True, apply batchnorm before activation in ConvModule layer of the conformer. default False cnn_out: int, optional the number of CNN channels before Conformer. default -1. cnn_layer_norm: bool, optional layer norm between Conformer and the first CNN. default False. ext_pw_out_channel: int, optional the number of channel for CNN before depthwise_seperable_CNN. If 0 then use linear. default 0. ext_pw_kernel_size: int, optional kernel size of N before depthwise_seperable_CNN. only work for ext_pw_out_channel > 0. default 1 depthwise_seperable_out_channel: int, optional the number of channel for depthwise_seperable_CNN. default 256. depthwise_multiplier: int, optional the number of multiplier for depthwise_seperable_CNN. default 1. chunk_se: int, optional 0 for offline SE. 1 for streaming SE, where mean is computed by accumulated history until current chunk_se. 2 for streaming SE, where mean is computed by only the current chunk. default 0. kernel_size: int, optional the number of kernels for depthwise_seperable_CNN. default 3. activation: str, optional FeedForward block activation. one of ["relu", "swish", "sigmoid"] default "relu". conv_activation: str, optional activation function used in ConvModule part of the conformer, default "relu". conv_glu_type: str, optional activation used use glu in depthwise_seperable_CNN, default "sigmoid" bias_in_glu: bool, optional if set to True, use additive bias in the weight module before GLU. default True linear_glu_in_convm: bool, optional if set to True, use GLULinear module, otherwise, used GLUPointWiseConv module. default to False. attention_glu_type: str only work for glu_in_attention !=0 default "swish". export: bool, optional if set to True, it removes the padding from convolutional layers and allow the onnx conversion for inference. default False. activation_checkpointing: str, optional a dictionarry of {"module","interval","offload"}, where "module": str accept ["transformer", "attention"] to select which module should do activation checkpointing. "interval": int, default 1, interval of applying activation checkpointing, interval = 1 means that we apply checkpointing on every layer (if activation), otherwise, we apply it every x interval. "offload": bool, default False, if set to True, we offload activation to cpu and reload it during backward, otherwise, we recalculate activation in backward. default "". extra_layer_output_idx: int the layer index to be exposed. relative_attention_bias_args: dict, optional use more efficient scalar bias-based relative multihead attention (Q*K^T + B) implemented in cmb.basics.embedding. [T5/ALiBi]RelativeAttentionLogitBias usage: relative_attention_bias_args={"type": t5/alibi} additional method-specific arguments can be provided (see transformer_base.py) time_reduction: int optional time reduction factor default 4 use_pt_scaled_dot_product_attention: whether to use pytorch scaled dot product attention in training. Default: False nemo_conv_settings: dict, optional A dictionary of settings for NeMo Subsampling. default: None usage: nemo_conv_settings= { "subsampling": dw_striding/striding/dw_striding_conv1d/striding_conv1d, "conv_channels": int, "subsampling_conv_chunking_factor": int, "is_causal": True/False } conv2d_extra_padding: str, optional Add extra padding in conv2d subsampling layers. Choices are (feat, feat_time, none, True) Default: none replication_pad_for_subsample_embedding: For batched-streaming decoding, use "replication" padding for the cache at start of utterance. Default: False attention_group_size: int, optional the number of groups to use for attention, default 1 (Multi-Head Attention), 1 = typical Multi-Head Attention, 1 < attention_group_size < attention_heads = Grouped-Query Attention attention_group_size = attention_heads = Multi-Query Attention extra_multi_layer_output_idxsNrrrrrfTFr!rrrrr r"r#rhrir1rjnum_langrkrl linear_units num_blocksr,rmr-r.rnror%r*r&r'r0r+r/r2r3r4r5r7r9extra_layer_output_idxr8rrrpr:rtrurv'replication_pad_for_subsample_embeddingryrzr<c' !%t||| | || |d|"|#%|&||_||_|_|$|_|j%zdks Jd|j%z|_t%   !fdt|D|_ ||_ ||_ | dtjdd dS) Nrg)r,rrrsrtruryrzrrcg|]Q}tdidddddddddd d d d d d dd ddddddRS)r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r7r8r9r:r;r)r).0_r/r8rkr7ryrlr.r4r-r0r1r2r3r'r&r,r9r*r%r+r5rr:s rK z-ConformerEncoder.__init__..hsV43&)M'9'95T4S*>)=  +?  ', (:'9!, "."6 *z *z&X *z%4O #0-!"!, #$)<(;%&(:'9'(.F-E)*"6+,9\8[-.&:%9/rLdev_typerF) persistent)r@rArrr+rr num_heads_krrangeencodersrrregister_bufferr`zeros)(rIrir1rjrrkrlrrr,rmr-r.rnror%r*r&r'r0r+r/r2r3r4r5r7r9rrr8rrrpr:rtrurryrzrJs( ` ``` ` `` ````````````` ` ` ` rKrAzConformerEncoder.__init__!sR          %)E$'1!5!5%=!    $%  & 3 4~ 44999 5:99 >-AA'4z**5  <'=#-J* ZRUKKKKKrLrc>|jr||SdSrW)r)rIrs rKinit_relative_attention_biasz-ConformerEncoder.init_relative_attention_biass0  - D55lCC C D DrLrdevicerPc|jd}|jd}||||j|j}||}||S|d}|}t jd||| dd| dk} | d} | |z} | S)Nrr)rr!) rrr1rjtosumr`rrrr) rIrrrPmax_audio_lengthrrrpadding_lengthpad_masks rKcalculate_hs_maskz"ConformerEncoder.calculate_hs_masks"<?\!_ !11 j$/4?  0226:: <% %xx{{ %<#3FCCCJJ    " "B   $ $Q ' '(%%a((00rLrc||}|||\}}}}}d}|j\}} } d} | | krd}| | zdkr | | | zz } nd} | dkr5tj|ddd| fdd} | |j}t|| }||d}tj|d| fdd}| d }t|| }|d }nd}| ||j|}| |}|jdko|du}|r|||||^}}n/t!|jD]\}}|||||| \}}}}|r<|jd}||d|}| dkr|ddd| ddf}||fS) zConformer Forward function Args: xs_pad: torch.Tensor input tensor masks: torch.Tensor post-embedding input lengths FiTrconstantNrr!rS)rrrFpadrrrsqueezerr]r^rrrr enumeratereshape)rIrrrrNrOrunfoldedori_bzrD max_seq_lenchunk_pad_sizeinput_tensor_padsubsampled_pad_mask extra_padded_subsamlped_pad_mask masks_unfoldrQ_simplified_pathrrlayer embed_dims rKrVzConformerEncoder.forwards''//595L5L E6 6 2 eUGU)/ [ H$q((!,+0E!F!"!!#$5 1aN";Z$$  022<3FGG ({CCL ',mm''#455'!^)>rBBHHJJ1 -4k    ,33  $&&  $ ,,l1<G #'"C"CL"Q"Q  '2 - Q2IT2Q   #}}\5%QQ L11%dm44  5(- ,C )))% aAA  D$*2.I'//IFFL!!+AAA/?/?,BC U""rL)rXrYrZr[rr\__annotations__r]r_r^rrrrAr`rrrrjitignorerarVrbrcs@rKrrysccJ$(9,,, $   !& $"#"#/2$% %& $)")&(35(*>B4948KQ8=$%:>OhLhLhL$s)OhL$s)O hL * hL  hLhLhLhLhLhLhLhLhLhL  !hL" #hL$*-%hL&"'hL()hL*+hL,-hL./hL01hL23hL4"5hL6 7hL89hL:!$;hL<(,Cy=hL>#&?hL@'+38nt&;AhLBChLD.2EhLF!cNT1GhLH&&GHIhLJ26KhLL"MhLN#'sCx.4"7OhLP QhLhLhLhLhLhLTD!LD  DDDD l,1L@E t@S * YY#lY#+0<Y# u|U\) *Y#Y#Y#Y#Y#Y#Y#Y#rLrceZdZdZ dd ed ed ed ed edededeffd Z ddej dej dzdedzde ej edzffdZ xZ S) WindowQformerzWindow-level QformerrrrrgT window_size num_queriesrrkrlrr,normalize_beforec dttjfdt |D|_tjtjd||_ rtj dnd|_ ||_ dS)Nc Fg|]}tjddS)rT)r$nheaddim_feedforwarddropoutr/ batch_first norm_first)rTransformerDecoderLayer)rrrkrlr,rrs rKrz*WindowQformer.__init__..sR   *))$0(% $/   rLrg-q=)eps) r@rAr ModuleListrdecoders Parameterr`rqueriesrF after_normr) rIrrrrkrlrr,rrJs `````rKrAzWindowQformer.__init__s          z**       |EK; $N$NOO 6F PBLE 2 2 2 2D 'rLN audio_embedrP embed_lenr<c|dd}|jd|jz}|dkr!tj|d|j|z fdd}tj|ddddfd|jfd|jf}|j\}}}||d|j|}|dd }|||z|jd}|j ||zdd} |j D]} | | |d| } |j | | } | ||jz}| ||d} | |fS) zforward decoderrrr!rr.N)r+strider)tgtmemorytgt_mask memory_mask) transposerrrrunfoldview contiguousr'rr%r() rIr)rPr*padding embed_chunkbszrslenqrrUs rKrVzWindowQformer.forward(s"++Aq11 #B'$*:: Q;;%a!1G!;> !&&sTz43CRHH L  d B 3 3] R RE!K$DQQQAA ? &""A  !T%55IffS$##I~rL)rrrrrrrgTrW) rXrYrZr[r\r]r^rAr`rrarVrbrcs@rKrrs   !!% ' ' ' ' '  '  ' ' ' ' ' ' ' ' 'L!% **\*lT!*: * u|S4Z' ( ********rLrc eZdZdZdededdffd ZdejddfdZ d ejddfd Z ddejd ejdzd e dejfdZ ddejd ejdzd e dejfdZ xZS)AudioEmbeddingzImage embedding.configkwargsr<Nc Bt||_t|dr|jn|j}d}d|_t|jtr`|j dddkrA|j dd}|Jtdi||_ |d}|d}ntd| Jd ||_||_| d d |_| d d |_| dd r-| di}||d<t%di||_nd|_| dd r|j Jd| di}d|j||dd t)jd d} |r(| |dD]} | |vs Jdt/di| |_nd|_| dd} | dkrt)j|||_n9| dkr|} d} |js|jrd n|j|_t)j||jz| g}t9d | D]=}|t)jt)j| | g>t)j||_t)j||jz| g}t9d | D]=}|t)jt)j| | g>t)j||_ ntd| d|j!|_!d|_"d|_#dS)Nn_embdnamecascadesr<rkrir#z(Remember to set values for audio_dim_outfreeze_audio_processorFdownsample_rater use_qformerqformer_configuse_conv_downsamplez6don't support use qformer and conv downsample togetherrtr|rr}rrprojection_clslinearmlprzprojection_cls = z, not implementedr)$r@rAr<hasattrr? hidden_size layer_idxraudio_processorrrrencoderr audio_dim_out audio_dim_inrCrDrqformerrrrrconv_dsLinearaudio_projectionlinear_downsample_raterextendGELU Sequentialaudio_projection_for_vision vocab_size input_embedsaudio_embed_sizes)rIr<r=rLrPencoder_confign_melsrFrtrrrHdim_projectiondepthlayersrrJs rKrAzAudioEmbedding.__init__Xs  '.vx'@'@XfmmfFX   v-t 4 4 *&**6488JFF#377$GGN!---+==n==DL*?;M#L1FF%b)) )((*T(((*"&,jj1I5&Q&Q#%zz*;Q?? ::mU + + #ZZ(8"==N.;N? +(::>::DLLDL ::+U 3 3 <''H(''"(,@"!E!E ,&*&:()!$45 gii" * * &" *112DEEEFA$6666P7666/,DLL DL$4h?? X % %$&Im[$I$ID ! ! u $ $)NElMdlM9M  ' -$*EE~VVF1e__ V V rwyy")NN*S*STUUUU$&M6$:D ! -$*EE~VVF1e__ V V rwyy")NN*S*STUUUU/1}f/ED , ,%ENEEE  !+ !%rLr\c||_dSrW)r\)rIr\s rKset_audio_embedszAudioEmbedding.set_audio_embedss(rLr]c||_dSrW)r])rIr]s rKset_audio_embed_sizesz$AudioEmbedding.set_audio_embed_sizess!2rLspeechaudio_attention_maskaudio_projection_modec<|jrEtj5|||\}}dddn #1swxYwYn|||\}}|j||d\}}|j1||d}|||\}}|jdkr|\}}} ||jz} | dkr#tj |ddd|j| z fdd}|d}| |||jz| |jz}|dkr| |} n/|dkr| |} ntd|d | S) zl arguments: input_embeds: audio features (B, T, D) B: num audios in a sequence N)rPrrrrgvisionzaudio_projection_mode = z not implemented)rCr`no_gradrOrRrSrrVrrrr3rUrZr) rIr\rhriaudio_featuresrrbsrfeat_dimr5audio_set_tensors rKget_audio_featuresz!AudioEmbedding.get_audio_featuress.  & U Y Y(, \CW(X(X% Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y%)LL?S$T$T !NE < # $ ^$ G G NA < #  a(($(LLeL$L$L !NE  &! + +$2$7$7$9$9 !B ;;G{{!""1d9GCD ""%))!,,G+00466466N !H , ,#44^DD   "h . .#??OO  R+@RRR  sAAArmc||d||}|dS)z arguments: audio_features: audio features (T, D) returns: audio_embeds: audio embeddings (num_audio_tokens, hidden_dim) r)rhri)rqrr)rIrmrhri audio_embedss rKrVzAudioEmbedding.forwardsK..  $ $Q ' '!5"7/  ##A&&&rL)Nrg)rXrYrZr[r rrAr`rrdrfr_rqrVrbrcs@rKr;r;UsPf&/f&3f&4f&f&f&f&f&f&P)U\)d))))3u|33333 59%- 4 4 l4 $lT14  # 4  4 4 4 4 r59%- '' '$lT1' # '  ''''''''rLr;)&rrtypingrrnumpyrr`torch.nn.functionalr functionalrr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperr2torch.distributed.fsdp.fully_sharded_data_parallelr transformersr 'vllm.model_executor.models.phi4mm_utilsr r r r rrrrrrrrrABCrerrr;rrLrKr}s_  XWWWWW))))))                          V'V'V'V'V'BIV'V'V'rxAxAxAxAxASWbixAxAxAv G#G#G#G#G#-G#G#G#T OOOOOBIOOOdy'y'y'y'y'RYy'y'y'y'y'rL