.`i `UddlmZmZmZddlmZmZmZmZm Z ddl Z ddl Z ddl mZddlmZddlmZmZddlmZddlmZmZmZddlmZdd lmZdd lmZdd l m!Z!dd l"m#Z#dd l$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5m6Z6m7Z7m8Z8m9Z9ddl:m;Z;mZ>m?Z?m@Z@ddlAmBZBddlCmDZDddlEmFZFddlGmHZHmIZIddlJmKZKmLZLmMZMmNZNmOZOmPZPmQZQddlRmSZSmTZTmUZUmVZVmWZWddlXmYZYmZZZm[Z[ddl\m]Z]Gddej^Z_Gd d!ej^Z`Gd"d#ej^ZaGd$d%ej^ZbGd&d'ZcGd(d)ej^ZdGd*d+eHZeGd,d-eHZfeeefzZgeehd.<Gd/d0ej^ZiGd1d2e=ZjGd3d4e;ejZkd5eele jmfd6enele2ffd7ZoGd8d9e9ZpGd:d;eej^eUeVeTeWZsdS)?)IterableMappingSequence) AnnotatedAnyLiteral TypeAliascastN) BatchFeature) GlmAsrConfigGlmAsrProcessor)WhisperFeatureExtractor) ModelConfigSpeechToTextConfig VllmConfig)BaseDummyOptions)$get_tensor_model_parallel_world_size) PromptType) get_act_fn)MMEncoderAttention)ColumnParallelLinearQKVParallelLinearRowParallelLinear)QuantizationConfig)ApplyRotaryEmb)MultiModelKeys)MULTIMODAL_REGISTRY)MultiModalDataDictMultiModalFieldConfigMultiModalKwargsItems)DictEmbeddingItems ModalityDataModalityDataItemsMultiModalDataItemsMultiModalDataParser)BaseDummyInputsBuilderBaseMultiModalProcessorBaseProcessingInfoPromptReplacement PromptUpdatePromptUpdateDetails)IntermediateTensors)cached_tokenizer_from_config)cached_processor_from_config) TensorSchema TensorShape)DEFAULT_CONV_PARAMSDEFAULT_MAX_AUDIO_LEN_SDEFAULT_MERGE_FACTOR!_flatten_audio_features_by_length#_get_audio_output_lengths_for_tower_group_audio_embeddings_normalize_chunk_counts)MultiModalEmbeddings SupportsLoRASupportsMultiModal SupportsPPSupportsTranscription)AutoWeightsLoaderinit_vllm_registered_model maybe_prefix)ISO639_1_SUPPORTED_LANGSc>eZdZdZdfd ZdedejfdZxZ S)GlmAsrEncoderRotaryEmbeddinga$ Rotary Position Embedding for GLM-ASR encoder. Computes rotary position embeddings on-demand for efficiency. Only caches inv_freq as a buffer; cos/sin are computed during forward to avoid wasted computation during initialization and ensure correct device placement. returnNcZtt|d|j|jz}t |drp|jri|jdd}|jdd}t||z}|jdd|_ nt|dd}|}d|_ ||_ ||_ d|tj d|d tj |z zz }|d |d dS)Nhead_dimrope_parameters rope_thetag@partial_rotary_factorg?attention_scalingrdtypeinv_freqF) persistent)super__init__getattr hidden_sizenum_attention_headshasattrrGgetintrJdimrFtorcharangefloatregister_buffer)selfconfigrFbaserIrXrN __class__s u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/glmasr.pyrQz%GlmAsrEncoderRotaryEmbedding.__init__TsD  J 2f6P P   6, - - )&2H ))--lGDDD$*$:$>$>'%% !h!6677C%+%;%?%?#S&&D " "6<99DC%(D "  $5<3#M#M#MPS#STU ZeDDDDDseq_lenctj||jj|jj}tj||j}||jzS)a9 Compute rotary position frequencies for given sequence length. Args: seq_len: The sequence length to compute embeddings for. Returns: Frequency tensor with shape [seq_len, dim/2]. Use .cos() and .sin() to get the rotary embedding components. )devicerM)rYrZrNrerMouterrJ)r]rcseqfreqss raforwardz$GlmAsrEncoderRotaryEmbedding.forwardrsNl DM0 8K    C//t---rb)rDN) __name__ __module__ __qualname____doc__rQrWrYTensorri __classcell__r`s@rarCrCJsqEEEEEE<.s.u|........rbrCcxeZdZdZ d dedzdeffd Zdejdejd ejd ejfd Z xZ S) GlmAsrEncoderAttentiona$ Optimized Multi-headed Grouped Query Attention for GLM-ASR encoder. Uses vLLM's QKVParallelLinear for fused projections, ApplyRotaryEmb for rotary position embeddings, and MMEncoderAttention for hardware-optimized attention computation with automatic backend selection. N quant_configprefixc dt||_|j|_|j|_t |d|j|_|j|jz|_t|_ |j|j z|_ td|j|j z|_ t|j|j|j|jd||d|_t!|j|jd||d|_t |dd}|r|dd }nt |dd }t'|j|z|_t+d |_t/|j |j|jd z|j |d  |_dS)Nnum_key_value_headsr1Tz .qkv_projbiasrtruz.o_projrGrIg?)enforce_enablegz.attn) num_heads head_sizescale num_kv_headsru)rPrQr^rSrTr{rRr~rFrtp_sizenum_heads_per_rankmaxnum_kv_heads_per_rankrqkv_projro_projrVrW rotary_dimrapply_rotary_embrattn)r]r^rtru rope_paramsrIr`s rarQzGlmAsrEncoderAttention.__init__s  !-3# )6+E  (DN: ;== "&.DL"@%(D,=,M%N%N" *   M N  %'''    (    %%%%     f&7>>  R$/OO4KS$Q$Q ! !$+F4KS$Q$Q !dm.CCDD .d C C C'-m-%3###     rb hidden_statesrotary_pos_emb_cosrotary_pos_emb_sinrDc|j\}}}||\}}|j|jz}|j|jz} ||| | gd\} } } | |||j|j} | |||j|j} | |||j|j} || dd|jf||| dd|jf<|| dd|jf||| dd|jf<| | | | } | ||d} | | \}}|S)@ Args: hidden_states: [batch_size, seq_len, hidden_size] rotary_pos_emb_cos: [seq_len, rotary_dim/2] - cosine of rotary embeddings rotary_pos_emb_sin: [seq_len, rotary_dim/2] - sine of rotary embeddings Returns: [batch_size, seq_len, hidden_size] rX.N) shaperrrFrsplitviewrrrr)r]rrr batch_sizerc_qkvq_sizekv_sizeqkv attn_outputoutputs rarizGlmAsrEncoderAttention.forwards"/!4 GQ}--Q(4=8,t}<))VWg6B)??1a FF:w(? O O FF:w(BDM R R FF:w(BDM R R %)$9$9 c$T_$$ %'9;M% % #  !%)$9$9 c$T_$$ %'9;M% % #  ! ii1a(( "&&z7B?? KK ,,  rbNrs rjrkrlrmrstrrQrYrnrirorps@rarrrrs37 : : )4/:  : : : : : : x1|1"L1"L 1  11111111rbrrc\eZdZdZ d dedzdeffd Zdejdejfd Z xZ S) GlmAsrEncoderMLPzk Optimized MLP for GLM-ASR encoder. Uses vLLM's parallel linear layers for better performance. NrsrtrucPt||_|j|_|j|_t |j|jd||d|_t|j|_ t|j|jd||d|_ dS)NTz.fc1rxz.fc2) rPrQr^rSintermediate_sizerfc1r hidden_actact_fnrfc2r]r^rtrur`s rarQzGlmAsrEncoderMLP.__init__s  !-!'!9'    "%???    !!233 $  "  %???    rbrrDc||\}}||}||\}}|SN)rrr)r]rrs rarizGlmAsrEncoderMLP.forward!sE88M22 q M22 88M22 qrbrrrps@rarrs37   )4/        :U\elrbrcxeZdZdZ d dedzdeffd Zdejdejd ejd ejfd Z xZ S) GlmAsrEncoderLayerz Optimized Transformer encoder layer for GLM-ASR. Combines attention and MLP with residual connections and layer norms. Nrsrtrucjt|j|_t|||d|_t |||d|_t|dd}tj |j||_ tj |j||_ dS)Nz .self_attnrtruz.mlplayer_norm_epsh㈵>eps) rPrQrSrr self_attnrmlprRnn LayerNorminput_layernormpost_attention_layernormr]r^rtrurr`s rarQzGlmAsrEncoderLayer.__init__.s !-/ %(((    $ %???    !)94@@!|D,<.QQQ(*  .) ) ) %%%rbrrrrDc|}||}||||}||z}|}||}||}||z}|S)r)rrr)rrrr)r]rrrresiduals rarizGlmAsrEncoderLayer.forwardIs !,,];; '11'  !=0 !55mDD //  =0 rbrrrps@rarr(s37   )4/        6|"L"L   rbrc,eZdZdZdZdejfdZdS)_GlmAsrEncoderOutputa Simple output container compatible with transformers' BaseModelOutput. This lightweight container holds the encoder output and is compatible with the transformers library's output format while being more efficient than a full dataclass. Attributes: last_hidden_state: Final layer hidden states from the encoder. Shape: [batch_size, seq_len, hidden_size] last_hidden_staterc||_dSrr)r]rs rarQz_GlmAsrEncoderOutput.__init__zs!2rbN)rjrkrlrm __slots__rYrnrQrbrarrksB  'I3%,333333rbrceZdZdZdgdiZ ddedzdeffd Zd ej d e ej ej ffd Z d ej d e fd Z dee eej fd eefdZxZS) GlmAsrEncodera Optimized GLM-ASR Audio Encoder with vLLM native implementation. This encoder processes audio features through convolutional layers followed by transformer layers with rotary position embeddings. Optimized for performance with: - QKVParallelLinear for fused attention projections - Tensor parallelism support via ColumnParallelLinear/RowParallelLinear - Quantization support - Flash Attention (SDPA) rq_projk_projv_projNrsrtruct|_tjjjdd|_tjjjddd|_tj fdtj D|_ tdd}tjj| |_t!|_dS) Nr1) kernel_sizepaddingrK)rstriderc >g|]}td|S)z.layers.r)r).0 layer_idxr^rurts ra z*GlmAsrEncoder.__init__..sQ     #!-$99i99   rbrrr)rPrQr^rConv1d num_mel_binsrSconv1conv2 ModuleListrangenum_hidden_layerslayersrRrnormrC rotary_embrs ``` rarQzGlmAsrEncoder.__init__s  Y         Y         m       "'v'?!@!@        !)94@@L!3HHH 7v>>rb input_lengthsrDcB|dzdz dzdz}|dzdz dzdz}||fS)z Compute the output length after convolutions. Args: input_lengths: Input sequence lengths [batch_size] Returns: Tuple of (output after conv1, output after conv2) rKrr1r)r]routput_lengths_conv1output_lengths_conv2s ra _get_feat_extract_output_lengthsz.GlmAsrEncoder._get_feat_extract_output_lengthssF!. 5 9a?!C!5u qkv_proj mapping.r)default_weight_loader))rrr)rrr)rrrz.bias weight_loader) -vllm.model_executor.model_loader.weight_utilsrdictnamed_parameterssetreplaceendswithrrRadd) r]rrstacked_params_mapping params_dict loaded_paramsname loaded_weight param_name weight_nameshard_idparamrs ra load_weightszGlmAsrEncoder.load_weightss]WWWWWW" " "  4002233 "%%% #* $ $ D-5K 4 41 Kd**||K<<==))d+.E.E#D) % 3  e]H=====))d+.E.E{**#D) '@U V V  e]333   d # # # #rbr)rjrkrlrmpacked_modules_mappingrrrQrYrntuplerrrirrrrorps@rarr~s   22237 )?)?)4/)? )?)?)?)?)?)?V:"\: u|U\) *::::(#Eel#E7K#E#E#E#EJ$HU3 3D-E$F$3s8$$$$$$$$rbrc2eZdZUdZeded<eeje ejze ddddhfed<eeje ejze dddhfed <eeje ejze d fed <d S) GlmAsrFeatureInputsz Dimensions: - num_chunks: Number of audio chunks (flattened) - nmb: Number of mel bins - num_audios: Number of original audio files audio_featurestype num_chunksnmb chunk_length dynamic_dimsrfeature_attention_mask num_audios chunk_countsN) rjrkrlrmr__annotations__rrYrnlistr0rrbrarrs " #### tEL)) L%~FVWWW Y& tEL)) L.?OPPP R tEL)) L!! #rbrczeZdZUdZdZeded<eee j e ddddhfed<dS) GlmAsrEmbeddingInputsz Dimensions: - bn: Batch size - naf: Number of audio features - hs: Hidden size (must match the hidden size of language model backbone) audio_embedsrbnnafhsr N) rjrkrlrmrrrrrrYrnr0rrbrarr2su%3D'. !222 U\ D%UG<<< >rbr GlmAsrInputsc`eZdZdZ d dededzdeffd Zdej d ej fd Z xZ S) GlmAsrMultiModalProjectora Projects audio encoder outputs to language model hidden space. This projector uses a two-layer MLP to map audio features from the encoder's intermediate size to the language model's hidden size. Uses vLLM's parallel linear layers for tensor parallelism support. Architecture: - Linear layer: intermediate_size -> hidden_size * 2 - Activation function (e.g., GELU) - Linear layer: hidden_size * 2 -> hidden_size Nrsr^rtrucBtt|jj|jjdz||d|_t|j |_ t|jjdz|jj||d|_ dS)NrKz .linear_1) input_size output_sizertruz .linear_2) rPrQr audio_configr text_configrSlinear_1rprojector_hidden_actactrlinear_2rs rarQz"GlmAsrMultiModalProjector.__init__Ss ,*<*6:%'''     f9::))59*6%'''     rbrrDc||\}}||}||\}}|Sr)rr!r")r]rrrs rariz!GlmAsrMultiModalProjector.forwardhsE==88 q// ==77 qrbr) rjrkrlrmr rrrQrYrnrirorps@rarrEs   37    )4/        *elu|rbrc`eZdZdZdefdZdedefdZdede fdZ de e e dzffdZdS) GlmAsrProcessingInfoz Processing information provider for GLM-ASR model. Provides access to model configuration, processor, and feature extractor needed for audio preprocessing and multimodal integration. rDc@|jtSr)ctx get_hf_configr r]s rar(z"GlmAsrProcessingInfo.get_hf_configwsx%%l333rbkwargsc 2|jjtfi|Sr)r'get_hf_processorr r]r*s rar,z%GlmAsrProcessingInfo.get_hf_processorzs(tx(CCFCCCrbc &|jdi|jSNr)r,feature_extractorr-s raget_feature_extractorz*GlmAsrProcessingInfo.get_feature_extractor}s$t$..v..@@rbNc ddiS)Naudiorr)s raget_supported_mm_limitsz,GlmAsrProcessingInfo.get_supported_mm_limitss rb)rjrkrlrmr r(objectr r,rr1rrrWr4rrbrar%r%os4|4444DDODDDDAfA9PAAAAcDj)Arbr%c teZdZdZdeeefdefdZ d dedeeefdeeefdzde fdZ dS) GlmAsrDummyInputsBuilderz Builder for dummy inputs used in profiling and testing. Generates dummy text prompts and audio data that match the expected format for GLM-ASR model inputs. Used for memory profiling and performance benchmarking. mm_countsrDct|dd}|j}|j|zS)Nr3r)rVinfor, audio_token)r]r8r  hf_processors raget_dummy_textz'GlmAsrDummyInputsBuilder.get_dummy_texts6]]7A.. y1133 '*44rbNrc mm_optionscR|j}|j}|dd}|r|dnd}t |jdt }t||z} d|| ||iS)Nr3r max_audio_len)lengthr  overrides) r:r1 sampling_raterVrRr,r3rW_get_dummy_audios) r]rcr8r>r0rCr audio_overridesr@ audio_lens raget_dummy_mm_dataz*GlmAsrDummyInputsBuilder.get_dummy_mm_datas !I;;==)7 ]]7A.. 5?I*..111T I & & ( (/;R   566  T++ Z?,  rbr) rjrkrlrmrrrWr=rrrGrrbrar7r7s5S(95c5555=A    38$ C!112T9         rbr7 hf_inputsrDc |d}|attjdtjd|dtjd|dtjdSttjdtjdtjdtjdS)a Configure multimodal field batching strategy for GLM-ASR. Determines how to batch audio inputs based on whether chunking is used. When chunk_counts is present, features are flattened across chunks; otherwise, they are batched normally. Args: hf_inputs: Dictionary of preprocessed inputs from HuggingFace processor. Returns: Dictionary mapping field names to MultiModalFieldConfig objects that specify batching behavior. rNr3rr)rrr r)rVrrbatchedflat_from_sizes)rHrs ra_glmasr_field_configrLs"==00L.6w??0@1$9#H1$$$/6w??     *27;;,4W==4I1y;6:{JKKH   Hk : : ; ; ; ;rbpromptmm_data mm_kwargs tok_kwargscd|vr|d|d<|dg}|rt|ts|gn|}|sa|j|}||}tt|gdS|jj di|}tdi|d|j i}t ||||} d| vr| d| d <nfd | vrbd | vr^| d } t| tjr merge_factor conv_paramsr rr1)_as_list_chunk_counts#_get_audio_output_lengths_from_maskritem_idxcr |}n<d}|||}|jd}ntd|dkrtdgt|z}t j|S)Nrrz>Either feature_attention_mask or audio_embeds must be providedzAudio is too short)embed_token_id)rVr ValueErrorrWr+select_token_id)r num_featuresrembed audio_tokensaudio_output_lengthsaudio_token_id out_mm_datas raget_replacement_glmasrzMGlmAsrMultiModalProcessor._get_prompt_updates..get_replacement_glmasr}s# 3H= *~>> +(2E#(;q>LL$Xq   !5666*+c,.?.??L&6- rbr3)rQtarget replacementr)r:r,rq get_vocabr(rRrVrr4r2get_data glmasr_utilsrrrTrrYstackr_rWsumitemrr^rw unsqueezer))r]r~r{rr\ tokenizervocabr^r;rrr rrr start_idxcountend_idxrzlengthsidxrrrrs @@@ra_get_prompt_updatesz-GlmAsrMultiModalProcessor._get_prompt_updatesFs /DI.HH1GHH I++-- ##%%((**i BB ;//  !&5Nv~7KLL fm5HII #,,.. !,1I!J!J"~66 +- ! -         ' 22<@@ ( (E'%/G1)G2CDD!$--1${400AAlKG)//GKKMM4F4F4H4H0I0IJJJ 'II (!%;!>AAlKG)//GKKMM4F4F4H4H0I0IJJJJ S        0  "2     rb)rjrkrlrmr%rZrrrr rWrerrr5rr rtrr}r$r rr*rrorps@rarWrWs U"6UUUUI3#  c &<<c6k"<38$ < CK( <  <<<<<<|//!(V 4/ ++ , ////T %T !(V 4T - T ,  T T T T T T T T rbrW)r: dummy_inputscDeZdZeZgdddgdZdddedeffd Ze d ed e d ed zfdZ d e fdZ ded ed zfdZded ejeejdfzfdZded efdZ d*dejdejded zdejd zded ejezf dZdejd ejd zfdZdeeeejfd eefdZe ded efd Ze ded!ed efd"Z e d#e!j"ded$ed%ed zd!e#d&d'ed(ed zd e$fd)Z%xZ&S)+GlmAsrForConditionalGenerationr gate_projup_proj)r gate_up_projrs)ru vllm_configruc ~t|jj}|j}|jj}||_||_||_||d5t|j |t|d|_ t||t|d|_ dddn #1swxYwY||5t||jt|ddg|_dddn #1swxYwY|jj|_dS)Nr3 audio_towerrmulti_modal_projectorlanguage_modelLlamaForCausalLM)r hf_configru architectures)rPrQ model_configrrtmultimodal_configr^_mark_tower_modelrrr@rrr_mark_language_modelr?rrmake_empty_intermediate_tensors)r]rrur^rtrr`s rarQz'GlmAsrForConditionalGeneration.__init__s )3"/ '4F !2(  # #K 9 9  ,#)#FM::   D  *C)#F,CDD***D &                & &{ 3 3  "<' ,#F,<==12 ###D                   ? ,,,s%,ACC C (-D!!D%(D%rQirDNcN|drdStd)Nr3z)<|begin_of_audio|><|pad|><|end_of_audio|>z Only audio modality is supported) startswithr)clsrQrs raget_placeholder_strz2GlmAsrForConditionalGeneration.get_placeholder_strs-   w ' ' ?>>;<<GlmAsrForConditionalGeneration._parse_and_validate_audio_inputszz.$77  #(n<XXX X$4d;;  !4"!)#)::.F#M#MND99     rb audio_input.c@|ddkrt|dS|d}|d}t|tr,tj|d}tj|d}|jd}t |d|}||j j j j }| |j }|jjj}|jjj}||z} |jd } | | z| z} | | kr|ddd| ddf}||d |}||} t)|jd t*} t)|jd t,}t/|j |d | |}t3| |}tj||}t;||S)Nrrrr rrr)rrLr1rrr)rrTrrYcatrr8rVrrrweightrMrr^rrSrreshaperrRr4r2r6rr5rflattentolistr7)r]rrr rraudio_hidden_statesrSr merge_ratiorcseq_len_truncatedrrrrmasked_audio_featureschunk_embeddingss ra_process_audio_inputz3GlmAsrForConditionalGeneration._process_audio_inputsF v . 0 0^455 5$%56!,-E!F nd + + N"Y~1===N%*Y/E1%M%M%M "#)!, . OON + +    (**1A1G1N1T*UU#..~>>Pk.:  K4F';6 &+A.$ 3{B w & &"5aaa9K:K9KQQQ6N"O 299      334GHHt{Nr)r]rrloaders rarz+GlmAsrForConditionalGeneration.load_weightsMs067 "4}EEE""7+++rbrcBt|}t|ddS)znGet the audio token from processor. Similar to get_placeholder_str but returns single token. r;r)r.rR)rrr\s ra_get_audio_tokenz/GlmAsrForConditionalGeneration._get_audio_tokenRs# 1>> y-;;;rb task_typect|}|j}t|dt}t ||jS)Nr@)max_audio_clip_s sample_rate)r.r0rRr3rrC)rrrr\r0rs raget_speech_to_text_configz8GlmAsrForConditionalGeneration.get_speech_to_text_config[sM1>> %7"9o?VWW!-)7    rbr3 stt_configlanguage) transcribe translaterequest_prompt to_languagecvt|}||} |dkr#|j||} | d| } n|dkr| d} nt d|d| dg} || dd } || }|d |id }tt|S) z@Get the generation prompt to be used for transcription requests.rztranslate the speech to rz4can you transcribe the speech into a written format?zUnsupported task type user)rolecontentFT)tokenizeadd_generation_promptr3)prompt_token_idsmulti_modal_data) r-rsupported_languagesrVrapply_chat_templaterrr r)rr3rrrrrrrr;full_lang_name_to user_contentmessagesrfr prompt_dicts raget_generation_promptz4GlmAsrForConditionalGeneration.get_generation_promptgs1>> **<88  # # # 7 ; ;K U U )VVCTVVLL , & &TTT LAiAABB B# ==>.. uD/  %++F33 0!(% 0  J ,,,rb)NN)'rjrkrlrArrrrrQ classmethodrWrrrr5rrrYrnrrr9rr,rirrrrrrrrnpndarrayrrrrorps@rarrs&3322$i0 BD   z 3      B=3=3=3:===[=        G'>G elC/0 0>G>G>G>G@%%4H%%%%<@-1 <<2D8  |d*    + +&A|A  AAAA ,HU3 3D-E$F,3s8,,,, r?r@whisperrAModulerCrrrrrrrrrrrr%r7rrnrrLrNrWregister_processorrrrbrars87777777777;;;;;;;;;;;;;; %%%%%%DDDDDDDD??????CCCCCCCCCC333333PPPPPP''''''<<<<<<XXXXXX GFFFFFMMMMMMDDDDDD////// .-----888888JJJJJJ>>>>>>>>ONNNNNNNNN------8.8.8.8.8.298.8.8.vvvvvvRYvvvr'''''ry'''T@@@@@@@@F33333333&ZZZZZBIZZZz,.     L    .0EE iEEE''''' '''T-*" " " " " 56JK" " " J"sEL()" #$ $%""""J/////!5///,v v v v v 78N Ov v v r(' ) h-h-h-h-h-I!:|=Rh-h-  h-h-h-rb