.`ivUddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z m Z mZddlZddlmZddlmZddlmZddlmZddlmZdd lmZmZmZmZdd l m!Z!m"Z"dd l#m$Z$dd l%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/dd l0m1Z1ddl2m3Z3ddl4m5Z5ddl6m7Z7m8Z8ddl9m:Z:m;Z;ddlm?Z?ddl@mAZA ddlBZBn#eC$reAdZBYnwxYwe&e+zZDe eEd<e(e-zZFe eEd<e)e.zZGe eEd<e deDZHe deFZIe deGZJe&e+ze(ze-zZKe eEd<e5eLZMGdde!ZNdS) N)AsyncGeneratorCallable)cached_property)Literal TypeAliasTypeVarcast)Request)PreTrainedTokenizerBase) EngineClient) RequestLogger) DeltaMessage ErrorResponseRequestResponseMetadata UsageInfo) OpenAIServingSpeechToTextRequest)OpenAIServingModels) TranscriptionResponse!TranscriptionResponseStreamChoiceTranscriptionResponseVerboseTranscriptionSegmentTranscriptionStreamResponseTranslationResponseTranslationResponseStreamChoiceTranslationResponseVerboseTranslationSegmentTranslationStreamResponse)VLLMValidationError) PromptType) init_logger) FlatLogprobsLogprob)SupportsTranscriptionsupports_transcription) RequestOutput) get_tokenizer)PlaceholderModulelibrosaSpeechToTextResponseSpeechToTextResponseVerboseSpeechToTextSegmentT)boundVS ResponseTypeceZdZdZddddddedededzd ed ed d ed effdZ d0dZ d0dZ e de efdZdededeeeeffdZ d1dedeeeeefzdede ededeef dZdededede eezde de!e"dffdeeze!e"dfze#zf d Z$ded!ee!e%dfd"e"d#e&d$ed%ed&d'e e'e e(zd(e e)e e*zde!e"dffd)Z+de,j-d*edee,j-fd+Z.d,e,j-d-ed.edefd/Z/xZ0S)2OpenAISpeechToTextzPBase class for speech-to-text operations like transcription and translation.F transcribe)return_tokens_as_token_ids task_typelog_error_stackenable_force_include_usage engine_clientmodelsrequest_loggerNr5r6r4 translater7r8cHt||||||j|_||_|j|j||_||_ tj |_ |jj r=ttt!|jj|jj|_|jr t&d|j||dS)N)r9r:r;r5r7)tokenizer_nametokenizer_modez6Overwriting default completion sampling param with: %s)super__init__ model_configget_diff_sampling_paramdefault_sampling_paramsr6 model_clsget_speech_to_text_config asr_configr8envsVLLM_MAX_AUDIO_CLIP_FILESIZE_MBmax_audio_filesize_mbsupports_segment_timestampr r r' tokenizerr@loggerinfo_warmup_audio_preprocessing_warmup_input_processor) selfr9r:r;r5r6r7r8 __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/entrypoints/openai/translations/speech_to_text.pyrBzOpenAISpeechToText.__init__Ls) ')'A+    (,'8'P'P'R'R$".BB  y  +E'%)%I" > 4 !'#'#4#>#'#4#CDN  '  KKH,    ((*** $$&&&&&returnc tttrdSt|jsdS t j}tdtj t|j j tj}tj||j j }ddlm}||j}d}t'|dr|j}n.t'|dr|j}t'|dr|j}|[tj||j j t1|d d t1|d d t1|d d}t j|z }td|dS#t2$rtdYdSwxYw)a:Warm up audio processing libraries to avoid first-request latency. The first call to librosa functions (load, get_duration, mel-spectrogram) triggers JIT compilation and library initialization which can take ~7s. This method warms up these operations during server initialization. Nz+Warming up audio preprocessing libraries...dtypeysrr)cached_processor_from_configfeature_extractoraudio_processorn_melsn_ffti hop_length)r[r\r`rbrcz-Audio preprocessing warmup completed in %.2fsz_Audio preprocessing warmup failed (non-fatal): %s. First request may experience higher latency.) isinstancer)r(r%rFtime perf_counterrNrOnpzerosintrH sample_ratefloat32 get_duration!vllm.transformers_utils.processorr]rChasattrr^r_featuremelspectrogramgetattr Exception exception) rR warmup_start dummy_audio_r] processorr^ audio_procwarmup_elapseds rTrPz.OpenAISpeechToText._warmup_audio_preprocessing}s g0 1 1  F&dn55  F, ,..L KKE F F F(3t'B#C#C2:VVVK${t7RSSSA      54T5FGGI $ y"566 E$-$?!!$566 E&6 :':;;E(2(D%!,O22!2"#4hDD!"3WcBB&'8,LL 3".00<?N KKG X X X X X      ?       sE0F&&$G Gc t|jsdSt|jdsdS ddlm}t j}tdtj t|j j tj}|j||j |jd|jdd }|d d d }|jd||}t j|z }td|dS#t($rtdYdSwxYw)a<Warm up input processor with dummy audio to avoid first-request latency. The first call to input_processor.process_inputs() with multimodal audio triggers multimodal processing initialization which can take ~2.5s. This method processes a dummy audio request to warm up the pipeline. Nget_generation_promptr)SamplingParamsz(Warming up multimodal input processor...rXenaudio stt_configrClanguager6request_prompt to_languageT) max_tokens temperature skip_clonewarmup) request_idpromptparamsz)Input processor warmup completed in %.2fsz[Input processor warmup failed (non-fatal): %s. First request may experience higher latency.)r%rFrovllm.sampling_paramsr}rfrgrNrOrhrirjrHrkrlr|rCr6input_processorprocess_inputsrsrt)rRr}rurv dummy_prompt dummy_paramsrwrzs rTrQz*OpenAISpeechToText._warmup_input_processors&dn55  Ft~'>??  F+  ; ; ; ; ; ;,..L KKB C C C(3t'B#C#C2:VVVK >??!?!..! @L*>L$33###4A ".00<?N KKC^ T T T T T      ?       sC6D''$EEcnddlm}||j}ttt |S)Nr) get_model_cls) vllm.model_executor.model_loaderrrCr typer$)rRrrFs rTrFzOpenAISpeechToText.model_clss<BBBBBB!M$"344 D./;;;rUrequest audio_datac K|j|j}|jr|j|jnd}t |dz |jkr"t ddt |dz tj|5}tj ||j j \}}dddn #1swxYwYtj ||}|j jo||j jk} | s|gn"||t#|} g} | D] } |j| |j |j||j|j|} |jdkrt/| t0s$t d d t3| jt7t0| }| d }t/|t:s$t d d t3|j|d d|d <| | | |fS)NizMaximum file size exceededaudio_filesize_mb) parametervalue)r\rZr verbose_jsonzExpected prompt to be a dictrdecoder_promptz!Expected decoder_prompt to be strz<|notimestamps|><|0.00|>) rFvalidate_languagerrlenrKrioBytesIOr)loadrHrkrmallow_audio_chunkingmax_audio_clip_s _split_audiorjr|rCr6rresponse_formatredictr__name__r getstrreplaceappend)rRrrrrbytes_r[r\durationdo_split_audiochunkspromptschunkr prompt_dictrs rT_preprocess_speech_to_textz-OpenAISpeechToText._preprocess_speech_to_texts >33G4DEE" DN , ,W-@ A A A  z??W $t'A A A%,-*oo/  Z # # IvLDO,GHHHEAr I I I I I I I I I I I I I I I '!333 O 0 <4?;; +M!0A0A!SWW0M0M # #E^99?!.!.&~':F&.88!&$//-6"*"6ll3 #400 !',22; 1?0F0F& 11 ,- NN6 " " " "  s$C  CCrtokens log_probs segment_class start_timecd}|jddd}|d|jjkr |dd}|f|z}g} d} |d|kr|d|kr ||dfz}d } td t |D].} || } | |kr|| d z |kr|| | }|d|z }|d|z }|j|d d}|d }t t|t | ||||zz|||zz|j|t |t tj |z |d d| | | z z }| || } d} | || d z | j z } 0| S) a Convert tokens to verbose segments. This method expects the model to produce timestamps as tokens (similar to Whisper). If the tokens do not include timestamp information, the segments may not be generated correctly. Note: No_speech_prob field is not supported in this implementation and will be None. See docs for details. g{Gz?rF)add_special_tokensrNrrzutf-8) idseekstartendrtextcompression_ratior avg_logprob) rMencode eos_token_idrangerdecoder r,rzlibcompressrlogprob)rRrrrrr BASE_OFFSET init_tokentokens_with_startsegmentslast_timestamp_startridxtokensliced_timestamp_tokensstart_timestamp end_timestampr text_bytescasting_segments rT_get_verbose_segmentsz(OpenAISpeechToText._get_verbose_segments?s!& ^**:%*PPQRS ":4 4 4CRC[F'MF2.0  R : - -2CB2G:2U2U 15Fr5J4L L  C 12233" A" AC&c*E ""'8q'AZ'O'O*;d}|j }d}j j!}|tE| d ks Jdt;| D]\}}|tG||znd}|23d{V}|jdkr|j$djsJ%tM|j$dj'||||j$dj}| (||(d|D|)|j$dj*6d+|}j dkrdtYt[j.| d}|jdkr%t_t`tc||}nt_tdtg||j4tk| | }ng|jdkr$t_t`tm|}n8t_tdto||j4tk| | }|S#tpj9$rdcYSt"$r} | cYd} ~ Sd} ~ wwxYw)zUBase method for speech-to-text operations like transcription and translation.N)rjsonrzFCurrently only support response_format`text`, `json` or `verbose_json`rz*Currently do not support verbose_json for z2verbose_json format doesn't support streaming case-)r)rrz$Error in preprocessing prompt inputsr)r lora_requestc \g|](\}}j|d|)S)rw)r)r9generate).0irrrsampling_paramsrRs rT z=OpenAISpeechToText._create_speech_to_text..sa%%%Av "++#!''A''!- ,%%%rUr<rz:`max_audio_clip_s` is set to None, audio cannot be chunkedrr)rrrrrcg|] }|j Sr)rsegs rTrz=OpenAISpeechToText._create_speech_to_text..s*H*H*H38*H*H*HrUr4r)rseconds)rusage)rrrrrzClient disconnected): _check_modelr9errored dead_errorrcreate_error_responserFrLmodelstreamr6_base_request_idrstaterequest_metadata_maybe_get_adaptersr ValueErrorrNrtmax_completion_tokensrC max_model_lenminto_sampling_paramsrElogprobs _log_inputsr enumeraterrrHrrfloatoutputsrtuple token_idsextendrrjoinrjmathceilr r-rr/rrrrrasyncioCancelledError)rRrrrrrerror_check_retrr duration_selist_result_generatordefault_max_tokenstotal_segments text_partssegments_typesrrchunk_size_in_srresult_generatorroprrfinal_responserrrs` @@@rT_create_speech_to_textz)OpenAISpeechToText._create_speech_to_texts!% 1 1' : :::::::  &" "   % 0$/ /  "*J J J--7   #~ 5 5N= 6--LW]LL   "n 4 4 4--D MM)>)>{)K)KMM 2jIII  B1AK  . 133G< **. O&011Q666P766*33H)I)I > >%%4C4OE#/000UX!1>>>>>>>"..@@!z!}5555 66',RZ]-D'E'E.;(/+5*,*Q-*@ 7!'--h777"))*H*Hx*H*H*HIIII"))"*Q-*<====!!1 0"77:&&D~--'"49Z#8#899 *n<<%)0d%HHH&&NN&*4!%%,%5%(__%3 &&NN*n<<%)!-@d-K-K-K%L%LNN%)2!%%,%5%(__%3 &&N" !% E E E--.CDD D D D 1 1 1--a00 0 0 0 0 0 0 1so55D++ E.5.E)#E.)E.4BH H:H5/H:5H:A9R1NGR11$S? S? S:4S?:S?r rraudio_duration_schunk_object_type)ztranslation.chunkztranscription.chunkresponse_stream_choice_classstream_response_classc hKttj} |j} d} d} |jp|j} | r|jr|jnd} |D]"}|23d{V}|jAt|j} |j ||j |j x}r| |z } t|j dksJ|j d}t|j}| t|jz } |j ||}n|||j|j}|||| |g| }|rt%| | | | z|_|d }d |d WV6$| rFt%| | | | z}|||| g| | }|d d }d |d WVt%| | | | z|_nU#t,$rH}t.d|j||}d |d WVYd}~nd}~wwxYwdWVdS)NrFr)content)delta)r finish_reason stop_reason)robjectcreatedchoicesr) prompt_tokenscompletion_tokens total_tokensT) exclude_unsetzdata: z )rr r!r"rr)r& exclude_nonezError in %s stream generator.zdata: [DONE] )rjrfrr8stream_include_usagestream_continuous_usage_statsprompt_token_idsrrFget_num_audio_tokensrHrCrrrrrrrrmodel_dump_jsonfinal_usage_inforsrNrtr6create_streaming_error_response)rRrr rrrrrr created_time model_namer$num_prompt_tokens include_usageinclude_continuous_usagerres audio_tokensoutput delta_message choice_datardata final_usagefinal_usage_chunkfinal_usage_datar s rT _speech_to_text_stream_generatorz3OpenAISpeechToText._speech_to_text_stream_generator5sv49;;'' ] 7W7;W  !(!F G 1 1 ! T &$91 .1 . !10.0.0.0.0.0.0.#+7,/0D,E,E)+/>+N+N,dot?P,,<>.=-s{++q0000 [^F$0$E$E$EM%V-=)>)>>%+3&B&B&W&W&W 'C&B"/*0*>(.(:''' 21%0 ,!, ( E0&/*;.?):=N)N''' !00t0DDD-4--------a"2!1h 6'"3&7!25F!F %:$9!,($% %%%!$5#D#D"&T$E$$ 6/55555551:/"3.1BB111  - -  & & &   H%%H*rkc|jjJd|jjd||jjz}||jjz}g}d}||jdkr||z|jdkr ||d|dfno||z|z }t ||z|jd}||||} ||d|| f| }||jdk|S)Nz!self.asr_config.max_audio_clip_s=z+ cannot be None to split audio into chunks.rr.)rHroverlap_chunk_secondshaperr_find_split_point) rRrrk chunk_size overlap_sizerr search_start search_end split_points rTrzOpenAISpeechToText._split_audios3/;; (t/ ( ( (<;;!4?#CC "T_%II  *"2&&&:~!1"!555 jabb1222z>L8LQ^Z-=b-ABBJ00\:VVK MM*S!K-%78 9 9 9A*"2&&& rUwav start_idxend_idxc|||}tj}d}|jj}|Jt dt ||z |D]6}||||z} | dzdz} | |kr||z}| }7|S)a.Find the best point to split audio by looking for silence or low amplitude. Args: wav: Audio tensor [1, T] start_idx: Start index of search region end_idx: End index of search region Returns: Index of best splitting point rNg?)rinfrHmin_energy_split_window_sizerrmean) rRrGrHrIsegment min_energy quietest_idxmin_energy_windowrwindowenergys rTrAz$OpenAISpeechToText._find_split_pointsi'(X   OH ,,,q#g,,)::rts 44444444%%%%%%444444444444000000------111111 VUUUUUUUFFFFFF                        0/////''''''######////////TTTTTTTT&&&&&&))))))555555+NNNN+++ **GGG+#8:M"MiMMM #==Y"68J!JYJJJ GC+,,, GC2333 GC*+++"#!! i X  Q Q Q Q Q Q Q Q Q Q s9B>>CC