.`igddlZddlmZmZddlmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZddlmZdd lmZdd lmZmZdd lmZmZmZdd lmZdd lm Z m!Z!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0ddl1m2Z2ddl3m4Z4ddl5m6Z6ddl7m8Z8m9Z9m:Z:m;Z;mGddeZ?Gdde?Z@dS)N)AsyncGeneratorMapping)AnyFinalcast)Request)Response) assert_neveroverride) EngineClient)ChatTemplateContentFormatOption) RequestLogger) ErrorResponse UsageInfo)EmbeddingServeContext OpenAIServing ServeContext)OpenAIServingModels)EmbeddingBytesResponseEmbeddingChatRequestEmbeddingCompletionRequestEmbeddingRequestEmbeddingResponseEmbeddingResponseData) RenderConfig) TokensPrompt) init_logger)EmbeddingRequestOutput PoolingOutputPoolingRequestOutput RequestOutput) PoolingParams)merge_async_iterators) chunk_list) EmbedDTypeEncodingFormat Endiannessencode_pooling_bytesencode_pooling_outputceZdZfdZedededzfdZdede fdZ edede e zezfdZ defd Zdefd Zded eed edeeedffd Zdeededeffd Zdedededeeefdzdedeeezdff dZedededzffd Zedededzffd ZxZ S)EmbeddingMixinctj|i||jj}t |o|j|_|r|jr|jnd|_dSN)super__init__ model_config pooler_configboolenable_chunked_processingsupports_chunked_processing max_embed_len)selfargskwargsr1 __class__s z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/entrypoints/pooling/embed/serving.pyr/zEmbeddingMixin.__init__9sw$)&))))7 ,0  EmE, , (  !.!< M ' ' ctxreturnNc Ktt|} ||j|_t |jt rt||j|j|jj |jj p|j |j |jj |jj |jjd{V\}|_nX|}||jj||jd{V|_dS#t(t*f$rF}t,d|t3|cYd}~Sd}~wwxYw)N) chat_templatechat_template_content_formatadd_generation_promptcontinue_final_messageadd_special_tokens)prompt_or_promptsconfigz$Error in preprocessing prompt inputs)rr_maybe_get_adaptersrequest lora_request isinstancer_preprocess_chatrenderermessagesr?r@rArBrCengine_prompts_get_completion_renderer render_promptinput_build_render_config ValueError TypeErrorlogger exceptioncreate_error_responsestr)r6r<_rKes r: _preprocesszEmbeddingMixin._preprocessHs (#.. 6#77 DDC #+';<< .2.C.CKMK("%+";"Ps?P141Q*-+*K+.;+M'*{'E/D / / ) ) ) ) ) )%3%% 88::+3+A+A&)k&744S[AA,B,,&&&&&&"4I& 6 6 6   C D D D--c!ff55 5 5 5 5 5 5 6sDD E71;E2,E72E7rGc||rd}n|jp|j}t||j|jS)N) max_lengthtruncate_prompt_tokensrC)_should_use_chunked_processingr5 max_model_lenrr]rC)r6rGr\s r:rQz#EmbeddingMixin._build_render_configgsV  . .w 7 7 BJJ+At/AJ!#*#A&9    r;cdtttjjjjjjjfd}dtdtffd }dksdkr |Sdksdkr|dkStdS) Nc >g}d}t D]U\}}t|t|  }|j}|||t |z }Vt ||}tjj j ||S)Nr)encoding_format embed_dtype endianness)index embedding) prompt_tokens total_tokensidcreatedmodeldatausage) enumeraterr)prompt_token_idsappendlenrr request_id created_time model_name) itemsnum_prompt_tokensidx final_resitemrprnr<rcrbrdfinal_res_batch_checkeds r:encode_float_base64z;EmbeddingMixin._build_response..encode_float_base64s13E ! "+,C"D"D ; ;Y,3!(7$/#- $-#=  T"""!S)9%:%::!!/.E %>(n  r; bytes_onlyr=c t\}}}|rdn*dtjjjj||di}t ||S)N)pooling_outputsrcrdmetadatari)contentheaders)r(jsondumpsrsrtrur) r}rrvrnrr<rcrdr{s r: encode_bytesz4EmbeddingMixin._build_response..encode_bytess$8 7'%%%% !GUE  "%.'*'7%(^$)%* !!   *'7KKK Kr;floatbase64bytes)r}) rlistr final_res_batchrGrbrcrdr2rr )r6r<r|rrcrbrdr{s ` @@@@r:_build_responsezEmbeddingMixin._build_responsets #'t,@'A3CV"W"W*-+*E"%+"9 !$!7          @ LT L.D L L L L L L L L L2 g % %H)D)D&&(( (  ' '?l+J+J>@@'0 y"4 5 5( (  2 2 #I|#&.XX*XXYXX #/ "M"M"M     #% -    "&!3!:!:#  -+ j!<< ";""    0 1 1 1 1r; input_ids input_textc:t|}t|ttfr||}|}|j d}|j}n d}|j}d} d} ||kr%t| |||||krD|rt d||n%t| d||t|| St|||S) z>Override to support chunked processing for embedding requests.Nzmaximum embedding input lengthzmaximum context lengthzThis model's {length_type} is {max_length_value} tokens. However, you requested {token_num} tokens in the input for embedding generation. Please reduce the length of the input.zThis model's {length_type} is {max_length_value} tokens. However, you requested {token_num} tokens in the input for embedding generation. Please reduce the length of the input or enable chunked processing.) length_typemax_length_value token_numzOInput length %s exceeds max_position_embeddings %s, will use chunked processingz"maximum position embeddings length)promptrp)rrrIrrr^rr5r_rRformatrTinforr._validate_input) r6rGrrrenable_chunkedrrrvalidation_error_msgchunked_processing_error_msgr9s r:rzEmbeddingMixin._validate_inputsy NN  g : #'#5  7 #'#5 O !0 )+++ (//$/)9"+0---!KK:!* %4;;(L-?&/< zINNN Nww&&w :FFFr; engine_promptrr prompt_indexc K|jd|}|||||j|j||||j|t |jddS)zACreate a generator for a single prompt using standard processing.-rrrr)rsrrHrrrrG)r6r<rrrrrequest_id_items r:_create_single_prompt_generatorz.EmbeddingMixin._create_single_prompt_generatorBs!^<fr|d kry||z }tA|}|j|}d|vr|d|dcS|d}t!| d||d d}| !||d|cS||vr0| !tt ||$|d|cSttDtFt z| |_$dS#tJ$r,}|tM|cYd}~Sd}~wwxYw)zCollect and aggregate batch results with support for chunked processing. For chunked requests, performs online aggregation to minimize memory usage. For regular requests, collects results normally. Nr)r<zResult generator not availablerrrr) weighted_sum total_weight chunk_countrsz9Expected PoolingRequestOutput for chunked embedding, got rmrfzUnsupported output type: )dtypez6prompt_token_ids cannot be None for chunked processingrrr)rmrpzChunked prompt z does not contain token IDsrsT)rsrpoutputsnum_cached_tokensfinishedz&Failed to aggregate chunks for prompt zResult not found for prompt )'rrrMrVr^rGr._collect_batchrrssplitintrerR IndexErrorrIr type__name__hasattrrrmrftorchTensortensorfloat32rprrtorangerrrqrr!rrrW)r6r<rprompt_aggregatorsshort_prompts_results result_idxresultpartsr aggregatorembedding_dataweightweighted_embeddingr num_promptsrrfinal_embeddingpooling_output_dataoriginal_promptoriginal_token_idspooling_request_outputrYr9s r:rzEmbeddingMixin._collect_batchsn(#..c 6!)112PQQQ==ckJJK ="WW333<<<<<<<<<#+112RSSS =? EG !,/,@N N N N N N N (j& 111"-33C88E0%(u{{8/D/Dq/H)I%J%J & 3000%/ 0 ");;;,0,-+,*0*;*A*A)*L*LQ*O ::*:6"4J!?J &f.BCC#997#F||477  v~v66 )/)< ==*0)A#99WV^8L8L8UWW  &nelCC).*%-***.6#99T  !!899F)7):):):)O)ORX)X&!.195G >22#>2226HH222~...&8...}---2----#-33C88E0%(r^^ & 3000%/ 09=,f99)*55Y-AbTVOc011K#K003 3  !333!3J!?J#-n#=L#-n#=L%0&|U\BB1&|c5\BB1)1,,+7*E/<.Q.Q.Q++.*S?LSLC S6ASA S-S S;!S60S;6S;)!r __module__ __qualname__r/r rrrZrrrQrr rrrr2r^rrrr rrWrrr"rr!rrr __classcell__r9s@r:r+r+8s      6 6  666X6<  ,F  <     H* H* X % 5H*H*H*XH*T/c////     + "+9+  + n1478 9++++ZIG9IG IG  IGIGIGIGIGIGV " $ &  sCx(4/      (<rs 33333333########## &&&&&&44444444------GGGGGG111111 GFFFFF322222))))))###### .-----888888222222 X  ^6^6^6^6^6]^6^6^6BQ.Q.Q.Q.Q.^Q.Q.Q.Q.Q.r;