-`i BddlZddlZddlZddlmZddlmZddlZddlm Z m Z m Z ddl m Z ddlmZddlmZmZddlmZdd lmZdd lmZdd lmZmZeeZ dd e dejdzdedefdZdej defdZ!dej defdZ"d e dej ddfdZ#dS)N) HTTPStatus)Any)FastAPIRequestResponse)envs) EngineClient)H11_MAX_HEADER_COUNT_DEFAULT%H11_MAX_INCOMPLETE_EVENT_SIZE_DEFAULT)SSLCertRefresher) init_logger)find_process_using_port)EngineDeadErrorEngineGenerateErrorFappsockenable_ssl_refreshuvicorn_kwargsc Ktd|jD]X}t|dd}t|dd}||)td|d|Y|dd}|dd}|t }|t}tj |fi|} || _ || _ | tj | } t|| tj} | t%| |jj| | |r|gnd |sdn&t-| j| j| j| j dfd } dd } | t8j| | t8j|  d{V| S#tj $r|d}tC|}|Bt"d||d|#td| $cYSwxYw#wxYw)z Start a FastAPI app using Uvicorn, with support for custom Uvicorn config options. Supports http header limits via h11_max_incomplete_event_size and h11_max_header_count. zAvailable routes are:methodsNpathzRoute: %s, Methods: %sz, h11_max_incomplete_event_sizeh11_max_header_count)sockets) ssl_contextkey_path cert_pathca_pathreturncrdSdSN)cancelstop) server_taskssl_cert_refresher watchdog_tasksm/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/entrypoints/launcher.pysignal_handlerz"serve_http..signal_handlerTsQ  &  # # % % % % % & &c KdSr!r+r)r'dummy_shutdownz"serve_http..dummy_shutdown[s  r)portz7port %s is used by process %s launched with command: %s z"Shutting down FastAPI HTTP server.)rN)%loggerinforoutesgetattrjoinpopr r uvicornConfigrrloadServer_add_shutdown_handlersasyncioget_running_loop create_task watchdog_loopstate engine_clientserver ssl ssl_keyfile ssl_certfile ssl_ca_certsadd_signal_handlersignalSIGINTSIGTERMr"CancelledErrorrwarningcmdlineshutdown)rrrrrouterrrrconfigserverloopr(r,r-processr$r%r&s @@@r' serve_httprRs: KK'(((HH%D11ufd++ ?dl  ,dDIIg4F4FGGGG%3$6$6'%%!*--.DdKK%,(M%#; ^C 2 2> 2 2F+HF("6F KKMMM ^F # #F3'''  # % %D$$]639;R%S%STTM""6<<$8PD<#Q#QRRK"   ')'    &&&&&&&&     FM>:::FNN;;;~   ! ! ! !f%)$//   NNJ**++      8999     ! s%<H""BK:KKKK*rOenginecbKd} tj|d{Vt||+)z # Watchdog task that runs in the background, checking # for error state in the engine. Needed to trigger shutdown # if an exception arises is StreamingResponse() generator. g@TN)r:sleepterminate_if_errored)rOrSVLLM_WATCHDOG_TIME_Ss r'r=r=tsK -m0111111111VV,,,-r)cV|jo|j }tjs |r d|_dSdSdS)a See discussions here on shutting down a uvicorn server https://github.com/encode/uvicorn/discussions/1103 In this case we cannot await the server shutdown here because handler must first return to close the connection for this request. TN)errored is_runningrVLLM_KEEP_ALIVE_ON_ENGINE_DEATH should_exit)rOrSengine_erroreds r'rVrVsJ^=F,=(=N  /"N"!""""r)rc|t|t|tdtffd }dS)a) VLLM V1 AsyncLLM catches exceptions and returns only two types: EngineGenerateError and EngineDeadError. EngineGenerateError is raised by the per request generate() method. This error could be request specific (and therefore recoverable - e.g. if there is an error in input processing). EngineDeadError is raised by the background output_handler method. This error is global and therefore not recoverable. We register these @app.exception_handlers to return nice responses to the end user if they occur and shut down if needed. See https://fastapi.tiangolo.com/tutorial/handling-errors/ for more details on how exception handlers work. If an exception is encountered in a StreamingResponse generator, the exception is not raised, since we already sent a 200 status. Rather, we send an error message as the next chunk. Since the exception is not raised, this means that the server will not automatically shut down. Instead, we use the watchdog background task for check for errored state. requestc|Kt|jjjt t jS)N)rOrS) status_code)rVrr>r?rrINTERNAL_SERVER_ERROR)r___rOs r'runtime_exception_handlerz9_add_shutdown_handlers..runtime_exception_handlersD ;$2    J$DEEEEr)N)exception_handler RuntimeErrorrrr)rrOrds ` r'r9r9s2 <((?++.//FFFFFF0/,+)(FFFr))F)$r:rFsockethttprtypingrr5fastapirrrvllmrvllm.engine.protocolr vllm.entrypoints.constantsr r vllm.entrypoints.sslr vllm.loggerr vllm.utils.network_utilsrvllm.v1.engine.exceptionsrr__name__r/boolrRr8r=rVr9r+r)r'rts ..........------211111######<<<<<<JJJJJJJJ X   %VV V -$ VV VVVVr - - - - - - " " " " " ""F"F"FD"F"F"F"F"F"Fr)