&`i0ddlZddlZddlZddlZddlmZmZddlZddlm cm Z ddl m Z ddlmZmZmZddlmZmZmZ ejeZdeeeefejefdeeeffdZGd d ZdS) N)OptionalUnion)aiohttp)SubprocessModuleSubprocessModuleConfig run_module) ResponseTypeget_http_session_to_modulemodule_logging_filenameheadersreturnchdt|tjrt|}fd|D}|S)z> Filter out hop-by-hop headers from the headers dict. > keep-alivetransfer-encodingproxy-authenticateproxy-authorizationteupgradetrailers connectioncHi|]\}}|v||S)lower).0keyvalueHOP_BY_HOP_HEADERSs u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/dashboard/subprocesses/handle.py z-filter_hop_by_hop_headers..0s= C 99;;0 0 0 U 0 0 0) isinstance multidictCIMultiDictProxydictitems)r filtered_headersrs @rfilter_hop_by_hop_headersr'sp   '9566 w--!--// r ceZdZdZejdZdejde e de fdZ de dee fd Zd Zd Zd Zd ZdejjfdZdZdZejfdejjdedejjfdZdejjdejjfdZdejjdejjfdZ dejjdejjfdZ!dS)SubprocessModuleHandlea_ A handle to a module created as a subprocess. Can send messages to the module and receive responses. It only acts as a proxy to the aiohttp server running in the subprocess. On destruction, the subprocess is terminated. Lifecycle: 1. In SubprocessModuleHandle creation, the subprocess is started and runs an aiohttp server. 2. User must call start_module() and wait_for_module_ready() first. 3. SubprocessRouteTable.bind(handle) 4. app.add_routes(routes=SubprocessRouteTable.bound_routes()) 5. Run the app. Health check (_do_periodic_health_check): Every 1s, do a health check by _do_once_health_check. If the module is unhealthy: 1. log the exception 2. log the last N lines of the log file 3. fail all active requests 4. restart the module TODO(ryw): define policy for health check: - check period (Now: 1s) - define unhealthy. (Now: process exits. TODO: check_health() for event loop hang) - check number of failures in a row before we deem it unhealthy (Now: N/A) - "max number of restarts"? (Now: infinite) spawnloop module_clsconfigcv||_||_||_d|_d|_d|_d|_d|_dS)Nr)r+r,r- incarnation parent_connprocesshttp_client_sessionhealth_check_task)selfr+r,r-s r__init__zSubprocessModuleHandle.__init__YsJ  $   DH !%r r/pidc.d|jjd|d|dS)Nz"SubprocessModuleHandle(module_cls=z, incarnation=z, pid=))r,__name__)r4r/r6s r str_for_statez$SubprocessModuleHandle.str_for_statels*uDO4Luu\guuoruuuur c^||j|jr |jjndS)N)r:r/r1r6r4s r__str__zSubprocessModuleHandle.__str__os3!!  $,Hdl..D   r c|j\|_}tj|jjstj|jj|j t|j |j|j |fd|j j d|j |_|j|dS)z; Start the module. Should be non-blocking. T-)targetargsdaemonnameN) mp_contextPiper0ospathexistsr- socket_dirmakedirsProcessrr,r/r9r1startclose)r4 child_conns r start_modulez#SubprocessModuleHandle.start_modulets(,';';'='=$*w~~dk455 0 K . / / /..    O,AAt/?AA/     r c>|jtjrg |jn+#t $rt d|jjdwxYw|j d|_n*t d|jjdtjd|jj}t||j j |j j |_|j||_dS)zs Wait for the module to be ready. This is called after start_module() and can be blocking. Module z) failed to start. Received EOF from pipe.Nz failed to start. Timeout after z seconds.)r0polldashboard_consts$SUBPROCESS_MODULE_WAIT_READY_TIMEOUTrecvEOFError RuntimeErrorr,r9rMr r-rI session_namer2r+ create_task_do_periodic_health_checkr3)r4 module_names rwait_for_module_readyz,SubprocessModuleHandle.wait_for_module_readysF   !1!V W W   %%''''   ".do6...    " " $ $ $#D  b$/2bb!1!Vbbb  o. #= /1I$ $  "&!6!6t7U7U7W7W!X!Xs A(A(cK|xjdz c_|jr |jd|_|jr9|j|jd|_|jr&|jd{Vd|_|jr"|jd|_dSdS)zR Destroy the module. This is called when the module is unhealthy. N) r/r0rMr1killjoinr2r3cancelr<s rdestroy_modulez%SubprocessModuleHandle.destroy_modules A   $   " " $ $ $#D  < L      L     DL  # ,*0022 2 2 2 2 2 2 2'+D $  ! *  " ) ) + + +%)D " " " * *r r cK|jdd{V}tj|jt |j|d{VS)aT Do internal health check. The module should respond immediately with a 200 OK. This can be used to measure module responsiveness in RTT, it also indicates subprocess event loop lag. Currently you get a 200 OK with body = b'success'. Later if we want we can add more observability payloads. zhttp://localhost/api/healthzNstatusr body) r2getrwebResponserer'r readr4resps r _health_checkz$SubprocessModuleHandle._health_checks-112PQQQQQQQQ{##;-dl;;yy{{""""""$   r cK|jjtd|jj|d{V}|jdkrtd|jdS)z Do a health check once. We check for: 1. if the process exits, it's considered died. 2. if the health check endpoint returns non-200, it's considered unhealthy. NzProcess exited with code z$Health check failed: status code is )r1exitcoderWrmrerks r_do_once_health_checkz,SubprocessModuleHandle._do_once_health_checks < ,R4<;PRRSS S'')))))))) ;#  SdkSSTT T  r c K |d{Vn#t$rt|jj|jj}td|jjd|jj d|d| d{V| | YdSwxYwtjdd{V)z Every 1s, do a health check. If the module is unhealthy: 1. log the exception 2. log the last N lines of the log file 3. restart the module TNrQz is unhealthy. Please refer to /z/ for more details. Failing all active requests.r^)rq Exceptionr r,r9r-logging_filenamelogger exceptionlog_dirrbrOr\asynciosleep)r4filenames rrZz0SubprocessModuleHandle._do_periodic_health_checks> # 002222222222   2O,dk.J  'do6''{*''-5''' ))+++++++++!!###**,,, -"" " " " " " " "! #s B(C  C request resp_typec,K|tjkr||d{VS|tjkr||d{VS|tjkr||d{VStd|)zQ Sends a new request to the subprocess and returns the response. NzUnknown response type: )r HTTP proxy_httpSTREAM proxy_stream WEBSOCKETproxy_websocket ValueError)r4r|r}s r proxy_requestz$SubprocessModuleHandle.proxy_requests  ) ) )11111111 1  + + +**733333333 3  . . .--g66666666 6>9>>???r cKd|j}|d{V}|j|j||t |jd4d{V }|d{V}tj |j t |j|cdddd{VS#1d{VswxYwYdS)z Proxy handler for non-streaming HTTP API It forwards the method, query string, headers, and body to the backend. http://localhostNF)datar allow_redirectsrd) path_qsrjr2r|methodr'r rrhrire)r4r|urlrf backend_resp resp_bodys rrz!SubprocessModuleHandle.proxy_https 322\\^^######+33 N -go>>! 4          *//11111111I;''#*1,2FGG(                              s$AC  CCcpKd|j}|d{V}|j|j||t |j4d{V }tj |j t |j}| |d{V|j 23d{V\}}||d{V&6|d{V|cdddd{VS#1d{VswxYwYdS)z~ Proxy handler for streaming HTTP API. It forwards the method, query string, and body to the backend. rN)rr )rer )rrjr2r|rr'r rrhStreamResponserepreparecontent iter_chunkswrite write_eof)r4r|rrfr proxy_respchunk_s rrz#SubprocessModuleHandle.proxy_streams322\\^^######+33 N -go>> 4            33#*1,2FGG4J$$W-- - - - - - - -"."6"B"B"D"D . . . . . . .heQ &&u----------#E&&(( ( ( ( ( ( ( (                              s#A-D%C6!>!@!@$,,W555555555!.FFFFFFF#x7#4#999,55ch??????????W%6%===,77AAAAAAAAAA %D#(%D%DEEEE "/%**,,,,,,,,,% & & & & & & & & & & & & & & & & & & & & & & & & & & & & & &- 2 2 2 NNBaBB C C C11111111 1 1 1 1 1 1 V V V LL<477<< = = =+55=T5UU U Vs\4E!rs """"""""///////////////    8 $ $ 4S>9#=c#BB C #s(^4]V]V]V]V]V]V]V]V]V]Vr