&`i C.ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z mZmZddlmZddlmZddlmZddlmZmZmZdd lmZe ed ejeZej d Z!Gd d eZ"GddZ#dS)N) defaultdict) AsyncIterable AwaitableCallableDictListOptionalTuple)ActorIDNodeIDWorkerID) BaseModel)ActorTableData)JOB_LOGS_PATH_TEMPLATE)DEFAULT_RPC_TIMEOUT GetLogOptionsprotobuf_to_task_state_dict)StateDataSourceClientz3Please install pydantic via `pip install pydantic`.z0.*worker-([0-9a-f]+)-([0-9a-f]+)-(\d+).(out|err)cPeZdZUeed<eed<eeed<eeed<dS)ResolvedStreamFileInfonode_idfilename start_offset end_offsetN)__name__ __module__ __qualname__str__annotations__r inty/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/dashboard/modules/log/log_manager.pyrrsJ LLLMMM3- r"rceZdZdefdZedefdZdeedeefdZ d#ded e d ede ee effd Z d ed eegeeefdeefdZdedeeeffdZdedeedee ded e deef dZded eegeeefded e fdZdede ded e fdZdddddddeddd deedeedeedeedee deed eeegeeefd e dedeedefd Zd!e ede ee effd"ZdS)$ LogsManagerdata_source_clientc||_dSNclient)selfr&s r#__init__zLogsManager.__init__+s ( r"returnc|jSr(r))r+s r#r&zLogsManager.data_source_client.s {r"node_ipcFK|j|d{VS)zResolve the node id in hex from a given node ip. Args: node_ip: The node ip. Returns: node_id if there's a node id that matches the given node ip and is alive. None otherwise. N)r* ip_to_node_id)r+r/s r#r1zLogsManager.ip_to_node_id2s0[..w777777777r"*rtimeout glob_filtercK|j|||d{V}||jS)aReturn a list of log files on a given node id filtered by the glob. Args: node_id: The node id where log files present. timeout: The timeout of the API. glob_filter: The glob filter to filter out log files. Returns: Dictionary of {component_name -> list of log files} Raises: ValueError: If a source is unresponsive. r3N)r* list_logs_categorize_log_files log_files)r+rr3r4replys r#r7zLogsManager.list_logs>sL k++G['+RRRRRRRR))%/:::r"options get_actor_fnc K|j}| ||jd{V}|||j|j|j|j|j||j |j |j  d{V}|j dk}|j |j|j||j|j|s|j nd|j|jd{V}|23d{V}|jWV6dS)zGenerate a stream of logs in bytes. Args: options: The option for streaming logs. Return: Async generator of streamed logs in bytes. N r log_filenameactor_idtask_idattempt_numberpidr<r3suffix submission_idstream)r log_file_name keep_alivelinesintervalr3rr)rr1r/resolve_filenamerr@rArBrCr3rDrE media_typer* stream_logrIrJrrdata)r+r;r<rresrHrF streamed_logs r# stream_logszLogsManager.stream_logsQsi/ ? ..w????????G)) )%O"1 %O>!/*         '83 {--K,!-%,6?GOO4)~.         #) $ $ $ $ $ $ $,# # # # # ##)&&sC' sub_job_idc,K|jtd{V}d}|D]}|j|kr|}n| td|ddS|j}|td|dtj |}||fS) zReturn the log file name and node id for a given job submission id. Args: sub_job_id: The job submission id. Returns: The log file name and node id. r6NzSubmission job ID not found.)NNzJob zH has no driver node id info. This is likely a bug. Please file an issue.)rE) r* get_job_inforrEloggerinfodriver_node_id ValueErrorrformat)r+rR job_infos target_jobjob_inforr?s r#_resolve_job_filenamez!LogsManager._resolve_job_filenames+22;N2OOOOOOOO  !  H%33% 4   KKDZDDD E E E:) ?>z>>>  .4:NNN  $$r" node_id_hex worker_id_hexrCrDcK||td|d|d|%|||d|d|d{V}n$|||d|d|d{V}g|d|dD]}|8t|d }||kr|cS>>>?? ?&-7 .(... -.. #+3 .... ((!66 #--// 7        {{}}l**r"rArBc Kd}d}d}d}|dvrtd|d|jdd|fg|d{V} t| jdkrt d |d |d |d} | jD]} | j|kr| } n| t d |d |dt| } | dd} | dd}| dd}| dd}|t d|d |d||t d|d|d|t d |d |d| d|d| |dkrdnd}||d}|t d|d|d ||d!|d"d}|d!|d#d}||||fS)$aResolve log file for a task. Args: task_id: The task id. attempt_number: The attempt number. suffix: The suffix of the log file, e.g. out or err. timeout: Timeout in seconds. Returns: The log file name, node id, the start and end offsets of the corresponding task log in the file. Raises: FileNotFoundError: If the log file is not found. ValueError: If the suffix is not out or err. NouterrSuffix z is not supported.rA=)filtersr3rz"Could not find log file for task: z (attempt z) with suffix: z)Could not find log file for task attempt:()ror task_log_infor@z)Could not find log file for task attempt.z) due to missing node info.z1For actor task, please query actor log for actor(z): e.g. ray logs actor --id z . Or set RAY_ENABLE_RECORD_ACTOR_TASK_LOGGING=1 in actor's runtime env or when starting the cluster. Recording actor task's log could be expensive, so Ray turns it off by default.z).Worker id = z , node id = z ,log_info = rw stdout_file stderr_filezMissing log filename info in z for task z ,attempt std_start_end) rYr*get_all_task_infolenevents_by_taskFileNotFoundErrorrBrget)r+rArBrDr3r?rrrr: task_eventttaskrolog_infor@ filename_keys r#_resolve_task_filenamez"LogsManager._resolve_task_filenamesT&    ' 'AvAAABB Bk33g./4         u# $ $ ) )#EWEE+EE11 2  #///+///  +:66HH[$// ((9d++88OT2288J-- ?#III+III    4$=!==?G===  #)))+))())6=))')) )/%}}] ||L$77  #,,,G,,),,   ||$8&$8$8$8$?? \\"4"4"4"4d;;  lJ>>r"Nrwr>r?rEc ZKd} d} | dvrtd| d|r4|tj||| |d{V\}}n}|r$|||| |d{V\}}} } nW| r|| d{V\}}n6|r4|td|||d|| |d{V}|(td|d|d |d |d |d | d | d|dt||| | } t d| | S)aReturn the file name given all options. Args: node_id: The node's id from which logs are resolved. log_filename: Filename of the log file. actor_id: Id of the actor that generates the log file. task_id: Id of the task that generates the log file. pid: Id of the worker process that generates the log file. get_actor_fn: Callback to get the actor's data by id. timeout: Timeout for the gRPC to listing logs on the node specified by `node_id`. suffix: Log suffix if no `log_filename` is provided, when resolving by other ids'. Default to "out". submission_id: The submission id for a submission job. Nrvryz is not supported. z=Node id needs to be specified for resolving filenames of pid rmz^Could not find a log file. Please make sure the given option exists in the cluster. node_id: z filename: z actor_id: z task_id: z pid: z suffix: z submission_id: z attempt_number:  )rrrrzResolved log file: ) rYrtr from_hexrr^rkrrrVrW)r+rr?r@rArBrCr<r3rDrErrrOs r#rKzLogsManager.resolve_filenameVsv>   ' 'BvBBBCC C  *.*F*F **L&'++%%%%%% !G\\  11    *.*D*D]*S*S$S$S$S$S$S$S !G\\   /),//"&!:!:#" ";""L  # 8% 8 8 , 8 8 ( 8 8 & 8 8  8 8$ 8 8%2 8 8&4 8 8 8   %!%!      /#//000 r"r9ctt}|D]}d|vr1|dr|d|8d|vr1|dr|d|md|vr1|dr|d|d |vr1|dr|d |d |vr|d |d |vr|d|d|vr|d|9d|vr|d|Zd|vr|d|{d|vr|d||d||S)zCategorize the given log files after filterieng them out using a given glob. Returns: Dictionary of {component_name -> list of log files} workerz.outrbz.errrcz core-workerz.log core_workerz core-driverdriverzraylet.rayletz gcs_server. gcs_server log_monitorinternalmonitor autoscalerzagent.agentz dashboard. dashboard)rlistendswithappend)r+r9resultlog_files r#r8z!LogsManager._categorize_log_filess T""! 4 4H8##):):6)B)B#|$++H5555X%%8+<+A;;;%(;7:; c49n ;;;;&.$.$y)H^4L*MMN.$ u  .$.$.$.$`%c%eCHo%%%%:%% }%c] %  %  % #%%%%N3+3+y)H^4L*MMN3+ 3+  3+3+3+3+jZ?Z?,/Z?9<Z?GJZ?Z?Z?Z?~"&&*"&!%(,! *'+[[[#[sm [ 3- [ # [! [c][ gY (>*B CC D [[[ }[ [[[[ztCyT#tCy.=Qr"r%)$loggingre collectionsrtypingrrrrrr r rayr r r ray._common.pydantic_compatrray.core.generated.gcs_pb2r ray.dashboard.modules.job.commonrray.util.state.commonrrrray.util.state.state_managerrModuleNotFoundError getLoggerrrVcompilerfrr%r!r"r#rs ######RRRRRRRRRRRRRRRRRR))))))))))111111555555CCCCCC ?>>>>>   S T TT  8 $ $RZ STT     Y    hhhhhhhhhhr"