&`i"ddlZddlZddlmZddlmZmZddlmZmZm Z m Z ddl Z ddl Z ddl mcmZddlmZddlmZddlmZddlmZmZmZmZmZmZdd lmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)ej*e+Z,eGd d eZ-d e.de e.e#e&ffdZ/dee.fdZ0 dde ee$dee.fdZ1dede%de2de.fdZ3e-j4dddeede e%de-de2de.f dZ5dede$de.fdZ6dede.fdZ7e-j8dde e%de.de%de-de.f d Z9e-j4dddee%de%de-de2de.f d!Z:de-de2fd"Z; e j<d#e d$e %Z=e j<d&dd'%Z>e j?e j@de jAe1e$jBe$jCg()e j@de.d*e>e=ed+,de.de.d-e e.d.eDfd/ZEe j?e j@de jAe1)e j<d0d1e jAe02e j<d3d4d5d67e j<d8eeFd9:e j<d;ded+,de.de.d ee.d>eFde2d.eDd-e.fd?ZGe jHd@e jIed+,dAZJeJ?dBCe=e>e jIed+,d.eDd-e.fdDZKeJ?dECe=e>e jIed+,d.eDd-e.fdFZLeJ?dGCe=e>e jIed+,d.eDd-e.fdHZMe j<dId3de2d6dJKZNe j<dLdeFedMNZOe j<dOdeDddPd6QZPe j<d#e d$e dR%ZQe j<dSdTde.ddUNZRe j<dVdWde.ddXNZSe j<dYd6ddZ[ZTe j<d\dd]d^_ZUe j<d`ddadb_ZVdd-e e.fdcZWdddddddee dddd]dadddfd-e e.dee e.dfe e.dge e.dhe e.die eFdje2dkeFd.eFdle eDdme.dne.doe.dpe e.dqeFdre e.f dsZXdtZYGdudve jZZ[e[eYwZ\e\?dxCe j@dyddz{e>eSeReNeOePeQeUeVe jIed+,dye.d-e e.dee e.dfe e.dje2dkeFdleDd.eFdne.doe.fd|Z]e\?d}Ce j<d~dde.ddNe j<ddde.ddNe>eSeReNeOePeQeTe jIed+,de e.die e.d-e e.dee e.dfe e.dje2dkeFdleDd.eFde2fdZ^e\?dCe j<ddd6e.de>eSeReNeOePeQeTe jIed+,die e.d-e e.dee e.dfe e.dje2dkeFdleDd.eFde2fdZ_e\?dCe j<d~drd6e.de>eNeOePeQe jIed+,dre e.d-e e.dje2dkeFdleDd.eFf dZ`e\?dCe j<d~dpd6e.de j<dddeFddNe>eNePeOeQeTe jIed+,dpe e.dqeFd-e e.dje2dleDdkeFd.eFde2fdZadS)N)datetime)Enumunique)DictListOptionalTuple) parse_address)tabulate) PublicAPI)StateApiClientget_log list_logssummarize_actorssummarize_objectssummarize_tasks) DEFAULT_LIMITDEFAULT_LOG_LIMITDEFAULT_RPC_TIMEOUT GetApiOptionsListApiOptions PredicateType StateResource StateSchemaSupportedFilterTyperesource_to_schema)RayStateApiExceptionceZdZdZdZdZdZdS)AvailableFormatdefaultjsonyamltableN)__name__ __module__ __qualname__DEFAULTJSONYAMLTABLEl/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/util/state/state_cli.pyrr's"G D D EEEr,rfilterreturnc *d}d}tt|D]T}||}|dkr d}||dzf}n:|dkr3t||dzkr8||dz}|dkr d}||dzf}nU|r|std|d|d|d ||d |d||dd}}}|dks|dksJt|d kst|d krtd|d |d |d |||fS) z?Parse the filter string to a tuple of key, preciate, and value.N=!z!=zThe format of a given filter zY is invalid: Cannot find the predicate. Please provide key=val or key!=val format string.rz! is invalid: Cannot identify key z or value, z3. Please provide key=val or key!=val format string.)rangelen ValueError)r. predicatepredicate_indexichar next_charkeyvalues r- _parse_filterr?/sIO3v;;  ay 3;;I !a%jO E S[[6{{a!e##q1u IC #$a!e*  O  @F @ @ @    ###$q!OA$667q!##$C   yD000 0 3xx1}}E a @F @ @#& @ @38 @ @ @   E ""r,c$dtDS)z0Return the available formats in a list of stringcg|] }|j Sr+)r>).0 format_enums r- z*_get_available_formats..bs A A A+K  A A Ar,)rr+r,r-_get_available_formatsrE`s A A A A AAr,excludedc*fdtDS)z~Return the available resources in a list of string Args: excluded: List of resources that should be excluded cRg|]#}|v|jdd$S)N_-)r>replace)rBerFs r-rDz,_get_available_resources..nsC     q00 S!!000r,)rrFs`r-_get_available_resourcesrNes.       r, state_dataschemadetailc N tj}dd|dzdz}g}g}||}|D] D]3\}} t | t rt j| d |<4t } g}|D]-} | | vr'| | .| fd|Dd|d t|d t||d d d dS)aDisplay the table output. The table headers are ordered as the order defined in the dataclass of `StateSchema`. For example, @dataclass class A(StateSchema): a: str b: str c: str will create headers A B C ----- Args: state_data: A list of state data. schema: The schema for the corresponding resource. Returns: The table formatted string. ========z List:  )rQr4indentcDg|]}|Sr+lower)rBheaderdatas r-rDz$get_table_output..s&AAAvd6<<>>*AAAr, z. Stats: ------------------------------ Total: z( Table: ------------------------------ Tplainz.3f)headers showindextablefmtfloatfmt)rnow list_columnsitems isinstancedictr"dumpsetkeysappendupperr6r ) rOrPrQtimerZr^r#colsr=valricolr[s @r-get_table_outputrpusl. <>>D (((( (7 2FG E   f  - -D C C  5 5HC#t$$ 5 Ic!444S 499;; , ,Cd{{syy{{+++ AAAAAAABBBB    J   %D7USSS    r,F)formatrQrqcrfd|D}|tjkrt||S|tjkrt j|ddddS|tjkrtj|S|tj krt||Std|dt)Nc:g|]}|Sr+)humanify)rBstaterPs r-rDz&output_with_format..s%EEEfooe,,EEEr,TF)rVexplicit_start sort_keys explicit_endzUnexpected format: z. Supported formatting: ) rr'rpr)r"rgr(r!dumpsr*r7rErOrPrqrQs ` r-output_with_formatr|sFEEEE*EEE  ((( FF;;; %%%y       ?' ' 'z*%%% ?( ( ( FF;;; @& @ @%;%=%= @ @   r,resourcec  t|dkrdS|d}|d}|d}|d=|d=tj|d}g}g}|D] D]3\}} t | t rtj| d |<4td D}| fd |Dt||d d d } tj } dd|j d| dzdz} d| d|d|d| d S)NrNo resource in the clusterclustersummary summary_byr4rUc6g|]}|Sr+rkrBr=s r-rDz)format_summary_output.. @@@##))++@@@r,cDg|]}|Sr+rXrBrZrs r-rDz)format_summary_output..&DDD&gfllnn-DDDr,Tr]left)r^r_r`numalignrSrTz Summary: r\- Stats: ------------------------------------  Table (group by z(): ------------------------------------ )r6r"rgvaluesrdrerfsortedrirjr rrbr> capitalize)rOr} cluster_data summariesrcluster_info_tabler#r^r=rn summary_tablerlrZrs @r-format_summary_outputrs :!++i(LY'Il+J\"Y<::: EG##%%FF  8 8HC#t$$ 8#yQ777 @@@@@AA DDDDGDDDEEEE w$6M <>>D I8>4466II$III IG SF            r,c t|dkrdS|d}|d}|d}|d=|d=tj|d}g}|D]\}D]3\}}t |t rtj|d|<4g} t dD} | fd | Dt| | d d } | d d} || d| tj } dd| dzdz}d |}d|d|d|d|d S)Nrrrrrr4rUc6g|]}|Sr+rrs r-rDz0format_object_summary_output..rr,cDg|]}|Sr+rXrs r-rDz0format_object_summary_output.. rr,Tr)r^r_r|z |r\rSz Object Summary: rTz rrz') ------------------------------------ )r6r"rgrdrerfrrirjr rKrrbjoin)rOrrrrtablescallsiter=rnr#r^table_for_callsiteformatted_callsiterlrZ table_stringrs @r-format_object_summary_outputrs :!++i(LY'Il+J\"Y<:::F&__..EE'  8 8HC#t$$ 8#yQ777 @@@@@AA DDDDGDDDEEE% 7dV    &--c599 +CC/ACCDDDD <>>D 24222 2W .- 999U%,,..999r,TrPrqrQ)relistr6r|rOrrPrqs r-format_get_api_outputr!s BJ55B#j//Q:N:NA2AAAA j$ ' '" \ 99j999J jt T T TTr,cjt|dkrdSd|D}t||||S)Nrrc6g|]}|Sr+rrs r-rDz*format_list_api_output..;rr,r)r6r|r{s r-format_list_api_outputr2sE :!++99j999J jv V V VVr,cB|tjkp|tjkSN)rr'r*rs r-_should_explainr?s _, , O/:O0OOr,z --timeoutz4Timeout in seconds for the API requests. Default is )r helpz --addresszqThe address of Ray API server. If not provided, it will be configured automatically from querying the GCS server.rM)type)rrequiredstable) stabilityaddresstimeoutc 4|stjd|dt|dd}td|dt |}t|} ||||ttj  }n3#t$r&}tj t|d }~wwxYwtt!||t#|tj d S) aGet a state of a given resource by ID. We currently DO NOT support get by id for jobs and runtime-envs The output schema is defined at :ref:`State API Schema section. ` For example, the output schema of `ray get tasks ` is :class:`~ray.util.state.common.TaskState`. Usage: Get an actor with actor id ``` ray get actors ``` Get a placement group information with ``` ray get placement-groups ``` The API queries one or more components from the cluster to obtain the data. The returned state snapshot could be stale, and it is not guaranteed to return the live data. Args: resource: The type of the resource to query. id: The id of the resource. Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. z-Missing argument 'ID'. Do you mean 'ray list z'?rJrIz*Create StateApiClient to ray instance at: z...r)r)r}roptions_explainNr)click BadParameterrrKloggerdebugr rgetrrr)r UsageErrorstrprintrr)r}rrrclientrr[rLs r-ray_getrWsDv   HH H H H   X--c37788H LLJgJJJKKK G , , ,FG,,,G'zz$_%9::    '''s1vv&&&' %h//"'    s=0B.. C8!CCz--formatr )r rz-fz--filteraHA key, predicate, and value to filter the result. E.g., --filter 'key=value' or --filter 'key!=value'. You can specify multiple --filter options. In this case all predicates are concatenated as AND. For example, --filter key=value --filter key2=value means (key==val) AND (key2==val2), String filter values are case-insensitive.T)rmultiplez--limitz4Maximum number of entries to return. 100 by default.)r rrz--detailzIf the flag is set, the output will contain data in more details. Note that the API could query more sources to obtain information in a greater detail.)ris_flagr limitct|dd}t|}t|}d|D}t ||||} |||dt |} n3#t$r&} tj t| d} ~ wwxYw|r|tj kr tj }tt| t||| dS) a7List all states of a given resource. Normally, summary APIs are recommended before listing all resources. The output schema is defined at :ref:`State API Schema section. ` For example, the output schema of `ray list tasks` is :class:`~ray.util.state.common.TaskState`. Usage: List all actor information from the cluster. ``` ray list actors ``` List 50 actors from the cluster. The sorting order cannot be controlled. ``` ray list actors --limit 50 ``` List 10 actors with state PENDING. ``` ray list actors --limit 10 --filter "state=PENDING" ``` List actors with yaml format. ``` ray list actors --format yaml ``` List actors with details. When --detail is specified, it might query more data sources to obtain data in details. ``` ray list actors --detail ``` The API queries one or more components from the cluster to obtain the data. The returned state snapshot could be stale, and it is not guaranteed to return the live data. The API can return partial or missing output upon the following scenarios. - When the API queries more than 1 component, if some of them fail, the API will return the partial result (with a suppressible warning). - When the API returns too many entries, the API will truncate the output. Currently, truncated data cannot be selected by users. Args: resource: The type of the resource to query. Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. Changes: - changed in version 2.7: --filter values are case-insensitive. rJrIrc,g|]}t|Sr+)r?)rBfs r-rDzray_list...s / / /1mA / / /r,)rrfiltersrQF)rraise_on_missing_outputrNr{)rrKrr rrrrrrrr'r)rrr) r}rqr.rrQrrrrr[rLs r-ray_listrsBfX--c37788H V $ $FG , , ,F / / / / /F G'{{ $)$V,,    '''s1vv&&&'&&O333 % %h//    s#&B B:!B55B:rcdS)z6Return the summarized information of a given resource.Nr+)ctxs r-summary_state_cli_grouprQs  Dr,tasks)namec xttt||ddtjdS)ajSummarize the task state of the cluster. By default, the output contains the information grouped by task function names. The output schema is :class:`~ray.util.state.common.TaskSummaries`. Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. FTrrrrr}N)rrrrTASKSrrrs r- task_summaryrYsX$  (-     #(        r,actorsc xttt||ddtjdS)aSummarize the actor state of the cluster. By default, the output contains the information grouped by actor class names. The output schema is :class:`ray.util.state.common.ActorSummaries `. Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. FTrrN)rrrrACTORSrs r- actor_summaryrxsX&  (-     #)        r,objectsc `ttt||dddS)aPSummarize the object state of the cluster. The API is recommended when debugging memory leaks. See :ref:`Debugging with Ray Memory ` for more details. (Note that this command is almost equivalent to `ray memory`, but it returns easier-to-understand output). By default, the output contains the information grouped by object callsite. Note that the callsite is not collected and all data will be aggregated as "disable" callsite if the env var `RAY_record_ref_creation_sites` is not configured. To enable the callsite collection, set the following environment variable when starting Ray. Example: ``` RAY_record_ref_creation_sites=1 ray start --head ``` ``` RAY_record_ref_creation_sites=1 ray_script.py ``` The output schema is :class:`ray.util.state.common.ObjectSummaries `. Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. FTrN)rrrrs r-object_summaryrsLL $ (-           r,z--followz>Streams the log file as it is updated instead of just tailing.)rrrrz--tailzANumber of lines to tail from log. Use -1 to fetch the whole file.)rrr rz --intervalzDThe interval in secs to print new logs when `--follow` is specified.)rrr rhiddenz8. If --follow is specified, this option will be ignored.z-ipz --node-ipz#Filters the logs by this ip addressz --node-idz-idzFilters the logs by this NodeIDz--errzUIf supplied, querying stderr files for workers/actors, else defaults to stdout files.)rr rz --encodingzutf-8zwThe encoding use to decode the log file. Accepts any encoding supported by Python's `codecs` module. Defaults to utf-8.)rr rz--encoding-errorsstrictzThe error handling scheme to use for decoding errors. Accepts any error handling scheme supported by Python's `codecs`module. Defaults to strict.c tj|}t|dS#ttf$r&}t jt|d}~wwxYw)zGet the head node ip from the ray address if possible Args: address: ray cluster address, e.g. "auto", "localhost:6379" Raises: click.UsageError: if node ip could not be resolved rN)services%canonicalize_bootstrap_address_or_dier ConnectionErrorr7rrr)rrLs r-_get_head_node_ipr se'@IIW%%a(( Z ('''s1vv&&&'s(+A"!AA"outnode_idnode_ipfilenameactor_idpidfollowtailintervalsuffixencodingencoding_errorstask_idattempt_number submission_idc|dkrtd|d||t|}tdid|d|d|d|d |d |d |d |d | d|d| d| d| d| d|d|D]}t|dddS)zEWrapper around `get_log()` that prints the preamble and the log linesrz#--- Log has been truncated to last zP lines. Use `--tail` flag to toggle. Set to -1 for getting the entire file. --- Nrrrrrrrr _intervalrrrerrorsrrrT)endflushr+)rrr)rrrrrrrrrrrrrrrrchunks r- _print_logr1sO& axx  Y$ Y Y Y   7?#G,,    T  Cv(v&~ $m!))$ e4(((((%))r,uA Get logs based on filename (cluster) or resource identifiers (actor) Example: Get all the log files available on a node (ray address could be obtained from `ray start --head` or `ray.init()`). ``` ray logs cluster ``` [ray logs cluster] Print the last 500 lines of raylet.out on a head node. ``` ray logs cluster raylet.out --tail 500 ``` Or simply, using `ray logs` as an alias for `ray logs cluster`: ``` ray logs raylet.out --tail 500 ``` Print the last 500 lines of raylet.out on a worker node id A. ``` ray logs raylet.out --tail 500 —-node-id A ``` [ray logs actor] Follow the log file with an actor id ABC. ``` ray logs actor --id ABC --follow ``` [ray logs task] Get the std err generated by a task. Note: If a task is from a concurrent actor (i.e. an async actor or a threaded actor), the log of the tasks are expected to be interleaved. Please use `ray logs actor --id ` for the entire actor log. ``` ray logs task --id --err ``` ceZdZfdZxZS)LogCommandGroupcd|_t||}|\}}}|&t|dg|zS|||fS)zTry resolve the command line args assuming users omitted the subcommand. This overrides the default `resolve_command` for the parent class. This will allow command alias of `ray ` to `ray cluster `. TNr)resilient_parsingsuperresolve_command)selfrargsrescmd_namecmd parsed_args __class__s r-rzLogCommandGroup.resolve_commandse !%gg%%c400%("#{ ;77**3 d0BCC Ck))r,)r$r%r&r __classcell__)r s@r-rrs8 * * * * * * * * *r,r)rr glob_filter*)rr c ||t|}t|||||} g} | D]\} }|D]}| | t | dkrT|rt d|n|rt d|t t | dtjdS| d}t||||||||| |  dS) uGet/List logs that matches the GLOB_FILTER in the cluster. By default, it prints a list of log files that match the filter. By default, it prints the head node logs. If there's only 1 match, it will print the log file. Example: Print the last 500 lines of raylet.out on a head node. ``` ray logs [cluster] raylet.out --tail 500 ``` Print the last 500 lines of raylet.out on a worker node id A. ``` ray logs [cluster] raylet.out --tail 500 —-node-id A ``` Download the gcs_server.txt file to the local machine. ``` ray logs [cluster] gcs_server.out --tail -1 > gcs_server.txt ``` Follow the log files from the last 100 lines. ``` ray logs [cluster] raylet.out --tail 100 -f ``` Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. N)rrrrrr2z Node ID: z Node IP: )rPrqr) rrrrrrrrrr) rrrdrjr6rr|rr)r)rrrrrrrrrrrlogslog_files_foundrI log_fileslog_filers r- log_clusterrsRD7?#G,,     DO -- 9! - -H  " "8 , , , , - ?q   ) 'g'' ( ( ( (  ) 'g'' ( ( (  d?;OPPPQQQq!H '      r,actorz--idz-az1Retrieves the logs corresponding to this ActorID.z--pidz-pidz0Retrieves the logs from the actor with this pid.errc t||tjddt||||||||| | rdnd dS)uGet/List logs associated with an actor. Example: Follow the log file with an actor id ABCDEFG. ``` ray logs actor --id ABCDEFG --follow ``` Get the actor log from pid 123, ip x.x.x.x Note that this goes well with the driver log of Ray which prints (ip=x.x.x.x, pid=123, class_name) logs. ``` ray logs actor --pid=123 —ip=x.x.x.x ``` Get the actor err log file. ``` ray logs actor --id ABCDEFG --err ``` Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. MissingParameter: if inputs are missing. Nz0At least one of `--pid` and `--id` has to be setoption)message param_typerr) rrrrrrrrrr)rMissingParameterr) rrrrrrrrrrrs r- log_actorrsxL {rz$F      &uu      r,workerz1Retrieves the logs from the worker with this pid.)rrrc >t||||||||| rdnd dS)aGet logs associated with a worker process. Example: Follow the log file from a worker process with pid=123 ``` ray logs worker --pid 123 --follow ``` Get the stderr logs from a worker process. ``` ray logs worker --pid 123 --err ``` Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. MissingParameter: if inputs are missing. rr) rrrrrrrrrNr) rrrrrrrrrrs r- log_workerr"lsJj  &uu      r,jobzNRetrieves the logs from a submission job with submission id,i.e. raysubmit_XXXc0t||||||dS)aGet logs associated with a submission job. Example: Follow the log file from a submission job with submission id raysumbit_xxx. ``` ray logs job --id raysubmit_xxx ``` Follow the submission job log. ``` ray logs jobs --id raysubmit_xxx --follow ``` Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. MissingParameter: if inputs are missing. )rrrrrrNr!)rrrrrrrs r-log_jobr%s8d # r,taskz3Retrieves the logs from the task with this task id.z--attempt-numberz1Retrieves the logs from the attempt, default to 0c <t||||||||rdnddS)aGet logs associated with a task. Example: Follow the log file from a task with task id = ABCDEFG ``` ray logs tasks --id ABCDEFG --follow ``` Get the log from a retry attempt 1 from a task. ``` ray logs tasks --id ABCDEFG -a 1 ``` Note: If a task is from a concurrent actor (i.e. an async actor or a threaded actor), the log of the tasks are expected to be interleaved. Please use `ray logs actor --id ` for the entire actor log. Raises: :class:`RayStateApiException ` if the CLI is failed to query the data. MissingParameter: if inputs are missing. rr)rrrrrrrrNr!) rrrrrrrrrs r-log_taskr(sGz% &uu      r,r)br!loggingrenumrrtypingrrrr rr"ray._private.services_privaterray._common.network_utilsr )ray._private.thirdparty.tabulate.tabulater ray.util.annotationsr ray.util.stater rrrrrray.util.state.commonrrrrrrrrrrray.util.state.exceptionr getLoggerr$rrrr?rErNboolrpr'r|rrr)rrrrtimeout_optionaddress_optioncommandargumentChoiceJOBS RUNTIME_ENVSfloatrintrgroup pass_contextrrrrlog_follow_optionlog_tail_optionlog_interval_optionlog_timeout_optionlog_node_ip_optionlog_node_id_optionlog_suffix_optionlog_encoding_optionlog_encoding_errors_optionrrLOG_CLI_HELP_MSGGrouprlogs_state_cli_grouprrr"r%r(r+r,r-rMs ............ (((((((((333333>>>>>>******                        :99999  8 $ $d.##.#%]   <  X. c]..c] .  .  . .. ...+*0...r,