&`i7) ddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZddlZddlZddlmZddlmZmZddlmZddlmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-dd l.m/Z/m0Z0ej1e2Z3d Z4ed e5d e5d e6de7fdZ8 eGddeZ9ededfde5d e e5d e:de7de ef dZ;ededfde5d e e5d e:de7de e f dZededfde5d e e5d e:de7de e,f dZ?ededfdee5dfd e e5d e:de7de e+f dZ@ededfde5d e e5d e:de7de e#f dZAeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de efd#ZBeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e$fd$ZCeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e"fd%ZDeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e fd&ZEeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e,fd'ZFeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e+fd(ZGeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e#fd)ZHeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e&fd*ZIeddeedddfd e e5de e ee5e%e*fd e:d e:d!e7d"e7de7de e fd+ZJ eddddddddd,ed-d.d/ddddfd e e5d0e e5d1e e5d2e e5d3e e5d4e e5d5e e:d6e7d7e:d e:d8e5d9e e5d:e e5d;e e5de7de e5ddff$d?ZKeddddefd e e5d0e e5d1e e5d@e e5d e:de e5e e5ff dAZL ededdfd e e5d e:d"e7de7de f dBZMededdfd e e5d e:d"e7de7de f dCZNededdfd e e5d e:d"e7de7de f dDZOdS)EN)contextmanager)fields)AnyDict GeneratorListOptionalTupleUnion)SubmissionClient)!get_address_for_submission_clientray_address_to_api_server_url) DeveloperAPI) DEFAULT_LIMITDEFAULT_RPC_TIMEOUT ActorStateClusterEventState GetApiOptions GetLogOptionsJobStateListApiOptions NodeState ObjectStatePlacementGroupState PredicateTypeRuntimeEnvState StateResourceSummaryApiOptionsSummaryResourceSupportedFilterType TaskState WorkerState dict_to_state)RayStateApiExceptionServerUnavailableaddressendpointtimeoutexplainc#K|sdVdSdtffd fddD} |D]}|dV|D]}|dS#|D]}|wxYw)a9A context manager to print warnings if the request is replied slowly. Warnings are printed 3 times Args: address: The address of the endpoint. endpoint: The name of the endpoint. timeout: Request timeout in seconds. explain: Whether ot not it will print the warning. Nelapsedc ptdt|dddddS)N(z / z? seconds) Waiting for the response from the API server address .)loggerinforound)r,r'r(r)s f/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/util/state/api.py print_warningz/warnings_on_slow_request..print_warningCsd  ,gq!! , ,g , , , ( , , ,     cLg|] }tj|z |z g!S))args) threadingTimer).0ir5r)s r4 z,warnings_on_slow_request..JsD  ! ]'A+GGGr6)r/)floatstartcancel)r'r(r)r*warning_timerstimerr5s``` @r4warnings_on_slow_requestrE.s   u        N #  E KKMMMM $  E LLNNNN  ^  E LLNNNN s A&&BceZdZdZ ddeedeeeefdeeeefffd Ze de e e fdefd Z d d ed ed ededef dZ d dededee dedee eeeeeeeeff dZ d!dedededededef dZdedefdZ d dede dededee eeeeeeeeef f dZ d dde!de"dededef dZ#xZ$S)"StateApiClientzLState API Client issues REST GET requests to the server for resource states.Nr'cookiesheaderscttd|sddi}t|}t|d||dS)aInitialize a StateApiClient and check the connection to the cluster. Args: address: Ray bootstrap address (e.g. `127.0.0.0:6379`, `auto`), or Ray Client address (e.g. `ray://:10001`), or Ray dashboard address (e.g. `http://:8265`). If not provided, it will be detected automatically from any running local Ray cluster. cookies: Cookies to use when sending requests to the HTTP job server. headers: Headers to use when sending requests to the HTTP job server, used for cases like authentication to a remote cluster. Nz\The Ray state CLI & SDK require the ray[default] installation: `pip install 'ray[default']`` Content-Typeapplication/jsonF)r'create_cluster_if_neededrIrH)requests RuntimeErrorr super__init__)selfr'rHrIapi_server_url __class__s r4rQzStateApiClient.__init__rs}$  >  ;%'9:G;7CC "%*      r6optionsreturnci}t|D]}|jdkrg|d<g|d<g|d<|jD]}t|dkrt d|d|\}}}|d||d||d|t ||j}| |||j<|S)Nfilters filter_keysfilter_predicates filter_valuesz+The given filter has incorrect input type, z). Provide (key, predicate, value) tuples.)rnamerXlen ValueErrorappendgetattr) clsrU options_dictfieldfilterfilter_kfilter_predicate filter_val option_vals r4 _make_paramzStateApiClient._make_params' G__ 6 6E zY&&.0 ]+46 0102 _-%o E EF6{{a''(F&FFF>D:H.  /66x@@@ !45<<=MNNN 188DDDD %*55J%+5 UZ(r6Fr(paramsr)_explainc t|j|||5d} |d|||}|jdkr1d|jddvr|ne#tjj $rN}d|j|d }|d z }|t||d |j d |jd z }t||d}~wwxYwdddn #1swxYwY | }nR#tjj$r;}td|j d|jd|jdt d|d}~wwxYw|ddurtd|d|ddS)N)r'r(r)r*GET)r)rkr&rLrKzFailed to make request to z. aFailed to connect to API server. Please check the API server log for details. Make sure dependencies are installed with `pip install ray[default]`. Please also check dashboard is available, and included when starting ray cluster, i.e. `ray start --include-dashboard=True --head`. z Response(url=z,status=)zFailed to parse Response(url=z , status=z, text='z')resultFKAPI server internal error. See dashboard.log file for more details. Error: msgdata)rE_address _do_request status_coderIgetraise_for_statusrN exceptionsRequestExceptionr%urlr$jsonJSONDecodeErrortext!_MAX_HTTP_RESPONSE_EXCEPTION_TEXT)rRr(rkr)rlresponseeerr_strs r4_make_http_get_requestz%StateApiClient._make_http_get_requestst&MHgx   $ ;$ ;H ;++#! ,(C//*#+//CCDD--///&7 ; ; ;Rt}RhRRRI#+G444X8<XXAUXXXX*733:! ;)$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;$ ;L }}HH"2   &n nn".nn8@ FhGhFh8innn   H  & &&,"5/,,  ))sNC$AA32C$3CA CCC$$C(+C(0DE6EEresourceidc||}tjdtjdtjdtjdtjdtjdtjdi}|vrtdj d |g|d <d g|d <|g|d <d|d<dj }| |||j |}|d} t| dkrdSfd| D} tjkr| Stjkrt| dkr| dS| St| dksJ| dS)aGet resources states by id Args: resource_name: Resource names, i.e. 'workers', 'actors', 'nodes', 'placement_groups', 'tasks', 'objects'. 'jobs' and 'runtime-envs' are not supported yet. id: ID for the resource, i.e. 'node_id' for nodes. options: Get options. See `GetApiOptions` for details. _explain: Print the API information such as API latency or failed query information. Returns: None if not found, and if found: - ActorState for actors - PlacementGroupState for placement groups - NodeState for nodes - WorkerState for workers - TaskState for tasks - JobState for jobs Empty list for objects if not found, or list of ObjectState for objects Raises: Exception: This doesn't catch any exceptions raised when the underlying request call raises exceptions. For example, it could raise `requests.Timeout` when timeout occurs. ValueError: if the resource could not be GET by id, i.e. jobs and runtime-envs. node_idactor_idplacement_group_id worker_idtask_id object_id submission_idz Can't get z by id.rY=rZr[Tdetail/api/v0/r(rkr)rlrqrNc0g|]}t|Sr#r;drs r4r=z&StateApiClient.get..Cs#===-8,,===r6)rjrNODESACTORSPLACEMENT_GROUPSWORKERSTASKSOBJECTSJOBSr_r]valuerr)r^) rRrrrUrlrkRESOURCE_ID_KEY_NAMEr(list_api_responserqs ` r4rxzStateApiClient.gets`!!'**    *  *,@  !;    !;    / / /@(-@@@AA A!5h!? @}'*e"##%$x.hn.. 77O 8   #8, v;;!  4====f=== }, , , M }* * *6{{aay M6{{aayr6T api_responsewarn_data_source_not_availablewarn_data_truncation warn_limitwarn_server_side_warningsc |r,|dd}|rtj||rB|d}|d} | |kr,tjd|d| d|jd| |z d |d |rK|d } |d } | t | kr(tjd t | d| d|r1|dg} | r| D]} tj| dSdSdS)aPrint the API warnings. Args: resource: Resource names, i.e. 'jobs', 'actors', 'nodes', see `StateResource` for details. api_response: The dictionarified `ListApiResponse` or `SummaryApiResponse`. warn_data_source_not_available: Warn when some data sources are not available. warn_data_truncation: Warn when results were truncated at the data source. warn_limit: Warn when results were limited. warn_server_side_warnings: Warn when the server side generates warnings (E.g., when callsites not enabled for listing objects) partial_failure_warningNnum_after_truncationtotalz1The returned data may contain incomplete result. z (z total from the cluster) z% are retrieved from the data source. z% entries have been truncated. Max of zG entries are retrieved from data source to prevent over-sized payloads. num_filteredrqz Limit last z entries (Total zs). Use `--filter` to reduce the amount of data to return or setting a higher limit with `--limit` to see all data. warnings)rxrwarnrr^)rRrrrrrr warning_msgsrrrrtwarnings_to_printwarning_to_prints r4_print_api_warningz!StateApiClient._print_api_warningWs0 * ,'++,EtLLL , l+++  #/0F#G  )E+++  K/KK38KK#>KK!#77KK#7 KKK     (7L)Dc$ii'' Rc$iiRR".RRR % 4 , 0 0R @ @   4(944$M"23333  4 4 4 444r6c |dd}|rtd|jd|d}|d}||krtd|d|jd|d dS) aZRaise an exception when the API resopnse contains a missing output. Output can be missing if (1) Failures on some of data source queries (e.g., `ray list tasks` queries all raylets, and if some of queries fail, it will contain missing output. If all queries fail, it will just fail). (2) Data is truncated because the output is too large. Args: resource: Resource names, i.e. 'jobs', 'actors', 'nodes', see `StateResource` for details. api_response: The dictionarified `ListApiResponse` or `SummaryApiResponse`. rNzFailed to retrieve all z from the cluster becausethey are not reachable due to query failures to the data sources. To avoid raising an exception and allow having missing output, set `raise_on_missing_output=False`. rr z{ from the cluster because they are not reachable due to data truncation. It happens when the returned data is too large (> zf) To avoid raising an exception and allow having missing output, set `raise_on_missing_output=False`. )rxr$r)rRrrrrrs r4_raise_on_missing_outputz'StateApiClient._raise_on_missing_outputs$''(A4HH  &8(.888 W%+,BC ( ( (&8%88(.88 + 888    ) (r6raise_on_missing_outputc2|rgSdj}||}||||j|}|r|||r||fd|dDS)aList resources states Args: resource: Resource names, i.e. 'jobs', 'actors', 'nodes', see `StateResource` for details. options: List options. See `ListApiOptions` for details. raise_on_missing_output: When True, raise an exception if the output is incomplete. Output can be incomplete if (1) there's a partial network failure when the source is distributed. (2) data is truncated because it is too large. Set it to False to avoid throwing an exception on missing data. _explain: Print the API information such as API latency or failed query information. Returns: A list of queried result from `ListApiResponse`, Raises: Exception: This doesn't catch any exceptions raised when the underlying request call raises exceptions. For example, it could raise `requests.Timeout` when timeout occurs. rrc0g|]}t|Srrrs r4r=z'StateApiClient.list..s#PPPq a**PPPr6rq)has_conflicting_filtersrrjrr)rr)rRrrUrrlr(rkrs ` r4listzStateApiClient.listsT  * * , , I.hn..!!'** 77O 8   # G  ) )(4E F F F  A  # #H.? @ @ @PPPP4Eh4OPPPPr6rlcd|ji}d|jd}||||j|}|r||||r|||d|ddS) aSummarize resources states Args: resource_name: Resource names, see `SummaryResource` for details. options: summary options. See `SummaryApiOptions` for details. raise_on_missing_output: Raise an exception if the output has missing data. Output can have missing data if (1) there's a partial network failure when the source is distributed. (2) data is truncated because it is too large. _explain: Print the API information such as API latency or failed query information. Returns: A dictionary of queried result from `SummaryApiResponse`. Raises: Exception: This doesn't catch any exceptions raised when the underlying request call raises exceptions. For example, it could raise `requests.Timeout` when timeout occurs. r)rz /summarizerF)rrqnode_id_to_summary)r)rrrr)rRrrUrrlrkr(summary_api_responses r4summaryzStateApiClient.summarys:W_-8hn888#::O ;   # J  ) )(4H I I I  V  # #H.Bu # U U U#H-.BCCr6)NNN)F)TTTT)%__name__ __module__ __qualname____doc__r strrrrQ classmethodr rrrjr@boolrrrrrr"r!rrrrxdictrrrrrrrr __classcell__)rTs@r4rGrGnsVV"&,0,0 " " #" $sCx.)" $sCx.) " " " " " " H% (E"F4[F <*<*<*<* <*  <*  <*<*<*<*F bbb b-( b  b              bbbbP04%)*.F4F4F4F4)- F4 # F4  F4$(F4F4F4F4P%%d%%%%X :Q:Q:Q :Q"& :Q  :Q              :Q:Q:Q:QD *D*D*D!*D# *D "& *D  *D *D*D*D*D*D*D*D*Dr6rGFrrlrVct|tj|t ||S)aIGet an actor by id. Args: id: Id of the actor address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state API requests made. _explain: Print the API information such as API latency or failed query information. Returns: None if actor not found, or :class:`ActorState `. Raises: RayStateApiException: if the CLI failed to query the data. r'r)r)rGrxrrrrr'r)rls r4 get_actorr/sC0 ' * * * . .b-"@"@"@8 /  r6ct|tj|t ||S)agGet a submission job detail by id. Args: id: Submission ID obtained from job API. address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state API requests made. _explain: Print the API information such as API latency or failed query information. Returns: None if job not found, or :class:`JobState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrxrrrrs r4get_jobrLsF0 ' * * * . . g&&& /  r6ct|tj|t ||S)aZGet a placement group by id. Args: id: Id of the placement group address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state APIs requests made. _explain: Print the API information such as API latency or failed query information. Returns: None if actor not found, or :class:`~ray.util.state.common.PlacementGroupState`. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrxrrrrs r4get_placement_grouprlsF0 ' * * * . .& g&&& /  r6ct|tj|t ||S)aIGet a node by id. Args: id: Id of the node. address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state APIs requests made. _explain: Print the API information such as API latency or failed query information. Returns: None if actor not found, or :class:`NodeState `. Raises: RayStateApiException: if the CLI is failed to query the data. rrr)rGrxrrrrs r4get_nodersF0 ' * * * . . g&&& /  r6ct|tj|t ||S)aMGet a worker by id. Args: id: Id of the worker address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state APIs requests made. _explain: Print the API information such as API latency or failed query information. Returns: None if actor not found, or :class:`WorkerState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrxrrrrs r4 get_workerrsF0 ' * * * . . g&&& /  r6z ray.ObjectRefct|tr|}n&|}t |t j|t||S)aGet task attempts of a task by id. Args: id: String id of the task or ObjectRef that corresponds to task address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state APIs requests made. _explain: Print the API information such as API latency or failed query information. Returns: None if task not found, or a list of :class:`~ray.util.state.common.TaskState` from the task attempts. Raises: RayStateApiException: if the CLI failed to query the data. rrr) isinstancerrhexrGrxrrr)rr'r)rlstr_ids r4get_taskrsx4"c$!!## ' * * * . .g&&& /  r6ct|tj|t ||S)aGet objects by id. There could be more than 1 entry returned since an object could be referenced at different places. Args: id: Id of the object address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout value for the state APIs requests made. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`~ray.util.state.common.ObjectState`. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrxrrrrs r4 get_objectsrsF6 ' * * * . . g&&& /  r6TrXlimitrrc t|tjt ||||||S)aqList actors in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("id", "=", "abcd")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `ActorState`) will be queried and returned. See :class:`ActorState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`ActorState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rXrrUrrl)rGrrrrr'rXrr)rrrls r4 list_actorsrsYF ' * * * / /    !8 0   r6c t|tjt ||||||S)ayList placement groups in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("state", "=", "abcd")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `PlacementGroupState`) will be queried and returned. See :class:`~ray.util.state.common.PlacementGroupState`. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`~ray.util.state.common.PlacementGroupState`. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4list_placement_groupsrEsVD ' * * * / /&'&   !8 0  r6c t|tjt ||||||S)aList nodes in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("node_name", "=", "abcd")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `NodeState`) will be queried and returned. See :class:`NodeState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of dictionarified :class:`NodeState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4 list_nodesrqVF ' * * * / /'&   !8 0  r6c t|tjt ||||||S)aList jobs submitted to the cluster by :ref:`ray job submission `. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("status", "=", "abcd")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `JobState`) will be queried and returned. See :class:`JobState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of dictionarified :class:`JobState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4 list_jobsrsVF ' * * * / /'&   !8 0  r6c t|tjt ||||||S)a}List workers in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("is_alive", "=", "True")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `WorkerState`) will be queried and returned. See :class:`WorkerState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`WorkerState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4 list_workersrVF ' * * * / /'&   !8 0  r6c t|tjt ||||||S)aqList tasks in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("is_alive", "=", "True")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `TaskState`) will be queried and returned. See :class:`TaskState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`TaskState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4 list_tasksrrr6c t|tjt ||||||S)azList objects in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("ip", "=", "0.0.0.0")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `ObjectState`) will be queried and returned. See :class:`ObjectState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`ObjectState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4 list_objectsr%rr6c t|tjt ||||||S)aList runtime environments in the cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. filters: List of tuples of filter key, predicate (=, or !=), and the filter value. E.g., `("node_id", "=", "abcdef")` String filter values are case-insensitive. limit: Max number of entries returned by the state backend. timeout: Max timeout value for the state APIs requests made. detail: When True, more details info (specified in `RuntimeEnvState`) will be queried and returned. See :class:`RuntimeEnvState `. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Returns: List of :class:`RuntimeEnvState `. Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrr RUNTIME_ENVSrrs r4list_runtime_envsrRsVF ' * * * / /"'&   !8 0  r6c t|tjt ||||||S)Nrrr)rGrrCLUSTER_EVENTSrrs r4list_cluster_eventsrsU ' * * * / /$'&   !8 0  r6outzutf-8strictrnode_ipfilenamerrpidfollowtailsuffixencodingerrorsrattempt_number _intervalfilter_ansi_codec#8Kt|}|rdnd}t|||||||||| | | | }i}t|D]#}t||j}| |||j<$|||d<t j|d|dtj |d 5}|j d krt|j | d D]}| || | }|V ddddS#1swxYwYdS) a Retrieve log file based on file name or some entities ids (pid, actor id, task id). Examples: .. testcode:: :hide: import ray import time ray.shutdown() ray.init() # Wait for the node to be registered to the dashboard time.sleep(5) .. testcode:: import ray from ray.util.state import get_log # Node id could be retrieved from list_nodes() or ray.nodes() node_id = ray.nodes()[0]["NodeID"] filename = "raylet.out" for l in get_log(filename=filename, node_id=node_id): print(l) .. testoutput:: :options: +MOCK [2023-05-19 12:35:18,347 I 4259 68399276] (raylet) io_service_pool.cc:35: IOServicePool is running with 1 io_service. [2023-05-19 12:35:18,348 I 4259 68399276] (raylet) store_runner.cc:32: Allowing the Plasma store to use up to 2.14748GB of memory. [2023-05-19 12:35:18,348 I 4259 68399276] (raylet) store_runner.cc:48: Starting object store with directory /tmp, fallback /tmp/ray, and huge page support disabled Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If not specified, it will be retrieved from the initialized ray cluster. node_id: Id of the node containing the logs . node_ip: Ip of the node containing the logs. (At least one of the node_id and node_ip have to be supplied when identifying a node). filename: Name of the file (relative to the ray log directory) to be retrieved. actor_id: Id of the actor if getting logs from an actor. task_id: Id of the task if getting logs from a non concurrent actor. For concurrent actor, please query the log with actor_id. pid: PID of the worker if getting logs generated by a worker. When querying with pid, either node_id or node_ip must be supplied. follow: When set to True, logs will be streamed and followed. tail: Number of lines to get from the end of the log file. Set to -1 for getting the entire log. timeout: Max timeout for requests made when getting the logs. suffix: The suffix of the log file if query by id of tasks/workers/actors. Default to "out". encoding: The encoding used to decode the content of the log file. Default is "utf-8". Use None to get binary data directly. errors: The error handling scheme to use for decoding errors. Default is "strict". See https://docs.python.org/3/library/codecs.html#error-handlers submission_id: Job submission ID if getting log from a submission job. attempt_number: The attempt number of the task if getting logs generated by a task. _interval: The interval in secs to print new logs when `follow=True`. filter_ansi_code: A boolean flag for determining whether to filter ANSI escape codes. Setting to `True` removes ANSI escape codes from the output. The default value is `False`. Return: A Generator of log line, None for SendType and ReturnType. Raises: RayStateApiException: if the CLI failed to query the data. streamfile) rrrrrrlinesinterval media_typer)rrrNrz /api/v0/logs/?T)r) chunk_size)rr)rrrrar]rNrxurllibparse urlencoderwr$r iter_contentdecode)r'rrrrrrrrr)rrrrrrrrSr rUrcrdrirchunks r4get_logrsn37;;N#/J #%GL22Wej11  !'1L $#+; '(   2 2  2 2 < ! !, / / 2 2     =C  &qv.. .^^t^44  E# hv FFKKKK                   s+ADDD glob_filterc| | Jdt|}|sd}i}|r||d<|r||d<|r||d<||d<tj|dtj|}||}|d d urtd |d |d d S)aListing log files available. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If not specified, it will be retrieved from the initialized ray cluster. node_id: Id of the node containing the logs. node_ip: Ip of the node containing the logs. glob_filter: Name of the file (relative to the ray log directory) to be retrieved. E.g. `glob_filter="*worker*"` for all worker logs. actor_id: Id of the actor if getting logs from an actor. timeout: Max timeout for requests made when getting the logs. _interval: The interval in secs to print new logs when `follow=True`. Return: A dictionary where the keys are log groups (e.g. gcs, raylet, worker), and values are list of log filenames. Raises: RayStateApiException: if the CLI failed to query the data, or ConnectionError if failed to resolve the ray address. Nz/At least one of node ip and node id is required*rrglobr)z /api/v0/logs?rqFrrrsrt) rrNrxrrrryr}r$) r'rrrr)rSrcrrs r4 list_logsrs< w228 32237;;N  L*") Y*") Y+* V%L NN (>(>|(L(LNN  AvvxxHU""" (uo ( (    F H %%r6ct|tjt |||S)aSummarize the tasks in cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout for requests made when getting the states. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Return: Dictionarified :class:`~ray.util.state.common.TaskSummaries` Raises: RayStateApiException: if the CLI is failed to query the data. rrr)rGrrrrr'r)rrls r4summarize_tasksrZsF2 ' * * * 2 2!'222 7 3  r6ct|tjt |||S)aSummarize the actors in cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout for requests made when getting the states. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Return: Dictionarified :class:`~ray.util.state.common.ActorSummaries` Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4summarize_actorsr{sF2 ' * * * 2 2!'222 7 3  r6ct|tjt |||S)aSummarize the objects in cluster. Args: address: Ray bootstrap address, could be `auto`, `localhost:6379`. If None, it will be resolved automatically from an initialized ray. timeout: Max timeout for requests made when getting the states. raise_on_missing_output: When True, exceptions will be raised if there is missing data due to truncation/data source unavailable. _explain: Print the API information such as API latency or failed query information. Return: Dictionarified :class:`~ray.util.state.common.ObjectSummaries` Raises: RayStateApiException: if the CLI failed to query the data. rrr)rGrrrrrs r4summarize_objectsr!sF0 ' * * * 2 2!'222 7 3  r6)Ploggingr9rr contextlibr dataclassesrtypingrrrrr r r rNray#ray.dashboard.modules.dashboard_sdkr ray.dashboard.utilsr rray.util.annotationsrray.util.state.commonrrrrrrrrrrrrrrrrr r!r"r#ray.util.state.exceptionr$r% getLoggerrr1rrr@rrErGintrrrrrrrrrrrrrrrrrrrrr!rr6r4r.s\ %%%%%%EEEEEEEEEEEEEEEEEE @@@@@@.-----,MLLLLLLL  8 $ $$'!''"'-2'=A''''T*}D}D}D}D}D%}D}D}D@"&   c]  j 8"&   c]  h >"&   c]  !" >"&   c]  i >"&   c]  k >"& "" c?"#" c]"" " i """"J"&   c]   + D!NR&$(,, c], d5m5H!HIJ K, , ,  , " ,, *,,,,^!NR&$((( c]( d5m5H!HIJ K( ( (  ( " (( ((((V!NR&$()) c]) d5m5H!HIJ K) ) )  ) " )) )_))))X!NR&$()) c]) d5m5H!HIJ K) ) )  ) " )) (^))))X!NR&$()) c]) d5m5H!HIJ K) ) )  ) " )) +))))X!NR&$()) c]) d5m5H!HIJ K) ) )  ) " )) )_))))X!NR&$()) c]) d5m5H!HIJ K) ) )  ) " )) +))))X!NR&$()) c]) d5m5H!HIJ K) ) )  ) " )) /))))X!NR&$( c] d5m5H!HIJ K     "  $Z& !!!""!&%$#'!%"#{{ c]{ c]{c]{sm { sm { c] { #{ { {{ {sm{ SM{C={{ !{"#{$sD$%{{{{|!!!!%& :&:& c]:& c]:&c]:&# :&  :&  #tCy. :&:&:&:&z !&$(  c] "   @!&$(  c] "   @!&$(  c] "   r6