&`i6 @dZddlZddlZddlmZmZddlmZmZm Z m Z ddl m Z m Z mZmZmZmZmZddlZddlmZmZmZddlmZddlmZdd lmZdd lm Z dd l!m"Z"e r dd l#m$Z$dd l!m"Z"ej%e&Z'd Z(dZ)eGddZ*eGddZ+eGddZ,eGddZ-de.de/de.fdZ0e)fdZ1 d(de-de2de2de fd Z3d!edej4fd"Z5d)d$Z6Gd%d#eZ7Gd&d'e7Z8dS)*z,Metadata exporter API for Ray Data datasets.N)ABCabstractmethod)asdict dataclassfield is_dataclass) TYPE_CHECKINGAnyDictListMappingOptionalSequence) EventLogTypecheck_export_api_enabledget_export_event_logger)ExportDatasetMetadata)Panel)OPERATOR_PANELS) DatasetState) DataContext)PhysicalOperatorunknowndc(eZdZUdZeed<eed<dS)SubStagezRepresents a sub-stage within an operator in the DAG. Attributes: name: The name of the sub-stage. id: The unique identifier of the sub-stage. nameidN)__name__ __module__ __qualname____doc__str__annotations__x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/_internal/metadata_exporter.pyrr)s. III GGGGGr&rceZdZUdZeed<eed<eed<eeed<eeed<eed<ee Z e eed <ee Z e e ed <eeZeeefed <d S) OperatoraXRepresents a data processing operator in the DAG. Attributes: name: The name of the operator. id: The unique identifier of the operator within the DAG structure, typically incorporating a position or index (e.g., "ReadParquet_0"). This is used for referencing operators within the DAG topology. uuid: The system-generated UUID of the physical operator instance. This is the internal unique identifier created when the operator instance is initialized and remains consistent throughout its lifetime. input_dependencies: List of operator IDs that this operator depends on for input. sub_stages: List of sub-stages contained within this operator. args: User-specified arguments associated with the operator, which may include configuration settings, options, or other relevant data for the operator. execution_start_time: The timestamp when the operator execution begins. execution_end_time: The timestamp when the operator execution ends. state: The state of the operator. rruuidexecution_start_timeexecution_end_timestatedefault_factoryinput_dependencies sub_stagesargsN)rr r!r"r#r$rfloatrlistr0r r1rdictr2r r r%r&r'r)r)6s& III GGG III"5/))) ''' JJJ$)E$$?$?$?S ???!&t!z5Topology.create_topology_metadata..xs)$$$&)cXooHSMooor&N)rrr*r0r2r+r,r-_sub_progress_bar_names_sub_rr)r7post_order_iterr)rrr0sanitize_for_struct_get_logical_argsrPENDINGhasattrrA enumerater1appendrr8) r9r:resultopop_idoperatorjsub_name sub_stage_ids ` r'create_topology_metadataz!Topology.create_topology_metadata`sB%%'' . .BRLE WU$$$$-/-B$$$))=)=)?)?@@%)#'"*/   Hr455 Y":T Y#,R-G#H#HYYKAx&+#5#5!#5#5L'..xX,/W/W/WXXXX   # #H - - - - r&N)rr r!r"rr4r8r r)r$ staticmethodr r#rRr%r&r'r7r7Vs !&d ; ; ;ItH~;;;' '+/0BC0G+H' '''\'''r&r7c|eZdZUdZeed<eed<eed<eed<eed<e eed<e eed<eed <d S) DatasetMetadataaMetadata about a Ray Data dataset. This class represents the metadata associated with a dataset, including its provenance information and execution details. Attributes: job_id: The ID of the job running this dataset. topology: The structure of the dataset's operator DAG. dataset_id: The unique ID of the dataset. start_time: The timestamp when the dataset is registered. data_context: The DataContext attached to the dataset. execution_start_time: The timestamp when the dataset execution starts. execution_end_time: The timestamp when the dataset execution ends. state: The state of the dataset. job_idtopology dataset_id start_time data_contextr+r,r-N) rr r!r"r#r$r7r3rrr%r&r'rUrUs{ KKKOOO"5/))) ''' JJJJJr&rUstruncate_lengthr;cFt||kr |d|dzS|S)N...)len)r[r\s r'_add_ellipsis_for_stringr`s. 1vv!/!"U** Hr&ct|tr fd|DSt|trt |St|t t frYg}t|D]E\}}|kr|dn$|t|F|S t|rtt|St t|S#t$r1tdt|j}t |cYSwxYw)aPrepares the obj for Struct Protobuf format by recursively going through dictionaries, lists, etc... - Dataclasses will be converted to dicts - Dictionary keys will be converted to strings - Lists, tuples, sets, bytes, bytearrays will be converted to lists cRi|]#\}}t|t|$Sr%)r#rE)r>kvr\s r' z'sanitize_for_struct..s0XXXDAqA+A??XXXr&r^z: ) isinstancer itemsr#r`rsetrIrJrErr ExceptionUNKNOWNtyper)objr\resirdunk_names ` r'rErEso#wGXXXXCIIKKXXXX C  G'_=== C(C ) )GcNN @ @DAqO## 5!!! JJ*1o>> ? ? ? ?  GC   I*6#;;HHH+CHHoFF F G G G!99T#YY%799H+HoFF F F F Gs+D>D8EETdataset_metadatainclude_data_contextinclude_op_argsc ddlm}ddlm}m}m}m}|}|} |jjD]} |} |r| | j || j | j | j | | j| j|j| j} | jD]} | j| | jD]3}||j |j }| j|4| j| |}|r'| t-|j||j|j|j||j|j|j|jdt8D}|j| |S)aConvert the dataset metadata to a protobuf message. Args: dataset_metadata: DatasetMetadata object containing the dataset's information and DAG structure. include_data_context: If DataContext will be exported include_op_args: If operator args will be exported Returns: The protobuf message representing the dataset metadata. r)Struct)rr)rr7)rrr*r2r+r,r-rCc,g|]}t|Sr%)_to_proto_dashboard_panel)r>ps r'r@z-dataset_metadata_to_proto..s!OOO!2155OOOr&)rXrVrYrZr+r,r-operator_panels)google.protobuf.struct_pb2rt.ray.core.generated.export_dataset_metadata_pb2rr)rr7rWr8updater2rrr*r+r, OperatorStateValuer-r0rJr1rErZrXrVrYrrCopyFrom)rprqrrrtProtoDatasetMetadata ProtoOperator ProtoSubStage ProtoTopologyproto_dataset_metadataproto_topologyrLr2proto_operatordep_id sub_stageproto_sub_stagerZs r'dataset_metadata_to_protorsG"2111112133"]__N'188vxx  ! KK &u!#!8!4-33BH==   + = =F  - 4 4V < < < < > >I+m^<O  % , ,_ = = = =  ''7777688LP/0@0MNNOOO11#.&#.!-B+>"/556F6LMMOOOOO   #,,^<<< !!r&panelc`tjt|j|j}|S)z+Convert Dashboard Panel to protobuf format.)rtitle)rDashboardPanelMetadatar#rr)r proto_panels r'rvrvs3'= ux==kK r&DatasetMetadataExporterc4tS)znGet the dataset metadata exporter instance. Returns: The dataset metadata exporter instance. )LoggerDatasetMetadataExportercreate_if_enabledr%r&r'get_dataset_metadata_exporterr(s ) : : < <ray.dashboard.modules.metrics.dashboards.data_dashboard_panelsr*ray.data._internal.execution.dataset_staterray.data.contextr9ray.data._internal.execution.interfaces.physical_operatorr getLoggerrrrjDEFAULT_TRUNCATION_LENGTHrr)r7rUr#intr`rErrrrvrrrr%r&r'rs22 ########>>>>>>>>>>>>NNNNNNNNNNNNNNNNNN  BAAAAADCCCCC((((((--,,,,,  8 $ $               7777777 7> 1111111 1h  6  c c    .GGGGGD"& L"L"%L"L"L" L"L"L"L"^   0    ====     c   DBBBBB$;BBBBBr&