&`izddlmZmZmZddlmZddlmZedGddZGdd eZ d S) )AnyDictOptional)Block) PublicAPIalpha) stabilityc ^eZdZdZdededededef dZdeee fdededed edef d Z d S) FilenameProvideraGenerates filenames when you write a :class:`~ray.data.Dataset`. Use this class to customize the filenames used when writing a Dataset. Some methods write each row to a separate file, while others write each block to a separate file. For example, :meth:`ray.data.Dataset.write_images` writes individual rows, and :func:`ray.data.Dataset.write_parquet` writes blocks of data. For more information about blocks, see :ref:`Data internals `. If you're writing each row to a separate file, implement :meth:`~FilenameProvider.get_filename_for_row`. Otherwise, implement :meth:`~FilenameProvider.get_filename_for_block`. Example: This snippet shows you how to encode labels in written files. For example, if `"cat"` is a label, you might write a file named `cat_000000_000000_000000.png`. .. testcode:: import ray from ray.data.datasource import FilenameProvider class ImageFilenameProvider(FilenameProvider): def __init__(self, file_format: str): self.file_format = file_format def get_filename_for_row(self, row, write_uuid, task_index, block_index, row_index): return ( f"{row['label']}_{write_uuid}_{task_index:06}_{block_index:06}" f"_{row_index:06}.{self.file_format}" ) ds = ray.data.read_parquet("s3://anonymous@ray-example-data/images.parquet") ds.write_images( "/tmp/results", column="image", filename_provider=ImageFilenameProvider("png") ) block write_uuid task_index block_indexreturnct)aVGenerate a filename for a block of data. .. note:: Filenames must be unique and deterministic for a given write UUID, and task and block index. A block consists of multiple rows and corresponds to a single output file. Each task might produce a different number of blocks. Args: block: The block that will be written to a file. write_uuid: The UUID of the write operation. task_index: The index of the write task. block_index: The index of the block *within* the write task. NotImplementedError)selfr r rrs y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/datasource/filename_provider.pyget_filename_for_blockz'FilenameProvider.get_filename_for_block3s $"!row row_indexct)aGenerate a filename for a row. .. note:: Filenames must be unique and deterministic for a given write UUID, and task, block, and row index. A block consists of multiple rows, and each row corresponds to a single output file. Each task might produce a different number of blocks, and each block might contain a different number of rows. .. tip:: If you require a contiguous row index into the global dataset, use :meth:`~ray.data.Dataset.iter_rows`. This method is single-threaded and isn't recommended for large datasets. Args: row: The row that will be written to a file. write_uuid: The UUID of the write operation. task_index: The index of the write task. block_index: The index of the block *within* the write task. row_index: The index of the row *within* the block. r)rrr rrrs rget_filename_for_rowz%FilenameProvider.get_filename_for_rowGs <"!rN) __name__ __module__ __qualname____doc__rstrintrrrrrrr r s((T""(+"9<"KN" """"(" #s(^"" "  "  " """"""rr c eZdZ ddeedeefdZdedededed ef d Zd e ee fdededed ed ef d Z ded efdZ dS)_DefaultFilenameProviderN dataset_uuid file_formatc"||_||_dS)N _dataset_uuid _file_format)rr%r&s r__init__z!_DefaultFilenameProvider.__init__is*'rr r rrrcD|d|dd|d}||SN_06_generate_filename)rr r rrfile_ids rrz/_DefaultFilenameProvider.get_filename_for_blockos9 BB*BBB+BBB&&w///rrrcL|d|dd|dd|d}||Sr-r0)rrr rrrr2s rrz-_DefaultFilenameProvider.get_filename_for_rowusD QQ*QQQ+QQQ9QQQ&&w///rr2cdd}|j ||jdz }||z }|j |d|jz }|S)Nr..r()rr2filenames rr1z+_DefaultFilenameProvider._generate_filenamesT   ) 4-000 0HG   ( /D-// /Hr)NN) rrrrr r+rr!rrrrr1r"rrr$r$hsOS(($SM(?G}(((( 00(+09<0KN0 0000 0 #s(^ 0 0 0  0  0  0 0 0 0##rr$N) typingrrrray.data.blockrray.util.annotationsrr r$r"rrr;s&&&&&&&&&& ****** W]"]"]"]"]"]"]"]"@/r