&`i! U d dl Z d dlmZmZmZmZmZ d dlmZ d dl m
Z
d dl m Z m
Z
mZ d dlmZ d dlmZmZ erd dlZdZee ed< d d lmZ d
e j vr e rd dlZd dl Zd dl Z d dl Zd d l m Z e ej! " d Z# e d
Z$e#e$k re j% & ej' ( d Z)ej* + d
e) Z,ej* - e, Z.e.e j e,j/ < e,j0 1 e. n# e$ r Z2e2ZY dZ2[2ndZ2[2ww xY w G d de Z3dS ) N)
TYPE_CHECKINGIterableListOptionalUnion)pyarrow_table_from_pydict)_check_pyarrow_version)Block
BlockAccessor
BlockMetadata)Dataset)
DatasourceReadTaskTRANSFORMERS_IMPORT_ERROR)is_datasets_availabledatasets_modules)parsedatasetsz4.0.0z __init__.pyc e Zd ZdZ dded defdZeded defd Z de
e fd Z de e
fd
Z dd ed
e
e dee fdZd S )HuggingFaceDatasourceah Hugging Face Dataset datasource, for reading from a
`Hugging Face Datasets Dataset `_.
This Datasource implements a streamed read using a
single read task, most beneficial for a
`Hugging Face Datasets IterableDataset `_
or datasets which are too large to fit in-memory.
For an in-memory Hugging Face Dataset (`datasets.Dataset`), use :meth:`~ray.data.from_huggingface`
directly for faster performance.
dataset)zdatasets.Datasetzdatasets.IterableDataset
batch_sizec > t t || _ || _ d S N)r _dataset _batch_size)selfr r s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/_internal/datasource/huggingface_datasource.py__init__z HuggingFaceDatasource.__init__D s%
% 0+ +
% returnc ddl }|j j }|j j }t |j }t
||j s:ddl m} |||| }|j |j k rg S n# t $ r g cY S w xY wddl }d| d| d| } | | }
|
j
|j d k r|
S g S ) zReturn list of Hugging Face hosted parquet file URLs if they
exist for the data (i.e. if the dataset is a public dataset that
has not been transformed) else return an empty list.r N) load_dataset)splitz$https://huggingface.co/api/datasets/z /parquet//ok)r info dataset_name config_namestrr%
isinstanceIterableDatasetr$ _fingerprint Exceptionrequestsget status_codecodesjson) clsr r r) r*
split_namer$ dsr0
public_urlresps r list_parquet_urls_from_datasetz4HuggingFaceDatasource.list_parquet_urls_from_datasetO s,
|0 l. ''
'8#;<<
- - - - - -
!\, :NNN?g&::: I ;
3<
3
3#
3
3&0
3
3 ||J'' x~d 3 3 399;; Is
A. . A=<A=c | j j S r )r dataset_size)r s r estimate_inmemory_data_sizez1HuggingFaceDatasource.estimate_inmemory_data_size~ s
}))r! c # K dd l }dd l}dd l}| j d | j D ]}t ||j |j t |j f s t dt | d t ||j rd|i}t |t rt | }t! j | }|V d S )Nr arrow)r z
Batch format z isn't supported. Only the following batch formats are supported: dict (corresponds to `None` in `dataset.with_format()`), pyarrow.Table, np.array, pd.DataFrame.item)numpypandaspyarrowr with_formatiterr r, Table DataFramedictarray
ValueErrortypendarrayr r for_block
to_default)r nppdrC batchblocks r
_read_datasetz#HuggingFaceDatasource._read_dataset s# ]..w77<<' =
E egmR\4%RSS
>DKK > > > %,,
( %&&
91%88!+E22==??EKKKK5 r! N parallelismper_task_row_limitc x t t d d d d }t | j || g}|S )N)num_rows
size_bytes input_files
exec_stats)rU )r r r rS )r rT rU meta
read_taskss r get_read_tasksz$HuggingFaceDatasource.get_read_tasks sa
"#5
&
r! )r r )__name__
__module__ __qualname____doc__r intr classmethodr
r: r r= r r
rS r r r] r! r r r 9 s & &EF & & & & & , JK, , , , [,\*Xc] * * * *"x " " " "N -1 %SM
h r! r )4systypingr r r r r $ray.air.util.tensor_extensions.arrowr ray.data._internal.utilr ray.data.blockr
r r ray.data.datasetr
ray.data.datasourcer r r r ImportError__annotations__transformers.utilsr modules importlibimportlib.metadataos
datasets.loadpackaging.versionr metadataversionDATASETS_VERSION'DATASETS_VERSION_WITHOUT_SCRIPT_SUPPORTpathjoinloadinit_dynamic_modulesdynamic_modules_pathutilspec_from_file_locationspecmodule_from_specr nameloader exec_moduleer rd r! r r sx
A A A A A A A A A A A A A A J J J J J J : : : : : : > > > > > > > > > > $ $ $ $ $ $ 4 4 4 4 4 4 4 4 OOO 48 8K0 7 7 7&" 988888,,1F1F1H1H,!!!! ++++++ !5!3!;!;J!G!GHH27%../ E E E#%7<<
2244m$ $ >99"$8 D )~ > >t D D %5CK " K # #$4 5 5 5 " " " ! "F F F F FJ F F F F Fs C3E EE E