Pi[" d dl mZ d dlmZmZmZ d dlZd dlm Z
d dl m Z
d dlZd dlmZ d dlmZ ej j e Ze G d dej Z G d d ej ZdS )
) dataclass)LiteralOptionalUnionN)Key)
table_castc e Zd ZU dZdZee ed< dZee e
ed< dZ ee j
ed< dZeeej e e e e e f ed< dZeej ed< dZed ed
< fd Z xZS )
ParquetConfiga
BuilderConfig for Parquet.
Args:
batch_size (`int`, *optional*):
Size of the RecordBatches to iterate on.
The default is the row group size (defined by the first row group).
columns (`list[str]`, *optional*)
List of columns to load, the other ones are ignored.
All columns are loaded by default.
features: (`Features`, *optional*):
Cast the data to `features`.
filters (`Union[pyarrow.dataset.Expression, list[tuple], list[list[tuple]]]`, *optional*):
Return only the rows matching the filter.
If possible the predicate will be pushed down to exploit the partition information
or internal metadata found in the data source, e.g. Parquet statistics.
Otherwise filters the loaded RecordBatches before yielding them.
fragment_scan_options (`pyarrow.dataset.ParquetFragmentScanOptions`, *optional*)
Scan-specific options for Parquet fragments.
This is especially useful to configure buffering and caching.
on_bad_files (`Literal["error", "warn", "skip"]`, *optional*, defaults to "error")
Specify what to do upon encountering a bad file (a file that can't be read). Allowed values are :
* 'error', raise an Exception when a bad file is encountered.
* 'warn', raise a warning when a bad file is encountered and skip that file.
* 'skip', skip bad files without raising or warning when they are encountered.
Example:
Load a subset of columns:
```python
>>> ds = load_dataset(parquet_dataset_id, columns=["col_0", "col_1"])
```
Stream data and efficiently filter data, possibly skipping entire files or row groups:
```python
>>> filters = [("col_0", "==", 0)]
>>> ds = load_dataset(parquet_dataset_id, streaming=True, filters=filters)
```
Increase the minimum request size when streaming from 32MiB (default) to 128MiB and enable prefetching:
```python
>>> import pyarrow
>>> import pyarrow.dataset
>>> fragment_scan_options = pyarrow.dataset.ParquetFragmentScanOptions(
... cache_options=pyarrow.CacheOptions(
... prefetch_limit=1,
... range_size_limit=128 << 20
... ),
... )
>>> ds = load_dataset(parquet_dataset_id, streaming=True, fragment_scan_options=fragment_scan_options)
```
N
batch_sizecolumnsfeaturesfiltersfragment_scan_optionserror)r warnskip on_bad_filesc H t d S N)super
__post_init__)self __class__s }/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/datasets/packaged_modules/parquet/parquet.pyr zParquetConfig.__post_init__V s
)__name__
__module__ __qualname____doc__r r int__annotations__r liststrr
datasetsFeaturesr r ds
Expressiontupler ParquetFragmentScanOptionsr r r
__classcell__)r s @r r
r
s ; ;z !%J
$$$#'GXd3i
''',0Hhx()000NRGXeBM4;T%[8I IJ
KRRREI8B$ABIII5z-Parquet._split_generators.. s0 eeeysD#QUQ\QdJdJddJdJdJdr ) r.
data_filesr0 download_configextract_on_the_flydownloaditemsinfor
openr$ r% from_arrow_schemapq read_schemapa ArrowInvalidr loggerr typer warningdebugappendSplitGeneratorr r/ ) r
dl_managerrB splits
split_namer9 filefes ` r _split_generatorszParquet._split_generatorsi s {% w u]a]h]s u uvv v8<
"5 (()?@@
!+!1!1!3!3 a a Jy!) ! i iD i !$ - - "191B1T1TUWUcdeUfUf1g1gDI . !" " " " " " " " " " " " " " " " " ? i i i ;3w>> "LL)r)r)r[_`a[b[b[k)r)rop)r)r s s s ! ![ 5 ? ? "NN+i+i+iQUVWQXQXQa+i+ief+i+i+i j j j j "LL)gd)g)gtTUwwO_)g)gcd)g)g)g h h hi y!) gkgrg}
MM(1zwX]N^___ ` ` ` ` ; *s4;3F/G/G3tyOaKbKb/b/b!)!2eeeeDI,>,D,D,F,Feee" "DI
s= C*=C C* C" "C*%C" &C**G9CGGpa_tablereturnc \ | j j t || j j j }|S r )rG r
r arrow_schema)r r[ s r _cast_tablezParquet._cast_table s+ 9 ) "(DI,>,KLLHr c # K |E d {V d S r r= )r r9 s r _generate_shardszParquet._generate_shards s$ r c # K | j j y| j j mt d | j j j D t | j j k r*t
d| j j d| j j d t | j j t r t j | j j n | j j }t j
| j j }t | D ]\ }} t! |d 5 }| | }|j r| j j p|j d j }t | || j j |dd D ]L\ } }
t, j |
g } t3 || | | fV Md d d n # 1 swxY w Y # t, j t f$ r} | j j d k r6t: d
| d t? | j d | | j j d
k r7t: ! d| dt? | j d | d n6t: " d| dt? | j d | d Y d } ~ d } ~ ww xY wd S )Nc 3 $ K | ] }|j V d S r )r: )r> fields r z+Parquet._generate_tables.. s$ NNUejNNNNNNr z)Tried to load parquet data with columns 'z ' with mismatching features '') default_fragment_scan_optionsr4 r )r r filterbatch_readaheadfragment_readaheadr zFailed to read file 'r5 r6 r zSkipping bad file 'r7 r8 )#r. r
r sortedrG r^ r0
isinstancer r" rJ filters_to_expressionr& ParquetFileFormatr enumeraterH
make_fragment
row_groupsr num_rows
to_batchesrL Table from_batchesr r_ rM r rN r rO r rP rQ )
r r9 filter_exprparquet_file_formatfile_idxrW rX parquet_fragmentr batch_idx record_batchr[ rY s
r _generate_tableszParquet._generate_tables se ; + 0C0ONNdi.@.MNNNNNRXY]YdYlRmRmmm H @S H Hrvr{ sE H H H
$+-t44
%B $T[%8 9 9 9$
! 2QUQ\Qr s s s ' . . Z ZNHd
Z$%% W':'H'H'K'K$'2 W%)[%;%f?O?Z[\?]?f
7@ , 7 7+5(, (;'20134 8 8 8
W
W 3I| (*x'<' G'B9G G'G G'"G #G''K =CKK N) r r r r
BUILDER_CONFIG_CLASSr2 rZ rL rt r_ ra r| r= r r r, r, Z s (
C
C
C DBH 'Z 'Z 'Z 'Z 'Zr r, ) dataclassesr typingr r r pyarrowrL pyarrow.datasetdatasetr& pyarrow.parquetparquetrJ r$ datasets.builderr datasets.tabler utilslogging
get_loggerr rN
BuilderConfigr
ArrowBasedBuilderr, r= r r r s9 ! ! ! ! ! ! + + + + + + + + + + % % % % % %
* *8 4 4 F F F F F H* F F F RbZ bZ bZ bZ bZh( bZ bZ bZ bZ bZr