Pi&ddlmZddlZddlmZddlmZmZmZm Z ddlm Z ddl m Z ddlmZGddZd Zhd Z dd Z ddZddZdS))SequenceN) _pandas_api)CodecTable concat_tablesschema)_feather) FeatherErrorc0eZdZdZddZd dZdZd dZdS) FeatherDataseta Encapsulates details of reading a list of Feather files. Parameters ---------- path_or_paths : List[str] A list of file names validate_schema : bool, default True Check that individual file schemas are all the same / compatible Tc"||_||_dSN)pathsvalidate_schema)self path_or_pathsrs c/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/pyarrow/feather.py__init__zFeatherDataset.__init__*s" .Nc>t|jd|}|g|_|j|_|jddD]J}t||}|jr||||j|Kt|jS)a, Read multiple feather files as a single pyarrow.Table Parameters ---------- columns : List[str] Names of columns to read from the file Returns ------- pyarrow.Table Content of the file as a table (of columns) rcolumnsN) read_tabler_tablesrrvalidate_schemasappendr)rr_filpathtables rrzFeatherDataset.read_table.s$*Q-999v k JqrrN ' 'DtW555E# 3%%dE222 L   & & & &T\***rc|j|js"td|d|jd|jdS)Nz Schema in z was different. z vs )requals ValueError)rpiecer s rrzFeatherDataset.validate_schemasGst{!!%,// GF%FF $ FF7<|FFGG G G GrcV|||S)a Read multiple Parquet files as a single pandas DataFrame Parameters ---------- columns : List[str] Names of columns to read from the file use_threads : bool, default True Use multiple threads when converting to pandas Returns ------- pandas.DataFrame Content of the file as a pandas DataFrame (of columns) r) use_threadsr to_pandas)rrr&s r read_pandaszFeatherDataset.read_pandasLs2 w//99#:%% %r)Tr)NT)__name__ __module__ __qualname____doc__rrrr)rrr r sl  ////++++2GGG %%%%%%rr c|jdkrdS|jtjtjfvrt d|dt d|d|jd)NrzColumn 'zg' exceeds 2GB maximum capacity of a Feather binary column. This restriction may be lifted in the futurez ' of type zU was chunked on conversion to Arrow and cannot be currently written to Feather format) num_chunkstypeextbinarystringr#)namecols rcheck_chunked_overflowr7`s ~ xCJLL#*,,///0D00011 1  @t @ @sx @ @ @   r>lz4zstd uncompressedctjr?tjr3t|tjjr|}tj|ru|dkrd}n|dkrd}ntdtj ||}|dkr7t|j j D]\}} ||} t| | n|}|dkrit|jtt#|jkrtd|td|td n>|t%jd rd }n%|#|t(vrtd |d t( t+j||||||dS#t.$rBt|t0r+ t3j|n#t2j$rYnwxYwwxYw)a Write a pandas.DataFrame to Feather format. Parameters ---------- df : pandas.DataFrame or pyarrow.Table Data to write out as Feather format. dest : str Local destination path. compression : string, default None Can be one of {"zstd", "lz4", "uncompressed"}. The default of None uses LZ4 for V2 files if it is available, otherwise uncompressed. compression_level : int, default None Use a compression level particular to the chosen compressor. If None use the default compression level chunksize : int, default None For V2 files, the internal maximum size of Arrow RecordBatch chunks when writing the Arrow IPC file format. None means use the default, which is currently 64K version : int, default 2 Feather file version. Version 2 is the current. Version 1 is the more limited legacy format rFr;Nz%Version value should either be 1 or 2)preserve_indexz'cannot serialize duplicate column namesz2Feather V1 files do not support compression optionz0Feather V1 files do not support chunksize option lz4_framer8z compression="z " not supported, must be one of ) compressioncompression_level chunksizeversion)r have_pandas has_sparse isinstancepdSparseDataFrameto_dense is_data_framer#r from_pandas enumeraternamesr7len column_namessetr is_available_FEATHER_SUPPORTED_CODECSr write_feather Exceptionstrosremoveerror) dfdestr?r@rArBr=r ir5r6s rrRrRssr2  " 2{~=>> B $$ a<<"NN \\!NNDEE E!"^DDD a<<$U\%788 2 24Ah&tS1111!|| u! " "SU-?)@)@%A%A A AFGG G  "&'' '  &'' ' !  5#5k#B#B KK%!:::C[CC'@CCDD D ud 1B)2G E E E E E E  dC    $8     s0F!!G-GG-G(%G-'G((G-TFc Bt||||jdd|i|S)a Read a pandas.DataFrame from Feather format. To read as pyarrow.Table use feather.read_table. Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. use_threads : bool, default True Whether to parallelize reading using multiple threads. If false the restriction is used in the conversion to Pandas as well as in the reading from Feather format. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str. **kwargs Additional keyword arguments passed on to `pyarrow.Table.to_pandas`. Returns ------- df : pandas.DataFrame The contents of the Feather file as a pandas.DataFrame )r memory_mapr&r&r.r')sourcerr&r\kwargss r read_featherr_sO6 +JJ ! ! !!* N N7B NFL N NOrctj|||}||St|ts4t dt|jd|D}ttd|r| |}nUttd|r| |}n!d|D}t d|d ||j d kr|Stt||kr|S||S) a Read a pyarrow.Table from Feather format Parameters ---------- source : str file path, or file-like object You can use MemoryMappedFile as source, for explicitly use memory map. columns : sequence, optional Only read a specific set of columns. If not provided, all columns are read. memory_map : boolean, default False Use memory mapping when opening file on disk, when source is a str use_threads : bool, default True Whether to parallelize reading using multiple threads. Returns ------- table : pyarrow.Table The contents of the Feather file as a pyarrow.Table )use_memory_mapr&Nz&Columns must be a sequence but, got {}c,g|]}t|Sr.)r1).0columns r zread_table..s777VDLL777rc|tkSr)intts rzread_table..s crc|tkSr)rTrhs rrjzread_table..s 18rcg|] }|j Sr.)r*)rcris rrezread_table.. s>>>AQZ>>>rz.Columns must be indices or names. Got columns z of types )r FeatherReaderreadrEr TypeErrorformatr1r*allmap read_indices read_namesrBsortedrOselect)r]rr\r&reader column_typesr column_type_namess rrrs* #z{DDDF{{}} gx ( (9@W 67799 987w777L 3!!< 0 011O##G,, S##\ 2 2 3 3O!!'**>>>>>N'.NN:KNNOO O~ G   ( ( ||G$$$r)NNNr;)NTF)NFT)collections.abcrrUpyarrow.pandas_compatr pyarrow.librrrrlibr2pyarrowr pyarrow._featherr r r7rQrRr_rr.rrrsN&%$$$$$ ------000000000000))))))?%?%?%?%?%?%?%?%D    <;;AE*+PPPPf48!OOOO@1%1%1%1%1%1%r