&`iT ddlZddlmZddlmZmZmZmZmZm Z m Z m Z ddl Z ddlZddlmZddlmZmZmZmZmZmZmZmZmZmZddlmZerddlm Z ddl!m"Z"m#Z#ej$e%Z&ed eGd d Z'eGd dZ(de)deefdZ*de)deefdZ+de)deefdZ,dee dee)geefffdZ-de)dddeefdZ.de)dddee dee)geeffdeefdZ/ d0de dde ee)de ee dee)geeffde(fdZ0deded ej"dee)e eej"fffd!Z1d"ee)e2fd#ej d$ej d%eede3f d&Z4 d1d(ed)e ej"d*e)dej5fd+Z6d,ee)ee)e3ffd-e7d#ej d.e8dej9f d/Z:dS)2N) dataclass) TYPE_CHECKINGAnyCallableDictListOptionalTupleUnion)convert_to_pyarrow_array) AggregateFnV2ApproximateQuantileApproximateTopKCountMaxMeanMinMissingValuePercentageStdZeroPercentage) PublicAPI)SchemaDataType TypeCategoryalpha) stabilityceZdZUdZdZejed<ejed<ejed<e e ed<dejfdZ d e j d e j fd Zd Zdejd e d eefdZd e fdZdS)DatasetSummaryzWrapper for dataset summary statistics. Provides methods to access computed statistics. Attributes: dataset_schema: PyArrow schema of the original dataset statistic_stats_matching_column_dtype_stats_mismatching_column_dtypedataset_schemacolumnstablecjddlm} ||S#tt t jf$r}t d|di}|j j D]}| |} |||<0#tt t jf$rNt j t|t j}|||<YwxYwt!j|cYd}~Sd}~wwxYw)zSafely convert a PyArrow table to pandas, handling problematic extension types. Args: table: PyArrow table to convert Returns: pandas DataFrame with converted data r) BlockAccessorz$Direct conversion to pandas failed (z)), attempting column-by-column conversiontypeN)ray.data.blockr' for_block to_pandas TypeError ValueErrorpa ArrowInvalidloggerwarningschemanamescolumnnullslennullpd DataFrame)selfr%r'e result_datacol_namecolnull_cols b/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/stats.py_safe_convert_tablez"DatasetSummary._safe_convert_table:s] 100000 - **511;;== =:r7 - - - NN9q999    K!L. A All8,,A,/MMOOK))!:r?AAA!xCrwyyAAAH,4,>,>,@,@K)))A < ,, , , , , , ,! -sA&/D2 AD-B'&D-'A'DD-DD-'D2-D2dfreturncx|j|jvr||jStjS)zSet the statistic column as index if it exists, else return empty DataFrame. Args: df: DataFrame to set index on Returns: DataFrame with statistic column as index, or empty DataFrame if column missing )STATISTIC_COLUMNr$ set_indexr9r:)r;rCs rA_set_statistic_indexz#DatasetSummary._set_statistic_indexYs5  BJ . .<< 566 6|~~c|||j}|||j}|jr"|jrt j|jgS||}| |j dS)aConvert summary to a single pandas DataFrame. Combines statistics from both schema-matching and schema-changing tables. Note: Some PyArrow extension types (like TensorExtensionType) may fail to convert to pandas when all values in a column are None. In such cases, this method attempts to convert column-by-column, casting problematic columns to null type. Returns: DataFrame with all statistics, where rows are unique statistics from both tables r$Tdrop) rHrBr!r"emptyr9r:rF combine_first reset_index sort_values)r; df_matching df_changingresults rAr,zDatasetSummary.to_pandasfs//  $ $T%F G G  //  $ $T%I J J    A!2 A<)>(?@@@ @**;77    [. / / [d[ # # rIr5c||jjvrdS|||j|g}||diS)aExtract a column from a PyArrow table if it exists. Args: table: PyArrow table to extract from column: Column name to extract Returns: DataFrame with 'statistic' and 'value' columns, or None if column doesn't exist NvaluerK)r3r4rBrFrename)r;r%r5rCs rA_extract_column_from_tablez)DatasetSummary._extract_column_from_tablesR + + +4  % %e , ,d.CV-L Myy&'!2y333rIcfdjjfD}|stddtj|d}d}|jdd | jd }|S) zGet all statistics for a specific column, merging from both tables. Args: column: Column name to get statistics for Returns: DataFrame with all statistics for the column cDg|]}|xS)N)rX).0r%r5rCr;s rA z3DatasetSummary.get_column_stats..sC   55eVDDDQ  RQQrIzColumn 'z' not found in summary tablesT) ignore_indexcn|}t|dkr |jdndS)Nr)dropnar7iloc)seriesnon_nulls rAfirst_non_nullz7DatasetSummary.get_column_stats..first_non_nulls1}}H'*8}}q'8'88=##d BrIF)sortrVrL) r!r"r.r9concatgroupbyrFapplyrPrQ)r;r5dfscombinedrcrTrCs`` @rAget_column_statszDatasetSummary.get_column_statss      14    OMMMMNN N9St444 C C C   T2  ? ? H U> " " []] [. / / [d[ # #  rIN)__name__ __module__ __qualname____doc__rFr/Table__annotations__rliststrrBr9r:rHr,r dictrXrjrIrArrs##%(***%'X---I #Y-----> r|        @4X4'*4 $4444$%s%%%%%%rIrcDeZdZUdZeeefed<eeed<dS)_DtypeAggregatorszContainer for columns and their aggregators. Attributes: column_to_dtype: Mapping from column name to dtype string representation aggregators: List of all aggregators to apply column_to_dtype aggregatorsN) rkrlrmrnrrrrprr rtrIrArvrvsC#s(^###m$$$$$$rIrvr5rDc t|dt|dt|dt|dt |ddt |dgt |t|dgS) aGenerate default metrics for numerical columns. This function returns a list of aggregators that compute the following metrics: - count - mean - min - max - std - approximate_quantile (median) - missing_value_percentage - zero_percentage Args: column: The name of the numerical column to compute metrics for. Returns: A list of AggregateFnV2 instances that can be used with Dataset.aggregate() Fon ignore_nullsTr)r{r|ddofg?)r{ quantilesr{)rrrrrrrrr5s rA_numerical_aggregatorsrs( e,,, T*** vD))) vD))) vDq111v#777&)))&t444  rIct|dt|dt|dt|gS)a|Generate default metrics for temporal columns. This function returns a list of aggregators that compute the following metrics: - count - min - max - missing_value_percentage Args: column: The name of the temporal column to compute metrics for. Returns: A list of AggregateFnV2 instances that can be used with Dataset.aggregate() FrzTr)rrrrrs rA_temporal_aggregatorsrsL e,,, vD))) vD)))&)))  rIcdt|dt|t|dgS)aGenerate default metrics for all columns. This function returns a list of aggregators that compute the following metrics: - count - missing_value_percentage - approximate_top_k (top 10 most frequent values) Args: column: The name of the column to compute metrics for. Returns: A list of AggregateFnV2 instances that can be used with Dataset.aggregate() Frzr )r{k)rrrrs rA_basic_aggregatorsrs< e,,,&)))6R((( rIc6ddlm}m}|jt|jt|jt|jt|jt|j t|j t|j t|j t|j t|jt|jt |jt |jt&iS)aGet default mapping from Ray Data DataType to aggregator factory functions. This function returns factory functions that create aggregators for specific columns. Returns: Dict mapping DataType or TypeCategory to factory functions that take a column name and return a list of aggregators for that column. Examples: >>> from ray.data.datatype import DataType >>> from ray.data.stats import _default_dtype_aggregators >>> mapping = _default_dtype_aggregators() >>> factory = mapping.get(DataType.int32()) >>> aggs = factory("my_column") # Creates aggregators for "my_column" rr)ray.data.datatyperrint8rint16int32int64uint8uint16uint32uint64float32float64boolstringrbinaryTEMPORALrrs rA_default_dtype_aggregatorsrs$98888888  /000011122 /--4# rIdtyperc  |r6tj|jrt |dgS|rt|S|rt|St|S#t$r=}t d|d|d|dt|cYd}~Sd}~wwxYw)a!Get aggregators using heuristic-based type detection. This is a fallback when no explicit mapping is found for the dtype. Args: column: Column name dtype: Ray Data DataType for the column Returns: List of aggregators suitable for the column type Frzz,Could not determine aggregators for column 'z ' with dtype z: z. Using basic aggregators.N) is_arrow_typer/typesis_null_physical_dtyperis_numerical_typeris_temporal_typerr Exceptionr1r2)r5rr<s rA_get_fallback_aggregatorsrDs#*     .RX%5%5e6K%L%L .V%8889 9  $ $ & & .)&11 1  # # % % .(00 0&f-- - *** (6 ( (PU ( (YZ ( ( (   "&)))))))) *s0A B! "B!/"B!B!! C(+2C#C(#C(dtype_agg_mappingcddlm}m}|D]a\}}t ||r||kr ||cSt ||t fr"||r ||cSbt||S)aGet aggregators for a specific column based on its DataType. Attempts to match the dtype against the provided mapping first, then falls back to heuristic-based selection if no match is found. Args: column: Column name dtype: Ray Data DataType for the column dtype_agg_mapping: Mapping from DataType to factory functions Returns: List of aggregators with the column name properly set rr)rrritems isinstancerris_ofr)r5rrrr mapping_keyfactorys rA_get_aggregators_for_dtyperds(98888888!2 7 7 9 9## W k8 , , #+1E1E76?? " " "  lC%8 9 9 #ekk+>V>V #76?? " " " %VU 3 33rIr3rr$cddlm}|std||j}t |t |jz }|rtd|dt }|r8|}|D]\}}||vr|||<n|}i} g} tt|j|j } |D]z} | | } | | turt d| d4|j| }t|| | <| t#| ||{t%| | S) aGenerate aggregators for columns in a dataset based on their DataTypes. Args: schema: A Ray Schema instance columns: List of columns to include. If None, all columns will be included. dtype_agg_mapping: Optional user-provided mapping from DataType to aggregator factories. Each value should be a callable that takes a column name and returns aggregators. This will be merged with the default mapping (user mapping takes precedence). Returns: _DtypeAggregators containing column-to-dtype mapping and aggregators Raises: ValueError: If schema is None or if specified columns don't exist in schema rrz4Dataset must have a schema to determine column typesNzColumns z not found in dataset schemazSkipping field 'z': type is None or unsupported)rwrx)rrr.r4setrcopyrrsziprobjectr1r2 from_arrowrrextendrrv)r3r$rr missing_colsdefaults final_mappingrvrwall_aggs name_to_typenamepa_type ray_dtypes rA_dtype_aggregators_for_datasetrs,+***** QOPPP,w<<#fl"3"33LPNLNNNOOO*++H!)..00 NN$$ % %DAq %%#$ a  %! OHFL&,7788LTTt$ ?g// NNRdRRR S S S 'H'00 #I24MRRSSSS '   rIaggrVagg_typecddlm}|tj|p!|j|}t|ts|s|rq|j|r|j n||nCdtt|D}fdt||DS||fiS)aFormat aggregation result into stat entries. Takes the raw aggregation result and formats it into one or more stat entries. For scalar results, returns a single entry. For list results, expands into multiple indexed entries. Args: agg: The aggregator instance value: The aggregation result value agg_type: PyArrow type of the aggregation result Returns: Dictionary mapping stat names to (value, type) tuples rrNc,g|]}t|Srt)rr)r[idxs rAr\z!_format_stats..s<<<3c#hh<<z!_format_stats..sE#E8&&e&&&;(?rI)rr get_agg_namer/ris_listr is_list_typerrq value_typeranger7r)rrVrrrlabelsrrs @@rA _format_statsrs("+*****!!H ""R&9h&9(&C&C&P&P&R&R%5=\=#x"8,,99;; H    = <<%E *;*;<<.s777C#(C777rI()) rrendswithgetget_target_columnfieldr)r setdefaultadd)rrrrxschema_matchingschema_changingr$ agg_lookupkeyrVrr>r original_typeformatted_stats stat_name stat_value stat_type stats_dicts rA_parse_summary_statsrsf"OOeeG87;777J &&(( U c>>c!2!2> nnS!!  ((**  ##C((-'--h77< 'UH==2A2G2G2I2I U U .I. I$- #=#=? ?I)=TJ ! !)R 0 0 : : H OW 44rIcol_datacol_typer>c|8 tj||S#tjtjf$rYnwxYwt ||pdS)a{Create a PyArrow array with fallback strategies. Uses convert_to_pyarrow_array from arrow_block.py for type inference and error handling when no specific type is provided. Args: col_data: List of column values col_type: Optional PyArrow type to use col_name: Column name for error messages (optional) Returns: PyArrow array Nr(r5)r/arrayArrowTypeErrorr0r )rrr>s rA_create_pyarrow_arrayr/se  8H8444 4!2?3    D  $Hh.B( C CCs 88r all_columnspreserve_typesc|stjiSt|}tj|i}t|D]}g}d}|D]L} ||| vr+|| |\} } || || }7|dM|r$||jvr||j } n|} t|| |||<tj|S)asBuild a PyArrow table from parsed statistics. Args: stats_dict: Nested dict of {stat_name: {col_name: (value, type)}} all_columns: Set of all column names across both tables original_schema: Original dataset schema preserve_types: If True, use original schema types for columns Returns: PyArrow table with statistics N) r/r%sortedkeysrrFappendr4rr)r) rrrr stat_names table_datar>r first_typerrVrrs rA_build_summary_tablerKs#" x|| ))**J 1:>J;''SS # & &I:i000",Y"7"Ax&&&%!)J%%%%  "h/*???&,,X66;HH!H4XxRR 8 8J  rI)NN)Nr);logging dataclassesrtypingrrrrrr r r pandasr9pyarrowr/$ray.air.util.tensor_extensions.arrowr ray.data.aggregater rrrrrrrrrray.util.annotationsrray.data.datasetrrrr getLoggerrkr1rrvrrrrrrrrrranytuplerArrayrrrrorrtrIrArs!!!!!!SSSSSSSSSSSSSSSSSSSSIIIIII                        +*****9''''''88888888  8 $ $ W \\\\\\\ \~  % % % % % % %  %34 +>>#$}*=.stM':*)D $%xtM7J0J'KK%))))X*c***mAT****@4 4 4 ()8SE4 ;N4N+OO4  - 4444F$( @@ X @ d3i @  U- .#]@S9S0T TU@  @@@@F-) -)"-).0k-) #uS"+%& &'-)-)-)-)`45S#X45Y45 45m$ 45  45454545pMODDD&r{3DFIDXDDDD8- S$sEz**+- - Y-  - X - - - - - - rI