&`iA6ddlZddlmZddlmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZededeefd Zeded edefd ZGd d ZGddeZGddeZGddZdeefdZdS)N)deque)AnyCallableDequeDictListOptionalUnion)Dataset) AggregateFnV2) DeveloperAPIvaluereturnc,|dS)z*Tokenize a string using a split on spaces. )split)rs p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/preprocessors/utils.pysimple_split_tokenizerr s ;;s   num_featuresct|}tj|}t |d}||zS)z6Deterministically hash a value into the integer space.)strencodehashlibsha1int hexdigest)rr encoded_value hashed_valuehashed_value_ints r simple_hashr"sQJJ%%''M< ..L<1133R88 l **rc PeZdZdZdddeeefdedeegeffdZdS) BaseStatSpeczEEncapsulates a statistical computation with optional post-processing.c|SNxs rzBaseStatSpec. arpost_process_fnstat_fnr- post_key_fnc0||_||_||_dSr&r.r-r/)selfr.r-r/s r__init__zBaseStatSpec.__init__s! .&rN) __name__ __module__ __qualname____doc__r r rrr3r'rrr$r$srOO %0K ' ' '}h./ '" ' seSj) ' ' ' ' ' 'rr$c ~eZdZdZddddeeeegeffdedeegefdeeffd Z xZ S) AggregateStatSpecz5Represents an AggregateFnV2 spec for a single column.c|Sr&r'r(s rr*zAggregateStatSpec./r+rN)r-column aggregator_fnr-r/r;c^t|||||_dSNr1)superr3r;)r2r<r-r/r; __class__s rr3zAggregateStatSpec.__init__+s< !+#     r) r4r5r6r7r r rrr r3 __classcell__r@s@rr9r9(s?? %0K $   ]HcUM5I,JJK "  seSj)            rr9c eZdZdZdddedeeegefdeeegefdedeef fd ZxZ S) CallableStatSpeczPRepresents a user-defined stat function that operates outside Dataset.aggregate.c|Sr&r'r(s rr*zCallableStatSpec.Dr+rr,r. stat_key_fnr/r-columnsclt|||||_||_dSr>)r?r3rGrF)r2r.rFr/r-rGr@s rr3zCallableStatSpec.__init__>sD _+     &r) r4r5r6r7rr rrr3rArBs@rrDrD;sZZ%0K ' ' ' 'hucz23 ' hucz23 ' " 'c ' ' ' ' ' ' ' ' ' 'rrDc jeZdZdZdZdZddddeegefded e eegefd e ed df d Z d dddege fdedeegefd e eegefd e ed df dZ ded eee ffdZd e efdZd e efdZd e efdZdZdZdS)StatComputationPlana\ Encapsulates a set of aggregators (AggregateFnV2) and legacy stat functions to compute statistics over a Ray dataset. Supports two types of aggregations: 1. AggregateFnV2-based aggregators, which are batch-executed using `Dataset.aggregate(...)`. 2. Callable-based stat functions, executed sequentially (legacy use case). c,t|_dSr&)r _aggregatorsr2s rr3zStatComputationPlan.__init__Xs16rc8|jdSr&)rLclearrMs rresetzStatComputationPlan.reset[s !!!!!rc|Sr&r'r(s rr*zStatComputationPlan.br+rN)r-r/r<r-r/rGrc ||D]8}||}|jt||||9dS)a Registers an AggregateFnV2 factory for one or more columns. Args: aggregator_fn: A callable (typically a lambda or class) that accepts a column name and returns an instance of AggregateFnV2. post_process_fn: Function to post-process the aggregated result. post_key_fn: Optional key generator to use to save aggregation results after post-processing. columns: List of column names to aggregate. )r<r-r/r;N)rLappendr9)r2r<r-r/rGr; agg_instances radd_aggregatorz"StatComputationPlan.add_aggregator^sj"  F(=00L   $ $!".$3 +!       rc|Sr&r'r(s rr*zStatComputationPlan.~r+rr.rFc b|jt|||||p|dS)a Registers a custom stat function to be run sequentially. This supports legacy use cases where arbitrary callables are needed and cannot be run via Dataset.aggregate(). Args: stat_fn: A zero-argument callable that returns the stat. post_process_fn: Function to apply to the result. stat_key_fn: post_key_fn: columns: )r.r-rGrFr/N)rLrSrD)r2r.r-rFr/rGs radd_callable_statz%StatComputationPlan.add_callable_statzsO,    /''6;         rdatasetci}|}|rn|j|}|D]O}|jj}|j||jn|}|||||<P|D]n}||j }|j D]J} | | }|| }|||||<Ko|S)aq Executes all registered aggregators and stat functions. AggregateFnV2-based aggregators are batched and executed via Dataset.aggregate(). Callable-based stat functions are run sequentially. Args: dataset: The Ray Dataset to compute statistics on. Returns: A dictionary of computed statistics. ) _get_aggregate_fn_list aggregate_get_aggregate_specsr.namer/r;r-_get_custom_stat_fn_specsrFrG) r2rYstats aggregators raw_resultspecstat_keypost_keyresultcols rcomputezStatComputationPlan.computes;1133  M**K8J1133 M M<,'3$$T[111! #'"6"6z(7K"L"Lh2244 I ID\\$"233F| I I++C00++C00"&"6"6vh7G"H"Hh I  rc$d|jDS)NcFg|]}t|t|jSr') isinstancer9r..0rcs r z>StatComputationPlan._get_aggregate_fn_list..s;   $ 122 L   rrLrMs rr[z*StatComputationPlan._get_aggregate_fn_lists&  )    rc$d|jDS)Nc<g|]}t|t|Sr')rkr9rls rrnz.s8   *TCT2U2U    rrorMs rr]z(StatComputationPlan._get_aggregate_specs&  !.    rc$d|jDS)Nc<g|]}t|t|Sr')rkrDrls rrnzAStatComputationPlan._get_custom_stat_fn_specs..s8   *TCS2T2T    rrorMs rr_z-StatComputationPlan._get_custom_stat_fn_specsrrrcLt|dkS)Nr)lenr_rMs rhas_custom_stat_fnz&StatComputationPlan.has_custom_stat_fns!4113344q88rcDt|S)z4 Iterates over all AggregatorSpecs. )iterr]rMs r__iter__zStatComputationPlan.__iter__sD--//000r)r4r5r6r7r3rPrrr r rrUrrXr rrhr[r9r]rDr_rwrzr'rrrJrJNs999"""%0K6:  } 45"  hucz23  c  @%0K6:    "c'" "  seSj)  hucz23  c      @#w#4S>####J ](;     d+<&=     40@+A    99911111rrJ callbackscfd}|S)z Wraps a base post-processing function with a sequence of callback functions. Useful when multiple post-processing steps need to be applied in order. c>|}D] }||}|Sr&r')rf processedcbbase_fnr{s rwrapperz$make_post_processor..wrappers4GFOO  & &B9 IIrr')rr{rs`` rmake_post_processorrs)  Nr)r collectionsrtypingrrrrrr r ray.datar ray.data.aggregater ray.util.annotationsr rrobjectrr"r$r9rDrJrr'rrrsDDDDDDDDDDDDDDDDDD,,,,,,------#$s) +v+S+S++++ ' ' ' ' ' ' ' ' &'''''|'''&I1I1I1I1I1I1I1I1X DN      r