&`i4I&ddlmZmZmZmZmZmZddlZddl Z ddl m Z m Z mZmZmZmZddlmZmZddlmZddlmZerddlmZed ed d Gd deZed ed d GddeZed ed d GddeZed ed d GddeZdS)) TYPE_CHECKINGAnyDictListOptionalTupleN)AbsMaxApproximateQuantileMaxMeanMinStd) PreprocessorSerializablePreprocessorBase)SerializablePreprocessor) PublicAPI)Datasetalpha) stabilityz$io.ray.preprocessors.standard_scaler)version identifierceZdZdZddeedeeeffd Zdddefd Z d e j fd Z de eeffd Zd e eefdefdZdZxZS)StandardScalerav Translate and scale each column by its mean and standard deviation, respectively. The general formula is given by .. math:: x' = \frac{x - \bar{x}}{s} where :math:`x` is the column, :math:`x'` is the transformed column, :math:`\bar{x}` is the column average, and :math:`s` is the column's sample standard deviation. If :math:`s = 0` (i.e., the column is constant-valued), then the transformed column will contain zeros. .. warning:: :class:`StandardScaler` works best when your data is normal. If your data isn't approximately normal, then the transformed features won't be meaningful. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import StandardScaler >>> >>> df = pd.DataFrame({"X1": [-2, 0, 2], "X2": [-3, -3, 3], "X3": [1, 1, 1]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 1 1 0 -3 1 2 2 3 1 Columns are scaled separately. >>> preprocessor = StandardScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.224745 -0.707107 1 1 0.000000 -0.707107 1 2 1.224745 1.414214 1 Constant-valued columns get filled with zeros. >>> preprocessor = StandardScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 0.0 1 0 -3 0.0 2 2 3 0.0 >>> preprocessor = StandardScaler( ... columns=["X1", "X2"], ... output_columns=["X1_scaled", "X2_scaled"] ... ) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 -1.224745 -0.707107 1 0 -3 1 0.000000 -0.707107 2 2 3 1 1.224745 1.414214 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Ncolumnsoutput_columnsct||_tj|||_dSNsuper__init__rr#_derive_and_validate_output_columnsrselfrr __class__s q/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/preprocessors/scaler.pyr!zStandardScaler.__init__U@  *N ^  datasetrreturnc|jt|j|jd|j|S)N) aggregator_fnrc$t|dS)Nr)ddof)r)cols r&z%StandardScaler._fit..bsc#A&6&6&6r()stat_computation_planadd_aggregatorr r)r$r)s r&_fitzStandardScaler._fit\s^ "11L 2    "1166L 2    r(dfc|dtjffd }|j||j<|S)Nscjd|jd}jd|jd}||tj|dd<|S|dkrd}||z |z S)Nzmean()zstd(rr)stats_namenpnan)r6s_means_stdr$s r&column_standard_scalerz@StandardScaler._transform_pandas..column_standard_scalerhss[!2!2!2!23FK 0qv 0 0 01E}v!!!zzJ%' 'r(pdSeriesr transformr)r$r4r?s` r&_transform_pandasz StandardScaler._transform_pandasgsT (bi ( ( ( ( ( (#%T\"2"<"<=S"T"T4  r(c@|j|jt|dddSN_fitted)rrrGrrgetattrr$s r&_get_serializable_fieldsz'StandardScaler._get_serializable_fieldsz+|"1tY55   r(fieldsrcn|d|_|d|_|d|_dSNrrrGrrgetrGr$rMrs r&_set_serializable_fieldsz'StandardScaler._set_serializable_fields3i( $%56zz),, r(c@|jjd|jd|jdSN (columns=z, output_columns=r8r%__name__rrrJs r&__repr__zStandardScaler.__repr__,.)mmDLmmUYUhmmmmr(rrY __module__ __qualname____doc__rstrrr!rr3rA DataFramerDrrrKintrSrZ __classcell__r%s@r&rrs AAF  S  8DI;N       I ,    BL& $sCx.    -tCH~-----nnnnnnnr(rz#io.ray.preprocessors.min_max_scalerceZdZdZddeedeeeffd Zdddefd Z d e j fd Z de eeffd Zd e eefdefdZdZxZS) MinMaxScaleraScale each column by its range. The general formula is given by .. math:: x' = \frac{x - \min(x)}{\max{x} - \min{x}} where :math:`x` is the column and :math:`x'` is the transformed column. If :math:`\max{x} - \min{x} = 0` (i.e., the column is constant-valued), then the transformed column will get filled with zeros. Transformed values are always in the range :math:`[0, 1]`. .. tip:: This can be used as an alternative to :py:class:`StandardScaler`. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MinMaxScaler >>> >>> df = pd.DataFrame({"X1": [-2, 0, 2], "X2": [-3, -3, 3], "X3": [1, 1, 1]}) # noqa: E501 >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 1 1 0 -3 1 2 2 3 1 Columns are scaled separately. >>> preprocessor = MinMaxScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 0.0 0.0 1 1 0.5 0.0 1 2 1.0 1.0 1 Constant-valued columns get filled with zeros. >>> preprocessor = MinMaxScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -2 -3 0.0 1 0 -3 0.0 2 2 3 0.0 >>> preprocessor = MinMaxScaler(columns=["X1", "X2"], output_columns=["X1_scaled", "X2_scaled"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 0.0 0.0 1 0 -3 1 0.5 0.0 2 2 3 1 1.0 1.0 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrrct||_tj|||_dSrrr#s r&r!zMinMaxScaler.__init__r'r(r)rr*cZfdttfD}|j|_S)Nc:g|]}jD] }||S)r).0Aggr/r$s r& z%MinMaxScaler._fit..s0MMM3 MMcc#hhMMMMr()r r aggregater9r$r) aggregatess` r&r3zMinMaxScaler._fits6MMMM3*MMM 'g'4  r(r4c|dtjffd }|j||j<|S)Nr6cjd|jd}jd|jd}||z }|dkrd}||z |z S)Nzmin(r8zmax(rrr9r:)r6s_mins_maxdiffr$s r&column_min_max_scalerz=MinMaxScaler._transform_pandas..column_min_max_scalers_K 0qv 0 0 01EK 0qv 0 0 01E5=DqyyI% %r(r@)r$r4rws` r&rDzMinMaxScaler._transform_pandassT &RY & & & & & &#%T\"2"<"<=R"S"S4  r(c@|j|jt|dddSrFrHrJs r&rKz%MinMaxScaler._get_serializable_fieldsrLr(rMrcn|d|_|d|_|d|_dSrOrPrRs r&rSz%MinMaxScaler._set_serializable_fieldsrTr(c@|jjd|jd|jdSrVrXrJs r&rZzMinMaxScaler.__repr__r[r(rr\rds@r&rfrfs ==~  S  8DI;N      I, BL  $sCx.    -tCH~-----nnnnnnnr(rfz#io.ray.preprocessors.max_abs_scalerceZdZdZddeedeeeffd Zdddefd Z d e j fd Z de eeffd Zd e eefdefdZdZxZS) MaxAbsScalera@Scale each column by its absolute max value. The general formula is given by .. math:: x' = \frac{x}{\max{\vert x \vert}} where :math:`x` is the column and :math:`x'` is the transformed column. If :math:`\max{\vert x \vert} = 0` (i.e., the column contains all zeros), then the column is unmodified. .. tip:: This is the recommended way to scale sparse data. If you data isn't sparse, you can use :class:`MinMaxScaler` or :class:`StandardScaler` instead. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import MaxAbsScaler >>> >>> df = pd.DataFrame({"X1": [-6, 3], "X2": [2, -4], "X3": [0, 0]}) # noqa: E501 >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 -6 2 0 1 3 -4 0 Columns are scaled separately. >>> preprocessor = MaxAbsScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.0 0.5 0 1 0.5 -1.0 0 Zero-valued columns aren't scaled. >>> preprocessor = MaxAbsScaler(columns=["X3"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -6 2 0.0 1 3 -4 0.0 >>> preprocessor = MaxAbsScaler(columns=["X1", "X2"], output_columns=["X1_scaled", "X2_scaled"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 -2 -3 1 -1.0 -1.0 1 0 -3 1 0.0 -1.0 2 2 3 1 1.0 1.0 Args: columns: The columns to separately scale. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrrct||_tj|||_dSrrr#s r&r!zMaxAbsScaler.__init__:r'r(r)rr*cFd|jD}|j||_|S)Nc,g|]}t|Srj)r )rkr/s r&rmz%MaxAbsScaler._fit..Bs:::cfSkk:::r()rrnr9ros r&r3zMaxAbsScaler._fitAs-::T\::: 'g'4  r(r4c|dtjffd }|j||j<|S)Nr6cJjd|jd}|dkrd}||z S)Nzabs_max(r8rrrs)r6 s_abs_maxr$s r&column_abs_max_scalerz=MaxAbsScaler._transform_pandas..column_abs_max_scalerGs7 $8qv$8$8$89IA~~ y= r(r@)r$r4rs` r&rDzMaxAbsScaler._transform_pandasFsT !RY ! ! ! ! ! !#%T\"2"<"<=R"S"S4  r(c@|j|jt|dddSrFrHrJs r&rKz%MaxAbsScaler._get_serializable_fieldsTrLr(rMrcn|d|_|d|_|d|_dSrOrPrRs r&rSz%MaxAbsScaler._set_serializable_fields[rTr(c@|jjd|jd|jdSrVrXrJs r&rZzMaxAbsScaler.__repr__br[r(rr\rds@r&r|r|s 99v  S  8DI;N      I, BL     $sCx.    -tCH~-----nnnnnnnr(r|z"io.ray.preprocessors.robust_scalerc eZdZdZdZddefdeedeeefde eede ffd Z d d d e fd Z dejfdZd eeeffdZdeeefde fdZdZxZS) RobustScaleraI Scale and translate each column using approximate quantiles. The general formula is given by .. math:: x' = \frac{x - \mu_{1/2}}{\mu_h - \mu_l} where :math:`x` is the column, :math:`x'` is the transformed column, :math:`\mu_{1/2}` is the column median. :math:`\mu_{h}` and :math:`\mu_{l}` are the high and low quantiles, respectively. By default, :math:`\mu_{h}` is the third quartile and :math:`\mu_{l}` is the first quartile. Internally, the `ApproximateQuantile` aggregator is used to calculate the approximate quantiles. .. tip:: This scaler works well when your data contains many outliers. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import RobustScaler >>> >>> df = pd.DataFrame({ ... "X1": [1, 2, 3, 4, 5], ... "X2": [13, 5, 14, 2, 8], ... "X3": [1, 2, 2, 2, 3], ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> ds.to_pandas() # doctest: +SKIP X1 X2 X3 0 1 13 1 1 2 5 2 2 3 14 2 3 4 2 2 4 5 8 3 :class:`RobustScaler` separately scales each column. >>> preprocessor = RobustScaler(columns=["X1", "X2"]) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 0 -1.0 0.625 1 1 -0.5 -0.375 2 2 0.0 0.750 2 3 0.5 -0.750 2 4 1.0 0.000 3 >>> preprocessor = RobustScaler( ... columns=["X1", "X2"], ... output_columns=["X1_scaled", "X2_scaled"] ... ) >>> preprocessor.fit_transform(ds).to_pandas() # doctest: +SKIP X1 X2 X3 X1_scaled X2_scaled 0 1 13 1 -1.0 0.625 1 2 5 2 -0.5 -0.375 2 3 14 2 0.0 0.750 3 4 2 2 0.5 -0.750 4 5 8 3 1.0 0.000 Args: columns: The columns to separately scale. quantile_range: A tuple that defines the lower and upper quantiles. Values must be between 0 and 1. Defaults to the 1st and 3rd quartiles: ``(0.25, 0.75)``. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. quantile_precision: Controls the accuracy and memory footprint of the sketch (K in KLL); higher values yield lower error but use more memory. Defaults to 800. See https://datasketches.apache.org/docs/KLL/KLLAccuracyAndSize.html for details on accuracy and size. i )g?g?Nrquantile_rangerquantile_precisionct||_||_||_t j|||_dSr)r r!rrrrr"r)r$rrrrr%s r&r!zRobustScaler.__init__sR  ,"4*N ^  r(r)rr*cjddjdgfdjD}|j|}i_jD]<}|d|d\}}}|jd|d<|jd|d<|jd |d<=S) Nrg?rc>g|]}t|jS))on quantilesr)r r)rkr/rr$s r&rmz%RobustScaler._fit..sF    ##'#:      r(zapprox_quantile(r8 low_quantile(median(high_quantile()rrrnr9) r$r)rp aggregatedr/low_qmed_qhigh_qrs ` @r&r3zRobustScaler._fits   "    "      |    'W& 3  < : :C#-.G.G.G.G#H E5&27DK.... /,1DK(#((( )39DK//// 0 0 r(r4c|dtjffd }|j||j<|S)Nr6cjd|jd}jd|jd}jd|jd}||z }|dkrtj|S||z |z S)Nrr8rrr)r9r:r; zeros_like)r6s_low_qs_medians_high_qrvr$s r&column_robust_scalerz.column_robust_scalersk";!&";";";Hg%Dqyy}Q'''LD( (r(r@)r$r4rs` r&rDzRobustScaler._transform_pandassT )BI ) ) ) ) ) )#%T\"2"<"<=Q"R"R4  r(c X|j|j|j|jt |dddS)NrG)rrrrrG)rrrrrIrJs r&rKz%RobustScaler._get_serializable_fieldss7|"1"1"&"9tY55    r(rMrc|d|_|d|_|d|_|d|_|d|_dS)NrrrrrG)rrrrrQrGrRs r&rSz%RobustScaler._set_serializable_fieldssOi( $%56$%56"()=">zz),, r(cP|jjd|jd|jd|jdS)NrWz, quantile_range=z), output_columns=r8)r%rYrrrrJs r&rZzRobustScaler.__repr__sL~& 7 7 7 7"1 7 7"1 7 7 7 r()rYr]r^r_DEFAULT_QUANTILE_PRECISIONrr`rfloatrrbr!rr3rArarDrrrKrSrZrcrds@r&rrfs9IIV"% /;.2"<   c eUl+ !c+         I,2BL" $sCx.    -tCH~-----       r(r)typingrrrrrrnumpyr;pandasrAray.data.aggregater r r r r rray.data.preprocessorrr&ray.data.preprocessors.version_supportrray.util.annotationsrray.data.datasetrrrfr|rrjr(r&rsBBBBBBBBBBBBBBBBOOOOOOOOOOOOOOOOLLLLLLLLKKKKKK******)(((((( W!0VWWWxnxnxnxnxn1xnxnXWxnv W!0UVVVknknknknkn/knknWVkn\ W!0UVVVenenenenen/enenWVenP W!0TUUU_ _ _ _ _ /_ _ VU_ _ _ r(