&`iddlZddlmZmZmZmZddlZddlZ ddl m Z ddl m Z ejeZe dGdde ZdS) N)AnyDictListOptional) Preprocessor) PublicAPIalpha) stabilityc eZdZdZdZ ddeededeej de d e f fd Z d e j d dfd Zd e j fdZd eefdZd eefdZdZdeeefd dffd ZxZS) Concatenatora Combine numeric columns into a column of type :class:`~ray.air.util.tensor_extensions.pandas.TensorDtype`. Only columns specified in ``columns`` will be concatenated. This preprocessor concatenates numeric columns and stores the result in a new column. The new column contains :class:`~ray.air.util.tensor_extensions.pandas.TensorArrayElement` objects of shape :math:`(m,)`, where :math:`m` is the number of columns concatenated. The :math:`m` concatenated columns are dropped after concatenation. The preprocessor preserves the order of the columns provided in the ``colummns`` argument and will use that order when calling ``transform()`` and ``transform_batch()``. Examples: >>> import numpy as np >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import Concatenator :py:class:`Concatenator` combines numeric columns into a column of :py:class:`~ray.air.util.tensor_extensions.pandas.TensorDtype`. >>> df = pd.DataFrame({"X0": [0, 3, 1], "X1": [0.5, 0.2, 0.9]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> concatenator = Concatenator(columns=["X0", "X1"]) >>> concatenator.transform(ds).to_pandas() # doctest: +SKIP concat_out 0 [0.0, 0.5] 1 [3.0, 0.2] 2 [1.0, 0.9] By default, the created column is called `"concat_out"`, but you can specify a different name. >>> concatenator = Concatenator(columns=["X0", "X1"], output_column_name="tensor") >>> concatenator.transform(ds).to_pandas() # doctest: +SKIP tensor 0 [0.0, 0.5] 1 [3.0, 0.2] 2 [1.0, 0.9] >>> concatenator = Concatenator(columns=["X0", "X1"], dtype=np.float32) >>> concatenator.transform(ds) # doctest: +SKIP Dataset(num_rows=3, schema={Y: object, concat_out: TensorDtype(shape=(2,), dtype=float32)}) When ``flatten=True``, nested vectors in the columns will be flattened during concatenation: >>> df = pd.DataFrame({"X0": [[1, 2], [3, 4]], "X1": [0.5, 0.2]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> concatenator = Concatenator(columns=["X0", "X1"], flatten=True) >>> concatenator.transform(ds).to_pandas() # doctest: +SKIP concat_out 0 [1.0, 2.0, 0.5] 1 [3.0, 4.0, 0.2] Args: columns: A list of columns to concatenate. The provided order of the columns will be retained during concatenation. output_column_name: The desired name for the new column. Defaults to ``"concat_out"``. dtype: The ``dtype`` to convert the output tensors to. If unspecified, the ``dtype`` is determined by standard coercion rules. raise_if_missing: If ``True``, an error is raised if any of the columns in ``columns`` don't exist. Defaults to ``False``. flatten: If ``True``, nested vectors in the columns will be flattened during concatenation. Defaults to ``False``. Raises: ValueError: if `raise_if_missing` is `True` and a column in `columns` or doesn't exist in the dataset. F concat_outNcolumnsoutput_column_namedtyperaise_if_missingflattenct||_||_||_||_||_dSN)super__init__rrrrr)selfrrrrr __class__s w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/preprocessors/concatenator.pyrzConcatenator.__init__YsD  "4 0 dfreturnct|jt|z }|r?d|jd|}|jrt|t|dSdS)NzMissing columns specified in 'z': )setrr ValueErrorloggerwarning)rrmissing_columnsmessages r _validatezConcatenator._validatehs{dl++c"gg5  (SSS/SS $ ( )))w''''' ( (rc|jr.|j}fd|D}n&|jj}|j}t jt||j ddj f<|S)NcPg|]"}tjfd|D#S)cg|]I}jtj|n+tj|jJSr)rnp atleast_1dastype).0elemrs r z=Concatenator._transform_pandas...zs_! :- d+++]40077 CCr)r( concatenate)r+rowrs rr-z2Concatenator._transform_pandas..xs_   %(    r)rr) r$rrto_numpyrdroppdSerieslistlocr)rr concatenateds` r_transform_pandaszConcatenator._transform_pandasss r < Gdl+4466L    (   LLdl+444:4FFL WWT\W * *.0YtL7I7I-J-Jqqq$))* rc|jSrr0rs rget_input_columnszConcatenator.get_input_columnss |rc|jgSr)rr:s rget_output_columnszConcatenator.get_output_columnss'((rcdddddd}g}|D]5\}}t||}||kr||d|6|jjdd|dS)Nr F)rrrrr=(z, ))itemsgetattrappendr__name__join)rdefault_valuesnon_default_arguments parameter default_valuevalues r__repr__zConcatenator.__repr__s". %   !#(6(<(<(>(> E E $I}D),,E %%%,, -C-CE-C-CDDD.)OODII6K,L,LOOOOrstatec|t|t|ds d|_dSdS)NrF)r __setstate__hasattrr)rrMrs rrOzConcatenator.__setstate__sD U###tY'' ! DLLL ! !r)r NFF)rE __module__ __qualname____doc__ _is_fittablerstrrr(rboolrr3 DataFramer$r8r;r=rLrrrO __classcell__)rs@rr r sdFFPL #/$(!&  c   !           (BL (T ( ( ( (BL249)DI))))PPP"!$sCx.!T!!!!!!!!!!rr )loggingtypingrrrrnumpyr(pandasr3ray.data.preprocessorrray.util.annotationsr getLoggerrEr r rrras,,,,,,,,,,,,......******  8 $ $ WZ!Z!Z!Z!Z!<Z!Z!Z!Z!Z!r