&`i ddlmZmZmZddlZddlmZddlm Z ddl m Z e dGdd eZ dS) )CallableListOptionalN) Preprocessor)simple_split_tokenizer) PublicAPIalpha) stabilityc eZdZdZdZ d deedeeegeefdeeeffd Z de j fd Z d Z xZS) TokenizeraReplace each string with a list of tokens. Examples: >>> import pandas as pd >>> import ray >>> df = pd.DataFrame({"text": ["Hello, world!", "foo bar\nbaz"]}) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP The default ``tokenization_fn`` delimits strings using the space character. >>> from ray.data.preprocessors import Tokenizer >>> tokenizer = Tokenizer(columns=["text"]) >>> tokenizer.transform(ds).to_pandas() # doctest: +SKIP text 0 [Hello,, world!] 1 [foo, bar\nbaz] If the default logic isn't adequate for your use case, you can specify a custom ``tokenization_fn``. >>> import string >>> def tokenization_fn(s): ... for character in string.punctuation: ... s = s.replace(character, "") ... return s.split() >>> tokenizer = Tokenizer(columns=["text"], tokenization_fn=tokenization_fn) >>> tokenizer.transform(ds).to_pandas() # doctest: +SKIP text 0 [Hello, world] 1 [foo, bar, baz] :class:`Tokenizer` can also be used in append mode by providing the name of the output_columns that should hold the tokenized values. >>> tokenizer = Tokenizer(columns=["text"], output_columns=["text_tokenized"]) >>> tokenizer.transform(ds).to_pandas() # doctest: +SKIP text text_tokenized 0 Hello, world! [Hello,, world!] 1 foo bar\nbaz [foo, bar\nbaz] Args: columns: The columns to tokenize. tokenization_fn: The function used to generate tokens. This function should accept a string as input and return a list of tokens as output. If unspecified, the tokenizer uses a function equivalent to ``lambda s: s.split(" ")``. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. FNcolumnstokenization_fnoutput_columnsct||_|pt|_t j|||_dSN)super__init__r rrr#_derive_and_validate_output_columnsr)selfr rr __class__s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/preprocessors/tokenizer.pyrzTokenizer.__init__BsP  .H2H*N ^  dfcdtjffd }|jddjf||j<|S)Nsc8|jSr)mapr)rrs rcolumn_tokenizerz5Tokenizer._transform_pandas..column_tokenizerQs55-.. .r)pdSerieslocr transformr)rrrs` r_transform_pandaszTokenizer._transform_pandasPs^ /  / / / / / /#%&DL"9"C"CDT"U"U4  rc|t|jd|j}|jjd|jd|d|jdS)N__name__z (columns=z, tokenization_fn=z, output_columns=))getattrrrr%r r)rnames r__repr__zTokenizer.__repr__Wsit+Z9MNN~& O O O O# O O6:6I O O O r)NN)r% __module__ __qualname____doc__ _is_fittablerstrrrrr DataFramer#r) __classcell__)rs@rr r s22hL AE.2   c  "(C5$s)+;"<=  !c+       BL       rr ) typingrrrpandasrray.data.preprocessorrray.data.preprocessors.utilsrray.util.annotationsrr rrr7s++++++++++......??????****** WQ Q Q Q Q Q Q Q Q Q r