&`i4ddlmZddlmZmZmZmZddlZddl m Z ddl m Z m Z ddlmZerddlmZed Gd d e Zed Gd d e ZdS)Counter) TYPE_CHECKINGCallableListOptionalN) Preprocessor) simple_hashsimple_split_tokenizer) PublicAPI)Datasetalpha) stabilityc eZdZdZdZ d dddeededee egeefdeeeffd Z d e j fd Z d ZxZS)HashingVectorizeraCount the frequency of tokens using the `hashing trick `_. This preprocessors creates a list column for each input column. For each row, the list contains the frequency counts of tokens (for CountVectorizer) or hash values (for HashingVectorizer). For HashingVectorizer, the list will have length ``num_features``. If ``num_features`` is large enough relative to the size of your vocabulary, then each index approximately corresponds to the frequency of a unique token. :class:`HashingVectorizer` is memory efficient and quick to pickle. However, given a transformed column, you can't know which tokens correspond to it. This might make it hard to determine which tokens are important to your model. .. note:: This preprocessor transforms each input column to a `document-term matrix `_. A document-term matrix is a table that describes the frequency of tokens in a collection of documents. For example, the strings `"I like Python"` and `"I dislike Python"` might have the document-term matrix below: .. code-block:: corpus_I corpus_Python corpus_dislike corpus_like 0 1 1 1 0 1 1 1 0 1 To generate the matrix, you typically map each token to a unique index. For example: .. code-block:: token index 0 I 0 1 Python 1 2 dislike 2 3 like 3 The problem with this approach is that memory use scales linearly with the size of your vocabulary. :class:`HashingVectorizer` circumvents this problem by computing indices with a hash function: :math:`\texttt{index} = hash(\texttt{token})`. .. warning:: Sparse matrices aren't currently supported. If you use a large ``num_features``, this preprocessor might behave poorly. Examples: >>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import HashingVectorizer >>> >>> df = pd.DataFrame({ ... "corpus": [ ... "Jimmy likes volleyball", ... "Bob likes volleyball too", ... "Bob also likes fruit jerky" ... ] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> vectorizer = HashingVectorizer(["corpus"], num_features=8) >>> vectorizer.fit_transform(ds).to_pandas() # doctest: +SKIP corpus 0 [1, 0, 1, 0, 0, 0, 0, 1] 1 [1, 0, 1, 0, 0, 0, 1, 1] 2 [0, 0, 1, 1, 0, 2, 1, 0] :class:`HashingVectorizer` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> vectorizer = HashingVectorizer(["corpus"], num_features=8, output_columns=["corpus_hashed"]) >>> vectorizer.fit_transform(ds).to_pandas() # doctest: +SKIP corpus corpus_hashed 0 Jimmy likes volleyball [1, 0, 1, 0, 0, 0, 0, 1] 1 Bob likes volleyball too [1, 0, 1, 0, 0, 0, 1, 1] 2 Bob also likes fruit jerky [0, 0, 1, 1, 0, 2, 1, 0] Args: columns: The columns to separately tokenize and count. num_features: The number of features used to represent the vocabulary. You should choose a value large enough to prevent hash collisions between distinct tokens. tokenization_fn: The function used to generate tokens. This function should accept a string as input and return a list of tokens as output. If unspecified, the tokenizer uses a function equivalent to ``lambda s: s.split(" ")``. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. .. seealso:: :class:`CountVectorizer` Another method for counting token frequencies. Unlike :class:`HashingVectorizer`, :class:`CountVectorizer` creates a feature for each unique token. This enables you to compute the inverse transformation. :class:`FeatureHasher` This preprocessor is similar to :class:`HashingVectorizer`, except it expects a table describing token frequencies. In contrast, :class:`FeatureHasher` expects a column containing documents. FNoutput_columnscolumns num_featurestokenization_fnrct||_||_|pt|_t j|||_dSN) super__init__rrr rr #_derive_and_validate_output_columnsr)selfrrrr __class__s u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/preprocessors/vectorizer.pyrzHashingVectorizer.__init__}sX  (.H2H*N ^  dfc dttdtffd }tjjD]\}}||j}||}g}tj D]9 | fd}d |_ | |:tj |dj||<|S)Ntokensreturnc>fd|D}t|S)Nc:g|]}t|jS)r r).0tokenrs r zKHashingVectorizer._transform_pandas..hash_count..s&WWWu[0ABBWWWrr)r" hashed_tokensrs r hash_countz7HashingVectorizer._transform_pandas..hash_counts+WWWWPVWWWM=)) )rc|Srr&)countsis rz5HashingVectorizer._transform_pandas..s 6!9rhash_axis)rstrrziprrmaprrangernameappendpdconcatvaluestolist) rr r+col output_col tokenizedhashed hash_columnsseriesr.s ` @r_transform_pandasz#HashingVectorizer._transform_pandass *tCy *W * * * * * * #4<1DEE M MOC3 D$899I]]:..FL4,-- , ,$<$<$<$<==)akk ##F++++Y|!<<<CJJLLBzNN rc t|jd|j}|jjd|jd|jd|d|jd S)N__name__ (columns=z, num_features=, tokenization_fn=, output_columns=))getattrrrrFrrrrfn_names r__repr__zHashingVectorizer.__repr__sp$. D>> import pandas as pd >>> import ray >>> from ray.data.preprocessors import CountVectorizer >>> >>> df = pd.DataFrame({ ... "corpus": [ ... "Jimmy likes volleyball", ... "Bob likes volleyball too", ... "Bob also likes fruit jerky" ... ] ... }) >>> ds = ray.data.from_pandas(df) # doctest: +SKIP >>> >>> vectorizer = CountVectorizer(["corpus"]) >>> vectorizer.fit_transform(ds).to_pandas() # doctest: +SKIP corpus 0 [1, 0, 1, 1, 0, 0, 0, 0] 1 [1, 1, 1, 0, 0, 0, 0, 1] 2 [1, 1, 0, 0, 1, 1, 1, 0] You can limit the number of tokens in the vocabulary with ``max_features``. >>> vectorizer = CountVectorizer(["corpus"], max_features=3) >>> vectorizer.fit_transform(ds).to_pandas() # doctest: +SKIP corpus 0 [1, 0, 1] 1 [1, 1, 1] 2 [1, 1, 0] :class:`CountVectorizer` can also be used in append mode by providing the name of the output_columns that should hold the encoded values. >>> vectorizer = CountVectorizer(["corpus"], output_columns=["corpus_counts"]) >>> vectorizer.fit_transform(ds).to_pandas() # doctest: +SKIP corpus corpus_counts 0 Jimmy likes volleyball [1, 0, 1, 1, 0, 0, 0, 0] 1 Bob likes volleyball too [1, 1, 1, 0, 0, 0, 0, 1] 2 Bob also likes fruit jerky [1, 1, 0, 0, 1, 1, 1, 0] Args: columns: The columns to separately tokenize and count. tokenization_fn: The function used to generate tokens. This function should accept a string as input and return a list of tokens as output. If unspecified, the tokenizer uses a function equivalent to ``lambda s: s.split(" ")``. max_features: The maximum number of tokens to encode in the transformed dataset. If specified, only the most frequent tokens are encoded. output_columns: The names of the transformed columns. If None, the transformed columns will be the same as the input columns. If not None, the length of ``output_columns`` must match the length of ``columns``, othwerwise an error will be raised. Nrrr max_featuresrct||_|pt|_||_t j|||_dSr) rrrr rrYr rr)rrrrYrrs rrzCountVectorizer.__init__sX  .H2H(*N ^  rdatasetr r#cdfdjfddjS)Nc dtjdttf fd } |d}d jD}|dD]<}|D]%\}}|D]}|||&=dtd tfd  fd | D}fd t j|DS) Nr r#c:fdfdjDS)Nc|j}|}t|Sr)applyrsumr)r> token_seriesr"r rs rget_token_countsz\CountVectorizer._fit..stat_fn..get_pd_value_counts..get_token_countss8#%c7==1E#F#FL)--//F"6??*rc*i|]}||gSr&r&)r'r>rcs r zVCountVectorizer._fit..stat_fn..get_pd_value_counts.. s*MMM..s334MMMr)r)r rcrs`@rget_pd_value_countszBCountVectorizer._fit..stat_fn..get_pd_value_countssB++++++ NMMM MMMMrpandas) batch_formatc,i|]}|tSr&r)r'r>s rrez9CountVectorizer._fit..stat_fn..sCCCsCCCCr) batch_sizecounternc`tt||Sr)rdict most_common)rkrls rroz:CountVectorizer._fit..stat_fn..most_commons&tG$7$7$:$:;;<<.stat_fn..s8 GT%677rc.i|]\}}||Sr&r&)r'r>r-key_gens rrez9CountVectorizer._fit..stat_fn..s7!S& fr) r:rTrr map_batchesr iter_batchesitemsupdaterSr<r5) rrrf value_counts total_countsbatchr>countersrk top_countsror[rs ` @rstat_fnz%CountVectorizer._fit..stat_fnsz N  Ng N N N N N N#..#(/LDCdlCCCL%22d2CC : :%*[[]]::MC#+::$S)009999:: =W = = = = =+2244J %(z%B%B rc|Srr&)rrr|s rr/z&CountVectorizer._fit..%sGGG$4$4rcd|dS)N token_counts(rJr&)r>s rr/z&CountVectorizer._fit..&s$:C$:$:$:r)r| stat_key_fnr)stat_computation_planadd_callable_statr)rr[r|s``@r_fitzCountVectorizer._fitsc      > "444444::L 5    rr cP g}t|j|jD]\}}|jd|d}d|D}|||jt}g}|D]6 | fd} | _| | 7|r1tj |dj ||<nggt|z||<| ||S)NrrJcg|]\}}|Sr&r&)r'r(counts rr)z5CountVectorizer._transform_pandas..0sTTT~uUTTTrc|Srr&)valr(s rr/z3CountVectorizer._transform_pandas..6s 3u:rr1r2)r5rrstats_ror6rrr8r9r:r;r<r=len) rr result_columnsr>r? token_counts sorted_tokensr@ token_columnsrCr(s @rrDz!CountVectorizer._transform_pandas,s:"4<1DEE . .OC;'=s'='='=>LTT9Q9Q9S9STTTM3 D$899==gFFIM& - -"'='='='=>># $$V,,,, 0!#=q!A!A!A!H!O!O!Q!Q:"$B:  ! !* - - - - rc t|jd|j}|jjd|jd|d|jd|jd S)NrFrGrHz, max_features=rIrJ)rKrrrFrrYrrLs rrNzCountVectorizer.__repr__Csp$. DrsB::::::::::::......LLLLLLLL******)(((((( WW W W W W W W W t W_ _ _ _ _ l_ _ _ _ _ r