Pix2dZddlZddlZddlZddlZddlmZddlmZm Z ddl m Z ddl m Z dejejefdejeddffd ZGd d ZGd d ZdS)z Tokenizer of Unicode characters, grapheme clusters and tailored grapheme clusters (of orthographies) given an orthography profile. N)reader)nfdgrapheme_pattern)errors)Profilepreturnc#@Ktj|d5}|D]D}|}|r,|dst jd|VE ddddS#1swxYwYdS)Nzutf-8)encoding#NFD)pathlibPathopen readlinesstrip startswith unicodedata normalize)rfplines f/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/segments/tokenizer.py iterlinesrs a  w  / /92LLNN 9 9D::<z"Rules.__init__..s,YYY>OdK d++[9YYY)_rules)selfrs r__init__zRules.__init__sYYSXYYY r&r c t|tttt|SN)listrr)clsfnames r from_filezRules.from_file!s/sDYu%5%5 6 6778899r&cL|jD]\}}|||}|Sr+)r'sub)r(sr#r$s rapplyz Rules.apply%s2!% ) ) D+a((AAr&N)r r) __name__ __module__ __qualname____doc__typingTuplestrr) classmethodr/r3rr&rrrsxZv|CH5ZZZZ:::[:r&rc eZdZdZddejejejfdej e gej e fdej e gej e fdej e gej e ffdZ e jdddd d fd e d e d ej ejddedejdde f dZdde fdZdZe jejfdZdZdZdS) TokenizeruW Class for Unicode character and grapheme tokenization. This class provides extended functionality for orthography-specific tokenization with orthography profiles. Parameters ---------- profile : string or pathlib.Path or Profile instance (default = None) Specifies an orthography profile to use. rules : string (default = None) Filename of a rules file. Notes ----- The tokenizer can be used for pure Unicode character and grapheme tokenization, i.e. it uses the Unicode standard grapheme parsing rules, as implemented in the Python regex package by Matthew Barnett, to do basic tokenization with the "\X" grapheme regular expression match. This grapheme match combines one or more Combining Diacritical Marks to their base character. These are called "grapheme clusters" in Unicode parlance. With these functions the Tokenizer is meant to do basic rudimentary parsing for things like generating unigram models (segments and their counts) from input data. When a profile is passed, the Tokenizer reads the orthography profile and calls a helper class to build a tree data structure, which stores the possible Unicode character combinations that are specified in the orthography profile (i.e. tailored grapheme clusters) that appear in the data source. For example, an orthography profile might specify that in source X is a single grapheme (Unicode parlance: tailored grapheme) and therefore it should be chunked as so. Given an orthography profile and some data to tokenize, the process would look like this: input string example: uubo uubo output string example: uu b o # uu b o >>> prf = Profile({'Grapheme': 'uu'}, {'Grapheme': 'b'}, {'Grapheme': 'o'}) >>> t = Tokenizer(profile=prf) >>> t('uubo uubo') 'uu b o # uu b o' See also the test orthography profile and rules in the test directory. An additional method "combine_modifiers" handles the case where there are Unicode Spacing Modifier Letters, which are not explicitly combined to their base character in the Unicode Standard. These graphemes are called "Tailored grapheme clusters" in Unicode. For more information see the Unicode Standard Annex #29: Unicode Text Segmentation: * http://www.unicode.org/reports/tr29/ Additionally, the Tokenizer provides functionality to transform graphemes into associated character(s) specified in additional columns in the orthography profile. A dictionary is created that keeps a mapping between source-specific graphemes and their counterparts (e.g. an IPA column in the orthography profile). Lastly, the Tokenizer can be used to transform text as specified in an orthography rules file. These transformations are specified in a separate file from the orthography profile (that specifics the document specific graphemes, and possibly their IPA counterparts) and the orthography rules should be applied to the output of a grapheme tokenization. In an orthography rules file, rules are given in order as regular expressions, e.g. this rule replaces a vowel followed by an followed by followed by a second vowel with first vowel second vowel, e.g.:: $ (a|á|e|é|i|í|o|ó|u|ú)(n)(\s)(a|á|e|é|i|í|o|ó|u|ú), \1 \2 \4 N errors_stricterrors_replace errors_ignorecd|_t|tr||_n|tj||_|sO|jrH|jjr<|jjj|jjjdzz }|r|}|rt|nd|_ |||d|_ dS)Nz.rules)strictreplaceignore) op isinstancerr/r.parentstemexistsrr'_errors)r(profilerr>r?r@r's rr)zTokenizer.__init__us  gw ' ' 1DGG  '00DG  TW] W])TW]-?(-JKF}} 05?eooe,,,4 #%#   r&F  # rCstringcolumnform)NFCNFKCr NFKDipar)rCrBrDr c  g}|D]} |rI|t | Mjr7|| |j||t | fd | fd|DS)a The main task of a Tokenizer is tokenizing! This is what happens when called. This function determines what to do given any combination of orthography profile and rules or not orthography profile or rules. Parameters ---------- string : str The input string to be tokenized. column : str (default = "graphemes") The column label for the transformation, if specified. form : None or unicode normalization form Normalize return value if form is not None. ipa : bool Tokenize IPA (work in progress) Returns ------- result : str Result of the tokenization. )rOerrorc|}jrj|n|}rt j|n|Sr+)joinrr'r3rr)wordresrPsegment_separatorr(s rppzTokenizer.__call__..pps`#((..4466C,0K@$+##C(((SC7;D;(s333 Dr&c3.K|]}|VdSr+r)r"rYr\s r z%Tokenizer.__call__..s+774bbhh777777r&) splitappendcombine_modifiersgrapheme_clustersrrE transformrJrX) r(rNrOrPrTr[ separatorrrZrYr\s ` ` ` @r__call__zTokenizer.__call__s+FLLNN B BD B 411$2H2HT2S2STTUUUU7BJJtF$,vBVWWYYYYJJt55c$ii@@AAAA E E E E E E E ~~77773777777r&c|fdt|DS)a Given a string as input, return a space-delimited string of Unicode characters (code points rendered as glyphs). Parameters ---------- string : str A Unicode string to be tokenized into graphemes. Returns ------- result : str String returned is space-delimited on Unicode characters and contains "#" to mark word boundaries. The string is in NFD. Notes ----- Input is first normalized according to Normalization Ford D(ecomposition). String returned contains "#" to mark word boundaries. c3BK|]}|VdSr+)rX)r"rYr[s rr^z'Tokenizer.characters..s2[[t/44T::[[[[[[r&)rXrr_)r(rNr[rds ` r characterszTokenizer.characterss>&~~[[[[s6{{GXGXGZGZ[[[[[[r&c*tj|S)a See: Unicode Standard Annex #29: UNICODE TEXT SEGMENTATION http://www.unicode.org/reports/tr29/ Given a string as input, return a list of Unicode graphemes using the "\X" regular expression. Parameters ---------- word : str A Unicode string to be tokenized into graphemes. Returns ------- result : list List of Unicode graphemes in NFD. )rfindallr(rYs rrbzTokenizer.grapheme_clusterss( '---r&c,|js Jd|tjkr0||jjvr"t d||jj||}|tjkr|Sg}|D]} |jj||}n&#t$r|j d|}YnwxYw|Gt|ttfr||v|||S)a Transform a string's graphemes into the mappings given in a different column in the orthography profile. Parameters ---------- word : str The input string to be tokenized. column : str (default = "Grapheme") The label of the column to transform to. Default it to tokenize with orthography profile. Returns ------- result : list of lists Result of the transformation. z3method can only be called with orthography profile.z Column {0} not found in profile.rC)rEr GRAPHEME_COL column_labels ValueErrorformattreeparse graphemesKeyErrorrJrFtupler,extendr`)r(rYrOrVouttokentargets rrczTokenizer.transforms-(wMMMMMw W) ) )fDGt |dvr-t|dkr||z}|dkr||dz|d<qt|dkr$t |dvr|r||dz|d<d}t|dkr|tj|dkrdt|dkr||d}tj|dddkr||z|dz|d<d}7|||zd}S|ddd}d}g}|t|kr}t ||dd vr-|||||dzz|d z }n ||||dz }|t|k}|S) al Given a string that is space-delimited on Unicode grapheme clusters, group Unicode modifier letters with their preceding base characters, deal with tie bars, etc. Parameters ---------- string : str A Unicode string tokenized into grapheme clusters to be tokenized into simple IPA. Lm)iirSkN)iai\)lenreversedrcategoryordr`) r(rsresulttempcountgraphemesegmentsirs rrazTokenizer.combine_modifiers&sEI ++  H QJE8}}!!k&:8&D&D&L&LMMZ77C NNQrs)  ////////$$$$$$9glC/09V5Ec4QUo5V9999$xxxxxxxxxxr&