`i&dZddlZddlZddlmZddlmZmZerddlZddl m Z ddl m Z e j eZdd iZd ZGd d ZGd de ZdgZdS)z Tokenization classes for CPMAnt.N)Optional)is_rjieba_availablerequires_backends)PreTrainedTokenizer)logging vocab_filez vocab.txtctj}t|dd5}|}dddn #1swxYwYt |D]\}}|d}|||< |S)z*Loads a vocabulary file into a dictionary.rutf-8encodingN ) collections OrderedDictopen readlines enumeraterstrip)r vocabreadertokensindextokens /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/cpmant/tokenization_cpmant.py load_vocabr$s  # % %E j# 0 0 0$F!!##$$$$$$$$$$$$$$$!&)) u T""e LsAA A ceZdZddZdZdS)WordpieceTokenizerc0||_||_||_dSN)r unk_tokenmax_input_chars_per_word)selfrr#r$s r__init__zWordpieceTokenizer.__init__0s "(@%%%ct|}t||jkr|jgSd}g}|t|krt|}d}||kr4d|||}||jvr|}n |dz}||k4| ||j|dz }n|||}|t|k|S)Nr)listlenr$r#joinrappend)r%rcharsstart sub_tokensend cur_substrsubstrs rtokenizezWordpieceTokenizer.tokenize5sU  u::5 5 5N# # c%jj  e**CJ#++uSy!122TZ''!'Jq #++ !!!$.111 !!*---c%jj  r'N)rr )__name__ __module__ __qualname__r&r5r'rrr/s;AAAA r'rc eZdZdZeZddgZdZ d%fd Ze dZ e dZ e dZ e de fdZdZdZfdZdZdeedefdZdZdZd&dedeedeefdZ d&d ee d!eee dee fd"Z d'd ee d!eee d#edee ffd$ ZxZS)(CpmAntTokenizera Construct a CPMAnt tokenizer. Based on byte-level Byte-Pair-Encoding. Args: vocab_file (`str`): Path to the vocabulary file. bod_token (`str`, *optional*, defaults to `""`): The beginning of document token. eod_token (`str`, *optional*, defaults to `""`): The end of document token. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. pad_token (`str`, *optional*, defaults to `""`): The token used for padding. unk_token (`str`, *optional*, defaults to `""`): The unknown token. line_token (`str`, *optional*, defaults to `""`): The line token. space_token (`str`, *optional*, defaults to `""`): The space token. input_idsattention_maskFrleftc .t|dg||_||_t||_|j| |jd<|j||jd<|j| =|j|=t jt|jd|_d|jD|_ t|j||_ tj d |||||||| | d | dS) Nrjieba rc|dSNr*r9xs rz*CpmAntTokenizer.__init__.. Z[\]Z^r'keyci|]\}}|| Sr9r9).0kvs r z,CpmAntTokenizer.__init__..s>>>A1>>>r')rr#) bod_token eod_token bos_token eos_token pad_tokenr# line_token space_token padding_sider9)rrVrWrencoderrrsorteditemsdecoderrwordpiece_tokenizersuperr&) r%r rVrWrXrYrZr#r[r\r]kwargs __class__s rr&zCpmAntTokenizer.__init__ls. $ +++""!*--  L5 S!\*5 T L % L $".vdl6H6H6J6JP^P^/_/_/_`` >>););)=)=>>> #5DLT]#^#^#^  !#%  r'c&|j|jSr")r^rVr%s r bod_token_idzCpmAntTokenizer.bod_token_id|DN++r'c&|j|jSr")r^rWrgs r eod_token_idzCpmAntTokenizer.eod_token_idrir'c|jdS)Nrr^rgs r newline_idzCpmAntTokenizer.newline_ids|D!!r'returnc*t|jSr")r,r^rgs r vocab_sizezCpmAntTokenizer.vocab_sizes4<   r'c0t|jfi|jSr")dictr^added_tokens_encoderrgs r get_vocabzCpmAntTokenizer.get_vocabsDL>>D$=>>>r'cg}tj|dD]/}||j|0|S)zTokenize a string.F)rGcutextendrbr5)r%text output_tokensrLs r _tokenizezCpmAntTokenizer._tokenizesR D%(( G GA  !9!B!B1!E!E F F F Fr'c nd|D}fd|D}tj|fi|S)zDecode ids into a string.cg|] }|dk| S)rr9)rRis r z+CpmAntTokenizer._decode..s4441Q!VVQVVVr'cVg|]%}|jk |jk|jk#|&Sr9) pad_token_id eos_token_id bos_token_id)rRrLr%s rrz+CpmAntTokenizer._decode..sI   A):$:$:qDDU?U?UZ[_c_pZpZpAZpZpZpr')rc_decode)r% token_idsrdres` rrzCpmAntTokenizer._decodes_44 444         uwwy33F333r'c||jvSr"rmr%rs rcheckzCpmAntTokenizer.checks $$r'rc,d|S)Nr))r-)r%rs rconvert_tokens_to_stringz(CpmAntTokenizer.convert_tokens_to_stringswwvr'cr|j||j|jS)z0Converts a token (str) in an id using the vocab.)r^getr#rs r_convert_token_to_idz$CpmAntTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHr'cB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)rarr#)r%rs r_convert_id_to_tokenz$CpmAntTokenizer._convert_id_to_tokens|t~666r'Nsave_directoryfilename_prefixctj|r6tj||r|dzndtdz}n |r|dznd|z}d}d|jvr|jd|jd<|jd=d|jvr|jd|jd<|jd=t jt|j d |_t|d d 5}|j D]H\}}||kr t d|d|}| |dz|dz }I dddn #1swxYwY|fS)N-r)r rrHrDrrCc|dSrJr9rKs rrMz1CpmAntTokenizer.save_vocabulary..rNr'rOwr r zSaving vocabulary to z\: vocabulary indices are not consecutive. Please check that the vocabulary is not corrupted!r*)ospathisdirr-VOCAB_FILES_NAMESr^rrr_r`rloggerwarningwrite)r%rrr rwriterr token_indexs rsave_vocabularyzCpmAntTokenizer.save_vocabularys 7== ( ( ]/!Q3!6!6rUfgsUt tJJ4CJ/C//n\J $,  #'<#4DL  S! 4<  #'<#5DL  T"".vdl6H6H6J6JP^P^/_/_/_`` *cG 4 4 4 &*l&8&8&:&:  "{K''NNN NNN(E UT\***                 }sA#E22E69E6 token_ids_0 token_ids_1cJ| |jg|zS|jg|z|jgz|zS)a1 Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CPMAnt sequence has the following format: - single sequence: `[BOS] Sequence`. Args: token_ids_0 (`list[int]`): The first tokenized sequence that special tokens will be added. token_ids_1 (`list[int]`): The optional second tokenized sequence that special tokens will be added. Returns: `list[int]`: The model input with special tokens. )r)r%rrs r build_inputs_with_special_tokensz0CpmAntTokenizer.build_inputs_with_special_tokenss<  %&4 4!"[0D4E3FFTTr'already_has_special_tokensc|r$t||dS|/dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNr*r)rcget_special_tokens_maskr,)r%rrrres rrz'CpmAntTokenizer.get_special_tokens_masks" & 7722'[]a3   "31#K 0 001QC7A3[AQAQ;QR RsqcC ,,,--r') r>r?r@rArBrrCrDrEr")NF)r6r7r8__doc__rvocab_files_namesmodel_input_namesadd_prefix_spacer&propertyrhrkrnintrqrur{rrr+strrrrrtuplerrboolr __classcell__)res@rr;r;Osi0*$&67 ( ( ( ( ( ( T,,X,,,X,""X"!C!!!X!???44444%%%tCySIII777cHSM]bcf]g8JNUU9U3;DI3FU cUUUU*sx..9.3;DI3F.ko. c..........r'r;)rrrtypingrtransformers.utilsrrrGtokenization_utilsrutilsr get_loggerr6rrrrr;__all__r9r'rrs%'& EEEEEEEEMMM555555  H % %!;/@~.~.~.~.~.)~.~.~.B  r'