`i}/dZddlZddlZddlmZddlmZddlmZej e Z ddd Z d Z Gd d eZd gZdS) z Tokenization classes for BioGPT.N)Optional)PreTrainedTokenizer)loggingz vocab.jsonz merges.txt) vocab_file merges_filec~t}|d}|ddD]}|||f|}|S)z Return set of symbol pairs in a word. word is represented as tuple of symbols (symbols being variable-length strings) rN)setadd)wordpairs prev_charchars /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/biogpt/tokenization_biogpt.py get_pairsr!sP EEEQIQRR 9d#$$$ Lc TeZdZdZeZddgZ dfd Zed Z d Z d Z d Z d Z d dZdZdZdZ d!deedeeedeefdZ d"deedeeededeeffd Zd!dedeedeefdZdZdZxZS)#BioGptTokenizera: Construct an FAIRSEQ Transformer tokenizer. Moses tokenization followed by Byte-Pair Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Merges file. unk_token (`str`, *optional*, defaults to `""`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `""`): The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token. When building a sequence using special tokens, this is not the token that is used for the beginning of sequence. The token used is the `cls_token`. eos_token (`str`, *optional*, defaults to `""`): The end of sequence token. When building a sequence using special tokens, this is not the token that is used for the end of sequence. The token used is the `sep_token`. sep_token (`str`, *optional*, defaults to `""`): The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for sequence classification or for a text and a question for question answering. It is also used as the last token of a sequence built with special tokens. pad_token (`str`, *optional*, defaults to `""`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc  ddl} n#t$rtdwxYwd|_| |_i|_i|_ t |d5} tj| |_ dddn #1swxYwYd|j D|_ t |d5} | ddd} dddn #1swxYwYd | D} tt| t!t#| |_i|_t)jd |||||d |dS) NrzqYou need to install sacremoses to use BioGptTokenizer. See https://pypi.org/project/sacremoses/ for installation.enutf-8encodingci|]\}}|| Sr").0kvs r z,BioGptTokenizer.__init__..xs>>>A1>>>r c`g|]+}t|dd,S)N)tuplesplit)r#merges r z,BioGptTokenizer.__init__..{s1???u% bqb)**???r) bos_token eos_token sep_token unk_token pad_tokenr") sacremoses ImportErrorlangsmcache_moses_tokenizercache_moses_detokenizeropenjsonloadencoderitemsdecoderreadr,dictziprangelen bpe_rankscachesuper__init__)selfrrr2r/r0r1r3kwargsr4 vocab_handle merges_handlemerges __class__s rrHzBioGptTokenizer.__init__\s         M    %'"')$ *w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3>>););)=)=>>> + 0 0 0 ;M"''))//55crc:F ; ; ; ; ; ; ; ; ; ; ; ; ; ; ;?????c&%F *<*<==>>           s,"A99A=A==0C99C=C=c*t|jS)zReturns vocab size)rDr=rIs r vocab_sizezBioGptTokenizer.vocab_sizes4<   rc0t|jfi|jSN)rAr=added_tokens_encoderrPs r get_vocabzBioGptTokenizer.get_vocabsDL>>D$=>>>rc||jvr%|j|}||j|<|j||dddS)Nr6TF)aggressive_dash_splits return_strescape)r8r7MosesTokenizertokenize)rItextr6moses_tokenizers rmoses_tokenizezBioGptTokenizer.moses_tokenizesc t1 1 1"g44$4??O/>D &t ,)$/88 %9   rc||jvr%|j|}||j|<|j||S)NrW)r9r7MosesDetokenizer detokenize)rItokensr6moses_detokenizers rmoses_detokenizez BioGptTokenizer.moses_detokenizesR t3 3 3 $ 8 8d 8 C C 1BD ( .+D1<TcTj|tdS)Ninf)rEgetfloat)pairrIs rz%BioGptTokenizer.bpe..s 1C1CD%PU,,1W1Wrkeyrr r* z z ) r+rFrminrErDindexextend ValueErrorappendjoin) rItokenr rbigramfirstsecondnew_wordijs ` rbpezBioGptTokenizer.bpesU3B3Z  E"I$6#88 DJ  :e$ $$ "6> ! ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:xx~~ :  D  5 sC'C/.C/Fc |r|}n|||j}g}|D]L}|rH|t ||dM|S)zReturns a tokenized string.rp)r,r_r6rslistr~)rIr]bypass_tokenizer split_tokensrws r _tokenizezBioGptTokenizer._tokenizes  8::<)>s)C)C$D$DEEErcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)r=rjr2)rIrws r_convert_token_to_idz$BioGptTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHrcB|j||jS)z=Converts an index (integer) in a token (str) using the vocab.)r?rjr2)rIrrs r_convert_id_to_tokenz$BioGptTokenizer._convert_id_to_tokens|t~666rcd|D}d|}|||j}|S)z:Converts a sequence of tokens (string) in a single string.cbg|],}|dddd-S)rprg)replace)r#ts rr.z.s6JJJa!))C$$,,VS99JJJrr)rvr,rer6)rIrcr]s rconvert_tokens_to_stringz(BioGptTokenizer.convert_tokens_to_stringsOKJ6JJJ&&(($$VTY77 rN token_ids_0 token_ids_1returncB| |jg|zS|jg}||z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A BioGPT sequence has the following format: - single sequence: ` X ` - pair of sequences: ` A B ` Args: token_ids_0 (`List[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `List[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) sep_token_id)rIrrseps r build_inputs_with_special_tokensz0BioGptTokenizer.build_inputs_with_special_tokenss;&  %&4 4 ![ 3&44ralready_has_special_tokensc|r$t||dS|/dgdgt|zzdgzdgt|zzSdgdgt|zzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`List[int]`): List of IDs. token_ids_1 (`List[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `List[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNr r)rGget_special_tokens_maskrD)rIrrrrNs rrz'BioGptTokenizer.get_special_tokens_masks$ & 7722'[]a3   "31#K 0 001QC7A3[AQAQ;QR RsqcC ,,,--rsave_directoryfilename_prefixc ztj|s td|ddStj||r|dzndt dz}tj||r|dzndt dz}t|dd 5}|tj |j d d d dzdddn #1swxYwYd}t|dd 5}t|j dD][\}} || kr td|d| }|d|dz|dz }\ dddn #1swxYwY||fS)NzVocabulary path (z) should be a directory-rrrwrrr*TF)indent sort_keys ensure_asciir'rc|dS)Nr r")kvs rrmz1BioGptTokenizer.save_vocabulary..,s Y[\]Y^rrnzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!rpr )ospathisdirloggererrorrvVOCAB_FILES_NAMESr:writer;dumpsr=sortedrEr>warning) rIrrr merge_filefrrwriter bpe_tokens token_indexs rsave_vocabularyzBioGptTokenizer.save_vocabularysqw}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  W\\ oM_s222QbcpQqq  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c *cG 4 4 4 +1$.2F2F2H2HN^N^+_+_+_  ' KK''NNM MMM(E SXXj11D8999                 :%%s%<4C<<DDBF..F25F2cB|j}d|d<|S)Nr7)__dict__copy)rIstates r __getstate__zBioGptTokenizer.__getstate__8s# ""$$d  rch||_ ddl}n#t$rtdwxYw||_dS)NrznYou need to install sacremoses to use XLMTokenizer. See https://pypi.org/project/sacremoses/ for installation.)rr4r5r7)rIdr4s r __setstate__zBioGptTokenizer.__setstate__=s]          M   s()rrrrr)FrS)NF)__name__ __module__ __qualname____doc__rvocab_files_namesmodel_input_namesrHpropertyrQrUr_rer~rrrrrintrrboolrstrr+rrr __classcell__)rNs@rrr.s((T*$&67 * * * * * * X!!X!???   EEE ***X    III777JN55953;DI3F5 c55552sx..9.3;DI3F.ko. c......6&&c&HSM&]bcf]g&&&&8       rr)rr;rtypingrtokenization_utilsrutilsr get_loggerrrrrr__all__r"rrrs'& 555555  H % %   ZZZZZ)ZZZz  r