`idZddlZddlmZddlmZddlmZeje Z ddiZ d Z Gd d eZ d gZdS) zTokenization classes for ESM.N)Optional)PreTrainedTokenizer)logging vocab_file vocab.txtct|d5}|}d|DcdddS#1swxYwYdS)Nrc6g|]}|S)strip).0ls |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/esm/tokenization_esm.py z#load_vocab_file.. s )))a ))))openread splitlines)rfliness rload_vocab_filers j#  *!##%%))5)))******************s2AAAc $eZdZdZeZddgZ dfd Zd ed e fd Z d e d efdZ dZ dZ d e d efdZd ed e fdZ ddeedeeed eefdZ ddedeeded eefdZdZed efdZxZS) EsmTokenizerz& Constructs an ESM tokenizer. input_idsattention_maskc Lt||_tt|j|_dt|jD|_t jd|||||d||j|_| |jdS)Nci|]\}}|| Sr r )rindtoks r z)EsmTokenizer.__init__..7sQQQ(#sS#QQQr) unk_token cls_token pad_token mask_token eos_tokenr ) r all_tokensdict enumerate _id_to_token _token_to_idsuper__init__unique_no_split_tokens _update_trie) selfrr'r(r)r*r+kwargs __class__s rr2zEsmTokenizer.__init__+s**55 4?!;!;<<QQi6P6PQQQ !       '+o# $566666rindexreturncB|j||jSNr/getr'r5r8s r_convert_id_to_tokenz!EsmTokenizer._convert_id_to_tokenG $$UDN;;;rtokencr|j||j|jSr;r0r=r'r5rAs r_convert_token_to_idz!EsmTokenizer._convert_token_to_idJ. $$UD,=,A,A$.,Q,QRRRrc *|Sr;)split)r5textr6s r _tokenizezEsmTokenizer._tokenizeMszz||rcl|j}||j|Sr;)r0copyupdateadded_tokens_encoder)r5 base_vocabs r get_vocabzEsmTokenizer.get_vocabPs3&++-- $3444rcr|j||j|jSr;rCrDs r token_to_idzEsmTokenizer.token_to_idUrFrcB|j||jSr;r<r>s r id_to_tokenzEsmTokenizer.id_to_tokenXr@rN token_ids_0 token_ids_1c|jg}|jg}||j||zS||z|zS|jtd||z|z|z|zS)Nz=Cannot tokenize multiple sequences when EOS token is not set!) cls_token_id eos_token_id ValueError)r5rUrVclsseps r build_inputs_with_special_tokensz-EsmTokenizer.build_inputs_with_special_tokens[sv ! !   ([(([(3..   &\]] ][ 3&4s::rFalready_has_special_tokensc|r|tdfd|DSdgdgt|zzdgz}||dgt|zdgzz }|S)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`list[int]`): List of ids of the first sequence. token_ids_1 (`list[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. NzYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.c*g|]}|jvrdndS)r)all_special_ids)rrAr5s rrz8EsmTokenizer.get_special_tokens_mask..s*WWW%$"666AAAWWWrrar)rZlen)r5rUrVr^masks` rget_special_tokens_maskz$EsmTokenizer.get_special_tokens_maskis$ & X& R XWWW;WWW WsqcC ,,,-3  " QC#k***aS0 0D rctj||r|dznddz}t|d5}|d|jdddn #1swxYwY|fS)N-rw )ospathjoinrwriter,)r5save_directoryfilename_prefixrrs rsave_vocabularyzEsmTokenizer.save_vocabularysW\\.O3c?S3H3Hacgr2rss *c " " 0a GGDIIdo.. / / / 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0}s.A55A9<A9c*t|jSr;)rcr,)r5s r vocab_sizezEsmTokenizer.vocab_sizes4?###r)rrrr r!r;)NF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr2intstrr?rErJrPrRrTlistrr]boolrerqpropertyrs __classcell__)r7s@rrr#s*$&67 7777778<#<#<<<<S#S#SSSS SSSSSS<<<<<<JN ; ;9 ;3;DI3F ; c ; ; ; ;in.6tnae c> $C$$$X$$$$$rr)rwrktypingrtokenization_utilsrutilsr get_loggerrtloggerrxrr__all__r rrrs$# 555555  H % %!;/*** m$m$m$m$m$&m$m$m$`  r