.`iUddlZddlZddlmZddlmZddlmZmZm Z ddl m Z ddl m Z ee zZeed<d ed efd ZGd d e ZdS)N)Path) TypeAlias) AutoTokenizerPreTrainedTokenizerPreTrainedTokenizerFast))get_sentence_transformer_tokenizer_config) TokenizerLike HfTokenizer tokenizerreturnctj}jjt t tdrFtj t5t j dddn #1swxYwYGfddj }dj j |_ ||_ |S)z By default, transformers will recompute multiple tokenizer properties each time they are called, leading to a significant slowdown. This proxy caches these properties for faster access. vocab_sizeNceZdZedeeffd Zedeeffd Zedeffd Z de eefffd Z deffd Z fdZ dS) -get_cached_tokenizer..CachedTokenizerr cSN)selftokenizer_all_special_idss f/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/tokenizers/hf.pyall_special_idsz=get_cached_tokenizer..CachedTokenizer.all_special_ids(s , ,cSrr)rtokenizer_all_special_tokenss rall_special_tokensz@get_cached_tokenizer..CachedTokenizer.all_special_tokens,s / /rcSrr)r max_token_ids rrz:get_cached_tokenizer..CachedTokenizer.max_token_id0s  rcSrr)rtokenizer_vocabs r get_vocabz7get_cached_tokenizer..CachedTokenizer.get_vocab4s " "rcSrr)r tokenizer_lens r__len__z5get_cached_tokenizer..CachedTokenizer.__len__7s  rctffSr)get_cached_tokenizer)rr s r __reduce__z8get_cached_tokenizer..CachedTokenizer.__reduce__:s')5 5rN)__name__ __module__ __qualname__propertylistintrstrrrdictr!r$r')rr rrr#r srCachedTokenizerr's  -T#Y - - - - -  -  0S  0 0 0 0 0  0  #        #tCH~ # # # # # # !S ! ! ! ! ! ! 6 6 6 6 6 6 6rr0Cached)copyrrr!lenmaxvalueshasattr contextlibsuppressNotImplementedErrorr __class__r()r cached_tokenizerr0rrrr#r s` @@@@@rr&r&sx y++ ) 9#,#? ))++O NNM--//00L y,''C  !4 5 5 C C|Y-ABBL C C C C C C C C C C C C C C C666666666666)-666, G (;(DFFO!0 sB88B<?B<c VeZdZedddddeezdededzdedzdef d ZdS) CachedHfTokenizerFN)trust_remote_coderevision download_dirpath_or_repo_idr>r?r@r c tj|g|R|||d|}nJ#t$r=}|s4dt|vsdt|vrd} t | ||d}~wwxYwt ||} t | trN| ddr8d|j D} | | t|S)N)r>r? cache_dirz,does not exist or is not currently imported.z*requires you to execute the tokenizer filezFailed to load the tokenizer. If the tokenizer is a custom tokenizer not yet available in the HuggingFace transformers library, consider setting `trust_remote_code=True` in LLM or using the `--trust-remote-code` flag in the CLI. do_lower_caseFc>i|]\}}||Sr)lower).0kvs r z5CachedHfTokenizer.from_pretrained..rs3"""!%A17799"""r) rfrom_pretrained ValueErrorr. RuntimeErrorr isinstancer/getspecial_tokens_mapitemsadd_special_tokensr&) clsrAr>r?r@argskwargsr eerr_msgencoder_configrPs rrKz!CachedHfTokenizer.from_pretrainedDsS %5#4!&   II   % >#a&&HH?3q66IIA#7++2# *C X   nd + + =0B0B U1 1  ="")2)E)K)K)M)M"""   ( (); < < <#I...s A#8AA#) r(r)r* classmethodr.rboolr rKrrrr=r=Cs #(##' 2/2/2/t2/ 2/ * 2/ Dj 2/ 2/2/2/[2/2/2/rr=)r7r2pathlibrtypingr transformersrrrvllm.transformers_utils.configrprotocolr r __annotations__r&r=rrrras TTTTTTTTTTTTTTTT######,/FF YFFF/K/K////d4/4/4/4/4/ 4/4/4/4/4/r