`iZdZddlmZddlmZddlmZddlmZddl m Z ej e Z d d d d Zd diZGddeZdgZdS)zTokenization classes for Qwen2.)Optional) AddedToken)PreTrainedTokenizerFast)logging)Qwen2Tokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filezqwen/qwen-tokenizericpeZdZdZeZddgZeZ d fd Z d de de e d e e fd Z xZS) Qwen2TokenizerFastac Construct a "fast" Qwen2 tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level Byte-Pair-Encoding. Same with GPT2Tokenizer, this tokenizer has been trained to treat spaces like parts of the tokens so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from transformers import Qwen2TokenizerFast >>> tokenizer = Qwen2TokenizerFast.from_pretrained("Qwen/Qwen-tokenizer") >>> tokenizer("Hello world")["input_ids"] [9707, 1879] >>> tokenizer(" Hello world")["input_ids"] [21927, 1879] ``` This is expected. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Not applicable to this tokenizer. bos_token (`str`, *optional*): The beginning of sequence token. Not applicable for this tokenizer. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskN <|endoftext|>c t|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}tjd|||||||d|dS)NFT)lstriprstripspecial normalized)r r r unk_token bos_token eos_token pad_token) isinstancestrrsuper__init__) selfr r r rrrrkwargs __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.pyrzQwen2TokenizerFast.__init__Ss'$)S)) JyudW\ ] ] ] ] )S)) JyudW\ ] ] ] ] )S)) JyudW\ ] ] ] ] )S)) JyudW\ ] ] ] ]   !#)  save_directoryfilename_prefixreturncb|jj||}t|S)N)name) _tokenizermodelsavetuple)r r%r&filess r#save_vocabularyz"Qwen2TokenizerFast.save_vocabularys+%**>*PPU||r$)NNNrNrr)N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr slow_tokenizer_classrrrr-r/ __classcell__)r"s@r#rr%s''R*$&67)!!!. . . . . . bcHSM]bcf]gr$rN)r3typingrtokenization_utilsrtokenization_utils_fastrutilsrtokenization_qwen2r get_loggerr0loggerr4MAX_MODEL_INPUT_SIZESr__all__rr$r#rBs&%,,,,,,>>>>>>......  H % %&/6aaaaa0aaaH r$