Pi:'ddlmZmZmZmZddlmZmZmZm Z ddl m Z m Z m Z mZmZieidddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(ZGd)d*eZd+S),)DictListOptionalTuple)ChatMLTemplateMessagePromptTemplatetruncate)&DEFAULT_QWEN2_TOKENIZER_BPE_CACHE_SIZE ENDOFTEXTIM_ENDQWEN2_SPECIAL_TOKENSQwen2Tokenizerz<|object_ref_start|>i^Pz<|object_ref_end|>i_Pz <|box_start|>i`Pz <|box_end|>iaPz<|quad_start|>ibPz <|quad_end|>icPz<|vision_start|>idPz<|vision_end|>iePz<|vision_pad|>ifPz <|image_pad|>igPz <|video_pad|>ihP iiP ijPz<|fim_prefix|>ikPz<|fim_middle|>ilPz<|fim_suffix|>imPz <|fim_pad|>inPioPipP)z <|repo_name|>z <|file_sep|>ceZdZdZedfddddeeeddedede ee fde e d e e d ed e ed e ed ede ede ffdZ dddeededeee eeffdZdZdZdZdZxZS)Qwen2_5Tokenizerad This class construct a Qwen2.5 tokenizer, based on GPT-2 byte-level BPE tokenization. See and . Args: path (str): Path to vocab.json file. merges_file (str): Path to merges.txt file. merges.txt contains all BPE merge operations, and this file is required to split a single word into byte-level BPE tokens. special_tokens (Dict[str, int]): Special tokens to add to the tokenizer. Default is QWEN2_5_SPECIAL_TOKENS. max_seq_len (Optional[int]): A max sequence length to truncate tokens to. Default: None prompt_template (Optional[PromptTemplate]): template used to format the messages based on their role. This is used to add structured text around the actual messages. The structured text is used in three scenarios: - Task-specific templates to gear models for a particular task that it will expect after training - Model-specific templates that are required whenever the model is prompted, such as the [INST] tags in Llama2 and in Mistral - Community standardized templates, such as :class:`~torchtune.data.ChatMLTemplate` The extra text will still get tokenized as normal text, not as special tokens. Default: None errors (str): Paradigm to follow when decoding bytes to UTF-8. Defaults to "replace". See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (Optional[str]): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Defaults to ``<|endoftext|>``. bos_token (Optional[str]): The beginning of sequence token. Defaults to None. eos_token (str): The end of sequence token. Defaults to ``<|endoftext|>``. pad_token (Optional[str]): The token used for padding. Defaults to ``<|endoftext|>``. bpe_cache_size (int): BPE token cache size in Qwen2Tokenizer. NOTE: large cache size will speed up tokenization, but the cache object will get really large for long running processes (esp. for texts of language that do not use space between word, e.g. Chinese); technically not a memory leak but appears as one. By default, we set the cache size equals to size of the official Qwen2 tokenizer. Example: >>> tokenizer = Qwen2Tokenizer( path="/path/to/vocab.json", merges_file="/path/to/merges.txt", special_tokens=QWEN2_SPECIAL_TOKENS) >>> tokenized_text = tokenizer.encode("Hello world!") >>> print(tokenized_text) [39, 385, 78, 675, 0, 2000] Nreplace)prompt_templateerrors unk_token bos_token eos_token pad_tokenbpe_cache_sizepath merges_filespecial_tokens max_seq_lenrrrrrrrc t||||||||| | |  |jd|_|jd|_dS)N) rrrrrrrrrrrrr)super__init__rtool_call_start_idtool_call_end_id) selfrrrrrrrrrrr __class__s w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/qwen2_5/_tokenizer.pyr"zQwen2_5Tokenizer.__init__Wso #)#+)  #'"5m"D $ 3N CT)add_eosmessagesr)returnc t|jtr Jd|j||n|}g}g}t|D]\}}|||}|jD]W} | ddkr2||| ddd@td| d|| ||||||j gt|z|j rt||j krn|r5| |j| |d |j r9t||j |r|jnd}t||j |rd nd}||fS) a Given a list of messages, return a list of tokens for the concatenated and formatted messages. Args: messages (List[Message]): The message list to tokenize. add_eos (bool): Wether to add the tokenizer's eos_id at the end of the sequence of messages. Default is True. Returns: Tuple[List[int], List[bool]]: The list of token ids and the list of masks. Raises: RuntimeError: If a message contains non-text content zUsing ChatMLTemplate with tokenize_messages will result in multiple <|im_*|> tokens wrapping each message.Please use a different template or set to None.NtypetextcontentFadd_bosr)z"Unsupported message content type: T) isinstancerr enumerate_tokenize_headerr/extendencode RuntimeError_tokenize_footermaskedlenrappendeos_idr ) r%r*r)templated_messagestokenized_messagesmaskimessagetokensitems r'tokenize_messagesz"Qwen2_5Tokenizer.tokenize_messagesws3*d2NCC   >  C #/   * * *   #$677  JAw**+=qAAF   <6))MM  O$)$)$'KT&\KK MM$//0BAFF G G G  % %f - - - KK(3v;;6 7 7 7 C(:$;$;t?O$O$O  "  % %dk 2 2 2 KKR ! ! !   O!)"D$4W6VdkkRV"" D$"2G4MDDNND!4''r(c$g}||}|jdkr|dks||dz jdkrB||d||dddn||dddnr|||j|jd krL|jrE||j||d dd|S) Nipythonruserz Fr0z assistant )role_add_message_start_tokensr6r7rGr<r#r%r*rArCrBs r'r5z!Qwen2_5Tokenizer._tokenize_headers!1+ <9 $ $Avv!a%-::..vv>>> KK 3UEKRR KK 5ueKTT  * *67< @ @ @|{**w* d5666 dkk$ukMMNNN r(cxg}||}|jdkr|t|dz ks||dzjdkrA||ddd||n||dddn|jdkrL|jrE||ddd||j|jdks|t|dz kr|||S)NrGrHz Fr0rJrK)rLr;r6r7_add_message_end_tokensrGr<r$rNs r'r9z!Qwen2_5Tokenizer._tokenize_footersH1+ <9 $ $CMMA%%%!a%)=)J)J KK 4eUKSS,,V4444 KK 4eUKSS|{**w* dkk$ukMMNNN d3444|{**a3x==13D.D.D,,V444 r(c||j|||ddddSNrKFr0)r< im_start_idr6r7)r%rCrLs r'rMz*Qwen2_5Tokenizer._add_message_start_tokenssG d&''' dkkT+++uekLLMMMMMr(c||j||ddddSrR)r< im_end_idr6r7)r%rCs r'rPz(Qwen2_5Tokenizer._add_message_end_tokenss@ dn%%% dkk$ukEEFFFFFr()__name__ __module__ __qualname____doc__QWEN2_5_SPECIAL_TOKENSr r r strrintrr r"rrboolrrEr5r9rMrP __classcell__)r&s@r'rr*s**`*@%) D59#'#'#,DDDDDDS#X D c] D".1DDC=DC=DDC=DDDDDDDH J(J(J(w-J( J( tCy$t*$ % J(J(J(J(X**NNNGGGGGGGr(rN)typingrrrrtorchtune.datarrr r !torchtune.models.qwen2._tokenizerr r r rrrZrr(r'rcs /...........LLLLLLLLLLLLF&V  6  f FffVV6Ff f!"f#$6%&)0IGIGIGIGIG~IGIGIGIGIGr(