Pi=xddlmZmZmZmZmZddlmZmZddl m Z ddl m Z m Z mZgdZGdde e ZdS) )AnyListMappingOptionalTuple)MessagePromptTemplate) Transform)ModelTokenizerSentencePieceBaseTokenizer#tokenize_messages_no_special_tokens)     c neZdZdZ ddedeedeefdZe dZ e dZ e d Z e d Z dd ededededeef dZdeedefdZd ddeededeeeeeffdZ ddeeefdedeeeffdZdS)GemmaTokenizera Gemma's implementation of the SentencePiece tokenizer Args: path (str): Path to pretrained tokenizer file. max_seq_len (Optional[int]): A max sequence length to truncate tokens to. Default: None prompt_template (Optional[PromptTemplate]): template used to format the messages based on their role. This is used to add structured text around the actual messages. The structured text is used in three scenarios: - Task-specific templates to gear models for a particular task that it will expect after training - Model-specific templates that are required whenever the model is prompted, such as the [INST] tags in Llama2 and in Mistral - Community standardized templates, such as :class:`~torchtune.data.ChatMLTemplate` The extra text will still get tokenized as normal text, not as special tokens. Default is None. Examples: >>> tokenizer = GemmaTokenizer("/path/to/spm_model") >>> tokenized_text = tokenizer.encode("Hello world!", add_bos=True, add_eos=True) >>> print(tokenized_text) [1, 31587, 29644, 102, 2] Npath max_seq_lenprompt_templatec|t||_d|j_|jg|_||_||_dS)Nr)r _spm_modelpad_ideos_id stop_tokensrr)selfrrrs u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/gemma/_tokenizer.py__init__zGemmaTokenizer.__init__-sC 5T::"#!K=&.c|jjSN)rrrs rrzGemmaTokenizer.eos_id? %%r c|jjSr")rbos_idr#s rr&zGemmaTokenizer.bos_idCr$r c|jjSr")rrr#s rrzGemmaTokenizer.pad_idGr$r c|jjSr")r vocab_sizer#s rr)zGemmaTokenizer.vocab_sizeKs ))r TFtextadd_bosadd_eostrim_leading_whitespacereturnc>|j||||S)N)r+r,r-)rencode)rr*r+r,r-s rr0zGemmaTokenizer.encodeOs/%% $; &   r token_idsc6|j|Sr")rdecode)rr1s rr3zGemmaTokenizer.decode]s%%i000r )r,messagesc|j||n|}t|||j|r|jndS)aTokenize a list of messages one at a time then concatenate them, returning a list of tokens and a list of masks. Example: >>> tokenizer = GemmaTokenizer(tokenizer_path, max_seq_len) >>> messages = [ Message(role="system", content="system message\n", masked=True), Message(role="user", content="user prompt\n", masked=True), Message(role="assistant", content="assistant response\n"), ] >>> # tokenize_messages encodes messages separately and concats >>> tokenizer.tokenize_messages(messages)[0] [1, 1788, 2643, 13, 1792, 9508, 13, 465, 22137, 2933, 2] >>> # Same result as encoding the full string in one go >>> tokenizer.encode(''.join([message.content for message in messages])) [1, 1788, 2643, 13, 1792, 9508, 13, 465, 22137, 2933, 2] Args: messages (List[Message]): A list of messages, each containing role, content, and masked attributes. add_eos (bool): Whether to append EOS after assistant message, default to True Returns: Tuple[List[int], List[bool]]: The tokenized messages N) tokenizerr4r&r)rr r&r)rr4r,templated_messagess rtokenize_messagesz GemmaTokenizer.tokenize_messagescs^L#/   * * *  3';")34;;t     r sample inferencect|d}||\}}||d<||d<|S)a Apply ``tokenize_messages`` to the "messages" field in the sample. Args: sample (Mapping[str, Any]): A sample with a "messages" field containing a List[Message] to tokenize inference (bool): Whether the template is being used for inference or not. Returns: Mapping[str, Any]: The sample with added "tokens" and "mask" fields and the "messages" field removed. r4tokensmask)popr8)rr9r:r4r<r=s r__call__zGemmaTokenizer.__call__sD::j))--h77 !xv r )NN)TTF)F)__name__ __module__ __qualname____doc__strrintr rpropertyrr&rr)boolrr0r3rrr8rrr?r rrrs6&*48 ///c]/".1 ////$&&X&&&X&&&X&**X* (-         "&  c     191 1111 . . . w-.  . tCy$t*$ % . . . . brNs7666666666666622222222222222 100RRRRR^YRRRRRr