Pi( ddlmZmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZmZddd d d d d ddddd ZGddeeZdS))AnyDictListMappingOptionalTuple)Message)PromptTemplate)truncate) Transform)ModelTokenizerSentencePieceBaseTokenizer}i}i}i}i}i}i}i}i}i }i }) <|endoftext|> <|assistant|>z<|placeholder1|>z<|placeholder2|>z<|placeholder3|>z<|placeholder4|> <|system|><|end|>z<|placeholder5|>z<|placeholder6|><|user|>c peZdZdZ ddedeeeefdeedeefdZ e dZ e d Z dd ed e de de deef dZddeede defdZd d ddeede de deeeee ffdZ ddeeefde deeeffdZdS) Phi3MiniTokenizera SentencePiece tokenizer configured with Phi3 Mini's special tokens. Args: path (str): Path to pretrained tokenizer file. special_tokens (Optional[Dict[str, int]]): mapping containing special text tokens and their registered token IDs. If left as None, this will be set to the canonical Phi3 special tokens. max_seq_len (Optional[int]): A max sequence length to truncate tokens to. Default: None prompt_template (Optional[PromptTemplate]): template used to format the messages based on their role. This is used to add structured text around the actual messages. The structured text is used in three scenarios: - Task-specific templates to gear models for a particular task that it will expect after training - Model-specific templates that are required whenever the model is prompted, such as the [INST] tags in Llama2 and in Mistral - Community standardized templates, such as :class:`~torchtune.data.ChatMLTemplate` The extra text will still get tokenized as normal text, not as special tokens. Default is None. Examples: >>> tokenizer = Phi3MiniTokenizer("/path/to/spm_model") >>> tokenized_text = tokenizer.encode("Hello world!", add_bos=True, add_eos=True) >>> print(tokenized_text) [1, 31587, 29644, 102, 2] Npathspecial_tokens max_seq_lenprompt_templatect||_||nt|_|jd|_|jd|_|jg|_||_||_dS)Nr) r _spm_modelPHI3_SPECIAL_TOKENSreos_idpad_id stop_tokensrr)selfrrrrs t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/phi3/_tokenizer.py__init__zPhi3MiniTokenizer.__init__=sn5T::-8NN>Q  )/: )/: !K=&.c|jjSN)r vocab_sizer!s r"r'zPhi3MiniTokenizer.vocab_sizeUs ))r$c|jjSr&)rbos_idr(s r"r*zPhi3MiniTokenizer.bos_idYs %%r$TFtextadd_bosadd_eostrim_leading_whitespacereturnc>|j||||S)Nr,r-r.)rencode)r!r+r,r-r.s r"r2zPhi3MiniTokenizer.encode]s/%% $; &   r$idsskip_special_tokenscg}|D]&}|r |dkr|dkr||'|j|S)a0Decode token IDs to strings. Args: ids (List[int]): The input token IDs to be decoded. skip_special_tokens (bool): Whether to show or skip special tokens in the decoded string. Default is True. Returns: str: The decoded text. ri@})appendrdecode)r!r3r4ids_for_decodetoken_ids r"r7zPhi3MiniTokenizer.decodeksf 0 0H# 0F(:(:x6?Q?Q%%h////%%n555r$)r-ignore_system_promptmessagesr:c p|j||n|}d}d}g}g}|ddd} |D]} |r | jdkr|r4||j|| j| jdkr;||jd|| jn| jd kr=||jd d}|| jnf| jdkr;||jd || jn td | jd | jd| | | | jgt| zg} | jD][} | ddkr6| || d ddddz} Dtd| d| |jdgz| z} | | | | jgt| z|r;|r9||j || jd}d}nd}|jrt||jkrn|jrVt||jkr>t||j|r|j nd}t||j|r| jnd}||fS)aTokenize a list of messages one at a time then concatenate them, returning a list of tokens and a list of masks. Example: >>> tokenizer = Phi3MiniTokenizer(tokenizer_path, max_seq_len) >>> messages = [ Message(role="system", content="system message\n", masked=True), Message(role="user", content="user prompt\n", masked=True), Message(role="assistant", content="assistant response\n"), ] >>> # tokenize_messages encodes messages separately and concats >>> tokenizer.tokenize_messages(messages)[0] [1, 1788, 2643, 13, 1792, 9508, 13, 465, 22137, 2933, 2] >>> # Same result as encoding the full string in one go >>> tokenizer.encode(''.join([message.content for message in messages])) [1, 1788, 2643, 13, 1792, 9508, 13, 465, 22137, 2933, 2] Args: messages (List[Message]): A list of messages, each containing role, content, and masked attributes. add_eos (bool): Whether to append EOS after assistant message, default to False ignore_system_prompt (bool): Whether to ignore system prompt, defaults to False. Raises: ValueError: If the role is not "user", "assistant", or "system". RuntimeError: If ``message["type"] != "text``. Returns: Tuple[List[int], List[bool]]: The tokenized messages NTF )r,r-systemuserr assistantrrzUnknown role 'z' for message: ''typer+content r1z"Unsupported message content type: r)rr2roler6r*maskedr ValueErrorrCextendlenrstrip RuntimeErrorrrr ) r!r;r-r:templated_messages start_of_turn end_of_turntokenized_messagesmasknew_line_token_idmessagetokensitems r"tokenize_messagesz#Phi3MiniTokenizer.tokenize_messagessT#/   * * *    !KKeUKKK)= = G#  (@(@ ,"))$+666 GN+++|v%%"))$*=j*IJJJ GN++++,,"))$*=o*NOOO"  GN++++))"))$*=l*KLLL GN++++ UW\UU7?UUU  % %&7 8 8 8 KK(3/@+A+AA B B BF  <6))#dkkY..s33 % %04 '2''FF'KT&\KKt29=>>ARRF  % %f - - - KK(3v;;6 7 7 7 &w &"))$+666 GN+++# $ %  C(:$;$;t?O$O$O   Y$6 7 74;K K K!)"D$4W6VdkkRV"" D$"2g4WGNNSWXXD!4''r$sample inferencect|d}||\}}||d<||d<|S)a% Apply ``tokenize_messages`` to the "messages" field in the sample. Args: sample (Mapping[str, Any]): A sample with a "messages" field containing a List[Message] to tokenize inference (bool): Whether the template is being used for inference or not. Returns: Mapping[str, Any]: The sample with added "tokens" and "mask" fields and the "messages" field removed. inference (bool): Whether the template is being used for inference or not. r;rSrP)poprU)r!rVrWr;rSrPs r"__call__zPhi3MiniTokenizer.__call__sD ::j))--h77 !xv r$)NNN)TTF)T)F)__name__ __module__ __qualname____doc__strrrintr r#propertyr'r*boolrr2r7r rrUrrrZr$r"rr!s<48%)48 ///!c3h0/c] / ".1 ////0**X*&&X& (-         "&  c     66$s)6$6#66662%* |(|(|(w-|( |( # |( tCy$t*$ % |(|(|(|(~rjs=<<<<<<<<<<<<<<<,,,,,,;;;;;;******222222   qqqqq qqqqqr$