Pi~3ddlZddlmZmZmZmZmZmZddlm Z m Z m Z ddl m Z ddlmZmZdZddd d d d d ddddddd ZdZdeeeez DZieeZGddee ZdS)N)AnyDictListMappingOptionalTuple)MessagePromptTemplatetruncate) Transform)ModelTokenizerTikTokenBaseTokenizerzs(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+iiiiiiiiii i ii ) <|begin_of_text|><|end_of_text|>z<|reserved_special_token_0|>z<|reserved_special_token_1|><|finetune_right_pad_id|> <|step_id|><|start_header_id|><|end_header_id|> <|eom_id|> <|eot_id|><|python_tag|> <|image|>z <|video|>c(i|]}dd|zdd|zS)z<|reserved_special_token_z|>i ).0is v/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/llama3/_tokenizer.py r &s= *A)))6A:c eZdZdZ d%dedeeeefdeedeefdZ dZ d ed efd Z e d efd Z e d efd Z d&d ededed eefdZ d&deededed efdZded eefdZded eefdZded eefdZddddededed eefdZdddeeded eeeeeffd Z d'd"eeefd#ed eeeffd$ZdS)(Llama3Tokenizera& tiktoken tokenizer configured with Llama3 Instruct's special tokens, as described in https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3 Args: path (str): Path to pretrained tiktoken tokenizer file. special_tokens (Optional[Dict[str, int]]): mapping containing special text tokens and their registered token IDs. If left as None, this will be set to the canonical Llama3 special tokens. max_seq_len (Optional[int]): maximum sequence length for tokenizing a single list of messages, after which the input will be truncated. Default is None. prompt_template (Optional[PromptTemplate]): template used to format the messages based on their role. This is used to add structured text around the actual messages. The structured text is used in three scenarios: - Task-specific templates to gear models for a particular task that it will expect after training - Model-specific templates that are required whenever the model is prompted, such as the [INST] tags in Llama2 and in Mistral - Community standardized templates, such as :class:`~torchtune.data.ChatMLTemplate` The extra text will still get tokenized as normal text, not as special tokens. Default is None. Examples: >>> tokenizer = Llama3Tokenizer("/path/to/tt_model") >>> tokenized_text = tokenizer.encode("Hello world!", add_bos=True, add_eos=True) >>> print(tokenized_text) [1, 31587, 29644, 102, 2] Npathspecial_tokens max_seq_lenprompt_templatec||nt|_||jd|_|jd|_|jd|_|jd|_|jd|_|jd|_|jd|_ |jd|_ |jd |_ |jd |_ |j|j |j g|_ t|d t|j|j|j |_||_||_t'jd |_t'jd|_dS)Nrrrrrrrrrrllama3_tiktoken)r$namepatternbos_ideos_idr%z <\|.*?\|>z/<\|start_header_id\|>.*?<\|end_header_id\|>\n\n)LLAMA3_SPECIAL_TOKENSr%_validate_special_tokensr,r-pad_idstep_idstart_header_id end_header_ideot_ideom_id python_tagimage_id stop_tokensrCL100K_PATTERNtt_modelr&r'recompile_special_token_regex_special_token_header_regex)selfr$r%r&r's r__init__zLlama3Tokenizer.__init__KsZ-8NN>S  %%''')*=> )*;< )*EF *=9  $23HI!01DE),7 ),7 -.>?+K8 !KdkB-"";;.     '.%'J|$<$<!+-: >, , (((r!cFdD]}||jvrt|ddS)zV Validate that required special tokens are passed into the tokenizer. )rrrrrrrz missing from special_tokensN)r% ValueError)r?tokens rr/z(Llama3Tokenizer._validate_special_tokens~sM  I IED/// E!G!G!GHHH0 I Ir!textreturncj|jd|jd|S)z@ Remove special tokens from the decoded string. )r=subr>)r?rDs r_remove_special_tokensz&Llama3Tokenizer._remove_special_tokenss7 (,, 044R>>   r!c|jjSN)r:base_vocab_sizer?s rrLzLlama3Tokenizer.base_vocab_sizes },,r!c|jjSrK)r: vocab_sizerMs rrOzLlama3Tokenizer.vocab_sizes }''r!Tadd_bosadd_eosc<|j|||S)N)rDrPrQ)r:encode)r?rDrPrQs rrSzLlama3Tokenizer.encodes! }##w#PPPr! token_idstruncate_at_eosskip_special_tokenscl|j||}|r||n|S)a Decode a list of token ids into a string. Args: token_ids (List[int]): The list of token ids. truncate_at_eos (bool): Whether to truncate the string at the end of sequence token. Default is True. skip_special_tokens (bool): Whether to show or skip special tokens in the decoded string. Default is True. Returns: str: The decoded string. )rTrU)r:decoderI)r?rTrUrVdecoded_strings rrXzLlama3Tokenizer.decodesK.--+.   # D ' ' 7 7 7 r!messagec|jg||jddz|jgz|dddzS)zT Tokenize header start, message role, and header end as list of ids FrPrQz )r2rSrolestripr3r?rZs r_tokenize_headerz Llama3Tokenizer._tokenize_headersf  ! "kk',,,..ukMM N!" #kk&%k?? @ r!c0|jr|jgn|jgS)z> Add eot or eom id at the end of the message. )eotr4r5r_s r _tokenize_endzLlama3Tokenizer._tokenize_ends!( > }}$+>r!c g}|jD]q}|ddkr4|||dddz }B|ddkr ||jgz }Zt d|d|jr |jg|z}|S)z9 Tokenize message content as list of ids typerDcontentFr\imagez"Unsupported message content type: )rfrSr^r7 RuntimeErroripythonr6)r?rZtokenized_bodyitems r_tokenize_bodyzLlama3Tokenizer._tokenize_bodysO X XDF|v%%$++O))++UE#.##f((4=/1"#VV #V#VWWW ? @"o.?Nr!)add_start_tokensadd_end_tokensrmrnc|r||ng}||}|r||ng}||z|z}|S)a Tokenize a message into a list of token ids. Args: message (Message): The message to tokenize. add_start_tokens (bool): Whether to prepend a tokenized header to the message. Default is True. add_end_tokens (bool): Whether to append eot or eom id at the end of the message. Default is True. Returns: List[int]: The list of token ids. )r`rlrc)r?rZrmrntokenized_headerrj tokenized_endtokenized_messages rtokenize_messagez Llama3Tokenizer.tokenize_messagesj$>NU400999SU,,W557EM**73332 ,~= M  r!rnmessagesc|j||n|}|jg}dg}t|}t|D]j\}}||dz kr|nd} ||| } || z}||jgt| zz}|jrt||jkrnk|r||jgz}|dgz}|jr9t||j|r|jnd}t||j|rdnd}||fS)a Tokenize a list of messages into a list of token ids and masks. Args: messages (List[Message]): The list of messages to tokenize. add_end_tokens (bool): Whether to append end tokens ids (end-of-seq, end-of-turn, end-of-message) at the end of the last assistant message. This value should be set to False for generation. Default is True. Examples: >>> # Tokenize a list of messages with default settings >>> messages = [ ... Message(role="user", content="Hello world!", masked=True), ... Message(role="assistant", content="How are you?", masked=False), ... ] >>> tokenizer = Llama3Tokenizer("/path/to/tt_model") >>> tokenizer.tokenize_messages(messages) ([1, 31587, 29644, 102, 1, 31587, 29644, 102, 2], [True, True, True, True, True, False, False, False, True]) >>> # Tokenize a list of messages with add_end_tokens set to False >>> tokenizer.tokenize_messages(messages, add_end_tokens=False) ([1, 31587, 29644, 102, 1, 31587, 29644], [True, True, True, True, True, False, False]) Returns: Tuple[List[int], List[bool]]: The list of token ids and the list of masks. NTrt) r'r,len enumeratersmaskedr&r-r ) r?rurntemplated_messagestokensmask num_messagesrrZadd_end_tokens_to_messagerrs rtokenize_messagesz!Llama3Tokenizer.tokenize_messagessqB#/   * * *  +v-.. #$677  JAw#$|a'7"7"7T &!% 5 5(A!6!! //FGN+c2C.D.DDED CKK43C$C$C  !t{m+F4&=D   V(*Q$++TFD$"2N4TDDPTUUDt|r!Fsample inferencecz|d}||| \}}||d<||d<|S)a Apply ``tokenize_messages`` to the "messages" field in the sample. Args: sample (Mapping[str, Any]): A sample with a "messages" field containing a List[Message] to tokenize inference (bool): Whether the template is being used for inference or not. Returns: Mapping[str, Any]: The sample with added "tokens" and "mask" fields and the "messages" field removed. rurtr|r})popr)r?rrrur|r}s r__call__zLlama3Tokenizer.__call__LsK::j))--h9}-UU !xv r!)NNN)TT)F)__name__ __module__ __qualname____doc__strrrintr r@r/rIpropertyrLrOboolrrSrXr r`rcrlrsrrrrrrr!rr#r#.s >48%)48 1 1 1 !c3h01 c] 1 ".1 1 1 1 1 fIII$ 3 3    ----X-(C(((X(  QQQQ Q c QQQQ!%$(   9  "      B    DI     ?W?c???? g$s)0"&# !!!! !  ! c !!!!: $ BBBw-B B tCy$t*$ % BBBBJrsg <<<<<<<<<<<<<<<<<<<<<<<<<<222222 L $*$*!'! " U.^1D1DD E E >>=_=qqqqqniqqqqqr!