Pi ddlmZmZmZmZmZmZddlmZm Z m Z ddl m Z ddl mZddlmZdddd d d d d ddddd ZdZeddD]Zedkr eeded<edz ZdZGdde eZdS))AnyDictListMappingOptionalTuple)MessagePromptTemplatetruncate)ModelTokenizer) Transform)GPT2BaseTokenizer頇iiiiiiiiiii) z <|dummy_0|> <|endoftext|>z<|fim_prefix|>z<|fim_middle|>z<|fim_suffix|>z <|dummy_1|>z <|dummy_2|>z <|dummy_3|> <|im_start|> <|im_end|> <|im_sep|>z<|endofprompt|>iiz<|dummy_z|>zs(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+c beZdZdZ ddededeeeefdeedeef dZ e d Z dd ed e d e de efdZdde ede defdZdedefdZdddde ed e de dee ee e ffdZdeeefdeeeffdZdS) Phi4TokenizeraQ TikToken tokenizer configured with Phi4 (14B) special tokens. Args: merges_path (str): Path to merges.txt file. vocab_path (str): Path to vocab.json file. special_tokens (Optional[Dict[str, int]]): Mapping containing special text tokens and their registered token IDs. If left as None, this will be set to the canonical Phi4 special tokens. max_seq_len (Optional[int]): Max sequence length to truncate tokens to. prompt_template (Optional[PromptTemplate]): Template used to format the messages based on their role. N merges_path vocab_pathspecial_tokens max_seq_lenprompt_templatec|pt|_|jd|_|jd|_|jd|_|jg|_||_||_t|||j|j|j|j|_ dS)Nrrz <|dummy_85|>) PHI4_SPECIAL_TOKENSreos_idbos_idpad_id stop_tokensrrrtokenizer_model)selfrrrrrs t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/phi4/_tokenizer.py__init__zPhi4Tokenizer.__init__6s-C0C),7 )/: ).9  K=&.0   K K K K   c|jjS)N)r# vocab_size)r$s r%r)zPhi4Tokenizer.vocab_sizeRs#..r'Ttextadd_bosadd_eosreturnc<|j|||S)N)r*r+r,)r#encode)r$r*r+r,s r%r/zPhi4Tokenizer.encodeVs"#**gw*WWWr'idsskip_special_tokenscTfd|D}|j|S)zDecode token IDs to strings.c6g|]}rd|cxkrdkn|S)ri).0token_idr1s r% z(Phi4Tokenizer.decode..]sL   ' -4x,J,J,J,J7,J,J,J,J ,J,J,Jr')r#decode)r$r0r1ids_for_decodes ` r%r8zPhi4Tokenizer.decode[sD        #**>:::r'rolec|jdg}|||dd||jd|S)NrFr+r,r)rextendr/append)r$r:tokenized_messagess r%_tokenize_headerzPhi4Tokenizer._tokenize_headerds["1.AB!!$++dE5+"Q"QRRR!!$"5l"CDDD!!r'F)r,ignore_system_promptmessagesrAc"|jr||n|}g}g}|D]}|r |jdkr||j}||||jgt |zg} |jD]Z} | ddkr5| || ddddz } Ctd| d|r,|jd kr!| |j d n+|jd kr | |j d || ||jgt | z|j rt ||j krn|j rVt ||j kr>t||j |r|jnd}t||j |r|jnd}||fS) Nsystemtyper*content Fr<z"Unsupported message content type: assistantr)rr:r@r=maskedlenrFr/rstrip RuntimeErrorr>rrr r) r$rBr,rAtemplated_messagesr?maskmessagetokenized_headertokensitems r%tokenize_messageszPhi4Tokenizer.tokenize_messagesjs[/3.B PD  * * *  )  G#  (@(@#44W\BB   % %&6 7 7 7 KK(3/?+@+@@ A A AF  <6))dkkY..s33UE*FF'KT&\KK A7<;66 d1,?@@@@,, d1,?@@@  % %f - - - KK(3v;;6 7 7 7 C(:$;$;t?O$O$O   Y$6 7 74;K K K!)"D$4W6VdkkRV"" D$"2g4WGNNSWXXD!4''r'samplect|d}||\}}||d<||d<|S)zR Apply `tokenize_messages` to the "messages" field in the sample. rBrQrN)poprS)r$rTrBrQrNs r%__call__zPhi4Tokenizer.__call__sD::j))--h77 !xv r')NNNNN)TT)T)__name__ __module__ __qualname____doc__strrrintr r&propertyr)boolrr/r8listr@r rrSrrrWr4r'r%rr(s   37%)48     !c3h0  c]  ".1     8//X/@DXXX"&X8<X cXXXX ;;$s);$;#;;;;"S"T""""%* 3(3(3(w-3( 3( # 3( tCy$t*$ % 3(3(3(3(jwsCx0WS#X5Fr'rN)typingrrrrrrtorchtune.datar r r torchtune.modules.tokenizersr torchtune.modules.transformsr 'torchtune.modules.transforms.tokenizersrrcurrent_dummy_indexranger6CL100K_PATTERNrr4r'r%ris]=<<<<<<<<<<<<<<<<<<<<<<<<<777777222222EEEEEE   ff%%H6>F:#6:::;1LNIr'