*`iVH(dZddlZddlmZddlmZmZmZmZm Z  ddl Z n #e $rdZ YnwxYw ddl Z n #e $rdZ YnwxYwddl mZmZddlmZmZddlmZejejeZGd d eZGd d eZdS) zRThis module provides the tokenizer info class to handle the tokenizer information.N)Enum)AnyDictListOptionalUnion)PreTrainedTokenizerBasePreTrainedTokenizerFast) XGRObject_core)loggingc"eZdZdZdZ dZ dZdS) VocabTypezThe type of the vocabulary. Used in TokenizerInfo. XGrammar supports three types of vocabularies: RAW, BYTE_FALLBACK, BYTE_LEVEL. rr N)__name__ __module__ __qualname____doc__RAW BYTE_FALLBACK BYTE_LEVELk/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/xgrammar/tokenizer_info.pyrrs= CMJrrceZdZdZejfdddddeeeee fdede e de eee e fd e d df d Z ed ed e fd Zed ed e fdZed ed e fdZedddd ede e de eee e fd dfdZed efdZed e fdZed e fdZed e fdZed eefdZed ee fdZed ee fdZd e fdZedeeee fde d dfdZede d ee effdZ d e fdZ!ede d dfd Z"dS)! TokenizerInfouThe tokenizer info contains the vocabulary, the type of the vocabulary, and necessary information for the grammar-guided generation. Note that although some tokenizers will encode the tokens in a special format, e.g. "<0x1B>" for "" in the ByteFallback tokenizer, and "Ġ" for " " in the Byte-Level BPE tokenizer, TokenizerInfo always decodes the vocabulary to the original format (e.g. "" and " "). Also note that some models (e.g. Phi-3 and Deepseek-V2) may pad the vocabulary to a multiple of 32. In this case, the model's vocab_size is larger than the tokenizer's vocabulary size. Please pass the model's vocab_size to the vocab_size parameter in the constructor, because this information is used to determine the size of the token mask. NF vocab_sizestop_token_idsadd_prefix_space encoded_vocab vocab_typerr r!returnc t|tr|g}|tj||j|||dS)aConstruct the tokenizer info. Parameters ---------- encoded_vocab : Union[List[bytes], List[str]] The encoded vocabulary of the tokenizer. vocab_type : VocabType, default: VocabType.RAW The type of the vocabulary. See also VocabType. vocab_size : Optional[int], default: None The size of the vocabulary. If not provided, the vocabulary size will be len(encoded_vocab). stop_token_ids : Optional[List[int]], default: None The stop token ids. If not provided, the stop token ids will be auto detected (but may not be correct). add_prefix_space : bool, default: False Whether the tokenizer will prepend a space before the text in the tokenization process. N) isinstanceint _init_handler rvalue)selfr"r#rr r!s r__init__zTokenizerInfo.__init__Hs_: nc * * .,-N   z/^M]       r tokenizerctdSt|dot|jtj}t|dod|jvod|jdv}|p|S)NFr,vocab_files_names vocab_filetiktoken)r0hasattrr&r,Encodingr.)r,has_tiktoken_encodingfilename_patterns r_is_tiktoken_tokenizerz$TokenizerInfo._is_tiktoken_tokenizerms  5!( ; ? ?! J  !2E E  I2 3 3 H ;; Hi9,GG  %8(88rctdS|d}|dkrdS||}|d}|dkS)aEChecking whether the tokenizer has byte-level whitespace conversion. Parameters ---------- tokenizer : PreTrainedTokenizerBase The huggingface tokenizer. Returns ------- is_byte_level : bool The tokenizer has byte-level whitespace conversion. NF r ruĠ)r0encode__len__convert_ids_to_tokens)r,new_ids new_tokenstokens r_is_byte_level_tokenizerz&TokenizerInfo._is_byte_level_tokenizersd  5""4(( ??  q 544W== 1 }rcjtdSt|dot|jtj}t|do8t|jdo#t|jjtjp.t|dot|jtj}|p|S)NFsp_modelr,tok) sentencepiecer1r&r@SentencePieceProcessorr,rA)r,has_sp_model_attrhas_nested_sp_model_attrs r_is_sentencepiece_tokenizerz)TokenizerInfo._is_sentencepiece_tokenizers  5$Iz:: z   D@ @  I{ + + _ +Z88 _9.79]^^$ Iu % % P9=-*NOO !!<$<` tokens, but they will not be considered in lm_head or generated by the model. model_vocab_size need to be provided for case 2 and 3. If not provided, it will be set to the tokenizer's vocabulary size. stop_token_ids : Optional[List[int]], default: None The stop token ids. If not provided, the eos_token_id of the tokenizer will be used. Returns ------- tokenizer_info : TokenizerInfo The tokenizer info. rzstop_token_ids cannot be emptyz+Cannot get the vocabulary of the tokenizer z/. The tokenizer should have a get_vocab method.Nr eos_token_idzWhen constructing TokenizerInfo from a huggingface tokenizer, stop_token_ids is neither provided by user nor found from the tokenizer. It will be automatically detected.r#r!)r#rr r!Frr@r,rAz<0x0A>TzUnsupported tokenizer type: )r&r'listlen ValueError get_vocabAttributeErrortypemaxvaluesitemsr backend_tokenizerto_strr1rIloggerwarningr_detect_metadata_from_hfr5rrr>rrFr@r,rAeos_idr)r,rr vocab_dictemsgmax_idtokenizer_vocab_sizer"r=idx backend_strmetadatar#r@rYs rfrom_huggingfacezTokenizerInfo.from_huggingfacesb nc * * .,-N nd + + ?N0C0Cq0H0H=>> > )",,..JJ ) ) )2d9oo222 S//q (  )Z&&(())"3z??FQJ??7#7 z) $**,, + +JE3Z%* c" i!8 9 9] O$5<<>>K%9n55):P:\&/&<%=NNNN= %==kJJH #L1%-!)* is converted back to "". )rd decoded_vocabres rrkzTokenizerInfo.decoded_vocabts |))rc|jjS)zThe stop token ids.)rdr res rr zTokenizerInfo.stop_token_ids|s|**rc|jjS)zThe special token ids. Special tokens include control tokens, reserved tokens, padded tokens, etc. Now it is automatically detected from the vocabulary.)rdspecial_token_idsres rrnzTokenizerInfo.special_token_idss|--rc4|jS)zDump the metadata of the tokenizer to a json string. It can be used to construct the tokenizer info from the vocabulary and the metadata string.)rd dump_metadatares rrpzTokenizerInfo.dump_metadatas|))+++rracrttj||S)a=Construct the tokenizer info from the vocabulary and the metadata string in json format. Parameters ---------- encoded_vocab : List[Union[bytes, str]] The encoded vocabulary of the tokenizer. metadata : str The metadata string in json format. )r_create_from_handler from_vocab_and_metadata)r"ras rrsz%TokenizerInfo.from_vocab_and_metadatas200   7 7 x P P   rr`ctj|}tj|}t |d|ddS)zDetect the metadata from the huggingface tokenizer backend string. For implementation use only. It returns {"vocab_type": VocabType, "add_prefix_space": bool}. r#r!)r#r!)r rrXjsonloadsr)r` metadata_strras rrXz&TokenizerInfo._detect_metadata_from_hfsQ*CCKPP :l++#H\$:;; (); <   rc4|jS)zSerialize the tokenizer info to a JSON string. Returns ------- json_string : str The JSON string. )rdserialize_jsonres rryzTokenizerInfo.serialize_jsons|**,,,r json_stringcpttj|S)afDeserialize a tokenizer info from a JSON string. Parameters ---------- json_string : str The JSON string. Returns ------- tokenizer_info : TokenizerInfo The tokenizer info. Raises ------ InvalidJSONError When the JSON string is invalid. DeserializeFormatError When the JSON string does not follow the serialization format of the tokenizer info. DeserializeVersionError When the __VERSION__ field in the JSON string is not the same as the current version. )rrrr deserialize_json)rzs rr|zTokenizerInfo.deserialize_jsons*.001D1U1UVa1b1bcccr)#rrrrrrrrbytesstrrr'boolr+ staticmethodr r5r>rFrbpropertyr#rr!rirkr rnrprsrrrXryr|rrrrr9s  "!* # %):>!&# # # T%[$s)34# # SM # !tCy#~!67 # #  # # # # J9*A9d999\9",C\2=/F=4===\=*%):> hOhOhO*hOSMhO!tCy#~!67 hO  hOhOhO\hOT2I222X2'C'''X'-$---X- %t%%%X%*tE{***X*+S +++X+.49...X. ,s,,,,  E%*-. :=    \ $  c  d38n    \  -----dcdoddd\dddrr)rruenumrtypingrrrrrrB ImportErrorr0 transformersr r baser r supportrenable_logging getLoggerrrVrrrrrrsXX 33333333333333MMMOOOOHHHJIIIIIII""""""""  8 $ $@VdVdVdVdVdIVdVdVdVdVds!++4>>