`i+ddlmZmZddlZddlmZmZmZmZm Z m Z m Z m Z ddl Z ddlmZddlmZmZmZddlmZmZddlmZGd d ZGd d ZdS) ) dataclassfieldN)AnyCallableDictHashableListOptionalTupleUnion)LMFormatEnforcerException)CharacterLevelParserForceStopParserCharacterLevelParserConfig)TokenizerPrefixTreeTokenizerPrefixTreeNode) TokenListc eZdZdZdeeeeefde eegefde eeefdedef dZ dS) TokenEnforcerTokenizerDatazTokenEnforcerTokenizerData contains all of the preprocessing for preparing the TokenEnforcer to work with a specific tokenizer. It does some calculations, so it is recommended to reuse it for multiple TokenEnforcersregular_tokensdecoder eos_token_id use_bitmask vocab_sizec*fd|D}||_t|j||_||_||_dd|jjjD|_ |_ ||_ dS)aA Create the tokenizer data that the TokenEnforcer needs. This can be reused for multiple TokenEnforcers if they work with the same tokenizer. :param regular_tokens: A list of tuples (token_id, token_string, is_new_word_token) for all the regular (not special) tokens in the tokenizer vocabulary. Note that token_string is expected to include leading / trailing whitespaces if relevant. :param decoder: A function that decodes a list of token ids into a string. :param eos_token_id: The token id(s) of the end-of-string token(s). c,g|]}|dk|S)r).0 token_tuplers r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/lmformatenforcer/tokenenforcer.py z7TokenEnforcerTokenizerData.__init__..s+"q"q"q;T_`aTbfpTpTp;TpTpTpc3@K|]}t|dk|VdS)r N)len)r token_strs r! z6TokenEnforcerTokenizerData.__init__..!s_*F*F qtu~qqDErErE)rErErErE*F*Fr#N) rrtokenizer_treerrjoinrootchildrenkeystokenizer_alphabetrr)selfrrrrrfiltered_regular_tokenss ` r!__init__z#TokenEnforcerTokenizerData.__init__s#r"q"q"q."q"q"q51$2E{T^__ ("$''*F*FTEXE]EfEkEkEmEm*F*F*F#F#F$&r#N) __name__ __module__ __qualname____doc__r r intstrboolrr r1rr#r!rr sss'!%eCdN&;!<'"DI;#34' %S$s)^4'# ' ! ''''''r#rc eZdZdZeGddZdedefdZde e de fd Z d e d d fd Zdedede deefdZd d de e fdZdS) TokenEnforcerzTokenEnforcer provides a token filtering mechanism, given a CharacterLevelParser and some information about the tokenizer. It is the main entry point for extending lm-format-enforcer to new inference libraries. See __init__() and get_allowed_tokens()cpeZdZUeed<edZedzed<eeZ e e ed<dS)TokenEnforcer.OutputTensorStateparserN)defaultallowed_tokens)default_factorycurrent_word_tokens) r2r3r4r__annotations__rr?rlistrAr r6rr#r!OutputTensorStater<)sb$$$$+05+>+>+> D(>>>).t)D)D)DT#YDDDDDr#rDtokenizer_datar=ci|_||_|j|_|j|_|j|_|j|_i|_|j|_|j|_t|j }||_ dS)z Create a new TokenEnforcer. :param tokenizer_data: Per tokenizer data that the token enforcer needs in order to operate. :param parser: A CharacterLevelParser that defines the allowed strings. )alphabetN) prefix_states root_parserr)rrrallowed_token_cacherrrr.config)r/rEr=rKs r!r1zTokenEnforcer.__init__0s~ LN!,;%- *7,;8: )5(3,^5VWWW r#token_sequencereturnct|}|dd}||jvr|j|jS||jvrGt|j}||j|<||||jS|j|}|||}||j|<||||jS)a Get a list of allowed tokens, given a list of tokens that were already generated. :param token_sequence: The tokens that were already generated, and the next token will be generated for. :return: A list of token ids that are allowed to be selected next. Nr=)tuplerHr?r:rDrI_compute_allowed_tokens_apply_new_characters)r/rL sent_tupleprev_step_tuplestateprev_step_state new_states r!get_allowed_tokensz TokenEnforcer.get_allowed_tokensDs>** $SbS/ + + +%j1@ @ D$6 6 6"334;K3LLE-2D z *  ( (U ; ; ;' '#0AO22?NSSI-6D z *  ( (Y ? ? ?+ +r# state_tokensrVr<c$ t|j|j}|j}|||jvr|j||_dS|j}||j|j j |||j rOt|j tr||j n||j |st#d||_| ||j|<dSdS#t$$rt&$rt)jt(j|t|}t)jd|dt|j|j|_t|j tr"|j|j YdS|j|j YdSwxYw)Nz+Parser reached state with no allowed tokens)levelz+Unknown LMFormatEnforcer Problem. Prefix: 'z' Terminating the parser. Please open an issue at https://github.com/noamgat/lm-format-enforcer/issues with the prefix and CharacterLevelParser parameters)rrrr= cache_keyrJr? shortcut_key_collect_allowed_tokensr)r+can_end isinstancerrCextendappend ValueErrorr Exceptionlogging basicConfigERRORr exception)r/rZrVr?r]r^prefixs r!rRz%TokenEnforcer._compute_allowed_tokensbs3% ?(1$2BDO(T(TN ..00I$d6N)N)N'+'? 'J$ <4466L  ( (t7J7OQ_am n n n|##%% =d/66="))$*;<<<<"))$*;<<<! Q !OPPP$2E $6D(333%$(     ? ? ?  gm 4 4 4 4\\$|"4"455F  @F@@@ A A A$-T-=t#O#OE $+T22 ?$++D,=>>>>>>$++D,=>>>>>> ?s AD$C D$$CH,HH tree_noder?r^c||j|}|j}t ||}t|tr|ddkrt|dksJ|\}} } } |j j } t| j td| | z } t| j | | z }|| | |j|dg}|D]<}||}|j|}||||d=dS)Nr json_freetext")rbtokensget_allowed_charactersr,r-set intersectionrarQr&r)json_freetext_tokensmin max_token_lenmaxlookup_allowed_tokensr? add_characterr_)r/r=rkr?r^allowed_charactersrelevant_characterscharacters_to_explore_cur_lenmin_lenmax_lencache min_remainingmax_allowed_len character next_parsernext_tree_nodes r!r_z%TokenEnforcer._collect_allowed_tokenssi.///#::<<'05577 #$7 8 8 E EFX Y Y lE * * N|A//Q/Q|$$))))+7 (Aw'rs)(((((((( NNNNNNNNNNNNNNNNNNNN111111ccccccccccMMMMMMMM ''''''''4PPPPPPPPPPr#