*`iI dZddlZddlZddlmZmZmZmZmZddl Z ddl m Z ddl m Z mZddlmZe jZ ded ed eeeffd Ze jd e Zded ed e jfdZde jd dfdZdddde jde jd eedeeed df dZGdde ZGdde ZdS)zaMatch the output of the LLM to the specified grammar, then generate the mask for the next token. N)ListLiteralOptionalTupleUnion) ArrayLike) XGRObject_core)CompiledGrammar batch_size vocab_sizereturnc4|tj|dz fS)zEReturn the shape of the bitmask: (batch_size, ceil(vocab_size / 32)). )mathceilr rs d/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/xgrammar/matcher.pyget_bitmask_shapers  *r/22 33dtypec`tjt||ttS)avAllocate the bitmask for the next token prediction. The bitmask is an int32 tensor on CPU with shape (batch_size, ceil(vocab_size / 32)). Users who have their own needs to manage CUDA memory can construct the tensor with get_bitmask_shape and bitmask_dtype themselves. The reason why we use int32 instead of uint32 is that old versions of PyTorch do not support uint32. Parameters ---------- batch_size : int The batch size of the bitmask. vocab_size : int The size of the vocabulary. Returns ------- bitmask : torch.Tensor The shape of the bitmask. r)torchfullr _FULL_MASK bitmask_dtypers rallocate_token_bitmaskr s'. :' J??S` a a aarbitmaskc:|tdS)z#Reset the bitmask to the full mask.N)fill_r)r!s rreset_token_bitmaskr$5s MM*r)rindiceslogitsr%c*|j|jkr"tdd|jd|jz|jjdkrddlm}|||||d S|jjdkrddlm}|||||d Sdd lm}|||||d S) a Apply the bitmask to the logits in-place. The bitmask is a 01 bitwise compressed tensor, where 0 means the token is masked and 1 means the token is not masked. It can be generated by allocate_token_bitmask and filled by fill_next_token_bitmask. After applying the bitmask, the masked logits will be set to -inf. The shape of logits and bitmask should be (batch_size, vocab_size) and (batch_size, bitmask_size) respectively. bitmask_size = ceil(vocab_size / 32). The operation is: .. code:: python for i in range(batch_size): for j in range(vocab_size): if get_bitmask_value(bitmask, i, j) == 0: logits[i, j] = -inf get_bitmask_value(bitmask, i, j) gets the j-th bit of the i-th row of the bitmask. Notes ----- Padding: This method allows additional padding on the vocabulary dimension of logits or bitmask. If padding exists, provide the real vocab size to the vocab_size parameter, and the operation will be applied to logits[..., :vocab_size] and bitmask[..., :ceil(vocab_size / 32)]. If vocab_size is not provided, the vocab size will be detected as min(logits.shape[-1], bitmask.shape[-1] * 32). Indices: Indices can be used to specify which logits in the batch to apply the bitmask to. It is especially useful when there are structured requests and unstructured requests mixed in the same batch by skipping masking the logits in the unstructured requests. When specified, the operation will be .. code:: python for batch_id in indices: for j in range(vocab_size): if get_bitmask_value(bitmask, batch_id, j) == 0: logits[batch_id, j] = -inf When indices is specified, the batch sizes of logits and bitmask do not need to be the same. As long as the indices are valid, the operation will be performed. Device: The logits and bitmask should be on the same device. If both them are on GPU, we launch a GPU kernel to apply bitmask. If both them are on CPU, we use a CPU implementation. The GPU kernel is optimized and should be preferred. In practice, the bitmask is allocated on CPU, and the logits is usually on GPU, so users should manually copy the bitmask to GPU before calling this function. Parameters ---------- logits : torch.Tensor The tensor to apply the bitmask to. bitmask : torch.Tensor The bitmask to apply. vocab_size : Optional[int], default: None The size of the vocabulary. If not provided, the vocab size will be detected as min(logits.shape[-1], bitmask.shape[-1] * 32). indices : Optional[List[int]], default: None A list of indices to specify which logits in the batch to apply the bitmask to. Should be unique. If None, apply the bitmask to all logits in the batch. z1logits and bitmask should be on the same device. zBut got logits.device: z, bitmask.device: cpur )apply_token_bitmask_inplace_cpucuda)"apply_token_bitmask_inplace_triton))apply_token_bitmask_inplace_torch_compileN) device ValueErrortype'kernels.apply_token_bitmask_inplace_cpur)*kernels.apply_token_bitmask_inplace_tritonr+1kernels.apply_token_bitmask_inplace_torch_compiler,)r&r!rr%r)r+r,s rapply_token_bitmask_inplacer3:sT~&& ?Y YYYY Z   }U""\\\\\\''WMMMMM  v % %bbbbbb**67JPPPPP       21&':wWWWWWrc HeZdZdZdddddedeeeeefde d ed df d Z dd d ede d e fdZ dd dee e fde d e fdZ ddd dedede d e fdZd e fdZd ded dfdZd e fdZd!dZed efdZed eefdZd e fdZdS)"GrammarMatcheraMatch the output of the LLM to the specified grammar, then generate the mask for the next token. This is the core class in the grammar-guided generation. This class maintains a stateful matcher that can accept tokens and strings, then match them to the specified grammar. The matcher can provide a bitmask for the next token prediction, so that the output of the LLM follows the specified grammar. Its state can be reset and rolled back by tokens. It also provides utilities for jump-forward decoding. After matching the whole grammar, the matcher will accept a stop token. The token mask at this time will only allow stop tokens. After accepting the stop token, the matcher will terminate, then it cannot accept any new token or generate a new token mask, meaning the generation is finished. Under the hood, it utilizes a pushdown automaton with backtracking to match the grammar, with optimizations specific to LLM token mask generation. NFr)override_stop_tokensterminate_without_stop_tokenmax_rollback_tokenscompiled_grammarr6r7r8rct|tstd|dkstjdt t|t r|g}|tj |j |||dS)a|Construct the grammar matcher. Parameters ---------- compiled_grammar : CompiledGrammar The initialization context for the grammar matcher. override_stop_tokens : Optional[Union[int, List[int]]], default: None If not None, the stop tokens to override the ones in the grammar. terminate_without_stop_token : bool, default: False Whether to terminate the matcher without accepting a stop token. max_rollback_tokens : int, default: -1 Deprecated. You don't need to set it and it's always unlimited (-1). The new Earley parser significantly reduces the number of states, so we can allow unlimited rollback. The maximum number of rollback tokens allowed. The rollback operation is useful for jump-forward decoding and speculative decoding. zCThe grammar should be compiled before passing it to GrammarMatcher.rz[max_rollback_tokens is deprecated. You don't need to set it and it's always unlimited (-1).N) isinstancer r.warningswarnDeprecationWarningint _init_handler r5_handle)selfr9r6r7r8s r__init__zGrammarMatcher.__init__s:*O<< dbcc c"b(( M""    *C 0 0 :$8#9     ($,#        r) debug_printtoken_idrDc8|j||S)aoAccept one token and update the state of the matcher. In the following cases, the matcher will not accept the token and return False: 1. The token does not match the grammar. 2. The matcher has terminated after accepting the stop token, but is trying to accept a new token. 3. The token id is out of range. 4. The token is a special token. The user should capture the return value and handle the cases where the token is not accepted. Parameters ---------- token_id : int The id of the token to accept. debug_print : bool, default: False Whether to print information about the internal state of the matcher. Helpful for debugging. Returns ------- accepted : bool Whether the token is accepted. )rA accept_token)rBrErDs rrGzGrammarMatcher.accept_tokens8|((;???r input_strc8|j||S)atAccept a string and update the state of the matcher. The whole string is considered as one step in rollback. It is used to complement the functionality of accept_token, and accept_token should always be used to accept tokens. Parameters ---------- input_str : Union[str, bytes] The string to be accepted. debug_print : bool, default: False Whether to print information about the internal state of the matcher. Helpful for debugging. Returns ------- accepted : bool Whether the string is accepted. )rA accept_string)rBrHrDs rrJzGrammarMatcher.accept_strings&|)))[AAArrr!indexc:|j|||S)a2Fill the bitmask for the next token prediction. The input bitmask can be generated by allocate_token_bitmask, and must be on CPU. bitmask[index] will be filled with the next token bitmask. This method does not change the matcher state. Parameters ---------- bitmask : ArrayLike The bitmask for the next token prediction. It supports torch.Tensor and other array-like objects, as long as they support the DLPack protocol. index : int, default: 0 The batch id of the bitmask. debug_print : bool, default: False Whether to print information about generated bitmask. Helpful for debugging. Returns ------- need_apply : bool Whether the bitmask need to be applied (not all-true). An optimization: if False, this means the bitmask is already all-true, so no need to apply it. Raises ------ RuntimeError If the bitmask is invalid (not on CPU, not int32, shape mismatch). )rAfill_next_token_bitmask)rBr!rKrDs rrMz&GrammarMatcher.fill_next_token_bitmasks@|33GUKPPPrc4|jS)aFind the jump-forward string for jump-forward decoding. This is the longest string that certainly conforms with the current grammar from the current matcher state. This string can become the output of the LLM without requiring LLM decoding. This method does not change the matcher state. Returns ------- jump_forward_string : str The jump-forward string. )rAfind_jump_forward_stringrBs rrOz'GrammarMatcher.find_jump_forward_string6s|44666rr num_tokensc:|j|dS)a;Rollback the matcher to a previous state by several tokens. Parameters ---------- num_tokens : int, default: 1 The number of tokens to rollback. It cannot exceed the current number of steps, nor can it exceed the specified maximum number of rollback tokens. N)rArollback)rBrQs rrSzGrammarMatcher.rollbackDs  j)))))rc4|jS)aXCheck if the matcher has terminated. If terminate_without_stop_token is False, the matcher will terminate if it has accepted the stop token. Otherwise, the matcher will terminate after matching the whole grammar. Returns ------- terminated : bool Whether the matcher has terminated. )rA is_terminatedrPs rrUzGrammarMatcher.is_terminatedOs|))+++rc4|jS)z'Reset the matcher to the initial state.)rAresetrPs rrWzGrammarMatcher.reset[s|!!###rcdS)zDepracated. Now max_rollback_tokens is always unlimited (-1). Get the maximum number of rollback tokens allowed. Returns ------- max_rollback_tokens : int The maximum number of rollback tokens. rrPs rr8z"GrammarMatcher.max_rollback_tokens_s rrc|jjS)a"The ids of the stop tokens used in the matcher. If specified, the provided stop tokens will be used. Otherwise, the stop tokens will be detected from the vocabulary. Returns ------- stop_token_ids : List[int] The ids of the stop tokens. )rAstop_token_idsrPs rr[zGrammarMatcher.stop_token_idsls|**rc4|jS)aPrint the internal state of the matcher. This is used for debugging. The representation of the internal state is subject to change. Returns ------- internal_state : str The internal state of the matcher. )rA_debug_print_internal_staterPs rr]z*GrammarMatcher._debug_print_internal_statexs|77999r)r)r )rN)__name__ __module__ __qualname____doc__r rrr?rboolrCrGstrbytesrJrrMrOrSrUrWpropertyr8r[r]rYrrr5r5sQ*AE-2#% 1 1 1 )1 'uS$s)^'<= 1 '+ 1 ! 1  1 1 1 1 fBG@@@S@$@4@@@@<RWBBBuS%Z'8B$B[_BBBB,01 QIN Q Q Q  Q), QBF Q  Q Q Q QD 7# 7 7 7 7 * *3 *t * * * * ,t , , , ,$$$$ S   X  +S + + +X + :S : : : : : :rr5c 6eZdZdZddeeedfddfdZ dded d e d e eed e ddf d Z e dded deed e dee fdZe dded deeeefd e dee fdZdS)BatchGrammarMatcherzA batch version of GrammarMatcher that can fill the next token bitmask for multiple matchers in parallel. It utilizes multiple threads to speed up the computation. It is especially useful when the batch size is large. auto max_threadsrNcT|tj|dS)aGConstruct the batch grammar matcher. Parameters ---------- max_threads : Union[int, Literal["auto"]], default: "auto" The maximum number of threads to use for parallel processing. If set to "auto", the max_threads will be set to std::thread::hardware_concurrency() / 2. N)r@r rg)rBris rrCzBatchGrammarMatcher.__init__s) %3K@@AAAAArFmatchersr5r!r%rDcXd|D}|j||||dS)aFill the next token bitmask for multiple matchers. Parameters ---------- matchers : List[GrammarMatcher] The list of matchers to fill the bitmask for. bitmask : ArrayLike Must be a 2-dimensional int32 tensor with shape (bitmask_batch_size, bitmask_size). Bitmask_batch_size could be larger than the actual batch size to allow padding. Bitmask_size equals to ceil(vocab_size/32), and could be computed through xgrammar.allocate_token_bitmask. indices : Optional[List[int]], default: None A list of indices to specify which rows in the bitmask to fill. If None, fill the bitmask [0:len(matchers))]. debug_print : bool, default: False Whether to print information about generated bitmask. Helpful for debugging. Raises ------ RuntimeError If the bitmask is invalid (not on CPU, not int32, shape mismatch). cg|] }|j SrYrA.0matchers r zEBatchGrammarMatcher.batch_fill_next_token_bitmask..CCCw7?CCCrN)rAbatch_fill_next_token_bitmask)rBrkr!r%rDmatcher_handless rrtz1BatchGrammarMatcher.batch_fill_next_token_bitmasks<@DC(CCC 22?GWVabbbbbrtokensc\d|D}tj|||S)aAccept a batch of tokens for multiple matchers. Parameters ---------- matchers : List[GrammarMatcher] The list of matchers to accept tokens for. tokens : List[int] The list of tokens to accept. debug_print : bool, default: False Whether to print information about generated bitmask. Helpful for debugging. Returns ------- accepted : List[bool] A list of booleans indicating whether each token was accepted by its corresponding matcher. Raises ------ RuntimeError If the sizes of matchers and tokens do not match. cg|] }|j SrYrnros rrrz:BatchGrammarMatcher.batch_accept_token..rsr)r rgbatch_accept_token)rkrvrDrus rryz&BatchGrammarMatcher.batch_accept_tokens46DC(CCC(;;OVU`aaarstringsc\d|D}tj|||S)aAccept a batch of strings for multiple matchers. Parameters ---------- matchers : List[GrammarMatcher] The list of matchers to accept tokens for. strings : List[Union[str, bytes]] The list of strings to accept. debug_print : bool, default: False Whether to print information about generated bitmask. Helpful for debugging. Returns ------- accepted : List[bool] A list of booleans indicating whether each string was accepted by its corresponding matcher. Raises ------ RuntimeError If the sizes of matchers and strings do not match. cg|] }|j SrYrnros rrrz;BatchGrammarMatcher.batch_accept_string..rsr)r rgbatch_accept_string)rkrzrDrus rr}z'BatchGrammarMatcher.batch_accept_strings4:DC(CCC(<<_gWbcccr)rh)NF)F)r^r_r`rarr?rrCrrrrbrt staticmethodryrcrdr}rYrrrgrgs B BE#wv*>$? BT B B B B (,! "c"c'("c"c$s)$ "c  "c  "c"c"c"cHQVbb'(b26s)bJNb dbbb\b:"dd'(deCJ'(dd d ddd\dddrrg)rarr<typingrrrrrr numpy.typingrbaser r compilerr int32rr?rtensorrTensorr r$r3r5rgrYrrrs2 88888888888888 """"""""""""""%%%%%% &4#4345c?4444 U\"M 2 2 2 bsbb bbbb4$!%#' _X_X_X L_X \_X _X d3i _X  _X_X_X_XDe:e:e:e:e:Ye:e:e:Prdrdrdrdrd)rdrdrdrdrdr