)`i8 dZddlmZmZddlmZmZmZddlm Z ddl m Z GddeZ Gd d e Z Gd d e ZGd de ZGdde ZGdde ZGdde ZdS)a3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. )ABCabstractmethod)AnyListOptional)Op) TensorTypecVeZdZdZdefdZededee fdZ de fdZ dS) LogitsProcessora LogitsProcessor defines high-level transformations that can be applied to logits or probabilities. Each processor is automatically legalized into low-level :class:`Op` or :class:`ParameterizedOp` that can be type-checked, validated, and fused for optimal performance. Users can extend this class to implement their own processors. Parameters ---------- **params : Any Processor-specific parameters at compile-time. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, TopK, Sample, TensorType >>> torch.manual_seed(42) >>> >>> # Create a pipeline that legalizes to a fused op. >>> pipe = LogitsPipe([ ... TopK(), # Top-k filtering on logits ... Sample() # Sample from the filtered distribution ... ], input_type=TensorType.PROBS) # assume the input is probabilities >>> >>> pipe LogitsPipe([TopK -> Sample], ops=[ProbsTopKOp -> ProbsSampleOp], compiled_ops=[FusedProbsTopKSampleOp]) Notes ----- Subclasses must implement the :meth:`legalize` method to convert the high-level processor into one or more low-level operators with specific input/output types paramsc ||_dS)z Initialize the processor. Parameters ---------- **params : Any Processor-specific parameters at compile-time. N)r )selfr s z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/logits_processor/processors.py__init__zLogitsProcessor.__init__9s input_typereturnct)a- Legalize the processor into a list of low-level operators. Parameters ---------- input_type : TensorType The expected input tensor type of the processor. Returns ------- List[Op] A list of low-level operators. )NotImplementedError)rrs rlegalizezLogitsProcessor.legalizeDs "!rcdd|jD}|jjd|dS)Nz, c3*K|]\}}|d|VdS)=N).0kvs r z+LogitsProcessor.__repr__..Vs0JJda!zzazzJJJJJJr())joinr items __class____name__)r params_strs r__repr__zLogitsProcessor.__repr__UsLYYJJdk6G6G6I6IJJJJJ .)99J9999rN) r% __module__ __qualname____doc__rrrr rr rstrr'rrrr r s@     ":"$r("""^" :#::::::rr cDeZdZdZdeffd ZdedeefdZ xZ S) Temperaturea7 Temperature scaling processor for logits. Scales logits by dividing by a temperature value. :attr:`TensorType.LOGITS` -> :attr:`TensorType.LOGITS` Parameters ---------- temperature : float or torch.Tensor, Runtime Temperature value for scaling. Must be positive. Can be a scalar or per-batch tensor. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, Temperature, Sample >>> torch.manual_seed(42) >>> pipe = LogitsPipe([Temperature()]) >>> logits = torch.randn(2, 2, device="cuda") >>> logits tensor([[ 0.1940, 2.1614], [ -0.1721, 0.8491]], device='cuda:0') >>> scaled_logits = pipe(logits, temperature=0.8) >>> scaled_logits tensor([[ 0.2425, 2.7017], [-0.2151, 1.0613]], device='cuda:0') r c :tjdi|dS)z_ Constructor for Temperature processor. No compile-time parameters are needed. Nrsuperrrr r$s rrzTemperature.__init__u( ""6"""""rrrcnddlm}|tjkrt d||di|jgS)L Legalize the processor into a list of low-level operators. r) TemperatureOpz/Temperature can only be applied to LOGITS, got r) operatorsr5r LOGITS ValueErrorr )rrr5s rrzTemperature.legalize{s^ -,,,,, * * *N*NN  ,, ,,--r r%r(r)r*rrr rr r __classcell__r$s@rr-r-Zst4####### .: .$r( . . . . . . . .rr-cVeZdZdZd deedeffd Zdede e fdZ xZ S) Softmaxa3 Softmax processor to convert logits to probabilities. Applies the softmax function. :attr:`TensorType.LOGITS` -> :attr:`TensorType.PROBS` Parameters ---------- enable_pdl : bool, optional, Compile-time Whether to enable PDL for the kernel implementation. Default is True. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, Softmax, Sample >>> torch.manual_seed(42) >>> pipe = LogitsPipe([Softmax()]) >>> logits = torch.randn(2, 2, device="cuda") >>> logits tensor([[ 0.1940, 2.1614], [ -0.1721, 0.8491]], device='cuda:0') >>> probs = pipe(logits) >>> probs tensor([[0.1227, 0.8773], [0.2648, 0.7352]], device='cuda:0') Notes ----- Can only appear once in a pipeline. N enable_pdlr c >tjdd|i|dS)aC Constructor for Softmax processor. Parameters ---------- enable_pdl : bool, optional, Compile-time Whether to enable PDL for the kernel implementation. Default is None, which means the kernel will be automatically enabled if PDL is supported on the device. r>Nrr/)rr>r r$s rrzSoftmax.__init__s- 99J9&99999rrrcnddlm}|tjkrt d||di|jgS)r4r) SoftmaxOpz+Softmax can only be applied to LOGITS, got r)r6rAr r7r8r )rrrAs rrzSoftmax.legalizesV )((((( * * *W:WWXX X ((DK(())r)N) r%r(r)r*rboolrrr rr rr:r;s@rr=r=s> : :8D> :C : : : : : : *: *$r( * * * * * * * *rr=cJeZdZdZd dedeffd Zdedee fdZ xZ S) TopKa* Top-k filtering processor. Keeps only the top-k highest probability tokens and masks out the rest. :attr:`TensorType.LOGITS` -> :attr:`TensorType.LOGITS` | :attr:`TensorType.PROBS` -> :attr:`TensorType.PROBS` Parameters ---------- joint_topk_topp : bool, optional, Compile-time Whether to enable joint top-k and top-p filtering when followed by TopP. Default is False. top_k : int or torch.Tensor, Runtime Number of top tokens to keep. Can be a scalar or per-batch tensor. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, TopK, Sample, TensorType >>> torch.manual_seed(42) >>> >>> # Top-k filtering on logits >>> pipe = LogitsPipe([TopK()], input_type=TensorType.LOGITS) >>> logits = torch.randn(2, 2, device="cuda") >>> logits tensor([[ 0.1940, 2.1614], [ -0.1721, 0.8491]], device='cuda:0') >>> topk_logits = pipe(logits, top_k=1) >>> topk_logits tensor([[ -inf, 2.1614], [ -inf, 0.8491]], device='cuda:0') >>> >>> # Top-k filtering on probabilities >>> pipe = LogitsPipe([TopK()], input_type=TensorType.PROBS) >>> probs = torch.randn(2, 2, device="cuda") >>> probs_normed = probs / probs.sum(dim=-1, keepdim=True) >>> probs_normed tensor([[ 4.4998, -3.4998], [-18.2893, 19.2893]], device='cuda:0') >>> topk_probs = pipe(probs_normed, top_k=1) >>> topk_probs tensor([[1., 0.], [0., 1.]], device='cuda:0') Notes ----- When applied to :attr:`TensorType.LOGITS`, sets non-top-k values to -inf. When applied to :attr:`TensorType.PROBS`, zeros out non-top-k values and renormalizes. See Also -------- :meth:`~flashinfer.sampling.top_k_mask_logits` :meth:`~flashinfer.sampling.top_k_renorm_probs` Fjoint_topk_toppr c >tjdd|i|dS)a Constructor for TopK processor. Parameters ---------- joint_topk_topp : bool, optional, Compile-time Whether to enable joint top-k and top-p filtering when followed by TopP. Default is False. rENrr/)rrEr r$s rrz TopK.__init__s- CCCFCCCCCrrrcddlm}m}|tjkr|di|jgS|tjkr|di|jgStd|)r4r) LogitsTopKOp ProbsTopKOpzTopK cannot be applied to Nr)r6rHrIr r7r PROBSr8)rrrHrIs rrz TopK.legalizes 98888888 * * * L//4;//0 0 :+ + +K..$+../ /F*FFGG Gr)F r%r(r)r*rBrrr rr rr:r;s@rrDrDs22h D D D D D D D D D H: H$r( H H H H H H H HrrDcDeZdZdZdeffd ZdedeefdZ xZ S)TopPa Top-p (nucleus) filtering processor. Keeps tokens with cumulative probability up to threshold p. :attr:`TensorType.PROBS` -> :attr:`TensorType.PROBS` Parameters ---------- top_p : float or torch.Tensor, Runtime Cumulative probability threshold in (0, 1]. Can be a scalar or per-batch tensor. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, Softmax, TopP, Sample >>> torch.manual_seed(42) >>> pipe = LogitsPipe([TopP()]) >>> probs = torch.randn(2, 2, device="cuda") >>> probs_normed = probs / probs.sum(dim=-1, keepdim=True) >>> probs_normed tensor([[ 0.0824, 0.9176], [-0.2541, 1.2541]], device='cuda:0') >>> topp_probs = pipe(probs_normed, top_p=0.9) >>> topp_probs tensor([[0., 1.], [0., 1.]], device='cuda:0') See Also -------- :meth:`~flashinfer.sampling.top_p_renorm_probs` r c :tjdi|dS)zX Constructor for TopP processor. No compile-time parameters are needed. Nrr/r1s rrz TopP.__init__0r2rrrcnddlm}|tjkrt d||di|jgS)r4r)TopPOpz'TopP can only be applied to PROBS, got r)r6rPr rJr8r )rrrPs rrz TopP.legalize6V &%%%%% ) ) )SzSSTT T%%%%&&rr9r;s@rrMrMst>####### ': '$r( ' ' ' ' ' ' ' 'rrMcDeZdZdZdeffd ZdedeefdZ xZ S)MinPa Min-p filtering processor. Keeps tokens with probability at least p times the maximum probability. :attr:`TensorType.PROBS` -> :attr:`TensorType.PROBS` Parameters ---------- min_p : float or torch.Tensor, Runtime Minimum probability threshold as a ratio of max probability. Must be in (0, 1]. Can be a scalar or per-batch tensor. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, Softmax, MinP, Sample >>> torch.manual_seed(42) >>> pipe = LogitsPipe([MinP()]) >>> probs = torch.randn(2, 2, device="cuda") >>> probs_normed = probs / probs.sum(dim=-1, keepdim=True) >>> probs_normed tensor([[ 0.0824, 0.9176], [-0.2541, 1.2541]], device='cuda:0') >>> minp_probs = pipe(probs_normed, min_p=0.05) >>> minp_probs tensor([[0.0824, 0.9176], [0.0000, 1.0000]], device='cuda:0') r c :tjdi|dS)zX Constructor for MinP processor. No compile-time parameters are needed. Nrr/r1s rrz MinP.__init__`r2rrrcnddlm}|tjkrt d||di|jgS)r4r)MinPOpz'MinP can only be applied to PROBS, got r)r6rVr rJr8r )rrrVs rrz MinP.legalizefrQrr9r;s@rrSrSBst:####### ': '$r( ' ' ' ' ' ' ' 'rrScJeZdZdZd dedeffd Zdedee fdZ xZ S) Samplea Sampling processor to generate token indices. Samples tokens from logits or probability distributions. :attr:`TensorType.LOGITS` -> :attr:`TensorType.INDICES` | :attr:`TensorType.PROBS` -> :attr:`TensorType.INDICES` Parameters ---------- deterministic : bool, optional, Compile-time Whether to use deterministic kernel implementation. Default is True. indices : torch.Tensor, optional, Runtime Indices for batched sampling when probability tensors are shared. generator : torch.Generator, optional, Runtime Random number generator for reproducible sampling. Examples -------- >>> import torch >>> from flashinfer.logits_processor import LogitsPipe, Sample, TensorType >>> torch.manual_seed(42) >>> >>> # Sampling from logits >>> pipe = LogitsPipe([Sample(deterministic=True)], input_type=TensorType.LOGITS) >>> logits = torch.randn(2, 5, device="cuda") >>> logits tensor([[ 0.1940, 2.1614, -0.1721, 0.8491, -1.9244], [ 0.6530, -0.6494, -0.8175, 0.5280, -1.2753]], device='cuda:0') >>> tokens = pipe(logits, top_k=1) >>> tokens tensor([0, 1], device='cuda:0') >>> >>> # Sampling from probabilities >>> pipe = LogitsPipe([Sample(deterministic=True)], input_type=TensorType.PROBS) >>> probs = torch.randn(2, 5, device="cuda") >>> probs_normed = probs / probs.sum(dim=-1, keepdim=True) >>> probs_normed tensor([[ 2.8827, 0.0870, 0.2340, -3.2731, 1.0694], [ 0.3526, 0.0928, 0.1601, -0.1737, 0.5683]], device='cuda:0') >>> tokens = pipe(probs_normed, top_k=1) >>> tokens tensor([0, 0], device='cuda:0') Notes ----- Outputs :attr:`TensorType.INDICES` - no operators can follow See Also -------- :meth:`~flashinfer.sampling.sampling_from_logits` :meth:`~flashinfer.sampling.sampling_from_probs` T deterministicr c >tjdd|i|dS)z Constructor for Sample processor. Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. Default is True. rYNrr/)rrYr r$s rrzSample.__init__s- ??}??????rrrcddlm}m}|tjkr|di|jgS|tjkr|di|jgStd|)r4r)LogitsSampleOp ProbsSampleOpzSampling cannot be applied to Nr)r6r\r]r r7r rJr8)rrr\r]s rrzSample.legalizes =<<<<<<< * * *"N11T[112 2 :+ + +!M00DK001 1JjJJKK Kr)TrKr;s@rrXrXrs55n @ @d @S @ @ @ @ @ @ L: L$r( L L L L L L L LrrXN)r*abcrrtypingrrropr typesr r r-r=rDrMrSrXrrrrbs $#######&&&&&&&&&&?:?:?:?:?:c?:?:?:D,.,.,.,.,./,.,.,.^5*5*5*5*5*o5*5*5*pLHLHLHLHLH?LHLHLH^/'/'/'/'/'?/'/'/'d-'-'-'-'-'?-'-'-'`OLOLOLOLOL_OLOLOLOLOLr