)`i N0dZddlmZmZmZmZddlZddlmZddl m Z m Z ddl m Z ddlmZmZd eejeefd eeejeeefffd ZGd d e ZGdde ZGdde ZGdde ZGdde ZGdde ZGdde ZGdde ZGdde ZGdde ZGd d!e ZGd"d#e Z Gd$d%e Z!dS)&a3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. )AnyOptionalTupleUnionN)get_sampling_module)_get_cache_bufdevice_support_pdl)ParameterizedOp) TaggedTensor TensorTypexreturnctt|tjr||jtjkrdndfSd|fS)Nrg) isinstancetorchTensordtypeint32)rs y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/logits_processor/operators.py_to_tensor_scalar_tuplers?!U\""5;..11C88aycBeZdZdZejZejZdede defdZ dS) TemperatureOpz Temperature scaling operator. :attr:`TensorType.LOGITS` -> :attr:`TensorType.LOGITS` Parameters ---------- temperature : float or torch.Tensor Temperature value for scaling. tensorkwargsrc ||}|d|d}t|\}}|*t|tr|dkrt d||}n|}|j|z }t||S)N temperatureTrequiredr4Temperature must be positive float or a tensor array)_validate_input_type _get_paramrrfloat ValueErrordatar )selfrr output_typermaybe_temperature_arrtemperature_val scaled_logitss r__call__zTemperatureOp.__call__4s//77 oomVdoKK 1H1U1U. (?E22 )6E6J6JSTT T ,/KK)K k1 M;777rN __name__ __module__ __qualname____doc__r LOGITSINOUTr rr,rrrr%sY    B  C8|8s8|888888rrcBeZdZdZejZejZde de de fdZ dS) SoftmaxOpa Softmax operator. Converts logits to probabilities using softmax function. :attr:`TensorType.LOGITS` -> :attr:`TensorType.PROBS` Parameters ---------- enable_pdl: bool, optional Whether to enable PDL for the fused kernel. rrrc ||}|jdd}|t|jj}t j|jd}t||S)N enable_pdl)dim) r"default_paramsgetr r&devicersoftmaxr )r'rrr(r9probss rr,zSoftmaxOp.__call__Ysk//77 (,,\4@@  +FK,>??J fkr222E;///rN) r.r/r0r1r r2r3PROBSr4r rr,r5rrr7r7HsY    B  C0|0s0|000000rr7cBeZdZdZejZejZdede defdZ dS) ProbsTopKOpag Top-k filtering operator for probabilities. Keeps top-k probabilities, zeros out others, and renormalizes. :attr:`TensorType.PROBS` -> :attr:`TensorType.PROBS` Parameters ---------- top_k : int or torch.Tensor Number of top tokens to keep. See Also -------- :meth:`~flashinfer.sampling.top_k_renorm_probs` rrrc ||}|d|d}t|\}}|*t|tr|dkrt dt d|jjd|jjd}t |j|||}t||S) Ntop_kTrr2top_k must be a positive integer or a tensor arraytop_k_renorm_probs_row_states_ zero_init) r"r#rrintr%rr&r>rtop_k_renorm_probsr ) r'rrr(rEmaybe_top_k_arr top_k_valrow_states_buffer renorm_probss rr,zProbsTopKOp.__call__ys//77 $??% :attr:`TensorType.LOGITS` Parameters ---------- top_k : int or torch.Tensor Number of top tokens to keep. See Also -------- :class:`~flashinfer.sampling.top_k_mask_logits` rrrc ||}|d|d}t|\}}|*t|tr|dkrt dt d|jjd|jjd}t |j|||}t||S) NrETrrrFtop_k_mask_logits_row_states_rHrI) r"r#rrrKr%rr&r>rtop_k_mask_logitsr ) r'rrr(rErMrNrO masked_logitss rr,zLogitsTopKOp.__call__s//77 $??% @ @  K      ,--?? K)5F  M;777rNr-r5rrrSrSsY"  B  C8|8s8|888888rrScBeZdZdZejZejZdede defdZ dS)TopPOpa Top-p (nucleus) filtering operator. Keeps tokens with cumulative probability up to threshold p, zeros out others, and renormalizes. :attr:`TensorType.PROBS` -> :attr:`TensorType.PROBS` Parameters ---------- top_p : float or torch.Tensor Cumulative probability threshold in (0, 1]. See Also -------- :meth:`~flashinfer.sampling.top_p_renorm_probs` rrrc 0||}|d|d}t|\}}|d|cxkrdksntdt |j||}t||S)Ntop_pTrrr /top_p must be float in (0, 1] or a tensor array)r"r#rr%rtop_p_renorm_probsr&r )r'rrr(r[maybe_top_p_arr top_p_valrPs rr,zTopPOp.__call__s//77 $??%,>,>,>Q,>,>,>,>NOO O*,,?? K)  L+666rNrQr5rrrYrYsY"  B  C 7| 7s 7| 7 7 7 7 7 7rrYcBeZdZdZejZejZdede defdZ dS)MinPOpa Min-p filtering operator. Keeps tokens with probability at least p times the maximum probability, zeros out others, and renormalizes. :attr:`TensorType.PROBS` -> :attr:`TensorType.PROBS` Parameters ---------- min_p : float or torch.Tensor Minimum probability threshold as ratio of max probability. See Also -------- :meth:`~flashinfer.sampling.min_p_renorm_probs` rrrc 6||}|d|d}t|\}}|d|cxkrdksntd|B|j|d|jdddzk}n.|j||jdddzk}|j}d||<||ddz } t| |S) Nmin_pTrrr /min_p must be float in (0, 1] or a tensor arrayr:)r;keepdim) r"r#rr%r& unsqueezemaxclonesumr ) r'rrr(rcmaybe_min_p_arr min_p_val min_p_mask masked_probsr@s rr,zMinPOp.__call__s///77 $??%,>,>,>Q,>,>,>,>NOO O  &))"-- BPT0U0UVW0XXJJ FKOODOAA!DDJ{((** $% j[!|//B/EEEE;///rNrQr5rrrarasY"  B  C0|0s0|000000rracBeZdZdZejZejZde de de fdZ dS) ProbsSampleOpa) Sampling operator for probabilities. Samples token indices from probability distribution using inverse transform sampling. :attr:`TensorType.PROBS` -> :attr:`TensorType.INDICES` Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. indices : torch.Tensor, optional Indices for batched sampling. generator : torch.Generator, optional Random number generator. See Also -------- :meth:`~flashinfer.sampling.sampling_from_probs` rrrc 4||}|jdd}|d|d}|d|d}t |j|||}t||SN deterministicTindicesFr generator)r"r<r=r#rsampling_from_probsr&r r'rrr(rrrsrtsampless rr,zProbsSampleOp.__call__+s//77 +//FF //)Ve/DDOOK%OHH %'';; K-  G[111rN) r.r/r0r1r rAr3INDICESr4r rr,r5rrrorosY*  B  C 2| 2s 2| 2 2 2 2 2 2rrocBeZdZdZejZejZde de de fdZ dS)LogitsSampleOpa Sampling operator for logits. Samples token indices from logits using Gumbel-max trick. :attr:`TensorType.LOGITS` -> :attr:`TensorType.INDICES` Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. indices : torch.Tensor, optional Indices for batched sampling. generator : torch.Generator, optional Random number generator. See Also -------- :meth:`~flashinfer.sampling.sampling_from_logits` rrrc 4||}|jdd}|d|d}|d|d}t |j|||}t||Srq)r"r<r=r#rsampling_from_logitsr&r rvs rr,zLogitsSampleOp.__call__Ss//77 +//FF //)Ve/DDOOK%OHH %''<< K-  G[111rN) r.r/r0r1r r2r3rxr4r rr,r5rrrzrz:sY*  B  C 2| 2s 2| 2 2 2 2 2 2rrzcjeZdZdZejZejZd de e de ffd Z de de de fd ZxZS) FusedTemperatureSoftmaxOpay Fused temperature scaling and softmax operator. :attr:`TensorType.LOGITS` -> :attr:`TensorType.PROBS` Parameters ---------- enable_pdl: bool, optional Whether to enable PDL for the fused kernel. temperature : float or torch.Tensor Temperature value for scaling. See Also -------- :meth:`~flashinfer.sampling.softmax` Nr9r<c >tjdd|i|dS)Nr9r5super__init__)r'r9r< __class__s rrz"FusedTemperatureSoftmaxOp.__init__xs+AAJA.AAAAArrrrc ||}|d|d}t|\}}|*t|tr|dkrt dt dd|jj}|j dd}|t|jj}t ||j|||} t| |S) NrTrrr!softmax_workspacerHr9)r"r#rrr$r%rr&r>r<r=r rr?r ) r'rrr(rr)r*workspace_bufferr9r@s rr,z"FusedTemperatureSoftmaxOp.__call__{s//77 oomVdoKK 1H1U1U. (?E22 )6E6J6JSTT T) fk.@  (,,\4@@  +FK,>??J#%%--  K !     E;///r)N)r.r/r0r1r r2r3rAr4rboolrrr r, __classcell__rs@rr~r~cs"  B  CBB8D>BCBBBBBB0|0s0|00000000rr~c^eZdZdZejZejZd de de ffd Z de de de fd Z xZS) FusedProbsTopKSampleOpa Fused top-k filtering and sampling operator for probabilities. Use rejection sampling to directly sample from the top-k probabilities. :attr:`TensorType.PROBS` -> :attr:`TensorType.INDICES` Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. top_k : int or torch.Tensor Number of top tokens to keep. indices : torch.Tensor, optional Indices for batched sampling. generator : torch.Generator, optional Random number generator. See Also -------- :meth:`~flashinfer.sampling.top_k_sampling_from_probs` Trrr<c >tjdd|i|dSNrrr5rr'rrr<rs rrzFusedProbsTopKSampleOp.__init__+GG}GGGGGGrrrrc ||}|jdd}|d|d}t |\}}|*t |t r|dkrtd|d|d}|d |d} t |j ||||| } t| |S) NrrTrErrrFrsFrt) r"r<r=r#rrrKr%rtop_k_sampling_from_probsr&r ) r'rrr(rrrErMrNrsrtrws rr,zFusedProbsTopKSampleOp.__call__s//77 +//FF $??% :attr:`TensorType.INDICES` Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. top_p : float or torch.Tensor Cumulative probability threshold. indices : torch.Tensor, optional Indices for batched sampling. generator : torch.Generator, optional Random number generator. See Also -------- :meth:`~flashinfer.sampling.top_p_sampling_from_probs` Trrr<c >tjdd|i|dSrrrs rrzFusedProbsTopPSampleOp.__init__rrrrrc ||}|jdd}|d|d}t |\}}|d|cxkrdksnt d|d|d }|d |d } t |j||||| } t| |S) NrrTr[rrr r\rsFrt) r"r<r=r#rr%rtop_p_sampling_from_probsr&r ) r'rrr(rrr[r^r_rsrtrws rr,zFusedProbsTopPSampleOp.__call__//77 +//FF $??%,>,>,>Q,>,>,>,>NOO O//)Ve/DDOOK%OHH %''AA K/9mY  G[111rrrrs@rrr.  B  CHHdHSHHHHHH2|2s2|22222222rrc^eZdZdZejZejZd de de ffd Z de de de fd Z xZS) FusedProbsMinPSampleOpu_ Fused min-p filtering and sampling operator for probabilities. Use rejection sampling to directly sample from the min-p probabilities. PROBS → INDICES Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. min_p : float or torch.Tensor Minimum probability threshold. indices : torch.Tensor, optional Indices for batched sampling. generator : torch.Generator, optional Random number generator. See Also -------- :meth:`~flashinfer.sampling.min_p_sampling_from_probs` Trrr<c >tjdd|i|dSrrrs rrzFusedProbsMinPSampleOp.__init__rrrrrc ||}|jdd}|d|d}t |\}}|d|cxkrdksnt d|d|d }|d |d } t |j||||| } t| |S) NrrTrcrrr rdrsFrt) r"r<r=r#rr%rmin_p_sampling_from_probsr&r ) r'rrr(rrrcrjrkrsrtrws rr,zFusedProbsMinPSampleOp.__call__rrrrrs@rrrrrrc^eZdZdZejZejZd de de ffd Z de de de fd Z xZS) FusedProbsTopKTopPSampleOpa9 Fused top-k, top-p filtering and sampling operator for probabilities. Use rejection sampling to directly sample from the probabilities, top-k and top-p filtering are applied jointly (rather than applying first -> renormalize -> second). :attr:`TensorType.PROBS` -> :attr:`TensorType.INDICES` Parameters ---------- deterministic : bool, optional Whether to use deterministic kernel implementation. top_k : int or torch.Tensor Number of top tokens to keep. top_p : float or torch.Tensor Cumulative probability threshold. indices : torch.Tensor, optional Indices for batched sampling. generator : torch.Generator, optional Random number generator. See Also -------- :meth:`~flashinfer.sampling.top_k_top_p_sampling_from_probs` Trrr<c >tjdd|i|dSrrrs rrz#FusedProbsTopKTopPSampleOp.__init__Prrrrrc |||}|jdd}|d|d}t |\}}|d|d}t |\} } |*t |t r|dkrtd| d| cxkrdksntd |d |d } |d |d } t |j | ||| | || } t| |S) NrrTrErr[rrFr r\rsFrt) r"r<r=r#rrrKr%rtop_k_top_p_sampling_from_probsr&r )r'rrr(rrrErMrNr[r^r_rsrtrws rr,z#FusedProbsTopKTopPSampleOp.__call__SsW//77 +//FF $??%,>,>,>Q,>,>,>,>NOO O//)Ve/DDOOK%OHH %''GG K          G[111rrrrs@rrr3s2  B  CHHdHSHHHHHH!2|!2s!2|!2!2!2!2!2!2!2!2rr)"r1typingrrrrrflashinfer.samplingrflashinfer.utilsrr opr typesr r rr$rKrrr7rCrSrYrarorzr~rrrrr5rrrs /........... 333333????????++++++++ U\5# %& 8EL !5#4 45 8 8 8 8 8O 8 8 8F000000008+7+7+7+7+7/+7+7+7\*8*8*8*8*8?*8*8*8Z"7"7"7"7"7_"7"7"7J+0+0+0+0+0_+0+0+0\%2%2%2%2%2O%2%2%2P%2%2%2%2%2_%2%2%2R2020202020202020j2222222222_222222j0202020202_020202f0202020202_020202fA2A2A2A2A2A2A2A2A2A2r