.`i+yUddlmZddlmZddlZddlmZddlmZddl m Z m Z ddl m Z mZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZeeZd Ze je d<dZ!e je d<dZ"Gddej#Z$dej%de&e'de'dej%dej%dzdej%dej%dedej%fdZ(dej%dej%dedej%fdZ) d1dej%d ej%d!e'd"e'd#e'dej%f d$Z*d!e'de&e'd%e+e'ej,fd&ej-dej%f d'Z.de'de&e'dej%dej%dej%dzdej%ded&ej-dej%fd(Z/e j0dg)d*Z1e j0dg)d+e jfd,Z2e j0d"d#g)d-e jfd.Z3e j0d/e jd+e jfd0Z4dS)2)Sequence)replaceN) init_logger)tltriton) LogprobsListsLogprobsTensors SamplerOutput)SamplingMetadata)apply_bad_words_with_drafts)apply_all_penalties)apply_top_k_top_p)Sampler)SpecDecodeMetadataPLACEHOLDER_TOKEN_IDGREEDY_TEMPERATUREc.eZdZdZdeffd Zdedejdzdejde d e f d Z d e dedejd ejd ejdejd e fdZe ddejde dee de dzd eeee edzff dZdejde ded ejfdZedejde dedejdeee d ejf dZe ddeee deee dzd eee fdZxZS)RejectionSamplerau The implementation strictly follows the algorithm described in https://arxiv.org/abs/2211.17192. However, we want to clarify the terminology used in the implementation: accepted tokens: tokens that are accepted based on the relationship between the "raw" draft and target probabilities. recovered tokens: tokens that are sampled based on the adjusted probability distribution, which is derived from both the draft and target probabilities. bonus tokens: If all proposed tokens are accepted, the bonus token is added to the end of the sequence. The bonus token is only sampled from the target probabilities. We pass in the bonus tokens instead of sampling them in the rejection sampler to allow for more flexibility in the sampling process. For example, we can use top_p, top_k sampling for bonus tokens, while spec decode does not support these sampling strategies. output tokens: Tokens are finally generated with the rejection sampler. output tokens = accepted tokens + recovered tokens + bonus tokens samplerct||_|jj}|d|_|d|_dS)N processedlogits)super__init__r logprobs_mode startswithis_processed_logprobs_modeendswithis_logits_logprobs_mode)selfrr __class__s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/v1/sample/rejection_sampler.pyrzRejectionSampler.__init__4s\   2 *7*B*B;*O*O''4'='=h'G'G$$$metadata draft_probsNrsampling_metadatareturnc |jtksJ|j}|j}|J||}||t |dd|jrdnd}|j} ||} | tj } | } |js| } | | ||} t| |j|} | dtj } t!|j|j|j|j|| | |} d}|j2||j|||jr| n| |jj| }t/| | S) a^ Args: metadata: Metadata for spec decoding. draft_probs (Optional[torch.Tensor]): Probability distribution for the draft tokens. Shape is [num_tokens, vocab_size]. Can be None if probabilities are not provided, which is the case for ngram spec decode. logits (torch.Tensor): Target model's logits probability distribution. Shape is [num_tokens + batch_size, vocab_size]. Here, probabilities from different requests are flattened into a single tensor because this is the shape of the output logits. NOTE: `logits` can be updated in place to save memory. sampling_metadata (vllm.v1.sample.metadata.SamplingMetadata): Additional metadata needed for sampling, such as temperature, top-k/top-p parameters, or other relevant information. Returns: SamplerOutput: Contains the final output token IDs and their logprobs if requested. Nr)max_num_logprobsTprocessed_logits raw_logits)rr(predict_bonus_tokenlogprobs_mode_override)dimdtype)sampled_token_idslogprobs_tensors) max_spec_len MAX_SPEC_LENbonus_logits_indicestarget_logits_indicesrrrr2totorchfloat32cloneapply_logits_processorsapply_sampling_constraintscu_num_draft_tokenssoftmaxrejection_sampledraft_token_idsnum_draft_tokensr+_get_logprobs_tensorsr3logprobsr )r"r&r'rr(r6r7 bonus_logitsbonus_sampler_outputbonus_token_idsraw_target_logits target_logits target_probsoutput_token_idsr3s r$forwardzRejectionSampler.forward;s>$ 4444'< ( > !!!23 #||%!!#!%.$#5#5 , /@ ##89-00??) . 2*//11M44 ,h  3   (   %,,5=,II +  $  %  !  (          - 9#99!2!%!@W FW$5>    .-    r%r+rIrEr2ctj|j}|jdd|dd<|j}|j} tj|tj} |tj| | <|tj| |<|} tj|jd| j | j } | d| dz } | | jddz | }d||tk<| | }|jr|n|j|}|j|||tjS)Nr)r1)devicer1r)max)r9 zeros_likecu_num_sampled_tokensr6r7r:r8arangeshaperOr1 unsqueezeflattenclamp_r;rr!rcompute_logprobsgather_logprobsint64)r"r+r&rrIrEr2rRr6r7 final_logitslogit_start_indicesoffsetsaccepted_logit_indicesaccepted_tokensaccepted_logitsaccepted_logprobss r$rCz&RejectionSampler._get_logprobs_tensorss!& 01O P P$,$B3B3$Gabb! (< ( >'emDDD .;.>.>u}.M.M *+-9__U]-K-K )* 4,  #B '&-%+    ) )! , ,w/@/@/C/C C '))  %%,*+ @OO..??  |++     u{ + +   r%rK vocab_sizediscard_req_indicesr3c |}|tk||kz d}|zdg dz}| }||}t|dkrd |< fdt|D}||fS)aMParse the output of the rejection sampler. Args: output_token_ids: The sampled token IDs in shape [batch_size, max_spec_len + 1]. The rejected tokens are replaced with `PLACEHOLDER_TOKEN_ID` by the rejection sampler and will be filtered out in this function. vocab_size: The size of the vocabulary. discard_req_indices: Optional row indices to discard tokens in. logprobs_tensors: Optional logprobs tensors to filter. Returns: A list of lists of token IDs. NrrNaxisFcVg|]%\}}||&Srb)tolist).0irow valid_masks r$ z1RejectionSampler.parse_output..s?   ,2AsC 1  % % ' '   r%) cpunumpyrsumcumsumrifilterrVtolistslen enumerate) rKrcrdr3output_token_ids_npoutput_logprobs cu_num_tokensfiltered_tensorsoutputsrms @r$ parse_outputzRejectionSampler.parse_outputs &/2244::<<)-AA * ,   'C*..a."8"8"?"?"A"A"H"H"J"JJM/66z7I7I7K7KLL .66}EEO " # #a ' '.3J* +    6?@S6T6T   ''r%cN|j }|jp|}|j}|r|||j}|j|rt |j}tj|j d}tj |d} | |} | |j d} ||||| |}|j0|j| } || t!d|jx} rt#|| ||j |S)Nro)rOT)rO non_blocking-inf) no_penaltiesbad_words_token_idsrK!_combine_outputs_with_spec_tokensspec_token_idsallowed_token_ids_maskrur9tensorrBrSrepeat_interleaver8rOapply_penalties masked_fill_floatr )r"rr(r& has_penaltiesany_penalties_or_bad_wordsrK num_requestsrBoriginal_indicesrepeat_indices_cpurepeat_indices token_maskrs r$r<z(RejectionSampler.apply_logits_processorssg .::  1 B] #-= % #EE !0     3 ?= ?0ABBL$|H,EeTTT $|LGGG !1!C!CDT!U!U /22}43N)))8^EUF !7C.EnU ##Jf >>>#4"G G   '+-=x?X    r%rc|jr|S|jJ|j|}|j|}|j|}|j|}t ||||||}|SN)rprompt_token_idspresence_penaltiesfrequency_penaltiesrepetition_penaltiesr ) rr(r&rrKrrrrs r$rz RejectionSampler.apply_penalties+s  ) M 1===,=nM.A.Q/CNS0EnU$         r%rc||Sg}t||D]u\}}t|dkr||tt|dz D]'}|g|d||(v|S)NrrNr)zipruappendrange)rKrresultoutspecrks r$rz2RejectionSampler._combine_outputs_with_spec_tokensGs  !# #-~>> 6 6IC4yyA~~ MM#   3t99q=)) 6 6 4r 4DG45555 6 r%)rbNr)__name__ __module__ __qualname____doc__rrrr9Tensorr r rLintr rC staticmethodrtuplelistrr|r<rr __classcell__)r#s@r$rrs,HHHHHHHl $l \D( l  l ,l  l l l l \/ / %/  / | / l / !</  / / / / b.037 "("(,"("(&c]"(*D0 "( tDI  44 5 "("("(\"(H* *,*% *  ****X +%  tCy/   \626tCy/T#Y$. d3i\r%rrArBr4r>r'rJrGr(r)cV|jdksJ| |jdksJ|jdksJ|jdksJt|}|jd} |jd} |j} |sJ||sJ|sJ|sJ|j| | fksJt j||dzftt j| } |j rd} n|j tk} |j s;| d}t|f| ||||| ||j r| St| ||j| }t#|||||||| }t%|f| |||||||| || |du | S)NrNrrr1rO)r0NO_DRAFT_PROBS)ndimrurTrO is_contiguousr9fullrint32 all_greedy temperaturer all_randomargmaxrejection_greedy_sample_kernelgenerate_uniform_probs generatorssample_recovered_tokensrejection_random_sample_kernel)rArBr4r>r'rJrGr( batch_size num_tokensrcrOrK is_greedy target_argmax uniform_probsrecovered_token_idss r$r@r@YsX  1 $ $ $ $  +"2a"7"7"7 7  #q ( ( ( (   ! ! ! !%&&J &q)J#B'J  F  ( ( * *** *  +";";"="=   =  % % ' ''' '  ( ( * *** *  *j!9 9 9 9 9z \A%&k #H %15GG  ' $$+++33 & }5            ' $# #+$ M2  #J=1"d*     r%rc|jdksJ|jdksJ|jr|S|jd}t|j||t d}||dd}|jt|j||}d}|j t|j ||}t|||S)aRProcess logits based on sampling metadata. This function applies temperature scaling to the logits, as well as top-k and top-p. For greedy decoding, it returns the original logits. Args: logits: Input logits tensor to be processed. cu_num_draft_tokens: Cumulative number of draft tokens. sampling_metadata: Metadata containing sampling parameters such as temperature and whether greedy sampling is used. Returns: torch.Tensor: Processed logits if non-greedy sampling is used, otherwise returns the original logits. rrNr) replace_from replace_torN) rrrTexpand_batch_to_tokensrrdiv_rUtop_ktop_pr)rr>r(rrrrs r$r=r=s* ;!      #q ( ( ( (# aJ(%' K KK %%b))*** E*&  #     E*&  #     VUE 2 22r%xryrrrc|jd}|jd|ksJ||}t|f|||||t|S)aExpand [batch_size] tensor to [num_tokens] tensor based on the number of tokens per batch in cu_num_tokens. For example, if x = [a, b, c] and cu_num_tokens = [2, 5, 6], then num_tokens = 6, and expanded_x = [a, a, b, b, b, c]. Args: x: [batch_size] tensor to expand. cu_num_tokens: [batch_size] tensor containing the cumulative number of tokens per batch. Each element represents the total number of tokens up to and including that batch. num_tokens: Total number of tokens. replace_from: int = 0 Value to be replaced if it is found in x. replace_to: int = 0 Value to replace with when replace_from is found. Returns: expanded_x: [num_tokens] tensor. r)MAX_NUM_TOKENS)rT new_empty expand_kernelr5)rryrrrr expanded_xs r$rrsq4J  q !Z / / / /Z((J:-  #  r%rrOctj|ftj|}d}t|D]H\}}|dkr ||z}||} | |||| |}I|S)a Generates a batch of uniform random samples, with optional seeding if available. This method creates a tensor of shape `(num_tokens, )` filled with uniform random values in the range [0, 1). If `generators` is provided, the requests with their own seeds will use the provided `torch.Generator` for reproducibility. The samples for the other requests will be generated without a seed. Args: num_tokens: int Total number of tokens. num_draft_tokens: List[List[int]] Number of draft tokens per request. generators: Optional[Dict[int, torch.Generator]] A dictionary mapping indices in the batch to `torch.Generator` objects. device: torch.device The device on which to allocate the tensor. Returns: uniform_rand: torch.Tensor A tensor of shape `(num_tokens, )` containing uniform random values in the range [0, 1). rrN generator)r9randfloat64rvgetuniform_) rrBrrOr start_idxreq_idxnend_idxrs r$rrsHJ  mM I 011    66 a-NN7++  )G+ , 5 5 5 J J J r%c t|}|jd} tj|| ftj|} | |jD]-\} } || dkr| | | .tj|} t||f| ||||| | tj | |du | S)Nrrrrr) rurTr9emptyr: exponential_ritems empty_likesample_recovered_tokens_kernelrnext_power_of_2)r4rBr>rAr'rJr(rOrrcqrkrrs r$rrTs %&&J#B'J Z m   A NN)4::<<33 9 A  " " aD    2 2 2*?;;"J #=> z**"d*     r%)do_not_specializecFtjd}|dntj||z}|sdS|dkrdntj||zdz } tj||z} | | z } d} t| D]a} | s]tj|| z| z}tj|| z| z}tj|||dzzz| z|||krd} b| s:tj||z}tj|||dzzz| z|dSdS)NrTrNFr program_idloadrstore)output_token_ids_ptrcu_num_draft_tokens_ptrdraft_token_ids_ptrtarget_argmax_ptrbonus_token_ids_ptr is_greedy_ptrr4rrrrrBrejectedposdraft_token_idtarget_argmax_idbonus_token_ids r$rrsmAG&-27=7;R3S3SI \\rw/F/PST/T'U'UIg-788G*H%&&    W%89%Ds%JKKN!w'89'Ds'JKK  H$w,2B'CCcI    !111  !4w!>??  7lQ.>#? ?BR R        r%rc tjd} tj|| z} | rdS| dkrdntj|| zdz }tj|| z}||z }d}t|D]}|stj||z|z}| rd}n tj|||z| zz|z}tj|||z| zz|z}tj||z|z}|dkr ||z |kr|}nd}tj||z|z}tj|| | dzzz|z||s:tj|| z}tj|| | dzzz|z|dSdS)NrrNFTr)rrrdraft_probs_ptrtarget_probs_ptrrrecovered_token_ids_ptruniform_probs_ptrrr4rcrrrrrrBrrr draft_prob target_prob uniform_probtoken_idrs r$rrsmAG /00I\\rw/F/PST/T'U'UIg-788G*H%&& W%89%Ds%JKKN  W#y3*&DD~U ' IOz#AANRK7#4y#@3#FGGLA~~+ ":l"J"J) 7#:Y#F#LMM H$w,2B'CCcI8     !4w!>??  7lQ.>#? ?BR R        r%rcztjd}|dkrd}ntj||zdz }tj||z}||z } tj||z} tj| |k|| } tjd|} tj||z| z| | | kdS)NrrN)mask)rrrwhererSr) output_ptr input_ptrcu_num_tokens_ptrrrrrrrrsrc_valoffsets r$rrsmAG!|| G-7!;<< g''122G9$Jgi')**Ghw,. GDDG Yq. ) )FHZ) #f ,gFZrst%$$$$$$ ######((((((((IIIIIIIIII444444DDDDDD<<<<<<BBBBBB******;;;;;; X  %'bl'''#$BL$$$ yyyyyryyyyx _\_3i _  __$_,_\_(_ \____D73 L7373(73 \ 73737373| %% |%<%% %  %  \ %%%%P443i4S%/)*4 L 4  \ 4444n))3i) ) \ )$),)() L) \))))Z ~.///( ( 0/( X ~.///9 L9 9 9 0/9 z ~|<===Q L QQQ>=Q,2C|2CL2C2C2C 2C2C2Cr%