.`i'1dZddlZddlmZddlmZddlmZddlm Z m Z ddl m Z ddl mZddlmZdd lmZdd lmZd ZGd d ejZdS)z>A layer that samples the next tokens from the model's outputs.N) LogprobsMode)is_pin_memory_available)LogprobsTensors SamplerOutput)SamplingMetadata)apply_bad_words)batched_count_greater_than)apply_all_penalties)TopKTopPSamplergh㈵>c eZdZdZddeffd Z ddejded e d edzd e f d Z e dejd ejde d ejfdZ e dejd ejfdZ ddejded edzd eejejdzffdZe dejd ejfdZe dejdedejd efdZe ddeeedeeedzd eeefdZdejded e d ejfdZe dejdedeeed ejfdZxZS)SampleraM A layer that samples the next tokens from the model's outputs with the following steps in order: 1. If logprobs are requested: a) If `logprobs_mode` is `raw_logprobs`, compute logprobs as the final logprobs to return. b) If `logprobs_mode` is `raw_logits`, clone the logits as the final logprobs to return. 2. Convert logits to float32. 3. Apply allowed token ids whitelist. 4. Apply bad words exclusion. 5. Apply logit processors which are not argmax-invariant, i.e. that can impact greedy sampling. a) Min tokens processor b) Logit bias processor 6. Apply penalties a) Repetition penalty b) Frequency penalty c) Presence penalty 7. Sample the next tokens. `sample` method performs the following steps: a) If not `all_random`, perform greedy sampling. If `all_greedy`, return the greedily sampled tokens and final logprobs if requested. b) Apply temperature. c) Apply logit processors which are argmax-invariant, by default the min_p processor. d) Apply top_k and/or top_p. e) Sample the next tokens with the probability distribution. f) If `all_random` or temperature >= epsilon (1e-5), return the randomly sampled tokens and final logprobs if requested. Else, return the greedily sampled tokens and logprobs if requested. 8. Gather the logprobs of the top `max_num_logprobs` and sampled token (if requested). Note that if the sampled token is within the top `max_num_logprobs`, the logprob will be eventually merged in `LogprobsProcessor` during output processing. Therefore, the final output may contain either `max_num_logprobs + 1` or `max_num_logprobs` logprobs. 9. Return the final `SamplerOutput`. raw_logprobs logprobs_modectt||_t |_||_dSN)super__init__r topk_topp_samplerr pin_memoryr)selfr __class__s j/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/v1/sample/sampler.pyrzSampler.__init__=sD !0!?!?133*FNlogitssampling_metadatapredict_bonus_tokenlogprobs_mode_overridereturnc|p|j}|j}|k|dkr||}nO|dkrI|jtjkr|}n|tj}|tj}||||}| ||\}} | | }| }|d} nT|dkr6tt j d|t j d} n| |||} |tj}t|d| } | S)Nr raw_logitsr) token_ids)sampled_token_idslogprobs_tensors)rmax_num_logprobscompute_logprobsdtypetorchfloat32clonetoapply_logits_processorssamplelongremptygather_logprobsint32r unsqueeze) rrrrrr num_logprobsrsampledprocessed_logprobsr$sampler_outputs rforwardzSampler.forwardCs/D$2D )9  #..#44V<< ,..<5=00#)<<>>LL#)99U]#;#;L5=))-- %':  '+kk&:K&L&L##  )-L ,,..  #   R  . A ek!nn     $33lg 4    **U[))'&//33-    rtemp all_randomc|stj|tkd|}||dS)Ng?dim)r(where _SAMPLING_EPSdiv_r2)rr8r9s rapply_temperaturezSampler.apply_temperaturesE @;tm3S$??D{{4>>a>00111rcT|ddS)Nr!r<)argmaxviewrs r greedy_samplezSampler.greedy_samples$}}}$$))"---rc2|p|j}|jr |jrJ|jrd}nM||}|jr1d}|j$|dkr|}n|dkr||}||fS|jJ|||j|j}|jj D]}| |}| ||j |j |j\}}|||fStj|jt"k|||} | |fS)zSample logits based on sampling metadata. The various logits processing functions called in this method may update the logits tensor in-place. Nprocessed_logitsr5)out)r all_greedyr9rFr%r& temperaturerA logitsprocsargmax_invariantapplyr generatorstop_ktop_pr(r>r?) rrrrrgreedy_sampledr5 processorrandom_sampledr4s rr-zSampler.samples/D$2D %0R5F5QRRR  ' :!NN!//77N + :%)"$5A$(:::-3**&*>>>-1-B-B6-J-J*%'999 ,888'' %13D3O   +6G - -I__V,,FF.2-C-C   (  #  # . . **  !!#55 5+  )M 9       ***rcD|dtjS)Nr!)r=r') log_softmaxr(r)rEs rr&zSampler.compute_logprobss!!b !>>>rlogprobsr3r"c|jtjksJtj||d\}}|d}|d|}t ||}tj||fd}tj||fd}|tj }t|||S)a Gather logprobs for topk and sampled/prompt token. Args: logprobs: (num tokens) x (vocab) tensor num_logprobs: minimum number of logprobs to retain per token token_ids: prompt tokens (if prompt logprobs) or sampled tokens (if sampled logprobs); 1D token ID tensor with (num tokens) elements Must be int64. Returns: Top-k int indices tensor, (num tokens) x (num_logprobs + 1) Top-k float logprobs tensor, (num tokens) x (num_logprobs + 1) Sampled token rank tensor, (num tokens) r!r<r;) r'r(int64topkr2gatherr catr+r1r)rWr3r" topk_logprobs topk_indicestoken_logprobs token_ranksindicess rr0zSampler.gather_logprobss0%+----&+jJJ )Y 51===9nm>>routput_token_idsspec_token_idsc>||Sdt||DS)Nc(g|]\}}|rg||n|Srf).0rIspecs r z=Sampler._combine_outputs_with_spec_tokens..s=   T" *McMDMMs   r)zip)rbrcs r!_combine_outputs_with_spec_tokensz)Sampler._combine_outputs_with_spec_tokenss<  !# #   !1>BB    rc|j}t|p|j }|j}|r|r|||j}|j(||jtd|rt||||j j D]}| |}| |||}|S)Nz-inf)bad_words_token_idsbool no_penaltiesrbrkrcallowed_token_ids_mask masked_fill_floatrrLnon_argmax_invariantrNapply_penalties)rrrrrmany_penalties_or_bad_wordsrbrSs rr,zSampler.apply_logits_processors s 0C $ % % K->-K)K #-=  #=  $EE !0     3 ?    1 H%PV-- X X X  K F$79I J J J+6K - -I__V,,FF%%f.?AQRR rcv|jr|S|jJt||j|j|j|j|Sr)roprompt_token_idsr presence_penaltiesfrequency_penaltiesrepetition_penalties)rrrbs rrtzSampler.apply_penalties.sS  ) M 1==="   .  0  1  2     r)r)FNr)__name__ __module__ __qualname____doc__rrr(Tensorrrnrr7 staticmethodrArFtupler-r&intrr0listrkr,rt __classcell__)rs@rr r s&&P++l++++++%*6: >> >,>" > !-t 3 >  >>>>@ 2  2l 2 2  2 2 2\ 2.el.u|...\.7; 8+8+ 8+,8+!-t 3 8+ u|U\D00 1 8+8+8+8+t??%,???\?)?,)?)?<)?  )?)?)?\)?V26   tCy/  T#Y$.   d3i    \  " ","" "  """"H   + tCy/      \     rr )r~r(torch.nnnnvllm.config.modelrvllm.utils.platform_utilsrvllm.v1.outputsrrvllm.v1.sample.metadatarvllm.v1.sample.ops.bad_wordsrvllm.v1.sample.ops.logprobsr vllm.v1.sample.ops.penaltiesr $vllm.v1.sample.ops.topk_topp_samplerr r?Moduler rfrrrsED ******======::::::::444444888888BBBBBB<<<<<<@@@@@@ k k k k k bik k k k k r