.`i: UddlZddlmZddlmZmZmZmZddlm Z m Z m Z m Z m Z ddlZddlZddlmZddlmZmZddlmZddlmZmZe r dd lmZdd lmZddlm Z dd l!m"Z"dd l#m$Z$dd l%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+m,Z,e$e-Z.e dZ/da0e/dze1d<dZ2de3de4fdZ5ej6dZ7de/fdZ8eGddZ9dede:e3de;dde dRd!e?d"e+d#e?de@e+eejAgejAfffd$ZBd"e+de+fd%ZC dSd"e+d'e?de@e?e?e?e?e?e?ffd(ZD dTd"e+d'e?d*e4de@e?e?e?e?ffd+ZE dRd,ejAd-e?d.e?de:e@e?e?ffd/ZF dSd0d1d2d3d'e?de4fd4ZGd5ejAd6e?dejAfd7ZHd8ejAdejAfd9ZId:e3d;e dd?e ZKd@e3dAe;e(de;e(fdBZLdCejAdDejMfdEZN dUdFejAdGe?dHe?dzdIe?dejAf dJZOd"e+dKejAdLejAde+fdMZPdNejAdFejAdOedPe3dejAf dQZQdS)VN)Callable) dataclassfieldfieldsmake_dataclass) TYPE_CHECKINGAnyLiteralProtocolget_args)runtime_checkable) VllmConfigget_layers_from_vllm_config)cdiv) KVCacheSpec MambaSpec)SchedulerOutput) InputBatch)get_kv_connector_cache_layout) init_logger)AttentionLayerBase)AttentionBackend AttentionImplAttentionMetadataCommonAttentionMetadatasubclass_attention_backend)NHDHND_KV_CACHE_LAYOUT_OVERRIDEvaluereturnc.|ttvSN)r KVCacheLayoutType)r!s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/v1/attention/backends/utils.pyis_valid_kv_cache_layoutr'/s H.// //cd}t$t}td||Stj}|t }n,t |sJtd||S)NzM`_KV_CACHE_LAYOUT_OVERRIDE` variable detected. Setting KV cache layout to %s.zT`VLLM_KV_CACHE_LAYOUT` environment variable detected. Setting KV cache layout to %s.)rlogger info_onceenvsVLLM_KV_CACHE_LAYOUTrr' cache_layouts r&get_kv_cache_layoutr03s 26L ,0  -    ,L466 ' 55555 7    r(r/c |adSr$)rr.s r&set_kv_cache_layoutr2Qs ,r(ceZdZUdZeed<edzed<eed<dZeed<e ddZ edzed <e ddZ edzed <dS) PerLayerParametersa Currently, FlashInfer backend only support models in which all layers share the same values for the following hyperparameters. Should not be used for trtllm-gen backend since it supports different values for the following hyperparameters. window_leftNlogits_soft_capsm_scaleF has_sinks)defaultcomparehas_same_window_leftshas_same_all_params) __name__ __module__ __qualname____doc__int__annotations__floatr8boolrr;r<r(r&r4r4VsT\!!!OOOIt).tU)K)K)K4$;KKK',uT5'I'I'IIIIIIr(r4 vllm_config layer_namescls_rcVt|t|}i}|D]{\}}|j}t ||sJt |dd}||dnd} t |dd} |j} t |dddu} t| | | | ||<||S)zc Scan layers in `layer_names` and determine some hyperparameters to use during `plan`. sliding_windowNrr r6sinks)rritemsimpl isinstancegetattrscaler4) rFrGrHlayersper_layer_paramskeylayerrM window_sizer5r6r7r8s r&get_per_layer_parametersrVhs)F 79llnn   Uz$%%%%%d$4d;; (3(?k!nnR !$(94@@:D'400< 2 (I! !  r(rRct|dks Jdt|}|dtfd|D_tfd|D_S)ad Currently, FlashInfer backend other than trtllm-gen only support models in which all layers share the same values for the following hyperparameters: - `window_left` - `logits_soft_cap` - `sm_scale` So this function asserts that all layers share the same values for these hyperparameters and returns the global values. rz'No attention layers found in the model.c38K|]}|jjkVdSr$)r5.0params global_paramss r& z/infer_global_hyperparameters..s?...s9,,$*-,,,,,,r()lenlistvaluesallr;r<)rR param_setsr\s @r&infer_global_hyperparametersrds  1 $ $ $&O $ $ $&--//00JqMM*-....FP...++M'),,,,,.8,,,))M% r(attn_chunk_sizecommon_attn_metadata block_sizec |j}|j}|j}|jj}|dd|ddz }|jd}tj|||z |zz | tj } ||| zz} dt|| z |z} tj | } | d tj | | z | } tj tj | z }tj | | |z dz }tj || z | }| ||dk<tj|||dz zz ||dk||dk<tj dztj }tj ||ddd|d<tj| d|tj }| || dz <||z }tj || ||ztj | | zz }||z}||zdksJd|d|||z}|dddftj|tj z}|d|jddz }tj tj|tj | |z}t'j|t'j| fd }||}t'j|}t'j|}t+|}t/|||d ||d t3||j||||jd |t'j| |fS) Nr rdtype)outzattn_chunk_size z is not divisible by block_size )maxc@|fdS)Nr )view) block_tablebatch_indices_torchblock_indices_torchvirtual_batchess r&z6make_local_attention_virtual_batches..Qs';00, d?Br(T)device non_blocking) query_start_loc_cpuquery_start_locseq_lensnum_reqsnum_actual_tokens max_query_len max_seq_lenblock_table_tensor slot_mappingcausal _seq_lens_cpu_num_computed_tokens_cpu)rwnumpy seq_lens_cpur~rxrushapenpminimumastypeint32rcumsumrepeatarangeemptyfullreshapecliptorch from_numpyrArmrtor_r{r)!rerfrgquery_start_loc_np seq_lens_nprpru q_seqlensactual_batch_sizeq_tokens_in_first_blocktokens_in_last_block local_blocks cu_num_blocks block_offsetsrrarangeseqlens_q_localcu_seqlens_q_localseqlens_k_localnum_computed_tokens_localk_seqstarts_absolute block_startspages_per_local_batch block_indices batch_indicesmake_block_tableblock_table_localrwrr}rqrrrss! @@@r&$make_local_attention_virtual_batchesrsO .AGGII&399;;K&9K ! 1 8F"122&);CRC)@@I#)!,!jK)3FG fRX+k_>#O44Ll&&(())K "/+..f4.PPTBB\"".@%))++,)6"!&!12K!L!L     r(c|jdkr|S|jJ|jJ|j}|j}|d|}|j}|j}t j||ddd}t j||}t j|dz|j |j }d|d<t j |d|dd<t| } t|} t!||dd |j|| | |j|j|jd|j|j }|S) NriT)right) minlength)rurkrdimcpu)rv) rxrwryrzr{r|r}r~rrrr)r|logits_indices_paddednum_logits_indicesrzrxr bucketizebincountrrurkrrArmitemsumrrryr}r~rrr) rfrrlogits_indicesrzrx request_idsnum_decode_tokensdecode_query_start_locdecode_max_query_lentotal_num_decode_tokenss r&1make_kv_sharing_fast_prefill_common_attn_metadatarjs)Q..$#  5 A A A  2 > > >0F-@*+>,>+>?N#,H*:O/./!""2ETRRRK{hGGG#[1 _3?;P!"1!&.?Q!G!G!G12204466;;==>>!"3"7"7"9"9">">"@"@AA2.255e$5OO%.1*(4/B)6*8!5!N    r(ridecode_thresholdc|j}|j}|j}|j}|j}||kr|dd|ddfS|dd|ddz }||k}||k|z} |d} | d} | } || } tj |s| dd| ddfS|| z }|| z }tj | s| |d| |dfS| | z }|| z }||| z }||z }| ||| ||fS)a Assuming a reordered batch, finds the boundary between prefill and decode requests. Args: common_attn_metadata: CommonAttentionMetadata object containing the batch metadata. decode_threshold: The maximum query length to be considered a decode. Returns: num_decodes: The number of decode requests. num_extends: The number of extend requests. num_prefills: The number of prefill requests. num_decode_tokens: The number of tokens in the decode requests. num_extend_tokens: The number of tokens in the extend requests. num_prefill_tokens: The number of tokens in the prefill requests. rriNr r) r|rzr{rwrrAargmaxrrany)rfrr|rz num_tokensrxry query_lensis_prefill_or_extend is_prefill first_extend first_prefill num_decodesrnum_prefills_or_extendsnum_prefill_or_extend_tokens num_extends num_prefillsnum_prefill_tokensnum_extend_tokenss r&"split_decodes_prefills_and_extendsrs*)6M#,H%7J*>O#0H(((Az1a// $ss';;J%(88j(,@@J'++--444<<AACCLNN$$+++3388::MK' 5::<< 9) * *<Q#4a;;&4#-0A#A 9Z   #  (    +-Km+L#om&DD47II  r(Frequire_uniformc|j}|j}|j}|j}||kr|r|dkr|d|dfS|dd|ddz }|d|krd|d|fS|rQt j||dk|dkzr||dz|ks Jd|d|dfS||dk}n||k}t j|s|d|dfS| d} t j|d| |ksJ| } || z } || } || z } | | | | fS)a Assuming a reordered batch, finds the boundary between prefill and decode requests. Args: common_attn_metadata: CommonAttentionMetadata object containing the batch metadata. decode_threshold: The maximum query length to be considered a decode. require_uniform: If True, requires that all decode requests have the same query length. When set, some queries may be considered prefills even if they are <= decode_threshold, in order to ensure uniformity. Returns: num_decodes: The number of decode requests. num_prefills: The number of prefill requests. num_decode_tokens: The number of tokens in the decode requests. num_prefill_tokens: The number of tokens in the prefill requests. rirNr ztokens not padded correctlyr) r|rzr{rwrrrbrrAr)rfrrr|rzrrxrrrrrrrs r&split_decodes_and_prefillsrs.)6M#,H%7J*>O((( )/144J)) $ss';;J!}...(Az)) 3 9jJqM1jAoF G G .jm+z999;X999Q A- -:a=0 "22 9Z *J))NN$$+++3388::M 9Z/3CC D DDD DKk)L' 6;;==#&77 '8:L MMr(rworkspace_sizerequest_offsetcg}dt|}}tj||ksJ||kr|d}}||krV|||x}z|kr3||z }|dz }||kr#|||x}z|k3|||z||zf||k|S)a Split the prefill requests into chunks such that the total sequence length of each chunk is less than or equal to the workspace size. Args: seq_lens_cpu: The sequence lengths of the prefill requests on CPU. workspace_size: The maximum workspace size (in tokens) per chunk. request_offset: The offset to add to the request indices. Returns: A list of tuples of (reqs_start, reqs_end) representing chunk boundaries. rri)r_rrbrappend) rrr chunk_boundsinstart chunk_totalss r&split_prefill_chunksr%sL c,qA 9\^3 4 4 9 9 ; ;;; ; a%%{!ee\!_-A-A-C-C(CDWW 1 K FA!ee\!_-A-A-C-C(CDWW U^3Q5GHIII a%% r( input_batchrscheduler_outputrct|j}fd|jD}tj|}|jd|}|dk}||k|z}||k|z} tj|jtj} d| | <d| |<t| } t| } tj|tj} d| | | | z<d| | | zd<| | k}| sdStj |d}tj | |d }||}d t||D}|D]I}||}||kr9||||||}|||<|}||k9Jd S) z Reorders the batch to split into prefill and decode requests; places all requests with <= decode_threshold tokens at the front of the batch. Returns: True if the batch was modified, False otherwise. c*g|]}j|SrE)num_scheduled_tokens)rZidrs r& z?reorder_batch_to_split_decodes_and_prefills..Us/68-b1r(NrrjriFstable)kindcNi|]"\}}t|t|#SrE)rA)rZsrcdsts r& z?reorder_batch_to_split_decodes_and_prefills..us*VVV83CHHc#hhVVVr(T)r_req_idsrarraynum_computed_tokens_cpuzerosrrrArrwhereargsortzip swap_statesget)rrrrzrnum_scheduled_tokens_npnum_computed_tokens_npr is_decode is_extend req_regionsrrtarget_regions needs_swap orig_indices sorted_order src_indices src_dest_maprrnext_dsts ` r&+reorder_batch_to_split_decodes_and_prefillsr@s(;&''H?N;{!::;23N;,../.J >>  u8J''*L:k*5HEEEL|,KVVs; 7U7UVVVL3Sjj  # #C - - -#''S11H #L C Sjj 4r(query batch_sizec,|dks Jd|d|jd}|jd}|jd}||zdksJd|d|||z}|||||S) z Reshapes the query tensor for the specified batch size, so that it has shape (batch_size, seq_len, num_heads, head_dim). zquery must be 3D, got Drrirz total_tokens=z is not divisible by batch_size=rrro)rr total_tokens num_headshead_dimseq_lens r&reshape_query_for_spec_decoder s 99;;!   DeiikkDDD   ;q>L AI{1~H * $ ) ) )<<<>L   L+*;A*> @QRS@T U UUr( name_prefix metadata_clsrc@||jz}t|||f}|S)zH Return a new subclass of `metadata_cls` with additional fields )bases)r=r)rrrnameWrappeds r&subclass_attention_metadatars,l33DT6,AAAG Nr(cBeZdZUdZejdzed<dZedzed<dS)KVSharingFastPrefillMetadataNrr) r=r>r?rrTensorrBrrArEr(r&rrs?155<$.555%)d )))))r(rprefixunderlying_attn_backendcn|}Gdd|}t|||}|S)Nc 4eZdZ ddedededeffd ZxZS)Gcreate_fast_prefill_custom_backend..FastPrefillAttentionBuilderFcommon_prefix_lenrf fast_buildr"ct|}t|||}Gdd|jt}|||S)NceZdZdZdS)|create_fast_prefill_custom_backend..FastPrefillAttentionBuilder.build..KVSharingFastPrefillAttentionMetadatac t|jD]+}t||jt ||j,|j|_|j|_dSr$)r __class__setattrrrOrr)selfmetadatarf_fields r&__init__zcreate_fast_prefill_custom_backend..FastPrefillAttentionBuilder.build..KVSharingFastPrefillAttentionMetadata.__init__se"();"<"<SSfk78V[3Q3QRRRR-B./C.UD+++r(N)r=r>r?r*rEr(r&%KVSharingFastPrefillAttentionMetadatar#s(VVVVVr(r+)rsuperbuildr%r)r'rrfr new_common_attn_metadatar(r+r%s r&r-zMcreate_fast_prefill_custom_backend..FastPrefillAttentionBuilder.buildsBBVWW %ww}}!#;ZH V V V V V", V V V98CWXX Xr()F) r=r>r?rArrDrr- __classcell__)r%s@r&FastPrefillAttentionBuilderrs| %  Y Y" Y#: Y  Y   Y Y Y Y Y Y Y Y Y Yr(r0)rattention_backend_cls builder_cls)get_builder_clsr)rrunderlying_builderr0 attn_backends r&"create_fast_prefill_custom_backendr6sp1@@BBYYYYY&8YYY<.5/L r(query_start_loc_p_cpuruc |jjdksJ|}i}d}d}dD]L}| |z }i||<|||d<|||d<t jtjtj t||}|||d<t||d} | ||d<td| dz} g} t|D]'\} } | t| (t j| t j } | ||d <|Qt j| ft$t j| }t j| ft$t j| }nr|| krZ|| t$|| t$|d | ||d | | |||d <|||d<N|||fS)Nr)numstotmlist mlist_lenirrj offsetlistrkrur batch_ptrtoken_chunk_offset_ptr)rutypediffrrrrrrrr_rm enumerateextendrangetensorrr PAD_SLOT_IDnelementresize_fill_copy_)r7ruseqlens nums_dictr@rABLOCK_Mr:r<r=MAX_NUM_PROGRAMSr>idxnums r&compute_causal_conv1d_metadatarSs ! ' , 5 5 5 5#((**GII!$N$NW$% '%) '6"$(HHJJOO$5$5 '5! 29SYY+?+?!F!FGG&+ '7# '*7344 *3 ';'tY//!3 !$ * *HC   eCjj ) ) ) )\*EK@@@ +5 '<(   !#[ FI&+Z!#[ F&&& " "!!##&666!!"23399+FFF&..$% $$$!I+$$U+++ iK %   *3 ';'7M '344 i!7 77r(rydcp_sizedcp_rankcp_kv_cache_interleave_sizecP|d}|Ntj|tj|jd|d}n(tj|ggtj|j}|tjdd|j d}||z|z|z}|||zz }tj |||zz d|}||z} | dS)zWhile using dcp, kv_cache size stored on each rank may be different, use this function to calculate split decode seq_lens of each dcp rank. Only consider dcp now, we can extend the case of cp based on this. rNr?rir ) sizerrrru unsqueezerrGrrrsqueeze) ryrTrUrV num_requests rank_offsetsseq_lens_tiledbase remainderdcp_local_seq_lenss r&get_dcp_local_seq_lensras3==##L LX_ M M M Yq\\ VL! $ $  |ZL HO     EK  **2..55a9KA9NOO  & '   & & 0I L#>>> #I  )  % %a ( ((r(rnew_slot_mappingc b|}|j|dt|jz}|jtjt|jtjz}||||jdz|j| z|j dz|j dz|}|S)a? Creates a new CommonAttentionMetadata with all query lengths increased by 1. Also all seq lens are increased by 1. This is useful e.g. in speculative decoding with draft models, where we extend each sequence by 1 token. The slot mapping is computed externally, as it requires more information. Nrjri)rxrwryr{r|r}r) rxr_rwrrrreplaceryr{rr|r})rfrrbcadnew_query_start_locnew_query_start_loc_cpunew_cads r&extend_all_queries_by_1ri<s C-7QS=P9Q9Q7Q0RR!5  C #$$EK999kk+3!/#..2B2BB'!+Oa'%  G Nr(rp kv_cache_specmamba_cache_modec |dvr|St|tsJtj|dz |jzd}tjd|jz|j}|d|z}tj |d|S)an Get the block table tensor for mamba kernels from the input common_attn_metadata.block_table_tensor given different mamba cache modes. - "all": input (#requests, cdiv(max_model_len, block_size)); output (#requests, cdiv(max_model_len, block_size)). - "none": input (#requests, 1 + num_speculative_blocks); output (#requests, 1 + num_speculative_blocks). - "align": input (#requests, cdiv(max_model_len, block_size)); output (#requests, 1 + num_speculative_blocks), which are the last 1 + num_speculative_blocks of each request. )rbnonerir)min)ru) rNrrclamprgrnum_speculative_blocksrurYgather)rpryrjrk start_indicesoffsetsindices_to_gathers r&mamba_get_block_table_tensorru\s(?**-33333 \m6 6   , 4 4[=O   *33A66@|K,=>>>r()r)ri)riF)riNri)R functoolscollections.abcr dataclassesrrrrtypingrr r r r rrrtyping_extensionsr vllm.configrrvllm.utils.math_utilsrvllm.v1.kv_cache_interfacerrvllm.v1.core.sched.outputrvllm.v1.worker.gpu_input_batchr vllm.envsr,/vllm.distributed.kv_transfer.kv_connector.utilsr vllm.loggerr/vllm.model_executor.layers.attention_layer_baservllm.v1.attention.backendrrrrrr=r*r%rrBrHstrrDr' lru_cacher0r2r4r`rBdictrVrdrAtuplerrrrrrrr rrrr6rurSrarirurEr(r&rs$$$$$$@@@@@@@@@@@@ //////????????&&&&&&========:999999999999$#####NNNNNN X  L)6:,t3::: 0C0D0000 :-&7----  JJJJJJJ J"*.s);?;P #! !"B3 223jLLL1LL "Hel^U\-I$J JK LLLL^8 18 8 8 8 8 z@@1@@ 3S#sC '(@@@@J!:N:N1:N:N:N 3S#  :N:N:N:N|LM,03EH %S/<@@@'@@ @@@@F @ @3 @5< @ @ @ @ VU\ Vel V V V V    sC}% &     *****8*** * *!"23* ****Z18 <18 L18181818l'( %)%)l%)%)Dj%)"% %)  \ %)%)%)%)P1 Ll @"?"?l"?"? "?  \ "?"?"?"?"?"?r(