.`i9 UdZddlZddlZddlZddlmZddlmZddlm Z m Z ddl Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZGd d eZejd efdZejd efdZde de d e fdZdaede fdzed<da ede fdzed<da!ede fdzed<da"ede fdzed<da#ede fdzed<da$ede fdzed<da%ede fdzed<da&ede fdzed<da'ede fdzed<dGdZ(d e)fdZ*ejd e+e)fdZ,d e j-d e j-fd!Z.d"Z/d#Z0d$Z1d%Z2d&e j-d'e3e j-e j-fd(e j-d)e j-d*e j-d e j-f d+Z4d,e j-d-e)d.e)d e j-fd/Z5d0e j-d1e j-d(e j-d,e j-d2e j-d3e j-d4e)d e j-fd5Z6d e j-fd6Z7d e)d7e)d e)fd8Z8d e)d9e)fd:Z9d;d;gZ:e j;dfd e j-d-e+e)d?ed e3e j-e j-ffd@Z=d e j-d7e j-fdAZ> dHdBe j?dCe j-dDedzfdEZ@gdFZAdS)IzmCompatibility wrapper for DeepGEMM API changes. Users of vLLM should always import **only** these wrappers. N)Callable)Enum)AnyNoReturn)logger)get_fp8_min_max)current_platform) has_deep_gemmcdivcJeZdZdZdZdZeddZed dZdS) DeepGemmQuantScaleFMTrreturnNct|dd}|dStjototdu}|s|j|_dStjdr|j n|j |_dS)z>Initialize the oracle decision and store it in the class cache _oracle_cacheNd) getattrenvsVLLM_USE_DEEP_GEMM_E8M0is_deep_gemm_supported_fp8_gemm_nt_implFLOAT32rr is_device_capability_familyUE8M0FLOAT32_CEIL_UE8M0)clscacheduse_e8m0s h/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/utils/deep_gemm.pyinit_oracle_cachez'DeepGemmQuantScaleFMT.init_oracle_cache&sot44   F  ( 0&(( 0"$.    # C  F ;C@@ (CII' c@t|dd}| Jd|S)z*Return the pre-initialized oracle decisionrNz2DeepGemmQuantScaleFMT oracle cache not initialized)r)rrs r! from_oraclez!DeepGemmQuantScaleFMT.from_oracle<s/ot44!!#W!!! r#rN)rr) __name__ __module__ __qualname__rrr classmethodr"r%r#r!rrsfG E   [ *[r#rrctjo'tjdptjd}tjot o|S)zReturn `True` if DeepGEMM is supported on the current platform. Currently, only Hopper and Blackwell GPUs are supported. Zr)r is_cudais_device_capabilityrrVLLM_USE_DEEP_GEMMr )is_supported_archs r!rrDsU )022-b11 =  7 < <  " L} L;LLr#ctstjddStttjddSt jrtjddStjddS)znReturn `True` if vLLM is configured to use DeepGEMM " "E8M0 scale on a Hopper or Blackwell-class GPU. z>DeepGEMM E8M0 disabled: DeepGEMM not supported on this system.FNz3DeepGEMM E8M0 disabled: _fp8_gemm_nt_impl not foundz*DeepGEMM E8M0 enabled on current platform.Tz0DeepGEMM E8M0 disabled on current configuration.)rr debug_once _lazy_initr info_oncerrr+r#r!is_deep_gemm_e8m0_usedr6Ps " # # L   uLLL NOOOu #EFFFt GHHH 5r#___c td)z-Placeholder for unavailable DeepGEMM backend.zDeepGEMM backend is not available or outdated. Please install or update the `deep_gemm` to a newer version to enable FP8 kernels.) RuntimeError)r7r8s r!_missingr;is  K  r#.r _grouped_impl_grouped_masked_impl_fp8_mqa_logits_impl_fp8_paged_mqa_logits_impl#_get_paged_mqa_logits_metadata_impl%_get_mn_major_tma_aligned_tensor_impl,_get_mk_alignment_for_contiguous_layout_impl'_transform_sf_into_required_layout_implct1t*t#ttt t tdStsdSd}tj |ds7tj tjdtj |<t!jd}t%|ddat%|ddat%|ddat%|ddat%|ddat%|d dat%|d dat%|d dat%|d dat(dS) z2Import deep_gemm and resolve symbols on first use.NDG_JIT_CACHE_DIR deep_gemm fp8_gemm_nt m_grouped_fp8_gemm_nt_contiguousfp8_m_grouped_gemm_nt_maskedfp8_mqa_logitsfp8_paged_mqa_logitsget_paged_mqa_logits_metadataget_mn_major_tma_aligned_tensor&get_mk_alignment_for_contiguous_layout!transform_sf_into_required_layout)rr<r=r>r?r@rBrCr osenvirongetpathjoinrVLLM_CACHE_ROOT importlib import_modulerrArr")DEEP_GEMM_JIT_CACHE_ENV_NAME_dgs r!r4r4|s %  $  +  + % 1 . : 7 C 2 > ??$6 :>>6 = = 357<<  +4 4  /0  !+ . .C]D99C!CTJJM"3(FMM"3(8$??!(.Dd!K!K*1 ,d++'-4 .--)4; 5t440/6 0$//+++-----r#cttjd}t|S)NrF)r4rVrWint get_num_sms)rYs r!r\r\s3LLL  !+ . .C s  ! !!r#cltttSt}||gSN)r4rBr;) mk_align_sizes r!rNrNs1LLL3;zz@BBM = ))r#xcftttSt|S)z6Wrapper for DeepGEMM's get_mn_major_tma_aligned_tensor)r4rAr;r`s r! get_col_major_tma_aligned_tensorrcs)LLL,4zz 0 3 33r#ctt t|i|Sd|vr |d}|d=nt}t|d| i|S)Nr6disable_ue8m0_cast)r4rr;r6)argskwargs use_ue8m0s r!rGrGskLLL ((((6))34 + , ,*,, d O9} O O OOr#cztt t|i|St|dt i|SNre)r4r<r;r6rfrgs r!rHrHsRLLL((((   &<&>&>"> BH  r#cztt t|i|St|dt i|Srj)r4r=r;r6rks r!rIrIsRLLL#((((   &<&>&>"> BH  r#cztt t|i|St|dt i|Srj)r4rCr;r6rks r!rOrOsRLLL.6(((( 2  &<&>&>"> BH  r#qkvweights cu_seqlen_ks cu_seqlen_kecntttSt|||||S)aCompute FP8 MQA logits for a single sequence without KV paging. Args: q: Query tensor of shape [M, H, D]. Casted to `torch.float8_e4m3fn` by caller. kv: Tuple `(k_fp8, k_scales)` where `k_fp8` has shape [N, D] with dtype `torch.float8_e4m3fn` and `k_scales` has shape [N]) with dtype `torch.float32`. weights: weights of shape [M, H], dtype `torch.float32`. cu_seqlen_ks: Start indices (inclusive) for valid K per query position, shape [M], dtype int32. cu_seqlen_ke: End indices (exclusive) for valid K per query position, shape [M], dtype int32. Returns: Logits tensor of shape [M, N], dtype `torch.float32`. )r4r>r;)rnrorprqrrs r!rJrJs30LLL#zz 2w l K KKr# context_lens block_sizenum_smscjtttSt|||S)aBuild scheduling metadata for paged MQA logits. Args: context_lens: Tensor of shape [B], dtype int32; effective context length per batch element. block_size: KV-cache block size in tokens (e.g., 64). num_sms: Number of SMs available. 132 for Hopper Returns: Backend-specific tensor consumed by `fp8_paged_mqa_logits` to schedule work across SMs. )r4r@r;)rtrurvs r!rLrL s/LLL*2zz .|Z Q QQr#q_fp8 kv_cache_fp8 block_tablesschedule_metadata max_model_lenc vtttSt|||||||dS)aCompute FP8 MQA logits using paged KV-cache. Args: q_fp8: Query tensor of shape [B, next_n, H, D]. Casted to `torch.float8_e4m3fn` by caller. kv_cache_fp8: Paged KV-cache in packed FP8+scale layout with shape [num_blocks, block_size, 1, D+4], dtype `torch.uint8`. The last 4 bytes per (block,pos) store the `float` dequant scale. weights: Tensor of shape [B * next_n, H], dtype `torch.float32`. context_lens: Tensor of shape [B], dtype int32; effective context length for each batch element. block_tables: Tensor of shape [B, max_blocks], dtype int32; maps logical block indices to physical blocks in the paged cache. schedule_metadata: Returned by `get_paged_mqa_logits_metadata`; used to distribute work across SMs. max_model_len: Maximum sequence length used to size the logits output. Returns: Logits tensor of shape [B * next_n, max_model_len], dtype `torch.float32`. NT) clean_logits)r4r?r;)rxryrprtrzr{r|s r!rKrK"sL<LLL!)zz %     r#c tjdtjtj|S)Ng@)torchpowceillog2absrbs r!_ceil_to_ue8m0rOs0 9S%*UZ%8%899 : ::r#yc(t|||zSr^r )r`rs r!_alignrSs 1::>r# element_sizec(t|d|zS)N)r)r`rs r!get_tma_aligned_sizerXs !R<' ( ((r#T)dynamicbackendFrhcltj}|dksJ|j\}}|\}}t jt ||t ||f|j|j}||d|d|f<| d|| d|z|} | ddd} t\} } | | z } |rt!| n| } | d | z z|}||d|d|f| | d | dfS) Nr)dtypedevicer)rT)dimkeepdimg-C6?g?r)r fp8_dtypershaperzerosrrrviewsizerfloatamaxclamprrtoview_as contiguous)r`rurhrmnblock_mblock_nx_paddedx_viewx_amaxr7fp8_maxsfx_scaleds r!per_block_cast_to_fp8r`s!*,,I 5577a<<<< 7DAq!GW{ 7  VAw//0HHRaR!V ]]2w a(8(8G(CW M MF ZZ\\   ! ! & &64 & @ @ F Ft L LF ""JAw ' B( 0   bB#(#'' 22H   H % %bqb"1"f - 8 8 : :BGG A A== r#c||}}||z||zz}d||zz|z }d|z S)a|Return a global difference metric for unit tests. DeepGEMM kernels on Blackwell/B200 currently exhibit noticeable per-element error, causing `torch.testing.assert_close` to fail. Instead of checking every element, we compute a cosine-style similarity over the whole tensor and report `1 - sim`. Once kernel accuracy improves this helper can be removed. rr)doublesum)r`r denominatorsims r! calc_diffrws^ 88::qxxzzqAq51q5=%%''K q1ukkmm k )C s7Nr# output_dtypeweightsupports_deep_gemmc|t}d}d}|o7|tjko'|jd|zdko|jd|zdkS)N@rrr)rrbfloat16r)rrr N_MULTIPLE K_MULTIPLEs r!"should_use_deepgemm_for_fp8_linearrsm !355 JJ  . EN * . LOj (A - . LOj (A - r#)rrrGrHrIrJrKrLrr6rr\rrcrNr&r^)B__doc__ functoolsrVrPcollections.abcrenumrtypingrrr vllm.envsr vllm.loggerr9vllm.model_executor.layers.quantization.utils.quant_utilsrvllm.platformsr vllm.utils.import_utilsr vllm.utils.math_utilsr rcacheboolrr6r;r__annotations__r<r=r>r?r@rArBrCr4r[r\listrNTensorrcrGrHrIrOtuplerJrLrKrrrDEFAULT_BLOCK_SIZEcompilesimple_compile_backendrrrr__all__r+r#r!rs  $$$$$$ ,+++++111111&&&&&&&&&&&D&&&R MMMMM 0CH048CH%,333+/ xS!D(///26hsCx(4/66626hsCx(4/6668<HS#X.5<<EEECG%xS'9D'@GGGJN,hsCx.@4.GNNNEI'#s();d)BIII2.2.2.2.j"S""""  *S ****4 44444 P P PL |L elEL()L\L, L , L  \ LLLL<R,R,/R:=R \RRRR** <*,*\*, * , * | ** \****Z;el;;;;ccc )C)s))))3Zt%5%LMMM-?SX |!%cLP 5< %&NM,  %,    &'++ Lt ,   r#