)`idZddlZddlZddlZddlmZmZmZmZddl Z ddl Z ddl Z ddl Z ddlZddlmZmZmZddlmZdEdefdZdeejdefd Zdeejfd Z dFdeejd edefd ZdZdedededeeeeffdZdGdZdejfdZdejdeejejffdZ dejdeejejffdZ!dZ"dZ#dZ$dZ%dZ&dZ'dZ(d Z)ej*ej*ej*fd!Z+ej*ej*ej*fd"Z, dHd(ed)ed*ed+ed,ee-d-eed.ee.d/e-dedeed0e-fd1Z/ dId(ed)ed*ed+ed,ee-d-eed.ee.d/e-d2e-dedeed0e-fd3Z0 dJd(ed)ed*ed+ed5ed,ee-d-eed.ee.d/e-dedeed0e-fd6Z1 dKd(ed)ed*ed+ed,ee-d-eed.ee.d/e-d7e-d2e-d5ededeed0e-fd8Z2Gd9d:Z3Gd;d<Z4 dLd>ed?e-d@e.dAe-dBe-f dCZ5dDZ6dS)Ma3 Copyright (c) 2023 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)TupleAnyListOptional) rearrangereducerepeat)round_upreturnc|tj}tj|}|jS)z Get L2 cache size in bytes for the given CUDA device. Args: device: CUDA device (int, torch.device, or None for current device). Returns: L2 cache size in bytes. )torchcudacurrent_deviceget_device_properties L2_cache_size)devicepropss l/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/testing/utils.pyget_l2_cache_sizer&s:~**,, J , ,V 4 4E  tensorscd}|D]O}t|tjr3|jr,|||zz }P|S)z Calculate total bytes of tensors residing on GPU. Assumes all tensors are on the same device. Args: tensors: List of torch.Tensor objects. Returns: Total bytes occupied by GPU tensors (CPU tensors are ignored). r) isinstancer Tensoris_cudanumel element_sizertotalts r_calculate_tensor_bytesr!6s\ E 22 a & & 219 2 QWWYY!1!11 1E Lrcg}t|tjr|jr||nt|t t fr(|D]$}|t|%nNt|tr9| D]$}|t|%|S)a Recursively extract all GPU-resident tensors from a nested structure of lists, tuples, and dicts. Args: obj: Object to extract tensors from (can be tensor, list, tuple, dict, or other). Returns: Flat list of tensors on GPU found in the structure. ) rr rrappendlisttupleextend_extract_gpu_tensorsdictvalues)objritemvs rr'r'HsG#u|$$44s C$ ' '4 7 7D NN/55 6 6 6 6 7 C  4 4 4A NN/22 3 3 3 3 Nr min_rotationsct|}t|}|dkrdS|dz}||krdStj||z dz}t ||S)a! Calculate the number of buffer copies needed to ensure cold L2 cache. The function uses conservative thresholds to account for: - LRU eviction being gradual (not all data evicted when capacity exceeded) - Cache associativity effects (some data may persist in non-conflicting sets) - Hardware prefetching behavior Returns 1 (no rotation needed) only when tensor size substantially exceeds L2 cache (>= 5x), ensuring cache effects are truly negligible. Args: tensors: List of tensors to consider for rotation (must be on GPU). device: Device for L2 cache query (None for current device). min_rotations: Minimum number of rotations when rotation is needed. Returns: Number of buffer copies needed (1 means no rotation needed). r)rr!mathceilmax)rrr.l2_size total_bytessafe_cache_threshold num_rotationss rcalculate_rotation_countr9_sw, ''G)'22Kaq#Q;***q I2[@AAAEM }m , ,,rct|tjr|jr|r&|Stj|| |j |j }| ||S|St|tr d|DSt|trtd|DSt|trd|DS|S)a( Deep clone a nested structure, cloning GPU tensors with detach().clone() while preserving scalars, booleans, and other non-tensor values. For non-contiguous tensors (e.g., created with as_strided), this function preserves the stride pattern using torch.empty_strided() + copy_(). This is important for backends like cuDNN that expect specific memory layouts. Args: obj: Object to clone (tensor, list, tuple, dict, or other). Returns: Cloned structure with GPU tensors cloned, other values preserved. dtyperc,g|]}t|S_clone_structure.0r+s r z$_clone_structure..s!7774 &&777rc34K|]}t|VdSNr?rAs r z#_clone_structure..s+<<%d++<<<<<r?)rBkr,s r z$_clone_structure..s'???41a#A&&???r)rr rr is_contiguousdetachclone empty_stridedsizestrider<rcopy_r$r%r(items)r*results rr@r@s9#u|$$ ;   "" zz||))+++,HHJJJJLL):   SZZ\\*** J C  7737777 C  <<<<<<<< C  ??399;;???? r input_args input_kwargsr8c|dkr||fgSg}|||ft|dz D]7}t|}t|}|||f8|S)a Create multiple copies of input_args and input_kwargs for buffer rotation. The first copy (index 0) uses the original args/kwargs. Subsequent copies clone all GPU tensors while preserving other values. Args: input_args: Positional arguments tuple. input_kwargs: Keyword arguments dict. num_rotations: Number of buffer copies to create. Returns: List of (args, kwargs) tuples, one for each rotation index. r0)r#ranger@)rSrTr8copies_ cloned_args cloned_kwargss r_create_rotated_buffer_copiesr[s"\*++ F MM:|,---=1$ % %44&z22 (66  {M23333 Mrrcj|i}t|t|z}|r |djS|S)a- Infer CUDA device from GPU tensors in input_args/input_kwargs. Args: input_args: Positional arguments tuple. input_kwargs: Keyword arguments dict (can be None). default: Default device if no GPU tensors found. Returns: Device string or torch.device. Nr)r'r)rSrTdefault gpu_tensorss r_infer_device_from_tensorsr_sD &z225I,5W5WWK%1~$$ Nrxc |ddksJtjdtjtj|S)imported from DeepGEMMr@)viewamaxr+r powr3log2abs)r`s r_ceil_to_ue8m0rjsa 66"::??   ! ! # #a ' ' ' ' 9S%*UZ%8%899 : ::rc2|dkr|ddzdksJ|j\}}||dd}|d|dd}t|dz }|d | dz z tj |||fS) rbr-r0rrc)dim-C6?|@?) rmrNshapererifloatrfclamprj unsqueezetor float8_e4m3fn)r`mnx_viewx_amaxsfs rper_token_cast_to_fp8r|s 5577a< s0 s1 t0 t1s0s1zs0 s1 t0 t1 -> s0 s1r4rnrdzs0 s1 -> (s0 t0) (s1 t1))t0t1z(s1 t0) (s0 t1) -> s0 s1 t0 t1zs0 s1 -> s1 s0zs1 s0 -> (s1 t0) (s0 t1)r~z,(s0 t0) (s1 t1) (s2 t2) -> s0 s1 s2 t0 t1 t2rrs2zs0 s1 s2 t0 t1 t2 -> s0 s1 s2z#s0 s1 s2 -> (s0 t0) (s1 t1) (s2 t2))rrt2z,(s0 t0) (s2 t1) (s1 t2) -> s0 s1 s2 t0 t1 t2zs0 s1 s2 -> s0 s2 s1z#s0 s2 s1 -> (s0 t0) (s2 t1) (s1 t2)g:0yE>)ndimlenr finforvtensorr4rfloat32rrrirsrgr3rhr ru)r` scale_shape tile_shapescale_major_moderfp8_infofp8_amaxrrrrx_tiledabs_maxx_scalescales_repeatedscales_permutedrrx_fp32x_fp8s r quantize_fp8r s 6D 6>>>CTCC>>> 3{## 6 6 6 6s: 6 6 6 6 6 6{5.//H|HLOOOH qyyBB s " "#CrRRRGW[[]],BEJJPPQUVVG(GiUZ 7;;==0I0I%J%JKKG%W.HRTVWWWOO #CrRRRGW[[]],BEJJPPQUVVG(GiUZ 7;;==0I0I%J%JKKG(1ABBO$!;rOO  B B s " "AbRTVG >eDkk (GiUZ 7;;==0I0I%J%JKKG%>2"QSOO  AbRTVG >eDkk (GiUZ 7;;==0I0I%J%JKKG(1GHHO$5 O/D( )F IIe) * *E '>rc|j}|dvs Jd||t|jksJ|dkr|dkr |j\}}n |j\}}t|t jd||}|dkrt|d}nt|d}t||zd }n|d kr|dkr |j\}}}n |j\}}}t|t jd ||| }|dkrt|d }nt|d}t||zd}|S)rrrr-rrrzs0 s1 -> s0 s1 1 1zs0 s1 -> s1 s0 1 1zs0 s1 t0 t1 -> (s0 t0) (s1 t1)r~z+(s0 t0) (s1 t1) (s2 t2)-> s0 s1 s2 t0 t1 t2rzs0 s1 s2 -> s0 s1 s2 1 1 1zs0 s1 s2 -> s0 s2 s1 1 1 1z,s0 s1 s2 t0 t1 t2 -> (s0 t0) (s1 t1) (s2 t2))rrrqrrur r)r`rrrrroutrs rdequantize_fp8rls 6D 6>>>CTCC>>> 3w}%% % % % % qyy s " "]FB]FB  DD  !AbR    s " ")=>>GG)=>>GG %EFF  s " " JBBB JBB  DD   9     s " ")EFFGG)EFFGG %STT JrcPtj|tj|tj|tjr@tj|tj|dSdS)z Set random seed for reproducibility during testing. Args: random_seed (int): Random seed to set. Returns: None N)r manual_seedrandomseednpr is_availablemanual_seed_all) random_seeds rset_seedrs k""" K INN; z  0 {+++ "";/////00rctj|stj|dz dg}nd}t j|dS)z Sleep after kernel run. Dynamically adjust sleep time up to 1 sec based on execution time. Args: execution_time (float): Kernel execution time in milliseconds. Returns: None rpg{Gz?N)r2isinfrmintimesleep)execution_time sleep_times rsleep_after_kernel_runrsK :n % %V^c13788  Jz Frc||krtd|r)|d|z|z z|z|z|z}|d|z|z z|z|z|z}n"d|z|z|z|z|z}d|z|z|z|z|z}||z} | S)a Calculate FLOPs for a given attention layer. Assumes all sequence lengths are the same within the batch Args: batch_size (int): Batch size. qo_seqlen (int): Sequence length of the query. Assumed same within the batch. kv_seqlen (int): Sequence length of the key and value. Assumed same within the batch. head_dim_qk (int): Head dimension of the query and key. head_dim_vo (int): Head dimension of the value. num_qo_heads (int): Number of query heads. causal (bool): Whether to use causal masking. FLOPs is halved for causal masking. Returns: total_flops (int): Total FLOPs for the layer. zFqo_seqlen must be less than or equal to kv_seqlen for causal attentionr-) ValueError) batch_size qo_seqlen kv_seqlen head_dim_qk head_dim_vo num_qo_headscausal bmm1_flops bmm2_flops total_flopss rattention_flopsrs29 T   Y 9}y( *      9}y( *      ^i/);lJ[X ^i/);lJ[X z)K rc|r'||krtd|rtjd|tjz|tjz |tj|z|z}tjd|tjz|tjz |tj|z|z}ndtj|tj|tjz|z|z}dtj|tj|tjz|z|z}||z}|S)a Calculate FLOPs for a given attention layer with actual sequence lengths where actual sequence lengths are provided as 1D tensors. Args: actual_seq_lens_q (torch.Tensor): Array of actual sequence lengths of the query. actual_seq_lens_kv (torch.Tensor): Array of actual sequence lengths of the key and value. head_dim_qk (int): Head dimension of the query and key. head_dim_vo (int): Head dimension of the value. num_qo_heads (int): Number of query heads. causal (bool): Whether to use causal masking. Note: Causal must be false for decode as this function assumes qo_seqlen == kv_seqlen. Returns: total_flops (int): Total FLOPs for the layer. zWactual_seq_lens_q must be less than or equal to actual_seq_lens_kv for causal attentionr-)anyrr dotrur) actual_seq_lens_qactual_seq_lens_kvrrrrrrrs r$attention_flops_with_actual_seq_lensrs4 $'99>>@@  e   & I&))%-888#&&u}556!$$U]33          I&))%-888#&&u}556!$$U]33           i"%%em44!$$U]33         i"%%em44!$$U]33        z)K rc ht|||||||}tj|s||z dz ndS)a Calculate TFLOPS per second for a given attention layer. Assumes all sequence lengths are the same within the batch. Args: batch_size (int): Batch size. qo_seqlen (int): Sequence length of the query. kv_seqlen (int): Sequence length of the key and value. head_dim_qk (int): Head dimension of the query and key. head_dim_vo (int): Head dimension of the value. num_qo_heads (int): Number of query heads. causal (bool): Whether to use causal masking. time (float): Execution time in milliseconds. Returns: tflops_per_sec (float): TFLOPS per second for the layer. eA)rr2isnan) rrrrrrrrfs rattention_tflops_per_secrGsK4   A"&D!1!1 :1t8c>>s:rct||||||}tj|s||z dz ndS)a Calculate TFLOPS per second for a given attention layer with actual sequence lengths. Does not assume all sequence lengths are the same within the batch. Args: actual_seq_lens_q (torch.Tensor): Array of actual sequence lengths of the query. actual_seq_lens_kv (torch.Tensor): Array of actual sequence lengths of the key and value. head_dim_qk (int): Head dimension of the query and key. head_dim_vo (int): Head dimension of the value. num_qo_heads (int): Number of query heads. causal (bool): Whether to use causal masking. ms (float): Execution time in milliseconds. Returns: tflops_per_sec (float): TFLOPS per second for the layer. rr)rr2rr+)rrrrrrmsrs r-attention_tflops_per_sec_with_actual_seq_lensrmsP2 -   A'+jnn =16688b=3  #=rc ||z|z|z|jz} ||z|z|z| jz} ||z|z|z| jz} ||z|z|z| jz}| | z| z|z}|dz }|dz }tj|s||z ndS)aA Calculate TB per second perf achieved for a given attention layer. Assumes all sequence lengths are the same within the batch. Args: batch_size (int): Batch size. qo_seqlen (int): Sequence length of the query. kv_seqlen (int): Sequence length of the key and value. head_dim_qk (int): Head dimension of the query and key. head_dim_vo (int): Head dimension of the value. num_qo_heads (int): Number of query heads. num_kv_heads (int): Number of key and value heads. time (float): Execution time in milliseconds. q_dtype (torch.dtype): Data type of the query. kv_dtype (torch.dtype): Data type of the key and value. o_dtype (torch.dtype): Data type of the output. Returns: tb_per_sec (float): TB per second for the layer. @@mBr)itemsizer2r)rrrrrr num_kv_headsrq_dtypekv_dtypeo_dtypeq_bytesk_bytesv_byteso_bytesr6 time_in_sec bytes_in_tbs rattention_tb_per_secrs@9$|3kAGDTTG9$|3kAHDUUG9$|3kAHDUUG9$|3kAGDTTGG#g-7K*K$K,0Jt,<,< E; $ $#Erc tj||z|z|jz} tj||z|z|jz} tj||z|z|jz} tj||z|z| jz} | | z| z| z}|dz }|dz }t j|s||z ndS)a Calculate TB per second perf achieved for a given attention layer with actual sequence lengths. Does not assume all sequence lengths are the same within the batch. Args: actual_seq_lens_q (torch.Tensor): Array of actual sequence lengths of the query. actual_seq_lens_kv (torch.Tensor): Array of actual sequence lengths of the key and value. head_dim_qk (int): Head dimension of the query and key. head_dim_vo (int): Head dimension of the value. num_qo_heads (int): Number of query heads. num_kv_heads (int): Number of key and value heads. time (float): Execution time in milliseconds. q_dtype (torch.dtype): Data type of the query. kv_dtype (torch.dtype): Data type of the key and value. o_dtype (torch.dtype): Data type of the output. Returns: tb_per_sec (float): TB per second for the layer. rrr)r sumrr+r2r)rrrrrrrrrrrrrrr6rrs r)attention_tb_per_sec_with_actual_seq_lensrs@  #$$|3kAGDTT   $%% 4{BXEVV   $%% 4{BXEVV   #$$|3kAGDTT W$w.8>>@@K*K$K,0Jt,<,< E; $ $#ErdFr>T dry_run_iters repeat_itersdry_run_time_msrepeat_time_msl2_flushl2_flush_size_mbl2_flush_devicesleep_after_run cold_l2_cachec   i td|||fDr/tjdtd||nd} ||nd} ||nd}n*| } t d}t |}|dzd z} t pt  fd }tj d }tj d }d}| r6t| d zd z}tj ||tj }d}tj |tj |t|D]"}| r||#|tj |||z }| t%dt||z }| t%dt||z }tj t|D]"}| r||#tj dt|D}dt|D}tj t|D]g}| r|||||||rt'|htj g}t|D]6}|||||7|S)a Benchmark kernel execution time using CUDA events (no CUDA graphs). This is the simplest benchmarking method. Best suited for kernels where launch overhead is negligible compared to execution time. The function performs: 1. A quick estimation phase (5 iterations) to determine iteration counts 2. Dry-run warmup iterations (not measured) 3. Measured iterations with per-iteration timing via CUDA events Iteration counts can be specified directly or derived from target durations: - If dry_run_iters/repeat_iters are provided, those counts are used directly. - Otherwise, counts are computed from dry_run_time_ms/repeat_time_ms. Args: fn (Callable): The kernel function to benchmark. dry_run_iters (int, optional): Number of warmup iterations (not timed). If None, computed from dry_run_time_ms. repeat_iters (int, optional): Number of measured iterations. If None, computed from repeat_time_ms. dry_run_time_ms (int): Target warmup duration in ms (default: 25). repeat_time_ms (int): Target measurement duration in ms (default: 100). sleep_after_run (bool): If True, sleep briefly after each iteration to reduce thermal throttling (default: False). input_args (tuple): Positional arguments to pass to fn. input_kwargs (dict, optional): Keyword arguments to pass to fn. cold_l2_cache (bool): If True, flush L2 cache before each iteration to ensure cold-cache performance measurements (default: True). Returns: List[float]: Per-iteration execution times in milliseconds. Example: Basic usage: >>> def my_kernel(a, b): ... return torch.matmul(a, b.T) >>> q = torch.randn(1024, 128, device="cuda") >>> k = torch.randn(1024, 128, device="cuda") >>> times = bench_gpu_time_with_cuda_event( ... fn=my_kernel, ... input_args=(q, k), ... ) >>> print(f"Median time: {np.median(times):.3f} ms") Note: This method does NOT use CUDA graphs, so each iteration incurs kernel launch overhead. For microbenchmarking where launch latency matters, consider using ``bench_gpu_time_with_cudagraph`` instead. .. deprecated:: The ``l2_flush``, ``l2_flush_size_mb``, and ``l2_flush_device`` parameters are deprecated. Use ``cold_l2_cache`` instead. Nc3K|]}|duV dSrEr>rBps rrFz1bench_gpu_time_with_cuda_event..8& P PQ1D= P P P P P PrZl2_flush, l2_flush_size_mb, and l2_flush_device are deprecated. Use cold_l2_cache instead.r-category stacklevelTrc4r idSdSrEr>fnhas_argsrSrTsrcall_fnz/bench_gpu_time_with_cuda_event..call_fnM5   B +l + + + + + BDDDDDr enable_timingrr1r0cNg|]"}tjd#STrr rEventrBrXs rrCz2bench_gpu_time_with_cuda_event..y+VVVQEJ$$4$88VVVrcNg|]"}tjd#Sr r r s rrCz2bench_gpu_time_with_cuda_event..z+TTT1%*"""66TTTr)rwarningswarnDeprecationWarningr_rboolr rr intemptyint8 synchronizerecordrVzero_ elapsed_timer4rr#)rrrrrrrrrrSrTr _do_l2_flush_l2_flush_size_mb_l2_flush_devicer5r start_event end_eventbuffer l2_flush_sizemeasurement_itersrXestimated_kernel_execution_time start_events end_eventsiter_idxmeasured_timesrs` `` @rbench_gpu_time_with_cuda_eventr)s)J  P P8-="O P P PPP;  )'     $,#7xxT 0@0L,,RU.=.I??v$ 5j,PVWW#$455$q[k:J54 #5#5H *"""66K   t 44I FW-..5< ]3C5:VVV J GIII J $ % %   LLNNN   J  ++.??$ As?5T#TUUVV 1c.3R"RSSTT  J = ! !   LLNNN  JWV% BUBUVVVLTTl@S@STTTJ J,''DD   LLNNNX%%''' 8##%%%  D "#B C C C JN,''YYl84AA*XBVWWXXXX ruse_cuda_graphc  789:;<=>? i td|||fDr/tjdtd||nd} ||nd}||nd}n*| } t d}t |}|dzd z} d d lm8d d lm}|d }t| dd dkrtdd dl m }n#ttf$r}t|trtjdt dntj|dt d| rt#||||| |  cYd}~St%||||| |  cYd}~Sd}~wwxYwd}8fd>8fd:8fd;8fd<:;<>fd7dt&t(t*t*tttfdt&t(t,t*t*tttttfdt&f78fd }t/ pt/ == fd}d}| r6t|d zd z}t1j||t0j!}|}d}| rXt0jt0j}|t0jt0j |5tCd"D] }| dddn #1swxYwYt0j|t0j"}t0j#|5|dddn #1swxYwY|j$}d#}t0j|t0jt0j%d$}t0j%d$} |&tC|D]"}| r|'|#| &t0j|(| |z }!| tSd%t||!z }| tSd%t||!z }t0jtC|D]"}| r|'|#t0jg}"g}#g}$8*8j+j,8*8j+j-8*8j+j.8*8j+j/8*8j+j081||||"|#tC|D]}| r|'82}%|82}&t0j|$3|%|&f|rti|!85d 868j+j,868j+j-868j+j.868j+j/868j+j087d&9d dl8}'ts|"d'(?d)?D}(i})|#D].}*|*d"}+|+|)vrg|)|+<|)|+3|*/g},d}-tu|$D]"\}.\}%}&|';|(|%}/|'<|(|&}0t{?fd*tC|/|0D}1g}2|1D]!}+|+|)vr|2>|)|+"|2std+|.t{9fd,|2D}3|-|3}-n|-|3krtd-|-d.|3td/|2D}4tSd0|2D}5|5|4z d1z }6|,3|6$|,S)2am Benchmark GPU time using CUPTI activity tracing for precise kernel timing. CUPTI (CUDA Profiling Tools Interface) provides hardware-level profiling that measures actual GPU kernel execution time, excluding CPU-side launch overhead. This gives the most accurate kernel performance measurements. Cold L2 cache is achieved via L2 flush between iterations. CUPTI measures per-iteration, so L2 flush works correctly regardless of ``use_cuda_graph``. Behavior: - Uses CUPTI (requires version >= 13, i.e., CUDA 13+) to trace kernel activities and compute per-iteration GPU time from recorded start/end timestamps. - Optionally captures operations in a CUDA graph (use_cuda_graph=True) for reduced launch overhead during measurement. - If CUPTI is unavailable, falls back to: - ``bench_gpu_time_with_cudagraph`` if use_cuda_graph=True (uses rotating buffers for cold L2) - ``bench_gpu_time_with_cuda_event`` otherwise (uses L2 flush for cold L2) Args: fn (Callable): The kernel function to benchmark. dry_run_iters (int, optional): Number of warmup iterations (not timed). If None, computed from dry_run_time_ms. repeat_iters (int, optional): Number of measured iterations. If None, computed from repeat_time_ms. dry_run_time_ms (int): Target warmup duration in ms (default: 25). repeat_time_ms (int): Target measurement duration in ms (default: 100). sleep_after_run (bool): If True, sleep briefly after each iteration (default: False). use_cuda_graph (bool): If True, capture and replay a CUDA graph (default: False). input_args (tuple): Positional arguments to pass to fn. input_kwargs (dict, optional): Keyword arguments to pass to fn. cold_l2_cache (bool): If True, flush L2 cache before each iteration to ensure cold-cache performance measurements (default: True). Returns: List[float]: Per-iteration GPU kernel execution times in milliseconds. Example: Basic CUPTI benchmarking (requires cupti-python >= 13): >>> def my_kernel(a, b): ... return torch.matmul(a, b.T) >>> q = torch.randn(1024, 128, device="cuda") >>> k = torch.randn(1024, 128, device="cuda") >>> times = bench_gpu_time_with_cupti( ... fn=my_kernel, ... input_args=(q, k), ... ) >>> print(f"Median GPU time: {np.median(times):.3f} ms") Note: Requires ``cupti-python`` package version >= 13.0.0: ``pip install -U cupti-python`` If CUPTI is not available, a warning is issued and the function automatically falls back to CUDA event or CUDA graph timing. .. deprecated:: The ``l2_flush``, ``l2_flush_size_mb``, and ``l2_flush_device`` parameters are deprecated. Use ``cold_l2_cache`` instead. Nc3K|]}|duV dSrEr>rs rrFz,bench_gpu_time_with_cupti..rrrr-rTrrrr)cupti)versionz cupti-python. z?CUPTI needs to be >= 13.0.0. Try 'pip install -U cupti-python'.)partialzhCUPTI is not installed. Try 'pip install -U cupti-python'. Falling back to CUDA events for benchmarking.z. Falling back to CUDA events for benchmarking. rrrrrrrSrTrcd}d}||fS)Nirr>) buffer_sizemax_num_recordss rfunc_buffer_requestedz8bench_gpu_time_with_cupti..func_buffer_requested&s% O++rc|jjjkr|jS|jjjkrdS|jjjkrdSdS)NMEMCPYMEMSET)kind ActivityKindCONCURRENT_KERNELnamer8r9activityr-s rset_kernel_namez2bench_gpu_time_with_cupti..set_kernel_name+sV =E.@ @ @= ]e07 7 78 ]e07 7 788 7rcT|jjjjjfvr|jSdSNr)r:r;r8r9bytesr>s r get_bytesz,bench_gpu_time_with_cupti..get_bytes3s. =U/68J8QR R R> !1rc@|jjjkr|jSdSrB)r:r;r8 copy_kindr>s r get_copy_kindz0bench_gpu_time_with_cupti..get_copy_kind9s$ =E.5 5 5% %1rc@|jjjkr|jSdSrB)r:r;r9valuer>s r get_valuez,bench_gpu_time_with_cupti..get_value?s# =E.5 5 5> !1rc ||j|j|j||||jfSrE)startendcorrelation_idr:)r?rDrGrJr@s rcollect_kernel_infoz6bench_gpu_time_with_cupti..collect_kernel_infoEsV OH % % N L  # M( # # Ih   Ih   M  rlauncheskernels activitiescH|D]}|jjjjjjjfvr||K|jjjjjfvr3||j|j |j |j |jfdSrE) r:r;r<r8r9r#RUNTIMEDRIVERrLrMrNcbid)rPrQrRr?rOr-s rfunc_buffer_completedz8bench_gpu_time_with_cupti..func_buffer_completedQs #  H}"4")")! 228<<===="*")#    /    rc4r idSdSrEr>rsrrz*bench_gpu_time_with_cupti..call_fnprrrrr~r1rr0c ^|dd|dd|dd|dd|d S)NrrXr1r>)kernels rgenerate_kernel_stringz9bench_gpu_time_with_cupti..generate_kernel_stringsB)MMfQiMM&)MMfQiMM&)MMMrc|dSrBr>)ls rz+bench_gpu_time_with_cupti..s QqTr)keycg|] }|d S)rr>)rBr`s rrCz-bench_gpu_time_with_cupti..s333aQqT333rc34K|]}|dVdSr-Nr>)rBisorted_launchess rrFz,bench_gpu_time_with_cupti..s-QQq)!,QQQQQQrz,No kernel activities recorded for iteration c3.K|]}|VdSrEr>)rBrHr^s rrFz,bench_gpu_time_with_cupti..s/"S"S#9#9!#<#<"S"S"S"S"S"SrzInconsistent kernel names: z != c3&K|] }|dV dS)r0Nr>rBrHs rrFz,bench_gpu_time_with_cupti..s&33!333333rc3&K|] }|dV dSrer>rjs rrFz,bench_gpu_time_with_cupti..s&11qad111111r.A)Arrrrr_rr-importlib.metadatar.rsplit Exception functoolsr1ModuleNotFoundErrorr UserWarningbench_gpu_time_with_cudagraphr)r$r%rrstrrr rrrrStream wait_streamcurrent_streamstreamrV CUDAGraphgraphreplayr rrrr4activity_enabler;rTr<rUr8r9activity_register_callbacks get_timestampr#ractivity_flush_allactivity_disablefinalizebisectsorted enumerate bisect_left bisect_rightsetr&rr)@rrrrrrrrrr*rSrTrrrrr5importlib_metadata_version cupti_versionr1er6rWrr!r"runnergsrXr#rr r$rPrQiter_timestamps start_cpuend_cpur launch_startscorr_id_to_kernelsrHcorr_idr( kernel_namesidxleft_idx right_idxcorr_ids iter_kernelscurrent_kernel_names min_startmax_endspan_msrOr-r^rDrGrJrr@rgs@` `` @@@@@@@@@rbench_gpu_time_with_cuptirs Z  P P8-="O P P PPP;  )'     $,#7xxT 0@0L,,RU.=.I??v$ 5j,PVWW#$455$q[k:1LLLLLL22>BB }""3''* + +b 0 0Q  &%%%%%%  +''' a, - -  Mz$      MDDD$      0+) /- /%)+         2+) /- /%)+         ;'T,,,            uUE3S89:eCsCc3FGH:J54 #5#5H FW-..5< ]3C5:VVVF A    J     ej//11222 Z  q ! !  1XX                     !!##//222 J " " Z  a    GIII                J GIII J*"""66K   t 44I $ % %   LLNNN  J  ++.??$ As?5T#TUUVV 1c.3R"RSSTT  J = ! !   LLNNN J:??? %,3444 %,3444 %,3444 %%ww'>:::O33?333M ..A$ , , ,*, w '7#**1----NL%.%?%?''! !i%%mY??'' w?? QQQQeHi6P6PQQQQQ  A AG,,,##$6w$?@@@ SQCQQRR R""S"S"S"Sl"S"S"SSS  /LL333 Z,ZZDXZZ33l33333 11L11111Y&#-g&&&& sPAC F1A*E>F!E>8F>F$MMM OO"%O" num_iters_within_graphc   i td|||fDr#tjdtd||nd} n| } t d}t pt t jd}t jd}d }d | r{t t z}t|d krtjd tdd } n*t||}|d krt | nd }  fd }dtf fd }t jt j}|t jt j|5t)dD] }| dddn #1swxYwYt j|t j}t j|5| r)|d kr#t)|D]}||z}||nt)|D] }| dddn #1swxYwYt jd}|t)|D]}||t j|||z }| t5d t||z }| t5d t||z }t jt)|D]}|t jdt)|D}dt)|D}t jt)|D][}|||||| rt7|\t jg}t)|D]9}||||||z :|S)a Benchmark GPU time using CUDA graphs with amortized kernel launch overhead. CUDA graphs capture a sequence of GPU operations and replay them with minimal CPU overhead. By running multiple iterations within a single graph, kernel launch latency is amortized, yielding measurements closer to pure GPU time. **Cold-L2 Benchmarking**: When ``cold_l2_cache=True``, the function uses **rotating buffers** to ensure cold L2 cache for each kernel invocation within the graph. Multiple copies of the GPU tensors in ``input_args``/``input_kwargs`` are created and rotated through during graph capture, ensuring each kernel invocation operates on different memory regions. The number of buffer copies is automatically calculated based on the device's L2 cache size. Args: fn (Callable): The kernel function to benchmark. dry_run_iters (int, optional): Number of warmup iterations (not timed). If None, computed from dry_run_time_ms. repeat_iters (int, optional): Number of measured iterations (graph replays). If None, computed from repeat_time_ms. dry_run_time_ms (int): Target warmup duration in ms (default: 25). repeat_time_ms (int): Target measurement duration in ms (default: 100). num_iters_within_graph (int): Number of kernel calls captured in the graph (default: 10). Higher values better amortize launch overhead but use more memory when rotating buffers. sleep_after_run (bool): If True, sleep briefly after each iteration (default: False). input_args (tuple): Positional arguments to pass to fn. GPU tensors in this structure will be cloned when ``cold_l2_cache=True``. input_kwargs (dict, optional): Keyword arguments to pass to fn. GPU tensors in this structure will be cloned when ``cold_l2_cache=True``. cold_l2_cache (bool): If True, use rotating buffers to ensure cold L2 cache for each kernel invocation within the graph (default: True). Returns: List[float]: Per-iteration execution times in milliseconds. Each time is the graph replay duration divided by ``num_iters_within_graph``. Example: Cold-L2 benchmarking (default, for memory-bound kernels): >>> def run_attention(q, k, v, o): ... flashinfer.single_prefill_with_kv_cache(q, k, v, o) ... >>> q = torch.randn(batch, heads, seq_len, head_dim, device="cuda") >>> k = torch.randn(batch, heads, seq_len, head_dim, device="cuda") >>> v = torch.randn(batch, heads, seq_len, head_dim, device="cuda") >>> o = torch.empty_like(q) >>> times = bench_gpu_time_with_cudagraph( ... fn=run_attention, ... input_args=(q, k, v, o), ... ) >>> print(f"Cold-L2 median time: {np.median(times):.3f} ms") Example: Hot L2 benchmarking (for compute-bound kernels): >>> times = bench_gpu_time_with_cudagraph( ... fn=lambda: torch.matmul(q, k.T), ... cold_l2_cache=False, ... ) Note: - When using ``input_args``/``input_kwargs``, the function must accept the tensors as arguments (not capture them from closure). - GPU tensors are automatically detected and cloned. Non-tensor arguments (scalars, booleans, etc.) are preserved across all copies. - Memory usage scales with the number of rotations needed to exceed L2 cache. See Also: - ``calculate_rotation_count``: Computes required buffer copies for cold-L2. .. deprecated:: The ``l2_flush``, ``l2_flush_size_mb``, and ``l2_flush_device`` parameters are deprecated. Use ``cold_l2_cache`` instead. Nc3K|]}|duV dSrEr>rs rrFz0bench_gpu_time_with_cudagraph..crrzl2_flush, l2_flush_size_mb, and l2_flush_device are deprecated. Use cold_l2_cache instead. For CUDA graphs, cold_l2_cache uses rotating buffers (not L2 flush) to ensure cold cache.r-rTrrr0rzfcold_l2_cache=True but no GPU tensors found in input_args/input_kwargs. Cold L2 benchmarking disabled.Fc4r idSdSrEr>rsrrz.bench_gpu_time_with_cudagraph..call_fnrrbuf_idxc.|\}}|i|dSrEr>)rargskwargsrrotated_copiess rcall_fn_with_rotationz.call_fn_with_rotations,%g. f DFrr~r1cNg|]"}tjd#Sr r r s rrCz1bench_gpu_time_with_cudagraph..rrcNg|]"}tjd#Sr r r s rrCz1bench_gpu_time_with_cudagraph..rr)rrrrr_rr rr r'rrrr9r[rrrurvrwrxrVryrzrr{rr4rr#)!rrrrrrrrrrrSrTr _do_rotate_devicerr r8r^rrrrXrr'rr#r$r%r&r(rrs!` `` @@rrsrss6x  P P8-="O P P PPP #  D(     "*!5XX4 " )\6JJGJ54 #5#5H*"""66K   t 44IMN#*:669M : :   {  q M1$      JJ4['JJMq  !> m"" #  s  J AMM%*++--...   1  q  A GIIII  J++A...  A   !      -!++!"899 / /"]2%%g.... / 122                   J $ % %    J  ++.??$ As?5T#TUUVV 1c.3R"RSSTT  J = ! !   JWV% BUBUVVVLTTl@S@STTTJ J,''DDX%%'''  8##%%%  D "#B C C C JN,''    " / / 80D E E$ %     s%HH H AKKK enable_cuptic  td|||fDr#tjdtd||n|}n|}| rt ||||||| | | | S| rt |||||| || | | St ||||||| | | S) a Unified GPU benchmarking interface with configurable timing backends. This is the recommended entry point for GPU kernel benchmarking. It provides a single interface that dispatches to the appropriate timing implementation based on the configuration flags. **Timing Backends** (in order of precedence): 1. **CUPTI** (``enable_cupti=True``): Most accurate, measures pure GPU kernel time via hardware profiling. Requires cupti-python >= 13. 2. **CUDA Graphs** (``use_cuda_graph=True``): Amortizes launch overhead by capturing and replaying multiple kernel calls. Good balance of accuracy and availability. 3. **CUDA Events** (default): Simplest method, measures launch + execution. Available everywhere but includes CPU overhead. **Cold-L2 Strategy** (automatically selected based on timing backend): .. list-table:: :header-rows: 1 * - Timing Backend - Cold-L2 Strategy - How it Works * - CUPTI - L2 Flush - Flush L2 cache before each iter * - CUDA Events (no CUDA Graphs) - L2 Flush - Flush L2 cache before each iter * - CUDA Events + CUDA Graphs - Rotating Buffers - Clone GPU tensors in input_args/input_kwargs and rotate through them use_cuda_graph (bool): If True, use CUDA graph timing (default: False). num_iters_within_graph (int): Kernel calls per graph (CUDA graph mode only, default: 10). input_args (tuple): Positional arguments to pass to fn. input_kwargs (dict, optional): Keyword arguments to pass to fn. cold_l2_cache (bool): If True, ensure cold L2 cache for each iteration (default: True). The strategy is automatically selected based on timing backend. Returns: List[float]: Per-iteration execution times in milliseconds. Example: Simple benchmarking with CUDA events (default): >>> times = bench_gpu_time(fn=lambda: my_kernel()) >>> print(f"Median: {np.median(times):.3f} ms") Example: CUDA graph benchmarking for reduced launch overhead: >>> def run_kernel(x, y, out): ... my_memory_bound_kernel(x, y, out) >>> times = bench_gpu_time( ... fn=run_kernel, ... input_args=(x, y, out), ... use_cuda_graph=True, ... ) Example: CUPTI benchmarking for most accurate GPU kernel time: >>> times = bench_gpu_time( ... fn=run_kernel, ... input_args=(x, y, out), ... enable_cupti=True, ... ) See Also: - ``bench_gpu_time_with_cuda_event``: Direct CUDA event timing. - ``bench_gpu_time_with_cudagraph``: Direct CUDA graph timing. - ``bench_gpu_time_with_cupti``: Direct CUPTI timing. .. deprecated:: The ``l2_flush``, ``l2_flush_size_mb``, and ``l2_flush_device`` parameters are deprecated. Use ``cold_l2_cache`` instead. c3K|]}|duV dSrEr>rs rrFz!bench_gpu_time..Grrrr-rN) rrrrrrr*rSrTr) rrrrrrrrSrTrr2)rrrrrrsr))rrrrrrrrrrr*rrSrTr_cold_l2_caches rbench_gpu_timersF P P8-="O P P PPP '  )'     &.%9}&  ('%+)+)!%(      ,'%+)#9+!%(     * #!'%'!$    rceZdZdZdZdS)empty_suppressc|SrEr>selfs r __enter__zempty_suppress.__enter__{s rcdSrEr>rrXs r__exit__zempty_suppress.__exit__~s rN__name__ __module__ __qualname__rrr>rrrrzs2     rrceZdZdZdZdS)suppress_stdout_stderrc.ttjd|_ttjd|_t j|_t j |_ tj t j|_ tj t j |_ t j|_t j |_tj|j|jtj|j|j |jt _|jt _ |S)Nw)openosdevnull outnull_file errnull_filesysstdoutfilenoold_stdout_fileno_undupstderrold_stderr_fileno_undupdupold_stdout_filenoold_stderr_fileno old_stdout old_stderrdup2rs rrz suppress_stdout_stderr.__enter__s  S11 S11'*z'8'8':':$'*z'8'8':':$!# (9(9(;(;!rrrrs2( " " " " "rr num_testssuppress_kineto_output trace_pathflush_l2with_multiple_kernelsc ttjdd}td}||r |stnt } | 5|s#t jddddnd} |s6t j t jj j g| n t } | 5tdD]i} t|D]A} |r3t j |t jd |B|s| j dddn #1swxYwYdddn #1swxYwY|rdSt!|t"t$fsJt!|t$}| d d d }t!|t"r|fn|}t-d|DsJ|s2|D]/t/fd|DdksJdd0|| |ddd}g}|D]Ɗd}d}|D]}|vr|d}|d}|D]V\}}||vrM|t5||d|z t|zz }|t|z }nW|||z |rt%|n|dS)NDG_NSYS_PROFILINGrgeAr0)waitwarmupactiver )rRscheduler-rr;cuda_time_totalr)sort_bymax_name_column_width c8g|]}t|tSr>)rrt)rBr=s rrCz bench_kineto..s"???$ 4%%???rcg|]}|vSr>r>)rBliner=s rrCz bench_kineto..s<<< <<NT) NNrrNNNFFr>NT) NNrrrNNNFr>NT)NNrrNNNFFFrr>NT)rFNTF)7__doc__r2rrtypingrrrrrrrnumpyrr einopsrrr flashinfer.utilsr rrrr!r'r9r@r(r[r_rjr|rrrrrrrrrbfloat16rrrrtr)rrsrrrrr r>rrrsf  ------------  ,,,,,,,,,,%%%%%%  c     T%,%7C$el!30DE'-'- %, '-=@'-'-'-'-'-T(((V%):= %t B(;el;;;; UU\UeEL%,4N.OUUUUU\eEL%,4N.O"___D444n000&   $111hHHHV#;#;#;L!>!>!>Z N ^ N(F(F(F(Ff N ^ N0F0F0F0Fj#&*%)!#'\\\\ \  \ tn \sm\c]\\\4.\\\\\B#&*%)! #'rrrr r  r tn rsmrc]rrrr4.rrrrrn "$#&*%)!#'^^^^ ^  ^  ^tn^sm^c]^^^4.^^^^^F#&*%)! "$#'SSSS S  S tn SsmSc]SSSS SS4.SSSSSl         " " " " " " " "N#("'\@\@\@! \@  \@  \@ \@\@\@\@~r