)`ik dZddlZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl m Z mZmZmZmZddlZddlZddlZdedefdZeejd d Zeejd d Zejd dZeejddZeejddZejddZ ejddZ!de "dDZ#de!"dDZ$ejdd dkZ%da&da'iZ(dZ)ej*dZ+dZ,e,defdZ-dZ.d ede/fd!Z0d"e d#eee fddfd$Z1d"e deeee ffd%Z2d&ej3defd'Z4d(e de fd)Z5d*e6d+e7deeeej3feee fffd,Z8 dMd-ed ed*e6d+e7d.eedeef d/Z9d0ed1e ddfd2Z:d(e de fd3Z;d4ed5edeefd6Z< dNd8e d9e d:ed;ede/f d<Z= dOd0ed>e/d?ed@e/dAeeeeefe fde f dBZ>dPdCed?ede?fdDZ@dEZAeAe.dQd(e dFedGedefdHZBd-ed*e6d+e7de7fdIZCd-ed ed*e6d+e7dFeddf dJZDd ed1e dFeddfdKZEdMd-edefdLZFdS)Ra3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)datetime)Path)AnyCallableDictTupleOptionalpathreturncvd|vr4|dttjS|S)z Replace %i with the current process ID in a path. This is useful for multi-process/multi-GPU environments where each process needs its own log file. z%i)replacestrosgetpid)r s j/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/api_logging.py_substitute_process_idr"s4 t||||D#bikk"2"2333 KFLASHINFER_LOGLEVEL0FLASHINFER_LOGDESTstdoutFLASHINFER_DUMP_DIRflashinfer_dumpsFLASHINFER_DUMP_MAX_SIZE_GB20FLASHINFER_DUMP_MAX_COUNT1000FLASHINFER_DUMP_INCLUDEFLASHINFER_DUMP_EXCLUDEc^g|]*}||+Sstrip.0ps r r(;-SSSS!''))SSSr,c^g|]*}||+Sr"r#r%s rr(r(<r)rFLASHINFER_DUMP_SAFETENSORS1Fzflashinfer.apictdkrTttjttjdzdSttjtj tdkrtj tj }nEtdkrtj tj}ntjtd}tjd}||t|d t_dS) z1Set up the logger based on environment variables.rNrstderra)modez %(message)sF)_API_LOG_LEVEL_logger addHandlerlogging NullHandlersetLevelCRITICALDEBUGhandlersclear _API_LOG_DEST StreamHandlersysrr0 FileHandler Formatter setFormatter propagate)handler formatters r _setup_loggerrFKs7.00111)A-... W]###   ' 33 ( " "' 33%m#>>>!-00I ### wGrcNtjdS)z:Get current timestamp in the format [YYYY-MM-DD HH:MM:SS].z[%Y-%m-%d %H:%M:%S])rnowstrftimer"rr_get_timestamprJms <>> " "#8 9 99rcLtdkrtdtdtdttrtdtrtdtt rtdt tddSdS) z;Warn users about security implications of Level 10 logging. P================================================================================aWARNING: FlashInfer API Logging is set to Level 10 (Tensor Dumping). This will dump ALL input and outputs including tensors for FlashInfer APIs to disk in the configured dump directory. Ensure that you are NOT processing sensitive data or that the dump directory is secure. To disable dumping, unset FLASHINFER_LOGLEVEL or set it to below 10. For more information, see https://docs.flashinfer.ai/logging.htmlzCurrent dump directory is: u⚠️ SAFETENSORS mode enabled: tensor stride/non-contiguity will NOT be preserved. Tensors will be saved as contiguous. Use torch.save (default) to preserve strides.zInclude filter: zExclude filter: N)r3print _DUMP_DIR_DUMP_SAFETENSORS_DUMP_INCLUDE_PATTERNS_DUMP_EXCLUDE_PATTERNSr"rr _warn_dumprSrs h  d    7I77888   i    " ? =%;== > > > ! ? =%;== > > > h'r func_namectr"tfdtDsdStr"tfdtDrdSdS)aN Check if a function should be dumped based on include/exclude filters. Uses fnmatch-style patterns (wildcards: * for any chars, ? for single char). Matching is case-sensitive. Parameters ---------- func_name : str The function name to check. For class methods, this is formatted as "ClassName.method_name" (e.g., "BatchDecodeWrapper.run"). Returns ------- bool True if the function should be dumped, False otherwise. Filter Logic ------------ 1. If FLASHINFER_DUMP_INCLUDE is set: - Function must match at least one include pattern - If it doesn't match any, return False (skip dump) 2. If FLASHINFER_DUMP_EXCLUDE is set: - If function matches any exclude pattern, return False (skip dump) 3. Otherwise, return True (dump the function) c3BK|]}tj|VdSNfnmatchr&patrTs r z(_should_dump_function..s/UUs7?9c22UUUUUUrFc3BK|]}tj|VdSrWrXrZs rr\z(_should_dump_function..s/QQ3wy#..QQQQQQrT)rQanyrR)rTs`r_should_dump_functionr_sr8UUUU>TUUUUU 5 QQQQ:PQQQ Q Q 5 4rfilepathrecordct|d5}|tj|dzddddS#1swxYwYdS)z Append a JSON record as a single line to a JSONL file. Parameters ---------- filepath : Path Path to the JSONL file record : Dict[str, Any] Record to append (will be serialized as single-line JSON) r1 N)openwritejsondumps)r`rafs r_append_to_jsonlris h  +  6""T)***++++++++++++++++++s+A  A A c|sdSd}t|d5}|D]}|}|r|} dddn #1swxYwY|rtj|SdS)am Read the last record from a JSONL file. For metadata.jsonl, this returns the most complete state (completed if available, otherwise inputs_saved). Parameters ---------- filepath : Path Path to the JSONL file Returns ------- Optional[Dict[str, Any]] The last record, or None if file is empty/doesn't exist Nr)existsrdr$rfloads)r` last_linerhlines r_read_jsonl_last_recordrps" ??  tI h  ! ! !D::<) isinstancetorchdtyperrzEnumry__name__r|rvintfloatboollisttupledict Exception)rvs r_serialize_valuerse eU[ ) ) &U ty ) ) ;;/>>%*>>  UCtDzzB C C L eT2 3 3 U ,UETE*  U ,E 5D5)     KK((      s*+D!AD!/7D!'A D!5+D!!!EEargskwargsci}i}t|D]N\}}d|}t|tjr|||<.ys+SSq/22SSSSSSrizDump size limit reached (z GB). Skipping dump for z1. Increase FLASHINFER_DUMP_MAX_SIZE_GB if needed. save_filec>i|]\}}||Sr" contiguousr&kvs r z)_dump_function_inputs..s3***.2a1<<>>***rinputs.safetensorszHsafetensors package not installed. Install with: pip install safetensors inputs.pt __module__z )input_tensor_keysoutput_tensor_keysinput_size_bytes input_size_mb safetensorsr signaturez )rpython inputs_saved) function_namemodule call_sequence timestamp process_idinput_metadataoutput_metadata tensor_infotensor_details tensor_formatfunction_signatureversionsexecution_statusrshaperstridedevicer __version__r flashinferdump_dirmetadata.jsonl session.jsonlzDumped inputs to: z (size: .2fz MB, total: /z dumps)zFailed to dump function call : )/r_r4debug _dump_count_DUMP_MAX_COUNTwarning_dump_call_counterrrHrIrrrrOmkdirrsumvalues_DUMP_MAX_SIZE_GB_dump_total_size_bytesrmdirrPsafetensors.torchrrr ImportErrorerrorrsavehasattrrrkeysinspectrrr?versionrrrrrri traceback format_exc)rrTrrrcall_seqrpid dump_namer input_tensorsr input_sizemax_size_bytesrtensors_contiguousrrrqflashinfer_versionsession_jsonl_pathers r_dump_function_inputsr3sJ ! + + S S S S   to%% =? = =y = = =   tB . . .,- y )9%%%*%%%%i0LNN++,>?? RC ikk JJcJJIJJHJJJ  ??Y.td333)FdF(S(S% ~SSM*>%?%?&($.!+{!;  !.?L]]Ww ,,#!#g&7&=&=">">"> *+!//$ $ 6  ")HY )..00  KCfl++V\**v}}//fm,, //H% &s + + A B B B B B B1CHZ  . . A A A1@HZ  . . . A #8}} $44h???")__>+X666 q *,  = = = K0 ; = =! = =$3 = = =   8}}  FiFF1FFGGG i**,,---ttttt s_)EQ 3 Q =AG>=Q >&H$$EQ ;N Q N%"Q $N%%B&Q R1AR,,R1rresultc Z t|}|std|dSi}i}t |t jr||d<nt |tr`t|D]O\}}t |t jr||d|<:t||d|<Pnt||d<td| D}|r`trAddlm}d|D} || t#|d z nt j||d z |d z } t'| } | || d <t)|| d d<|| d d<|dz | d d<| d d|z| d d<| d ddz | d d<d| d<d| vri| d<|D]j\} } t)| jt#| jt)| t#| jd| d| <kt5| | tt6dz }t5|| t8|z atd|d|dz dd| d ddddStd|dS#t<$r[}td |d!|ddl}t| Yd}~dSd}~wwxYw)"al Add function outputs to an existing dump directory (crash-safe). This function is called AFTER successful execution to append outputs to the dump that was created before execution. Parameters ---------- dump_dir : str Path to the dump directory created by _dump_function_inputs result : Any Function return value Dump directory not found: Nrresult_c34K|]}t|VdSrWrrs rr\z)_dump_function_outputs..s+UU033UUUUUUrrrc>i|]\}}||Sr"rrs rrz*_dump_function_outputs..s3&&&*.!QAq||~~&&&routputs.safetensors outputs.ptrrrroutput_size_bytesroutput_size_mbrtotal_size_bytes total_size_mb completedrrrrzDumped outputs to: z (output size: rz MB, total dump size: z MB)z%metadata.jsonl not found or empty in zFailed to dump outputs to r)!rrlr4rr~rrrrrrrrrPrrrrrrprrrrrrrirOrrrrr)rr dump_pathoutput_tensorsrritem output_sizerrmetadata_jsonl_pathrrrqrrrs r_dump_function_outputsr sv X.NN !!  MMAxAA B B B F fel + + A'-zz||N8 $ $  & & A$V,, L L4dEL11L48HHJJN=Q==115Ed5K5KOMaMM22  L )9(@(@OH %UU^=R=R=T=TUUUUU   E E777777&&2@2F2F2H2H&&&" ,c)>S2S.T.TUUUU >9|+CDDD(*::*+>??  *9H& '<@ATATAVAV.sI  1 Q4 . .      r)strictzKey mismatch: actual c3TK|]"}t||V#dSrWr$)r&rrr!rr s rr\z#_compare_results..s:XXA#F1Ix{D$GGXXXXXXrzValue mismatch: actual )r~rrrr4rrallcloseabsmaxrrrlenallziprr)rrr r!diffs```` rr%r%s &%,''/Jx,N,N/ <8> ) ) OOU&,UUX^UU   5 <8> ) ) OOU&,UUX^UU   5~fhTEEE X%**,,00227799D OO>>> ? ? ?5t FT5M * *z(T5M/R/R v;;#h-- ' ' OOT3v;;TTS]]TT   5     FHT:::      FD ! ! j4&@&@  ;;==HMMOO + + OOU UUHMMOOUU   5XXXXXXXQWXXXXXX X   OOUfUU8UU V V V5trcudacompare_outputsrrunobject_registryc t|}|std||dz }|std|t|}|t d||d}|dz } |dz } | r$t jt| d } nf| r@ d d lm } | t| d } n0#t$rtddwxYwtd|| D]\} }| || | <g}i}| di}d}| D]O} | dr8t!| dd}t%||}P|D]O} | dr8t!| dd}t%||}Pt'|dzD]}d|} | | vr|| | '| |vr)|t+|| Tt,d|d|d| D]8} | dr!| dd}| | ||<9|D]I} | dr2| dd}||vrt+|| ||<Jt,d|d|t,dt5|dt7||||d}i}i}|r|dz }|dz }|r$t jt|d }nT|r@ d d lm } | t|d }n#t$rtddwxYw| D]\} }| ||| <| d i}||d!<||d"<|r| d#}| d$}| d%}d}d}|||f} |d&r|d'd(}!t;||!}"|"rt=|"rt5|d kr |ddng}# t,d)|!d*|d+|d,|"|#i|}|||| <d}$|$|d-<|rd.|d/<|S#t>$r>}%t, d0|!d1|%t|%|d2<|cYd}%~%Sd}%~%wwxYwn|| |vr|| }|d'd}&tC||&r0tE||&}t5|d kr |ddng}nBt,d3|d4|&n!t,d5|d+|d6|t;||}|r t,d7|d'|d8||i|}$|$|d-<|ri}'tG|$t j$r|$|'d9<nktG|$tJt6fr8tM|$D]'\}}(tG|(t j$r|(|'d:|<(ntG|$tNr|$}'d.})|rtQ|'|})|)|d/<|)rt,d;nt,d<n#t>$ri}%t, d=|%d dl)}*t, |**t|%|d2<Yd}%~%nEd}%~%wwxYwt,d>|d'|n|st,d?|S)@a Replay a function call from a dumped directory. This function: 1. Loads metadata.jsonl to get function info 2. Loads inputs.pt to get input tensors 3. Moves tensors to specified device (default: cuda) 4. Reconstructs the function call 5. Optionally executes the function (if run=True) 6. Optionally compares with saved outputs Parameters ---------- dump_dir : str Path to the dump directory compare_outputs : bool If True, load and compare with saved outputs device : str Target device for tensors. Options: - "cuda" (default): Load to cuda:0 - "cpu": Load to CPU - "cuda:N": Load to specific CUDA device run : bool If True, try to resolve and execute the function object_registry : Optional[Dict[Tuple[int, int], Any]] Registry of stateful objects mapped by (process_id, self_id) tuple. This composite key ensures objects from different processes don't collide in multi-GPU environments where different processes may have objects at the same memory address. Returns ------- result : dict Dictionary containing: - 'args': Positional arguments (tensors on specified device) - 'kwargs': Keyword arguments (tensors on specified device) - 'metadata': Full metadata - 'execution_result': Result of execution (if run=True) - 'comparison_match': Boolean indicating if result matched expected (if run=True and compare_outputs=True) If compare_outputs=True, also includes: - 'expected_tensors': Expected output tensors - 'expected_metadata': Expected output metadata rrzmetadata.jsonl not found in Nzmetadata.jsonl is empty in rrrr) map_locationr) load_file)rz`Dump was saved with safetensors but package not installed. Install with: pip install safetensorsz2Neither inputs.pt nor inputs.safetensors found in rrrr/zMissing argument z in dump.rrz Replaying z from z Args: z , Kwargs: )rrrrrrexpected_tensorsexpected_metadatarrrz .__init__r{zInstantiating z (PID: z, ID: z)...execution_resultTcomparison_matchzFailed to instantiate rexecution_errorzObject z has no method z Object (PID: z) not found in registry.z Executing ...rrzReplay comparison passed!zReplay comparison FAILED.zExecution failed: z&Skipping execution: could not resolve z\Automatic function resolution disabled. Pass run=True to execute, or manually call function.)+rrlFileNotFoundErrorrp ValueErrorrloadrrr5rrtor r startswithrrr*rangeappendrr4rr infor+rendswithrrrrrr r~rrrrr%rr)+rr0rr1r2rr rrTinputs_pt_pathinputs_safetensors_pathrr5rrqrrr max_arg_idxidxr kwarg_name result_dictexpected_outputsroutputs_pt_pathoutputs_safetensors_pathrrrrr registry_key class_namecls_obj real_argsr:r method_nameactual_outputsrmatchrs+ rreplay_from_dumprWs dXI     I GX G GHHH$&66  % % ' 'K Ix I IJJJ&':;;HAxAABBB)I,N'*>>  3~#6#6UKKK ' ' ) )    3 3 3 3 3 3%Ic*A&B&B5QQQMM   8    K K K   %**,,// V#YYv.. c D F\\"2B77NK!!##00 >>& ! ! 0ciinnQ'((Ck3//K""$$00 >>& ! ! 0ciinnQ'((Ck3//K;? # #  Qjj -   KK c* + + + + N " " KK*>#+>?? @ @ @ @ OO<<<< = = = KK    !!##44 >>( # # 4Xr22J!.s!3F: ""$$MM >>( # # MXr22J''%7s8K%L%Lz" LL9i99x99::: LLFCIIFFfkkmm1D1DFFGGG+/6x"X"XKO;#l2#,/D#D  ! ! # # $z#o*>*>USSS   % , , . .  777777#,9011%$$$     !< ,1133 6 6KC$*IIf$5$5 S ! !",,'8"==*: &'+: '( r ll8,, ,,y))\\,//   &0L!!+..2 '__S11 ,KDD+x00+-0IIMMQRRrI+ _Z__ __RY___ &gy;F;;*6Aa99a>root_dirc t|}|std|g}|D]B}|r,|dz r||C|dg}t|}t d|d|i}t|D]\}} t d|dzd |d | j d  tt| d |d | } t| | d<|| {#t$ra} td| j d| |t| t| dYd} ~ d} ~ wwxYw|S)a Replay a sequence of API calls from a root dump directory. This function iterates through all dump directories in the root directory, sorted by timestamp/sequence number, and replays them in order. Parameters ---------- root_dir : str Path to the root directory containing dump subdirectories device : str Target device for execution (default: "cuda") Returns ------- list List of results from replay_from_dump calls zRoot dump directory not found: rc|jSrW)r|)xs rz!replay_sequence..sr)rzFound z dumps to replay from [r/rz ] Replaying r=T)r0rr1r2rzFailed to replay r)rrN)rrlr>iterdiris_dirrDsortr+r4rErr|rWrrr) rXr root_path dump_dirsrresultstotalr2rrresrs rreplay_sequencerfs&XI     N L( L LMMMI!!#### ;;== #d%55==?? #   T " " "NN''N(((G  NNE LLA%AAxAABBB35O ++II 8 FQFFFFHMFFFGGG I#H $ / C"(mmC O NN3     I I I MMBhmBBqBB C C C NNSVVXGG H H H H H H H H I NsAE## G-AG  Gc .tdkrdSg}|d|td|d ddlm}|d|n%#t $r|dYnwxYwt jj}|r|d |n|d  t jj rTt jj }|r|d |n+|d n|d n0#t $r#}|d|dYd}~nd}~wwxYwt j rt j }|d|t|D]} t j |}t j|}|ddz|dz} |d|d||d|dd|dd| d#t $r&}|d|d|dYd}~d}~wwxYwn|d|dt jn/#t $r"}|d|Yd}~nd}~wwxYw|d|dtd|dS)z5Log system information once at module initialization.rNrMz, FlashInfer API Logging - System Informationr/rzFlashInfer version: z!FlashInfer version: zCUDA toolkit version: z5CUDA toolkit version: zcuDNN version: zcuDNN version: zcuDNN version: zcuDNN version: zNumber of GPUs: rLz GPU rz Compute capability: r{z (SM)z : B B B B B B LLD0BDD E E E E > > > LL< = = = = = >})  R LL@,@@ A A A A LLP Q Q Q 9~#0022 ? % 4 < < > >  ALL!B=!B!BCCCCLL!?@@@@ =>>> 9 9 9 LL71777 8 8 8 8 8 8 8 8 9 : " " $ $ @ :2244L LL:L:: ; ; ;<(( = = =$z99!< ? ? ?  <):<<==== AAA ?A??@@@@@@@@A LL LL MM$))E""#####s A?>L?B!L B!!AL&B E32L3 F =FLF  A)L BJ#"L# K-K LK;L L;L66L;levelindentc td|z}||dSt|tjr!||jjd|jd|jdSt|tjr|dkr|dS|d g}| |d t|j | |d t| | |d |j | |d |j| |d|j| |d||dkr d}|jrit%tjdrOt)jt,5tj}dddn #1swxYwY|r| |dn|dkr|j tjtjtjtjtjtjfvrw|}| |d| !d| |d|"!d| |d|#!dtj$|%!}| |d|tj&|%!}| |d|n |j tj'tj(tj)tj*tj+fvr| |d| !| |d|"!| |d|#!dn2#t,$r%} | |d| dYd} ~ nd} ~ wwxYw| |dd,|St%|dr|jjdkr|dkr|d S|d!g}| |d"t[|j.||dz| |d#t[|j/||dz| |d$|j0t%|d%r&|j1| |d&|j1| |dd,|St|tdrtg|dkr|d'S|dkr|d(tg|d)S|d*g}ti|D]4\} } | |d+| d,t[| ||dz5| |dd,|St|trtg|dkr|d-S|dkr|d.tg|d/S|d0g}ti|D]4\} } | |d+| d,t[| ||dz5| |dd,|St|tjrtg|dkr|d1S|dkr|d2tg|d3S|d4g}|6D]A\} } | |dto| d5t[| ||dzB| |d6d,|St|tpt>trttfr||St|tvr|to|S |to|S#t,$r|d7ty|jd8cYSwxYw)9aX Format a value for logging based on the log level. Parameters ---------- value : Any The value to format level : int The logging level (1, 2, or 3) indent : int The indentation level for nested structures Returns ------- str Formatted string representation of the value  NNoner{z (value=rir/z Tensor(...)zTensor(z shape=z stride=z dtype=z device=z requires_grad=z is_contiguous=Fis_current_stream_capturingz6 [statistics skipped: CUDA graph capture in progress]rz min=z.6fz max=z mean=z nan_count=z inf_count=z [statistics error: ]rc __class__ FP4TensorzFP4Tensor(...)z FP4Tensor(z data=z scale=z scale_start_index=original_shapez original_shape=z[]z [list with z items]r]z []: z()z (tuple with z items)(z{}z {dict with z keys}{r})=r~rzrrrr|rvrrrDrrrrr requires_grad is_contiguousis_cudarr/ contextlibsuppressrr~numelfloat16float32float64bfloat16 float8_e4m3fn float8_e5m2rminrr*meanisnanrisinfint8int16int32int64uint8rp _format_valuedatascalescale_start_indexrrr+rrrr}rrcomplexrry)rvrxry indent_strrq is_capturing val_float nan_count inf_countrrrrvals rrrhs "J }""""%##  X5?3 X Xej X X%+ X X X %&&? A:: --- -'''( @@E%+,>,>@@AAA DDU5<<>>-B-BDDEEE 99EK99::: ;;U\;;<<< IIE4GIIJJJ KKE4G4G4I4IKKLLL A::- G % =PWUZ9V%W%WP#,Y77PP',z'M'M'O'O PPPPPPPPPPPPPPP #LL%]]][[]]Q&&{   +) '%*KKMM  %V%V)--//:N:N:P:P%V%V%VWWW %V%V)--//:N:N:P:P%V%V%VWWW )OO)..2B2B2G2G2I2IOOO%*K $:$:$>$>$@$@$E$E$G$G  %K%K %K%KLLL$)K $:$:$>$>$@$@$E$E$G$G  %K%K %K%KLLLL     ) %N%N%))++:J:J:L:L%N%NOOO %N%N%))++:J:J:L:L%N%NOOO )SS%++--2D2D2F2F2K2K2M2MSSS G G G EEEEEFFFFFFFF G  %%%&&&yyuk"" u'?;'N'N A:: 000 0***+  P P- E6A:"N"N P P      R R=eVaZ#P#P R R     QQ8OQQRRR 5* + + Q0D0P LLJOO9MOO P P P %%%&&&yy%  u::?? $$$ $ A:: @@SZZ@@@ @!!!" ''  GAt LLPP!PP dE6A:(N(NPP      %%%&&&yy%  u::?? $$$ $ A:: AAc%jjAAA A!!!" ''  GAt LLPP!PP dE6A:(N(NPP      %%%&&&yy%  u::?? &&& & A:: AAc%jjAAA A"""#   HC LLUUcUUmCQR .S.SUU      &&&'''yy%#udG455&%e%%%%,+d5kk+++>+d5kk+++ >>>==tE{{3======>sO*: Log to specified file path - Use %i in path for process ID substitution (e.g., "log_%i.txt" -> "log_12345.txt") Level 10 Tensor Dumping (additional variables): FLASHINFER_DUMP_DIR : str (default: "flashinfer_dumps") - Directory where tensor dumps are saved FLASHINFER_DUMP_MAX_SIZE_GB : float (default: 20) - Maximum total size of dumps in GB FLASHINFER_DUMP_MAX_COUNT : int (default: 1000) - Maximum number of function call dumps FLASHINFER_DUMP_SAFETENSORS : int (default: 0) - 0: Use torch.save format (preserves stride/contiguity) - 1: Use safetensors format (no pickle, but loses stride info) FLASHINFER_DUMP_INCLUDE : str (default: "") - Comma-separated list of patterns to include for dumping (fnmatch-style) FLASHINFER_DUMP_EXCLUDE : str (default: "") - Comma-separated list of patterns to exclude for dumping (fnmatch-style) Examples -------- Basic usage: >>> @flashinfer_api ... def my_function(x, y): ... return x + y Notes ----- - Key header lines include a timestamp in the format: [YYYY-MM-DD HH:MM:SS] (e.g., "FlashInfer API Call: function_name", "FlashInfer API Logging - System Information") - When FLASHINFER_LOGLEVEL=0, the decorator has truly zero overhead as it returns the original function unchanged. - Function names and inputs are logged BEFORE execution: - Level 1: Function name only - Levels 3-5: Function name + inputs with metadata This means critical debugging information is preserved even if the function crashes (e.g., CUDA illegal memory access, out-of-bounds, etc.). - Outputs are logged AFTER successful execution for levels 3 and 5. - **CUDA Graph Compatibility**: At level 5, tensor statistics (min/max/mean/nan_count) are automatically skipped during CUDA graph capture to avoid synchronization issues. The message "[statistics skipped: CUDA graph capture in progress]" will be logged. - The %i pattern is automatically replaced with the process ID for multi-process environments. - The logger does not propagate to the root logger to avoid duplicate logs. rNc|SrWr")rhs rr\z flashinfer_api..sQrrhr cFtjfd}|S)Nc j}d}|r^t|ddrH |djj}d|vs|dvr|d|}t|d}n#t$rYnwxYwd}t dkrl t ||||}|rtd|n7#t$r*}t d |d |Yd}~nd}~wwxYw t d kr,ttd |n3t d kr(tt d}t ||||n7#t$r*}t d|d|Yd}~nd}~wwxYw |i|} t d kr&tt d}t|||n7#t$r*}t d|d|Yd}~nd}~wwxYwt dkrh|rf t||td|n7#t$r*}t d|d |Yd}~nd}~wwxYw|S)NrrWrapper)BatchMLAPagedAttentionWrapperr{rL)rzInputs dumped to: z[DUMP ERROR (inputs) in rr/rr}z[LOGGING ERROR in z (pre-execution)]: z (outputs)]: zOutputs dumped to: z[DUMP ERROR (outputs) in )rrridrr3rr4rrrJrrrr rE) rrrTrrQrreffective_levelrrhs rwrapperz2flashinfer_api..decorator..wrapperse IG Q55 !%a!2!;J J..*A33(2$?$?I$?$? "$T!W++ DH##P49dFG   H G &E8&E&EFFF PPPMM"NY"N"N1"N"NOOOOOOOOP V!Q&&MM)++NN9NN$q(('*.!&<&.decorators9   > > > >   > @r)r3r)rrs rflashinfer_apirsfR <;  BXB(BBBBH | 9T??rrW)rr)Fr/FN)r/)r)G__doc__rzrYrrrfr6rr?rpathlibrtypingrrrrr rrrrrrenvironr r3r=rOrrr _DUMP_INCLUDE _DUMP_EXCLUDErrQrRrPrrr_session_jsonl_initialized getLoggerr4rFrJrSrr_rirprrurrrrrr rrr%rWrrfrwrrrrrr"rrrs    77777777777777       RZ^^$93??@@&&rz~~6JH'U'UVV  JNN02D E E E"*..)FMMNN#bjnn%@&IIJJ 8"==  8"== SS]-@-@-E-ESSSSS]-@-@-E-ESSSJNN#@#FF#M " ' , - -> ::::: 0%S%T%%%%P +t +T#s(^ + + + + +dxS#X/G@55<5C5555 $ C$ C$ $ $ $ N# ## 4U\! "DcN 23####V" tt tt t  t c] t c] ttttnh.Sh.#h.$h.h.h.h.Vcc:3sx?Q(CG44 44&+4:?4 4444r"<@ \\\\ \  \ d5c?C#789 \  \\\\~>>c>3>D>>>>BD$D$D$P k>k>k>Sk>#k>ck>k>k>k>\(h(e(T(d((((V1$ 1$"1$*/1$9=1$FI1$ 1$1$1$1$h$S$#$c$d$$$$.TTTXTTTTTTr