)`iB|ddlZddlZddlZddlZddlZddlZddlmZmZddl m Z m Z ddl m Z ddlmZmZmZmZmZmZmZmZddlZddlmZddlmZd Zd efd Ze d GddZe d d GddZ e d d GddZ!e d GddZ"e d GddZ#ee#e"fZ$e GddZ%e GddZ&Gdd eZ'ej(d)d!efd"Z)e Gd#d$Z*e d%d&Z+Gd'd(Z,dS)*N)ABCabstractmethod) dataclassfield) lru_cache)AnyCallableDictListSetTupleUnionOptional) delay_kernel)loggerz0.1 is_modulectjddd}tdd}d|d|}|rd|St jt jt j td|d zS) Nr _.v_trtllm_fused_moe_zflashinfer.tuning_configs.tuning_configsz.py) torchcudaget_device_namereplace_nvfp4_cutlass_versionospathjoindirnamerealpath__file__)rdev_name cutlass_ver config_names h/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/autotuner.pyget_config_pathr*sz))!,,44S#>>H(00c::K?k??X??K 9K999w|| GOOBG,,X66 7 7  %    T)slotsceZdZUdZeedfed<eedfed<eeedfefed<eed<e dZ e eed <d Z d efd Z d S)DynamicTensorSpecat A specification for a dynamic tensor dimension. Args: input_idx: A list of the indices of the input tensors. dim_idx: A list of the indices of the dimensions to tune. The length of input_idx and dim_idx must be the same. For every tensor mapped to the input_idx, their dimension mapped to the dim_idx must be the same. gen_tuning_buckets: A tuple of values to try or a function generating values. map_to_tuning_buckets: A function to map dimensions to valid values during inference. tensor_initializers: A list of functions to initialize the tensors. . input_idxdim_idxgen_tuning_bucketsmap_to_tuning_bucketscdSNr5r+r)zDynamicTensorSpec.:sr+default_factorytensor_initializerscx|j2dtt|jD|_dSdS)Ncg|]}dS)c`tj||dzdz |SN)device rrandtoshapesdtyper>s r)r6z..@s.Jvf555:Q>"U))r+r5).0rs r) z3DynamicTensorSpec.__post_init__..?s/((((((r+)r9rangelenr/selfs r) __post_init__zDynamicTensorSpec.__post_init__<sN  # +((s4>2233 (((D $ $ $ , +r+returnct|j|jt|jt r|jnt |jt |jfSr4)hashr/r0 isinstancer1tupleidr2rKs r)__hash__zDynamicTensorSpec.__hash__Fs^ d5u==1''/004-..    r+N)__name__ __module__ __qualname____doc__r int__annotations__rr rr9r rMrTr5r+r)r.r.(s  S#X 38_eCHox78888####*/% *M*M*MhMMM  #       r+r.)r, unsafe_hashc2eZdZUdZeed<eed<eed<dS)ConstraintSpecz A specification for a constraint on a tensor dimension. Args: input_idx: The index of the input tensor. dim_idx: The index of the dimension to constrain. infer_shape: A function to infer the shape of the dimension. r/r0 infer_shapeN)rUrVrWrXrYrZr r5r+r)r]r]Us<NNN LLLr+r])kw_onlyr[cPeZdZUdZdZeedfed<dZee dfed<dS) TuningConfigaYConfiguration for autotuning. This class specifies all the tuning configurations for a single tuning process. Args: dynamic_tensor_specs (Tuple[DynamicTensorSpec]): Specifications for how different tensor dimensions should be tuned to optimize performance. Each spec defines: - Which input tensor dimension is dynamic - How to generate tuning values - How to map dimensions to valid values during inference Example: >>> config = TuningConfig( ... dynamic_tensor_specs=( ... DynamicTensorSpec( ... input_idx=[0], ... dim_idx=[1], ... gen_tuning_buckets=(32, 64, 128), ... map_to_tuning_buckets=lambda x: ((x + 31) // 32) * 32 ... ), ... ) ... ) constraint_specs (Tuple[ConstraintSpec]): Specifications for constraints on tensor dimensions. Each spec defines: - Which input tensor dimension is constrained - How to infer the shape of the dimension based on other dimensions Example: >>> config = TuningConfig( ... constraint_specs=( ... ConstraintSpec( ... input_idx=1, ... dim_idx=2, ... infer_shape=lambda shapes: shapes[0][0] * 2 ... ), ... ) ... ) r5.dynamic_tensor_specsconstraint_specsN) rUrVrWrXrbr r.rZrcr]r5r+r)raradsV$$L;=% 13 67<<<35eNC/055555r+ra)r[c eZdZUeed<dZdS) StaticDimvalc|jSr4)rfrKs r)_optzStaticDim._opt xr+N)rUrVrWrYrZrhr5r+r)reres. HHHr+rec8eZdZUdZeed<eed<eed<dZdS) DynamicDimzRange of one dimensionminoptmaxc|jSr4)rmrKs r)rhzDynamicDim._optrir+N)rUrVrWrXrYrZrhr5r+r)rkrksF  HHH HHH HHHr+rkcdeZdZUdZeeeed<eeeed<dZ dZ dS)OptimizationProfilez$Ranges of all tensors, all dimensionrEr9c*|Sr4)get_opt_shapesrKs r) get_hash_keyz OptimizationProfile.get_hash_keys""$$$r+cg}|jD].}|td|D/t|S)z.Only the opt shapes are considered as hash keyc6g|]}|Sr5)rhrGds r)rHz6OptimizationProfile.get_opt_shapes..s $9$9$9!QVVXX$9$9$9r+)rEappendrR)rL opt_shapests r)rsz"OptimizationProfile.get_opt_shapessV  < shapeN) rUrVrWrrFrZr>r r|r5r+r)r~r~s> ; L 9r+r~c eZdZedeejdedeefdZ dZ e d deejded e de fd Z d Zd S) TunableRunnerinputsprofilerNcdgS)arOne tactic corresponding to one cuda kernel normally, but how to interpret the meaning of tactic is pure internal details of the runner. The autotuner will just pass the tactic value to the forward w/o any knowledge on what the tactic means. tactic==-1 has special meaning, means the fallback kernel which should be able to implement any shapes This fallback tactic is needed for 2 reasons: * when the autotuner cannot find a valid tactic in it's cache. * in eager mode, w/o autotunning the custom op should have at least one kernel, which makes the autotuning process an optional process, such that user can opt out. We choose not to have a standalone can_implement function, the tactics returned by get_valid_tactics should return valid kernel for these given input tensors. r5)rLrrs r)get_valid_tacticszTunableRunner.get_valid_tacticss &t r+c |j|fi|Sr4)forward)rLrkwargss r)__call__zTunableRunner.__call__st|F--f---r+rFtacticdo_preparationc t)adForward pass for tunable runners. Args: inputs: List of input tensors (position-only argument) tactic: Integer ID specifying which implementation tactic to use. -1 (default) represents the fallback tactic that must be implemented to handle any input shapes when autotuning is disabled. do_preparation: When True, allows one-time setup operations to be performed before tactic evaluation begins. These operations are excluded from the performance measurements during autotuning. Notice that anything prepared in this phase should be persistent in the forward and can be accessed by the following forward calls. Returns: Any: Output of the forward pass )NotImplementedError)rLrrrrs r)rzTunableRunner.forwards 2"!r+chtt|jSr4)rPrR__dict__valuesrKs r)rTzTunableRunner.__hash__s&E$-..0011222r+N)rF)rUrVrWrr rTensorrqrYrrboolrrrTr5r+r)rrs5<(3F c^(...$ ""U\""" " """^"433333r+r tune_modec#Ktj}|t_|o| }|rtjd dV|t_|rtjddSdS#|t_|rtjdwwxYw)Nz*[Autotuner]: Autotuning process starts ...z$[Autotuner]: Autotuning process ends) AutoTunergetis_tuning_moderinfo)rold_modeautotune_enableds r)autotuners}}-H%.IMMOO" 1\B @AAA@ )1 &  @ K> ? ? ? ? ? @ @*2 &  @ K> ? ? ? ? @s B7Cc eZdZUdZdZeed<eeZ e e e e fed<eeZe e e ee eeffed<eeZe e efed<eeZe e efed<d e fd Zd S) AutoTunerStatisticsaStatistics collected by the AutoTuner. Attributes: cache_misses (int): Number of cache misses requiring fallback cache_miss_config_collection (Dict[str, Set[OptimizationProfile]]): Collection of configs that caused cache misses failed_profiling_count (Dict[str, int]): Number of failed profiling attempts per operation tuned_op_total_configs (Dict[str, int]): Total configurations tried per operation tuned_op_successful_configs (Dict[str, int]): Successful configurations per operation r cache_missesr7cache_miss_config_collectionfailed_profiling_counttuned_op_total_configstuned_op_successful_configsrNcd}|d|jdz }|jr\|dz }t|jD]0\}}|d|dz }t|tD] }|d|dz } 1|jr|d z }t|jD]}|j|}|j|d }t|j |t}|d kr||z d znd }|d|dz }|d |dz }|d |dz }|d|dz }|d kr|dz }|j |D] } |d| dz } |d|ddz }|S)z7Return a string representation of collected statistics.zCache misses:  zCache miss config collection: z z: )keyz - Config: zTuned operations: rdz - Total configs tried: z - Successful configs: z - Failed profiling count: z% - Failed profiling combinations: z - z - Success rate: z.1fz% ) rrsorteditemsstrrkeysrrrJrset) rL stats_stropprofilesrtotal successfulfailed success_rate failed_keys r)__str__zAutoTunerStatistics.__str__s  ;d&7;;;;  , > : :I &t'H'N'N'P'P Q Q > > H\"\\\) %hC888>>G!='!=!=!==II>  & J . .IT8==??@@ J J3B7!=AA"aHH T8< %>%>%>> ILIIIII r+N)rUrVrWrXrrYrZrdictrr rr rRrr rrqrrrr5r+r)rr sL#:?%PT:U:U:U $sCJ"7UUU d###D SsM+>>? @@$$$.3U4-H-H-HDcNHHH27%2M2M2Mc3hMMMr+rmaxsizectd} tj|}|j}n#tt f$rd}YnwxYw|^t |d|d|df}||vr6tjd|dd||d||ddfStjd|dd S) NT)rrrz![Autotuner]: Loading configs for z from file.z1 from file failed; Using default configs instead.FrrN) r* importlib import_module best_configs ImportErrorAttributeErrorrrr)r module_namemodulerks r)load_from_filer<s!D111K(55*  (  QQQ( ) )   KJAJJJ K K Ka+\!_Q-?E E KbCbbb  s.AAceZdZdZdZd!dZedZded e e d e e j d ed e eeeeff d Zded e e d ede e jd e e eff dZde e jd e e j fdZde de e jded efdZd ede e jd e efdZeedde e j d ed e fdZedede d e e j d ed e f dZde jde eded e jfdZdede e jd e e jfdZd"dZ d"d Z!dS)#raAutoTuner for optimizing TensorRT-LLM operations. This class handles automatic performance tuning of tensor operations by profiling different implementations and caching the best performing configurations. Args: warmup (int): Number of warmup iterations before profiling (default: 3) repeat (int): Number of profiling iterations for averaging (default: 10) stream_delay_micro_secs (int): Delay on CUDA stream before the profiled kernel runs in microseconds (default: 1000) Nrr?c||_||_||_i|_d|_t |_d|_dS)NFT)repeatwarmupstream_delay_micro_secsprofiling_cacherrstatsprofiling_debug)rLrrrs r)__init__zAutoTuner.__init__]sE  '>$!#)** #r+cD|jt|_|jSr4) _instancer)clss r)rz AutoTuner.getis = %KKCM}r+ custom_oprunners input_shapes tuning_configrNc|D]y}t||||}tjdddkr|jst |}|cS||jvrdg|j|RcSzdS)aSearch for cached profiling results matching the current configuration. Args: custom_op (str): The name of the custom operation to be tuned runners (List[TunableRunner]): List of candidate implementations to profile profile (OptimizationProfile): Optimization profile Returns: A tuple containing: [is_cache_hit, runner_id, tactic, stored_profile] #FLASHINFER_AUTOTUNER_LOAD_FROM_FILE01Tr)r_get_cache_keyr environrrrr)rLrrrrr cache_keyoutputs r) search_cachezAutoTuner.search_cacheos$ > >A!001lMI DcJJcQQ+R( 22 d222=d29======3"!r+rc t||}|js}|||||\}}} } ||} |sRt jd|d|t jdt ||d||| | fSt|dks Jdtd|Ds Jd| ||} t| |j j |<i} |D]?}dtj|jjD| |<@| D]f}|||}|||||\}}} }|st)d }d \}} t+|D]_\}}|||}| |}d |vrt|dkr ||fd d d||D]} |j|||fi|}n#t0$r}||}t jd|d|d|t jd|d|d|d|||j jvrt7|j j|<|j j|t ||||t)d }Yd}~nd}~wwxYw||kr|}||} }a|t |||||}|| |f|j|<|j j|ddz|j j|<t jd||d| d|h|||||\}}} }||| fS)aChoose the best runner and tactic combination through performance profiling. Args: custom_op (str): The name of the custom operation to be tuned runners (List[TunableRunner]): List of candidate implementations to profile tuning_config (TuningConfig): Configuration for the tuning process inputs (List[torch.Tensor]): Input tensors for profiling **kwargs: Arbitrary keyword arguments, will be passed to get_valid_tactics and forward method of each runner Returns: Tuple[TunableRunner, int]: A tuple containing: - The selected runner implementation - The best tactic ID for that runner (-1 if using fallback) Note: The method profiles different implementations and tactics to find the optimal combination based on performance measurements. It caches results to avoid redundant profiling of the same configuration. Although runners[0] with tactic=-1 is always treated as the fallback runner. Runner authors are suggested to provide a fallback implementation for each runner to avoid potential issues. z([AutoTunner]: Using fallback tactic for z with input shapes zutoTunner]: Generated keyrzAt least one runner is requiredc8g|]}t|tSr5)rQr)rGrs r)rHz(AutoTuner.choose_one..s"BBBQJq-00BBBr+z3All Given runners must be subclass of TunableRunnerch|] }|j Sr5)name)rGparams r) z'AutoTuner.choose_one..s''''$ '''r+inf)NNrrT)rrz[Autotuner]: Skipping tactic rz", due to failure while profiling: z#[Autotuner]: Failed when profiling , shapes=z. Error occurred: Nrz&[Autotuner]: profiling chosen runner: z for ) rR_get_input_sizesrrrdebugrrrJall_generate_optimization_profilesrrinspect signaturer parametersr_prepare_input_tensorsrsfloat enumerater_profile_single_kernel Exceptionwarningrraddrrr)rLrrrrrr is_cache_hit runner_idrstored_profilerunnerrrunner_arg_names_maprptensorsrmin_timer_id valid_tacticsrunner_arg_namestac time_measurederErs r) choose_onezAutoTuner.choose_ones"<T226::;; " ">B>O>O7L-?? ;L)V^Y'F    kykk]ikk A)2J2J9V]^_V`bnp}2~2~AA6> !7||a!BBB'BBBCC  A  C77 vNN7:8}} ))4 "  A''(/(9!)(D(D(O(V(V(X(X'''  # #: : A11!V<$'+;;;M@R@RUV@V@V'L"TLLVLLL,::9,GD,G !7C--39--MM )999%)%:%:7%C%CF"N n n nC n nkl n n #L ua u u# u uX^ u urs u u ) 0QQQORuu A) L J=iHLL ) 8 8$-q!2B2B2D2Dm!"!"-2%LLMMMMMM-9.)833'4H04cvI=:>( ) 8 8!79#5q7G7G7I7I=!!I8A&!6LD(3 >BB9aPPSTTJ:9ELnASnnV\nnclnn #'"3"3 w m# # 9fay!6))sH++ L 5CL L cd|D}|S)Ncg|]D}t|tjr|ntjdES))r)rQrrsizeSize)rGinputs r)rHz.AutoTuner._get_input_sizes..sO   'uel;; QEJJLLLDAQAQ   r+r5)rLrsizess r)rzAutoTuner._get_input_sizess'       r+rrc tj}t|jD] }||fd|i|||jdkrt|jtjd}tjd}| |t|j D] }||fd|i|| ||| ||j z } | |} tjd|d|d| d | | S) aOProfile a single kernel implementation for performance measurement. Args: runner (TunableRunner): The runner implementation to profile inputs (List[torch.Tensor]): Input tensors for the kernel tactic (int): Tactic ID to use for this profiling run Returns: Average execution time in milliseconds Note: The method performs warmup runs, then measures multiple iterations to get an average execution time. Stream synchronization and delays are used to ensure accurate timing. rrT) enable_timing)streamz[Autotuner]: profiling rrz , avg_time )rrcurrent_streamrIr synchronizerrEventrecordr elapsed_timerrr) rLrrrrrrstartendavg_timerEs r)rz AutoTuner._profile_single_kernel!s$**,,t{## 4 4A F6 3 3& 3F 3 3 3 3  '! + + 5 6 6 6   t 44jT22 F ###t{## 4 4A F6 3 3& 3F 3 3 3 3 & !!!%%c**T[8&&v.. ]f ] ]v ] ] ] ]S[ ] ]   r+c ttd|Ddgt|z}g}g}|jD]L}tj|js+t |jttfs Jdt|j t|j ks4Jdt|j dt|j t|j t|j ks4Jdt|j dt|j t|j D]\}}|j ||j |<tj|jrO||j |j d|j d} n|j} ttt!| } t| dks Jdd t#| t| d dt%d fzd D} ||j |j | | fNt)jd|D} | D]} t-j|} t|D]f\}\}}} }| |}|}| |}t1t|D]-}t3|||| j ||||<.g|jD]V}|| x}x}}t3|||| j |j |j <W|| t;jd| |S)aGenerate optimization profiles for autotuning. Args: tuning_config (TuningConfig): Tuning configuration inputs (List[torch.Tensor]): List of input tensors Returns: List of OptimizationProfile objects representing different configurations Note: This method performs a cartesian product of all possible dimension combinations specified in dynamic_tensor_specs. cg|]J}t|tjrd|Dnt dgKS)c,g|]}t|Sr5)re)rGxs r)rHzHAutoTuner._generate_optimization_profiles...gs444aYq\\444r+r)rQrrrre)rGr{s r)rHz=AutoTuner._generate_optimization_profiles..esb    "!U\22(44166884444#A,,    r+Nz`The given dynamic dimension must provide a opt value generation function or a list of opt valueszHThe number of input indices and dimension indices must be the same, got z and zJThe number of tensor initializers and input indices must be the same, got rz$Empty tuning buckets are not allowedci|]\}}|| Sr5r5)rGv1v2s r) z=AutoTuner._generate_optimization_profiles..s.BBr+rrT)strictcg|] }|d S)rr5rws r)rHz=AutoTuner._generate_optimization_profiles..s'D'D'D!"'D'D'Dr+z [Autotuner]: generated profile: )rqrJrbr isfunctionr1rQlistrRr/r0r9rrErhrrziprry itertoolsproductcopydeepcopyrIrkrcr^rsrr)rLrr base_profilegenerated_profiles dynamic_dimsspeciidxrzopt_shapes_max dim_grids opt_pointrposr/r0 _opt_shapes opt_value min_value max_valueconstraint_specs r)rz)AutoTuner._generate_optimization_profilesPs}(+       FS[[    9;.0 !6" " D%d&=>> *'$CC  s   t~&&#dl*;*;;;;I[^_c_m[n[nIIuxy}zFvGvGII<;;t/00C4G4GGGGW]`aeay]z]zWWBEFJFTBUBUWWHGG$DN33 T T38<8PQR8S 055!$"9:: 5!44 'q(9:4<?KPPRR "4 vc*oo6677Jz??Q&&&(N&&&!jn 5 5u GPTN   ~zJ     %'D'D|'D'D'DE " A AI l++AJSKK  FFi.+&cN % *95 s9~~..A9C!9i::AHYq\*71:66 $1#A  4C4O4O$$&&55 I y)Y??23O4KLL  % %a ( ( ( L?A?? @ @ @ @!!r+rrEcdtd|D}|jD]X}|||jd|jd||jd|jd<Y|jD]}d||j|j<t d|DS)aFind the nearest optimization profile for given inputs User can define their own nearest profile generation method to reduce the host overhead. Args: shapes: Tuple of input tensor shapes tuning_config: Tuning configuration Return: Tuple: A tuple containing: - attributes: Tuple of runner attributes, sorted. - profile: Tuple of input tensor shapes c34K|]}t|VdSr4)r"rGrs r) z2AutoTuner._find_nearest_profile..s(<.s(<rQreryrfrkrm)rLr?r@rArFr>rErxs r)_create_tensor_likezAutoTuner._create_tensor_likes #% % %A!Y'' % ae$$$$"!Z00000 ae$$$${65&111r+rcd}g}t|jD]f\}}td|Dr+|||||j|p|}n||}||g|S)Nc`tj||dzdz |Sr=rArDs r)r6z2AutoTuner._prepare_input_tensors..s. Jvf - - - 2Q 6 "U))r+c3@K|]}t|tVdSr4)rQrkrws r)r:z3AutoTuner._prepare_input_tensors..s,88:a,,888888r+)rrEanyrCr9ry)rLrrdefault_initializerrr,rtensors r)rz AutoTuner._prepare_input_tensorssgn-- # #DAq88a88888 #111I/2I6I   NN6 " " " "r+c8|jdS)zClear the profiling cache.N)rclearrKs r) clear_cachezAutoTuner.clear_caches ""$$$$$r+c,t|_dS)zReset all statistics counters.N)rrrKs r)reset_statisticszAutoTuner.reset_statisticss(** r+)rr?r)rNN)"rUrVrWrXrr classmethodrrr rr rr rarrYrqrrrrrrrrr<rr|r rCrrLrNr5r+r)rrOs  I $ $ $ $[ ""m$"EJ' " $ " tS#22 3 """"BF*F*m$F*$ F* U\" F* }c! "F*F*F*F*PtEL'9d5:>N-#--1%,-?-IL- ----^b")b"37 3Eb" ! "b"b"b"b"HYt=5:&=7C= ===[=:      EJ'  $      [  2"\215c2IQ2 22228*484F el &%%%%++++++r+r)T)- contextlibr&rrr$r abcrr dataclassesrr functoolsrtypingrr r r r r rrrflashinfer.tllm_utilsrjit.corerrrr*r.r]rarerkr|rqr~rcontextmanagerrrrrr5r+r)rXs  ########((((((((IIIIIIIIIIIIIIIIIIII /.....   t      ) ) ) ) ) ) ) ) X 4(((       )(  4T***(6(6(6(6(6(6(6+*(6V t t J !" !!!!!!! !&   5353535353C535353p  @ @ @ @ @ @ /////// /d 4$H+H+H+H+H+H+H+H+H+H+r+