)`iE|dZddlZddlZ ddlZeedredkr edddlTddlTddl Tn#e$rddl Tddl Tddl TYnwxYw ddl mZmZmZn#e$r ddl mZmZmZYnwxYweeZd eDZd Zdd Zdd ZdZdZdZdZdS)zN Utilities for selecting CUTLASS library kernels based on problem description NCUTLASS_IGNORE_PACKAGETz+Disabling attempt to import cutlass_library)*)get_valid_schedules)generate_data_types_from_math_instructionfix_alignmentsci|]\}}|| Sr .0kvs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/data/cutlass/python/cutlass_library/heuristics.py rBs 4 4 4daQ 4 4 4cf|}|D]}|D]P\}}t|trt|||<+t|t rt |||<Q|d}|D]g}|D]P\}}t|trt|||<+t|t rt |||<Qht|d5}tj ||dddddS#1swxYwYdS)a  Utilitiy function to write heuristics results to a json file for debug args: problems_with_configs: List of problems provided to the heuristic, with a list of operations added to each problem dict outfile_path: Outfile path returns: None configsw)indentN) copyitems isinstanceDataType DataTypeNames LayoutTypeShortLayoutTypeNamesopenjsondump) problems_with_configs outfile_pathpc_copypr rrcfs r$serialize_heuristics_results_to_jsonr'Ds " & & ( ('  ) )a ''1 Ax 'Q! a $ $'#A&! lG ))''))))$!Q a " " )q!!A$$ : & & )%a(!A$ ))  L#$!Igq####$$$$$$$$$$$$$$$$$$sD&&D*-D*Fc b| t} | |||||||||| |  S)a Get heuristic-suggested GEMM kernel configurations for a single GEMM problem. args: m, n, k: GEMM dimensions batch_count: batch count layouts: tuple of layouts of type LayoutType use_fast_acc: Use fast accumulation for FP8. Ignored for other precisions count: Number of configs to return provider: Heuristics provider to use returns: A list of dictionaries containing the suggested kernel configurations and additional info from the input required to define a Cutlass GemmOperation, with the following keys: - 'cta_tile_m', 'cta_tile_m', 'cta_tile_k': CTA tile size - 'instr_tile_m', 'instr_tile_n', 'instr_tile_k': Instruction tile size - 'stages': kernel pipeline stage count - 'cluster_m', 'cluster_n', 'cluster_k': cluster size - 'layout_a', 'layout_b': input tensor layouts of type LayoutType - 'alignment_a', 'alignment_b': input tensor alignments, in count of elements - 'dtype_a', 'dtype_b', 'dtype_acc': dtypes of a, b, and accumulator, of type DataType - 'swizzle_size' : suggested threadblock swizzle - 'split_k_slices': number of partitions of the k dimension for splitK - 'raster_order': raster order for CTAs over output tiles ('along_m' or 'along_n') N)voidC use_fast_acccount)MatmulHeuristics get_configs) mnr batch_countlayoutsdtypes alignment_a alignment_br)r*r+providers rget_single_gemm_configr6`sU2!!H   aA{FG[R]ejzFNS  T TTrcg}|D]}|} |d}|d}|d}|d}|d} |d} |d} n3#t$r&} td| d |d } ~ wwxYw|d d } |d d}|dd}|dd }|dd}|dd}|dd}| t t jkrtd| t| dkrtd| Dstd| td| D} | | | || n| | g}td|D}n0#t$r#}td|d }~wwxYw|dd t|d!z}|d"d t|dz}t|||||||||dk||| }||d#<|||S)$a Get heuristic-suggested GEMM kernel configurations for a set of GEMM problems. args: problems: List of dictionaries describing GEMM problems with the following keys: - 'm', 'n', 'k': Matrix dimensions (required) - 'dtype_a': Data type of matrix A (required) - 'dtype_b': Data type of matrix B (required) - 'dtype_c': Data type of matrix C (default: None) - 'dtype_d': Data type of matrix D (required) - 'dtype_acc': Compute data type (default 'f32') - 'layout': Operation layout (e.g. 'tnt') - 'alignment_a': Memory access granularity of A, in units of elements (default: 16 bytes equivalent elements) - 'alignment_b': Memory access granularity of B, in units of elements (default: 16 bytes equivalent elements) - 'alpha': Scalar multiplier for A*B (default: 1.0) - 'beta': Scalar multiplier for C (default: 0.0) - 'batch_count': Number of GEMM operations in batch (default: 1) - 'use_fast_acc': Enable fast accumulation for FP8 on Hopper (default: True) provider: Heuristics provider to use count: Number of configurations to return per problem (defualt: 1) returns: A copy of the input dictionary, with key `configs` added containing the selected gemm configs r.r/r dtype_adtype_bdtype_dlayoutzMissing required parameter z for problem N operationgemmr0r dtype_accf32dtype_calphag?betagr*TzUnsupported operation c3K|]}|dvV dS)ntNr )r r%s r z#get_gemm_configs..s&$?$?1Q$Y$?$?$?$?$?$?rzDlayout must be a 3-character string containing only 'n' or 't', got c3RK|]"}|dkr tjn tjV#dS)tN)rRowMajor ColumnMajor)r ls rrFz#get_gemm_configs..s6``TU188J''9O``````rc30K|]}t|VdS)N) dtype_map)r dts rrFz#get_gemm_configs..s&88rYr]888888rzUnsupported data type: r3rr4r)rKeyError_LOGGERerrorgetOperationKindNames OperationKindGemm ValueErrorlenalltuplelower DataTypeSizer6append)problemsr5r+retproblemr.r/r r8r9r:r;er<r0r>r@rArBr*r1 dtype_listr2rNr3r4rs rget_gemm_configsrc}sI2 #**gllnnG  #,a #,a #,a "g "g "gx ff     mmK!KK'KKLLL   K00I++mQ//K K//Ikk)T**G KK % %E ;;vs # #D;;~t44L&}'9::: ; ;; < << KK1  $?$?$?$?$?!?!?  f^dff g gg``Y_`````G MMOOW]]__ioo6G6G\c\ou|vCvCvEvEGNGTGTGVGVWj88Z88888ff     mm2b22333  ++mSL4K-KLLK++mSL4K-KLLK$Q1k7FKYdfjlofoq}@EGOPPG GIJJw *s08A B !BB(BH99 I&I!!I&c d}d}|t}g}g}|D]>}|d|dg|d|dg|dd t|d zgf}|d |d |d |d|d f\} } } } } |ddzdk}|r d|dzn|d|d|ddzg}t|| | | tjt j}|j|j|dr tj n|j | |j |j dg}|d|rdndz|d|df}t|d|dz|d|dz|ddz|dzgdgd||||d|d|df}g}|r,| tjt jgn+| tjt jgt)||g|g||t*jt*jgt0jD],}| || |-@||fS)+ Generate CUTLASS operations based on the list of configs provided by the heuristic provider args: manifest: manifest argument to which to add operations, or None to just return the operations without a manifest (for pruning an existing manifest) cuda_version: Cuda compiler version for generating cutlass operations kernel_configs: list of configs generated by the heuristic returns: (configs, operations): a list of heuristic-provided kernel configs along with a one-to-one corresponding list of the generated operations deNlayout_ar3layout_br4layout_drOr:r8r9r>r@ cluster_mrr cta_tile_m cta_tile_n cta_tile_kr))a_typeb_typec_typed_typeacc_typeepi_typer cluster_n cluster_k)rorr cluster_shape)tile_schedulers gemm_kind)Manifestr\MathInstruction OpcodeClassTensorOp MathOperation multiply_add element_a element_brvoidelement_accumulatorTileDescriptionr]KernelScheduleTypeTmaWarpSpecialized2SmSm100EpilogueScheduleTypeTmaWarpSpecialized2SmTmaWarpSpecialized1SmSm100TmaWarpSpecialized1SmCreateGemmUniversal3xOperatorTileSchedulerTypeDefaultStreamKGemmKind Universal3x)manifest cuda_versionkernel_configsmin_ccmax_ccr operationsconfigr;rrr element_c element_dis_2sminstruction_shapemath_instruction data_typestile_multipliertile_description schedulesos r&generate_sm100_from_heuristics_configsrso & & zzH '*..fj!6-#89F:r@r:)rrr)element_source element_dest)alignment_bitsrlrmrnrrkrvrwrxi(#r*)rr is_alignedrinstantiation_levelr;r{enable_fp8_fast_acc)r{)rz)r|r\rrJr}r~rrrrrrrrrrXrr]rr)rrrrrrrrrr;rrrrrdummy_instr_shaperrdummy_warp_countrrstream_k_schedulesrs r%generate_sm90_from_heuristics_configsrs3.&& zzH '*44f',vi7H*IISPWW]^kWlo{}CDM}NpOXOSVXVJj!6-#89F:.os&CCT%4-CCCCCCr;@)r5r+c3K|]}d|vV dS)90Nr rs rrFz.vs& B BD44< B B B B B Brrc3&K|] }d|vpd|vV dS)r101Nr rs rrFz.xs/ X X$ETM -u} X X X X X Xrc&i|]\}}|dk ||S)rr r s rrz=filter_manifest_and_write_heuristics_file..|s#RRR11 >>q!>>>rcJg|]\}}d|i| S)operation_name)procedural_name)r r%rproblem_without_configss r z=filter_manifest_and_write_heuristics_file..}sPWWWeiefhi*A,=,=,?,?`CZ`^_`WWWr^$z!No valid configurations generated)rheuristics_problems_filerloadheuristics_gpur,any architecturessplit set_cta_div_nrcheuristics_configs_per_problemrheuristics_restrict_kernelsrrrzipadd_kernel_filterr Exceptionwrite_profiler_testlist_to_csvheuristics_testlist_file)rargsheuristics_problemsr&rmmhr!all_configs_and_operationsrr`problem_configsproblem_operationswith_problem_sizer<rs @r)filter_manifest_and_write_heuristics_filer\s+ D )3//'1)A,,'''''''''''''''$..$2E2K2KRVRe#S!!!#CCT%7%=%=c%B%BCCCCCb*+>TXTwxxx!*& 4 4g B BD$6$<$#>AAABBBB #9 7 8 88 !;T=Z[[[ ##s :>>c |}|D]g}|D]P\}}t|trt|||<+t|t rt |||<Qht|dd5}|d}tj ||}| | |ddddS#1swxYwYdS)a( Write a list of configs to a testlist to be consumed by cutlass_profiler args: configs_list: List of kernel configs along with runtime arguments and any other columns to include in the CSV, expressed as a list of dictionaries outfile_path: Outfile path returns: None rr)modenewliner) fieldnamesN) rrrrrrrrkeyscsv DictWriter writeheader writerows) configs_listr"profiler_testlistr%r rofilek_nameswriters rrrsQ#'')) ''a ''1 Ax 'Q! a $ $'#A&! '  LsB///(5"''))G ^Eg 6 6 6F  &''' ((((((((((((((((((sAC88C<?C<)FTrN)Nr)__doc__rrbuiltinshasattrr ImportErrorcutlass_library.librarycutlass_library.generator#cutlass_library.heuristics_providerlibrary generatorheuristics_provider sm90_utilsrrrlogging getLogger__name__rQrrrMr'r6rcrrrrr rrrs(B  $/// WX/00E5Kt5S5S +C D DD''''))))33333$$$######$     H % % 4 4m1133 4 4 4 $$$8TTTT:G G G G TEEEPKKKZ)$)$)$V(((((s!-:A A AA/.A/