)`idZddlZddlmZddlmZddlmZmZm Z ddl Z ddl m Z ddl mZdd l mZdd l mZmZmZmZmZmZmZdd lmZdd lmZmZmZmZmZm Z dBdZ! dCde j"de#de#de#de j"f dZ$defdZ%defdZ&defdZ'defdZ(defdZ)defdZ*dee+de+defdZ,ej-dDd e+fd!Z.e dEd$e j"d%ee j"de#d&e/d'e/d(e/d)ee/de e j"e j"ffd*Z0e de j"de j"fd+Z1e1Z2e dFd,e j"d-e j"d.ee j"de#d/e#d'e/de j"fd0Z3e d1e j"d2e#de j"fd3Z4e dCd1e j"d2e#d4e#fd5Z5Gd6d7eZ6e e6j7d"ddfd8Z8e d9Z9e d:Z:e dGde#de j"fd?Z;e dCd@Z||zdz |z|z}|dzdzdz}||zS)Nr) total_row total_columnrow_size padded_row padded_columns o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/fp4_quantization.py _compute_swizzled_layout_sf_sizer%/s:h&*x7(BJ!A%!+a/M  %% unswizzled_sfmn sf_vec_sizereturnc|dz}|dzdz dzdz}||zdz |z|z}||z }||z |z}|dkr|dkr|Stjj|d|d|fddS)agPad scale factors tensor to meet alignment requirements. Args: unswizzled_sf (torch.Tensor): Input scale factors tensor with dtype uint8. m (int): M dimension. n (int): N dimension. sf_vec_size (int, optional): Scale factor vector size. Defaults to 16. Returns: torch.Tensor: Padded scale factors tensor. rrrrconstant)modevalue)torchnn functionalpad contiguous) r(r)r*r+factorr" padded_colpad_rowspad_colss r$_pad_scale_factorsr:5s1_Fs7Q;3&#-Jv:>f,6JA~HQ;.H1}}Qx"&& AxH5Ja'  *,, r&c,ttdS)N100)gen_fp4_quantization_modulerrr&r$!gen_fp4_quantization_sm100_moduler>R &'8% @ @@r&c,ttdS)N103)r=rrr&r$!gen_fp4_quantization_sm103_modulerBVr?r&c,ttdS)N90)r=rrr&r$ gen_fp4_quantization_sm90_modulerEZs &'7 > >>r&c,ttdS)N110)r=rrr&r$!gen_fp4_quantization_sm110_modulerH^r?r&c,ttdS)N120)r=rrr&r$!gen_fp4_quantization_sm120_modulerKbr?r&c,ttdS)N121)r=r rr&r$!gen_fp4_quantization_sm121_modulerNfr?r& nvcc_flags device_archc td|tjdz tjdz tjdz tjdz tjdz tjdz tjdz g|d d td rd nd gzd d td rd nd gtjdz tjdz dz gS)Nfp4_quantization_z-nv_internal/tensorrt_llm/thop/fp4Quantize.cppz'nv_internal/tensorrt_llm/thop/fp4Op.cppz'nv_internal/cpp/kernels/quantization.cuz#nv_internal/cpp/common/envUtils.cppz!nv_internal/cpp/common/logger.cppz&nv_internal/cpp/common/stringUtils.cppz(nv_internal/cpp/common/tllmException.cppz -DENABLE_BF16z -DENABLE_FP8z12.8z -DENABLE_FP4 nv_internalinclude)extra_cuda_cflags extra_cflagsextra_include_paths)r jit_envFLASHINFER_CSRC_DIRr)rOrPs r$r=r=js )K))  '= >  '*S S  '*S S  '*O O  '*M M  '*R R  '*T T %  6v>> FNNB    6v>> FNNB  '- 7  '- 7) C /   r&r<backendc tttttt d}||vrt d|||tdd d4d tj d ttj d td td tdtdttdttj tj fffd }td d5d tj d ttj d td td tdttj tj ff d}tdd d6dtj dtj dtdtj ffd }td d6dtj dtj dtdtj fd}tdddtj dtj ffd }tddtj dtj fd}tdd d7d tj d ttj d td tdttj tj ff fd }td d7d tj d ttj d td tdttj tj ff d!} td"d d8d tj d#tj d ttj dttj tj fffd$ } td" d8d tj d#tj d ttj dttj tj ffd%} td&dd'tj d(tj d#tj dttj tj fffd) } td&d'tj d(tj d#tj dttj tj ffd*} td+d d9d-tj d.tj d/ttj d td0td tdtj ffd1 }td+ d9d-tj d.tj d/ttj d td0td tdtj fd2}t!|||||| | 3S):N)rMrJrGrAr<rDzInvalid backend: zflashinfer::fp4_quantize_sm100rS) mutates_argsr'FTinput global_scaler+ sf_use_ue8m0is_sf_swizzled_layoutis_sf_8x4_layout enable_pdlr,c |t|j}tjg|jdd|jddzRtj|j}||jdz}|jd} |rt|| |z|rdnd} n|| z|z} tj| ftj|j} |||| ||||| || fS)a~Quantize input tensor to FP4 format. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype fp16/bf16/fp8_quantized. global_scale (torch.Tensor, optional): Global scale factor of shape [1] and dtype float32. sf_vec_size (int, optional): Scale factor vector size. Defaults to 16. sf_use_ue8m0 (bool, optional): Whether to use UE8M0 format for scale factors. Defaults to False. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. is_sf_8x4_layout (bool, optional): Whether to use 8x4 layout or 128x4 layout for scale factors. Defaults to False. enable_pdl (Optional[bool], optional): Whether to enable PDL (Programmatic Dependent Launch). If None, automatically detects based on device capability. Defaults to None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [M, K/2] with dtype FLOAT4_E2M1X2 - Scale factors tensor with shape determined by layout and sf_vec_size Ndtypedevicer) rrir1emptyshapeuint8numelr% fp4_quantize) r^r_r+r`rarbrcout_valr)k out_sf_sizeout_sfmodules r$fp4_quantize_sm100z7get_fp4_quantization_module..fp4_quantize_sm100s<  +EL99J+ 5ek#2# 5 B1 4 5 5+<    KKMMU[_ , KO  /:1 #*:%CQQKKa%;.Kk^5;u|TTT       !   r&c|j\}}|||dzgtj|||z|zgtjfS)Nrfrh)rl new_emptyr1int64int32)r^r_r+r`rar)rqs r$_fake_fp4_quantize_sm100z=get_fp4_quantization_module.._fake_fp4_quantize_sm100sW{1 OOQQKu{O ; ; OOQUk12%+O F F  r&z!flashinfer::mxfp4_dequantize_host weightscale group_sizectj|jd|jddzftj|j}|||||S)Nrrrfrg)r1rkrlfloat32rimxfp4_dequantize_host)r}r~routrts r$rz:get_fp4_quantization_module..mxfp4_dequantize_hostskk \!_fl1o1 2-=    $$         r&cz||jd|jddzgtjSNrrrfrwrxrlr1r)r}r~rs r$_fake_mxfp4_dequantize_hostz@get_fp4_quantization_module.._fake_mxfp4_dequantize_hosts>  \!_fl1o1 2%-    r&z(flashinfer::block_scale_interleave_sm100)rSr(c|dkr |jdnd}t|jd|jdd}tj||zf|j|j}|||S)aSwizzle block scale tensor for FP4 format. Args: unswizzled_sf (torch.Tensor): unswizzled block scale tensor with dtype uint8 or bfloat16. Returns: torch.Tensor: output tensor for swizzled block scale with dtype uint8 or bfloat16. rrrrerrg)dimrlr%r1rkrhriblock_scale_interleave_sm100)r( num_expertsexpert_out_sizerrts r$rzAget_fp4_quantization_module..block_scale_interleave_sm100s1>0A0A0C0Cq0H0Hm)!,,a :   #]%8%._fake_block_scale_interleave_sm100sE&&   #m&9!&< < B C5;'   r&z&flashinfer::fp4_batched_quantize_sm100c"|j\}}}tj|||dzftj|j}tj|t |||zdftj|j} ||||||||fS)uQuantize a batched tensor to FP4 (E2M1x2) with per-block scale factors. This function converts a float/bfloat16 (or FP8-quantized) input tensor into a packed FP4 tensor using the E2M1 format (two 4-bit values per byte), along with per-block scale factors. Scale factors are encoded as UE4M3 by default, or UE8M0 when requested, and an optional global scale can be applied. Args: input (torch.Tensor): Input tensor of shape [B, M, K] with dtype torch.float16, torch.bfloat16, or an FP8-quantized dtype supported by the kernel. global_scale (torch.Tensor, optional): Global scale factor of shape [1] and dtype float32. sf_vec_size (int, optional): Scale-factor vector size and alignment unit along K. Supported/expected values: - 16 (NVFP4 path; supported) - 32 (MXFP4 path; not supported yet) Defaults to 16. sf_use_ue8m0 (bool, optional): Scale-factor encoding type. False → UE4M3 (default), True → UE8M0. Returns: Tuple[torch.Tensor, torch.Tensor]: - self_fp4 (torch.Tensor): Packed FP4 tensor in E2M1x2 format of shape [B, M, K // 2] with dtype torch.uint8 (two FP4 lanes per byte). - self_block_scale_factors (torch.Tensor): Block scale factors with dtype uint8 (UE4M3 or UE8M0), laid out as a flat buffer of shape [B, ceil(M / 128) * 128 * ceil(K / sf_vec_size / 4) * 4]. Notes: - K must be even (because outputs pack two FP4 values per byte). - For best performance, K should be a multiple of sf_vec_size; the scale-factor buffer is aligned to sf_vec_size along K, pads M to multiples of 128, and rounds (K / sf_vec_size) up to a multiple of 4 for storage. - The batch dimension B is preserved for both outputs. rfrgr)rlr1rkrmrir%fp4_batched_quantize) r^r_r+r`br)rqrprsrts r$fp4_batched_quantize_sm100z?get_fp4_quantization_module..fp4_batched_quantize_sm100%sZ+1a+ 16N+<     0A4DcJJ K+<    ##          r&c |j\}}}||||dzgtj||t |||zdgtjfS)Nrfrwr)rlrxr1rmr%)r^r_r+r`rr)rqs r$ _fake_fp4_batched_quantize_sm100zEget_fp4_quantization_module.._fake_fp4_batched_quantize_sm100gso+1a OOQ16N%+O > > OO4Q[8H#NNOk     r&z.silu_and_mul_scaled_nvfp4_experts_quantize_sm100wsL{ 1f aK ;!###%N!%N%N%N###{"u%!+a/!QMc)C/Q16& LLL xek    99 KKAqAv & &   q8|^ < < JJq1uf % %       1a((%**5+>??DD x3A r1a  &--aAq!Q?? }$$r&c|j}|j\}}}|dz}d}||zdksJd|d||z} | dzdzdz} | dz} |dzd zd z} tj|||dz|tj } tj|| | |tj }|tj|| d z| dzd dd}|ddd d dd}| |fS)Nrfr'rrrrrrrrr|rr) rirlr1rkrmrzrrr)r^rr_rirr)rrqr+rrrrrrs r$6_fake_silu_and_mul_scaled_nvfp4_experts_quantize_sm100z[get_fp4_quantization_module.._fake_silu_and_mul_scaled_nvfp4_experts_quantize_sm100s3 { 1f aK ;!###%N!%N%N%N###{"u%!+a/!QMc)C/Q16& LLL xek   &**5+>??DD x3A r1a  &--aAq!Q??  &&r&z*flashinfer::scaled_fp4_grouped_quant_sm100 input_tensorinput_global_scalec|j}|j\}}}d}||zdksJd|d||z}|dzdzdz} | dz} |dzdzdz} tj|||d z|tj } tj|| | |tj } | ||z|d z| || z| |||z|||d | d d d} | tj || dz| dzd dd} | ddd dd d} | | fS)af Quantize input tensor to FP4 and return quantized tensor and scale, for grouped gemm inputs (e.g., grouped_gemm_nt_masked for flashinfer). Args: input: The input tensor to be quantized to FP4, with shape (l, m, k) l is number of groups, m is number of tokens per group, k is number of features. input_global_scale: A scalar scaling factor for the entire tensor, with shape (l,). Outputs: output: The quantized tensor in FP4, with shape (m, k // 2, l) but the physical layout is (l, m, k // 2). `// 2` is because two fp4 values are packed into an uint8. output_scales: The blockscale tensor in FP8-E4M3, with shape (32, 4, rm, 4, rk, l) but the physical layout is (l, rm, rk, 32, 4, 4). Note: For the shape of output_scales, `32 * 4 * rm` is a padded m to nearest multiple of 128. `4 * rk` is a padded `k // 16` to nearest multiple of 4. These layout constants are required by the NVIDIA Blackwell MMA operations. r'rrrrrrrrfrFrr|rr)rrrrirr)rqr+rrrrrrrts r$scaled_fp4_grouped_quant_sm100zCget_fp4_quantization_module..scaled_fp4_grouped_quant_sm100s8$$1a ;!###%N!%N%N%N###{"u%!+a/!QMc)C/Q16& LLL xek    99 KKAqAv & &   q8|^ < <   a!eQ ' '       1a((&**5+>??DD x3A r1a  &--aAq!Q?? }$$r&c|j}|j\}}}d}||zdksJd|d||z}|dzdzdz} | dz} |dzdzdz} tj|||d z|tj } tj|| | |tj } | d d d} | tj|| dz| dzd dd} | ddd d d d} | | fS)Nr'rrrrrrrrfrrr|r) rirlr1rkrmrzrrr)rrrrirr)rqr+rrrrrrs r$$_fake_scaled_fp4_grouped_quant_sm100zIget_fp4_quantization_module.._fake_scaled_fp4_grouped_quant_sm100sA $$1a ;!###%N!%N%N%N###{"u%!+a/!QMc)C/Q16& LLL xek   1a((%**5+>??DD x3A r1a  &--aAq!Q?? }$$r&z0flashinfer::e2m1_and_ufp8sf_scale_to_float_sm100r e2m1_tensorufp8_scale_tensorglobal_scale_tensor ufp8_typec Htj|jd|jddzftjd}||d||||||S)aConvert E2M1 format tensor and UFP8 scale factors to float tensor. This function performs dequantization by converting a packed FP4 tensor in E2M1 format back to float values using the associated UFP8 scale factors and global scale. Args: e2m1_tensor (torch.Tensor): Packed FP4 tensor in E2M1 format of shape [M, K/2] with dtype uint8. ufp8_scale_tensor (torch.Tensor): Scale factors tensor in UFP8 format with dtype uint8. global_scale_tensor (torch.Tensor, optional): Global scale factor of shape [1] and dtype float32. sf_vec_size (int, optional): Scale factor vector size. Defaults to 16. ufp8_type (int, optional): UFP8 scale factor type (0 for UE8M0, 1 for E4M3). Defaults to 1. is_sf_swizzled_layout (bool, optional): Whether scale factors use swizzled layout. Defaults to True. Returns: torch.Tensor: Dequantized float tensor of shape [M, K] with dtype float32. rrrfcpurgre)r1zerosrlr$e2m1_and_ufp8sf_scale_to_float_sm100rreshape)rrrr+rrarrts r$rzIget_fp4_quantization_module..e2m1_and_ufp8sf_scale_to_float_sm100/s8k  q !;#4Q#7!#; <-    33 OO    ! ! # # + +B / /  # # % %    !    r&cz||jd|jddzgtjSrr)rrrr+rras r$*_fake_e2m1_and_ufp8sf_scale_to_float_sm100zOget_fp4_quantization_module.._fake_e2m1_and_ufp8sf_scale_to_float_sm100[sB$$  q !;#4Q#7!#; rE ValueErrorbuild_and_loadrr1Tensorrintboolrrr)r[backend_modulesrur{rrrrrrrrrrrrrts @r$get_fp4_quantization_modulersm10000. Oo%%6W66777 %_W % ' ' 6 6 8 8F( 04"&*!&%)55|5u|,55 5 $ 5  5TN5 u|U\) *55555 5n67704"&*   |  u|,      $  u|U\) *    87  + |   $9::    |       ;: 2|  .@AA |    BA 0 04" <<|<u|,<< < u|U\) * <<<<< <|>??04"   |  u|,      u|U\) *    @?  F04>%>%|>%l>%u|,>% u|U\) * >%>%>%>%>% >%@TUU04''|'l'u|,' u|U\) * '''VU'448%l8%!L8%l8% u|U\) * 8%8%8%8%8% 8%tBCC%l%!L%l% u|U\) * %%%DC%4:7;&* &&\& <&&el3& &  & $ & &&&&& &PHII7;&*   \   <  &el3      $       JI   -%A-Q3#=9i'E   r&FTr^r_r`rarbrcc ,|dkr|dkrtd|ddk}|r|dd}|jd|zdksJ|t |j}t |j\}} t|| |||||||\} } | d|jd|zf} |r,| dd} | dd} | | fS)aQuantize input tensor to FP4 format. This function implements FP4 quantization that converts input tensors to a compressed FP4 format with associated scale factors. It supports various input data types and scale factor layouts. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype fp16/bf16/fp8_quantized. global_scale (torch.Tensor, optional): Global scale factor of shape [1] and dtype float32. sf_vec_size (int, optional): Scale factor vector size. Defaults to 16. sf_use_ue8m0 (bool, optional): Whether to use UE8M0 format for scale factors. Defaults to False. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. is_sf_8x4_layout (bool, optional): Whether to use 8x4 layout or 128x4 layout for scale factors. Defaults to False. enable_pdl (Optional[bool], optional): Whether to enable PDL (Programmatic Dependent Launch). If None, automatically detects based on device capability. Defaults to None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [M, K/2] with dtype FLOAT4_E2M1X2 - Scale factors tensor with shape determined by layout and sf_vec_size Raises: NotImplementedError: If any of the following features are requested but not implemented: - BFloat16 input when BFloat16 is not enabled - FP8 input when FP8 is not enabled - sf_vec_size other than 16 or 32 r'r|z sf_vec_size can only be 16 or 32rrrer) NotImplementedErrorstride transposerlrrirrrur) r^r_r+r`rarbrcis_column_majormajorminorx_qsfs r$rorots;Hb[B..!"DEEEll2&&!+O(B'' ;r?[ (A - - - -' 55 )%,77LE5)U*;E*;*;<<OO GC RRK78 9 9B"mmB## \\"b ! ! 7Nr&c|jtjks'|jtjksJd|jt |j\}}|dz|z}t ||S)aSwizzle block scale tensor for FP4 format. This function swizzles the block scale tensor to optimize memory access patterns for FP4 operations. The output needs to be padded in the m dimension to be a multiple of 128. Args: unswizzled_sf (torch.Tensor): Input tensor with dtype uint8 or bfloat16. Returns: torch.Tensor: Swizzled tensor with the same shape as input. Raises: AssertionError: If input dtype is not uint8 or bfloat16. z+Input dtype must be uint8 or bfloat16, got )rhr1rmbfloat16rrirr)r(rrrPs r$block_scale_interleavers$ u{**m.AU^.S.S.SJ]5HJJ/T.SS*-*>??LE5RZ%')K &{ 3 3 P P  r&rrrrcttjd\}}|dz|z}t|||||||S)aConvert E2M1 format tensor and UFP8 scale factors to float tensor. This function performs dequantization by converting a packed FP4 tensor in E2M1 format back to float values using the associated UFP8 scale factors and global scale. Args: e2m1_tensor (torch.Tensor): Packed FP4 tensor in E2M1 format of shape [M, K/2] with dtype uint8. ufp8_scale_tensor (torch.Tensor): Scale factors tensor in UFP8 format with dtype uint8. global_scale_tensor (torch.Tensor, optional): Global scale factor of shape [1] and dtype float32. sf_vec_size (int, optional): Scale factor vector size. Defaults to 16. ufp8_type (int, optional): UFP8 scale factor type (0 for UE8M0, 1 for E4M3). Defaults to 1. is_sf_swizzled_layout (bool, optional): Whether scale factors use swizzled layout. Defaults to True. Returns: torch.Tensor: Dequantized float tensor of shape [M, K] with dtype float32. cuda:0r)rr1rirr) rrrr+rrarrrPs r$e2m1_and_ufp8sf_scale_to_floatrsn6* XLE5RZ%')K &  **  r&repilogue_tile_mcbt||}|||jS)z; PyTorch equivalent of trtllm-gen `shuffleMatrixA` )rtori)rr row_indicess r$shuffle_matrix_ars- 3<QQK  |':;; <>@@@Kqvvxx)9)9););RtLLKE4 $;r&c@t|tj|tjdtjdg|jdddS)aj Dequantize input tensor from MXFP4 format. Parameters: a_fp4 (torch.Tensor): Quantized tensor of shape [M, K/2] with dtype uint8 (FLOAT4_E2M1X2) a_sf (torch.Tensor): Scale factors tensor with shape determined by layout and sf_vec_size (uint8) Returns: torch.Tensor: Dequantized tensor of shape [M, K] with dtype float. reg?)rir|rT)rrrr1rmrtensorri)rrs r$mxfp4_dequantizer~sw * %%   $$,,R00 cU5<000     r&r|r}r~rcttjd\}}|dz|z}t||||S)a Dequantize input tensor from MXFP4 format on host. Parameters: weight (torch.Tensor): Quantized tensor of shape [M, K/2] with dtype uint8 (FLOAT4_E2M1X2) scale (torch.Tensor): Scale factors tensor with shape determined by layout and sf_vec_size (uint8) group_size (int, optional): Group size for dequantization. Defaults to 32. Returns: torch.Tensor: Dequantized tensor of shape [M, K] with dtype float. rr)rr1rirr)r}r~rrrrPs r$rrs`$* XLE5RZ%')K &{ 3 3 I I   r&ct|j\}}|dz|z}t||||d\}}||fS)a. Quantize batched input tensor to NVFP4 format. Parameters: a (torch.Tensor): Input tensor of shape [B, M, K] with dtype fp16/bf16. a_global_sf (torch.Tensor): Global scale factor of shape [1] with dtype float32. sf_vec_size (int, optional): Scale factor vector size. Defaults to 16. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [B, M, K/2] with dtype FLOAT4_E2M1X2 - Scale factors tensor with shape determined by layout and sf_vec_size rF)rrirr)rrr+rrrPrrs r$nvfp4_batched_quantizersa&*!(33LE5RZ%')K-k::UU  KE4 $;r&ct|j\}}|dz|z}t||||\}}||fS)a. quantize batched input tensor to NVFP4 format with mask. Parameters: a (torch.Tensor): Input tensor of shape [B, M, K] with dtype fp16/bf16. a_global_sf (torch.Tensor): Global scale factor of shape [1] with dtype float32. mask (torch.Tensor): Mask tensor to apply before quantization. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [B, M, K/2] with dtype FLOAT4_E2M1X2 - Scale factors tensor with shape determined by layout and sf_vec_size r)rrirr)rrrrrrPrrs r$scaled_fp4_grouped_quantizersf"*!(33LE5RZ%')K-$$  E4 $;r&)r)r')r<rrr)>r functoolsenumrtypesrtypingrrrr1 api_loggingr jitr r rYr r rrrrr jit.cpp_extrutilsrrrrrrr%rrr:r>rBrErHrKrNstrr=cacherrrornvfp4_block_scale_interleaverrrrrrrrrrrrr&r$r s !!!!!!(((((((((( ''''''211111&&&&EG<$',/>A \:A7AAAAA7AAAA?'????A7AAAAA7AAAAA7AAAADICG> ggggggT,0"&"!%>> <>5<(>> >  >  >> 5< %&>>>>B%,5<: 637"& '''|'"%,/' '  '  ' \''''T=5<=#=%,====..,......0t " 9999x"* L < \ 8:r&