)`i>ddlZddlmZddlmZmZddlZddlmZddl m Z ddl m Z m Z mZdd Zejd Ze dd ejdededeedeejejff dZe dd ejdejdedejfdZdS)N)SimpleNamespace)OptionalTuple)flashinfer_api)#gen_mxfp8_quantization_sm100_module)device_support_pdlregister_custom_opregister_fake_opc>||zdz |z|z}|dzdzdz}||zS)Nr) total_row total_columnrow_size padded_row padded_columns o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/fp8_quantization.py _compute_swizzled_layout_sf_sizers:h&*x7(BJ!A%!+a/M  %%cttdd ddtjdt dt d tt d ttjtjff fd }td ddtjdt dt d ttjtjffd }td d ddtjdtjdt d tjffd }td  ddtjdtjdt d tjfd}t||S)Nz flashinfer::mxfp8_quantize_sm100) mutates_argsT inputis_sf_swizzled_layout alignment enable_pdlreturncf|jjdkrtj|jtj|j}|r+t |jd|jddzd}n|dz}tj|ftj|j} ||||||fS|t|j}||jdz}|jd}||zdz |z|z} tjg|jdd| Rtj |j}|rt || dzd}n|| zdz}tj|ftj|j} ||||||||fS) aKQuantize input tensor to MxFP8 format. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype fp16/bf16/fp8_quantized. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. alignment (int, optional): sfVecSize. Defaults to 32. Note that alignment is not used in the host kernel. enable_pdl (Optional[bool], optional): Whether to enable PDL (Programmatic Dependent Launch). If None, automatically detects based on device capability. Defaults to None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [M, K] with dtype FLOAT8_E4M3 - Scale factors tensor with shape determined by layout and sf_vec_size cpudtypedevicerrrr N) r&typetorchemptyshapeuint8rnumelmxfp8_quantize_hostr float8_e4m3fnmxfp8_quantize) rrrr out_val out_sf_sizeout_sfmkpadded_kmodules rmxfp8_quantize_sm100zAget_mxfp8_quantization_sm100_module..mxfp8_quantize_sm100s. <  % %k%+U[VVVG$ 2>KNEKNb$8# $kkmmr1 [+u{5<XXXF  & &%     F? "!/ ==  R0A BAI )i7)CHk-%+crc"-H--)|G % 1>q(b.RUVV (lb0 [+u{5<XXXF  ! !%    F? "rc|j\}}|||gtj|||zdzgtjfS)Nr%r)r+ new_emptyr)int64int32)rrrr4r5s r_fake_mxfp8_quantize_sm100zGget_mxfp8_quantization_sm100_module.._fake_mxfp8_quantize_sm100[sQ {1 OOQF%+O 6 6 OOQUb[MO = =  rz'flashinfer::mxfp8_dequantize_host_sm100)r scale_tensorctj|jtj|j}|||||S)aDequantize input tensor from MxFP8 format. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype FLOAT8_E4M3. scale_tensor (torch.Tensor): Scale factors tensor with shape determined by layout and sf_vec_size. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. Returns: torch.Tensor: Dequantized float tensor of shape [M, K] with dtype float32. r$)r)r*r+float32r&mxfp8_dequantize_host)rr?routr7s rmxfp8_dequantize_host_sm100zHget_mxfp8_quantization_sm100_module..mxfp8_dequantize_host_sm100gsL&k%+U]5<PPP$$    !     rct||jd|jdgtjS)Nrrr:)r;r+r)rArr?rs r!_fake_mxfp8_dequantize_host_sm100zNget_mxfp8_quantization_sm100_module.._fake_mxfp8_dequantize_host_sm100s-  A A?u}UUUr)r8rDTrN)TrT) rbuild_and_loadr r)Tensorboolintrrr r)r8r>rDrGr7s @r#get_mxfp8_quantization_sm100_modulerNs 0 2 2 A A C CF* '+%) ;#;#|;##;#;#TN ;# u|U\) * ;#;#;#;#;# ;#z899'+   |  #     u|U\) *    :9  1'+|l $   0?@@'+VV|VlV $V  VVVA@V 1$?   rTrrrrr r!cd}|jd|zdksJ|t|j}t||||\}}||fS)aQuantize input tensor to MxFP8 format. This function implements MxFP8 quantization that converts input tensors to a compressed MxFP8 format with associated scale factors. It supports various input data types and scale factor layouts. Args: input (torch.Tensor): Input tensor of shape [M, K] with dtype fp16/bf16/fp8_quantized. is_sf_swizzled_layout (bool, optional): Whether to use swizzled layout for scale factors. Defaults to True. alignment (int, optional): sfVecSize. Defaults to 32. enable_pdl (Optional[bool], optional): Whether to enable PDL (Programmatic Dependent Launch). If None, automatically detects based on device capability. Defaults to None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing: - Quantized tensor of shape [M, K] with dtype FLOAT8_E4M3 - Scale factors tensor with shape determined by layout and sf_vec_size rr'r)r+r r&rNr8)rrrr sf_vec_sizex_qsfs rr0r0sp.K ;r?[ (A - - - -' 55 133HH  GC 7Nrr?cHt|||S)avDequantize input tensor from MxFP8 format. This function performs dequantization by converting a packed FP8 tensor in MxFP8 format back to float values using the associated scale factors. Args: input (torch.Tensor): Packed FP8 tensor in MxFP8 format of shape [M, K] with dtype FLOAT8_E4M3. scale_tensor (torch.Tensor): Scale factors tensor with shape determined by layout and sf_vec_size. is_sf_swizzled_layout (bool, optional): Whether scale factors use swizzled layout. Defaults to True. Returns: torch.Tensor: Dequantized float tensor of shape [M, K] with dtype float32. )rNrDrFs rrBrBs+* / 0 0 L L   r)r rHrI) functoolstypesrtypingrrr) api_loggingrjit.fp8_quantizationrutilsr r r rcacherNrKrLrMr0rBrrrr[s!!!!!!"""""""" ''''''EEEEEE&&&&  xxxv#'!% !! <!!! !  5< %& !!!!H#' <,  \ r