Pix!UddlZddlmZmZddlmZmZmZmZm Z m Z ddl Z ddl m Z mZmZddlmZmZgdZGddeZGd d eZGd d eZe jeege je je je jhZ e jd e jde j de j!diZ"ee e j#efe e$e$ffe%d<ej&dej'dej(dej)dej*dej+dej,de jde jde j de j!di Z-ee e j#efe e$e$ffe%d<iZ.ee e j#efe e$e$ffe%d<ej&dej'dej(d ej)d!ej*d"ej+d#ej,d$iZ/ee e j#efe e$e$ffe%d%<e j0d&e j1d'e j2d(e j3d)e j4d*e j5d+e j6d,iZ.e-7e j0de j1de j2de j3de j4de j5de j6die/7e j8de j9de j:d e j;d!e j<d"e j=d#e j>d$ie-7e j8de j9de j:de j;de j<de j=de j>die"7e.e"7e/e-?e"?ksJd-Z@d.eAdDZBe jCDd/d0ZEeeEZFGd1d2e jGjHZIGd3d4e jGjHZJd5ZKd6ZLe jM dd7e jNd8e e$d9fd:e jNd;ee jNdee e$eOfd?e jNfd@ZPeF dd7e jNd8ee$d:e jNd;ee jNdee e$eOeQfd?e jNfdAZRd7e jNd8ee$d:e jNd;ee jNd=e e$eOfd>e e$eOfd?e jNfdBZS dd7e jNd8ee$d:e jNd;ee jNdee e$eOeQfd?e jNfdCZT dd7e jNd8e e$d9fd:e jNd;ee jNd=ee e$eOfd>ee e$eOfd?e jNfdDZU dd7e jNd8ee$d:e jNd;ee jNdee e$eOeQfd?e jNfdEZV dd7e jNd8e e$d9fd:e jNd;ee jNd=ee e$eOfd>ee e$eOfd?e jNfdFZW de jXdGd7e jNd8e e$d9fd:e jNd;ee jNdHe j#d=ee e$eOfd>ee e$eOfdee e$eOeQfde e$eOfde e$eOfdee e$eOfde e$eOfdee e$eOfdee e$eOfdQed?e jNfdRZaddej`fd7e jNd8e e$d9fd:e jNd;ee jNdPe j#d=ee e$eOfd>ee e$eOfdQed?e e jNe jNffdSZbddej`fd7e jNd8e e$d9fd:e jNd;ee jNdPe j#d=ee e$eOfd>ee e$eOfdQed?e e jNe jNffdTZce jMdddde j!dUfd7e jNdVed8e e$dWe j#d=ee e$eOfd>ee e$eOfdXeeOdYee j#dZee j#d[eQd?e e jNe jNffd\Zde jM dd7e jNdVed8e e$dWe j#d=ee e$eOfd>ee e$eOfdXeeOdYee j#dZee j#d?e e jNe jNffd]Ze dd7e jNdVed8e e$dWe j#d=ee e$eOeQfd>ee e$eOeQfdXeeOdYee j#dZee j#d?e e jNe jNffd^Zfdddddd_ej`fd`e jNdae jNdVed8e e$d9fdWe j#d=ee$d>ee$dXeeOdYee j#dZee j#dbeQdQed?e e jNe jNffdcZgeF dd7ee jNdVehd8ee$dWe j#d=ee e$eOeQfd>ee e$eOeQfdXeeOdYee j#dZee j#d[eQd?e e jNe jNffddZidee jNdfe$dge$d?e e jNe jNe jNffdhZjd7ee jNd8ee$dWe j#d?e e jNe jNe jNe jNffdiZkd7e jNd8ee$dWe j#dje jNdke jNdle jNdme jNd?e jNfdnZl dd7e jNd8ee$dWe j#dje jNdke jNdle jNdme jNdras,,,awqzz,,,r6torchaoFRAGMENTczeZdZdZedejdejfdZedejdejfdZdS)_RoundzF Implementation of generic round operation with backward STE. xreturnc*tj|SN)torchround)ctxrfs r7forwardz_Round.forwards{1~~r6gyc|Srir5rlrns r7backwardz_Round.backwards r6N) r.r/r0r1 staticmethodrjTensorrmrqr5r6r7reres \%,5<\r6receZdZdZedejdejdejfdZedejdejfdZ dS) _RoundToFloat8zH Implementation of `tensor.to(float8_dtype)` with backward STE. rf float8_dtypergc,||Sri)to)rlrfrvs r7rmz_RoundToFloat8.forwardsttL!!!r6rnc |dfSrir5rps r7rqz_RoundToFloat8.backwards 4xr6N) r.r/r0r1rrrjrsdtypermrqr5r6r7rurus" "EK"EL"""\"%,5<\r6ruc@|tvr3tj|jtj|j}}n+|t vrt d|t |\}}||}||}||ksJd|d|||ksJd|d|||fS)aGet quant_min and quant_max args based on dtype and also verify bounds. Args: dtype: Target quantization dtype (e.g., torch.uint8, torch.int8, or FP8 types) quant_min: Minimum quantized value, or None to use dtype default quant_max: Maximum quantized value, or None to use dtype default Returns: Tuple[int/float, int/float]: Validated (quant_min, quant_max) values Raises: ValueError: If dtype is unsupported AssertionError: If quant_min/quant_max are out of bounds for dtype zUnsupported dtype: Nz9quant_min out of bound for dtype, quant_min_lower_bound: z quant_min: z9quant_max out of bound for dtype, quant_max_upper_bound: z quant_max: ) FP8_TYPESrjfinfominmaxrG ValueError)rz quant_min quant_maxquant_min_lower_boundquant_max_upper_bounds r7_get_and_check_qmin_qmaxrs  K   " K   " 5 - - -6u667777Nu7U44) ) - - - - Q"7 Q QEN Q Q . - - - - - - Q"7 Q QEN Q Q . - - i r6c ~t|t|ksJg}g}d}tt|D]}||||kr||dkr||||zdks"Jd|d||d|d|||||||z|||||dz|dz }|||||dkr|||dz }||fS)aGiven block_size and input size find the parameters for reduction: Output: shape_for_reduction: the shape we use to `view` input to prepare it for reduction reduction_dims: the dims we'll do reduction over Example:: Input: block_size: (3, 3, 2, 10) input_size: (3, 3, 10, 10) Output: shape_for_reduction: (3, 3, 5, 2, 10) reduction_dim: [0, 1, 3, 4] rrHzExpecting input size at z dimension: z" to be divisible by block_size at rI)lenrangeappend) block_size input_sizeshape_for_reductionreduction_dimscur_dimr`s r7_get_reduction_paramsrs z??c*oo - - - -NG 3z?? # # a=JqM ) )jma.?.?a=:a=0A555J1JJ*Q-JJklJJ{EFG{HJJ655 & &z!} 1 'E F F F  & &z!} 5 5 5  ! !'A+ . . . qLGG & &z!} 5 5 5!}!!%%g... qLGG  ..r6inputr.scale zero_point output_dtyperrrgc ,t|||||||S)ag Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization output_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype Note: How can block_size represent different granularities? let's say we have a Tensor of size: (3, 3, 10, 10), here is the table showing how block_size represents different granularities: granularity type | block_size per_tensor | (3, 3, 10, 10) per_axis (axis=0) | (1, 3, 10, 10) per_axis (axis=1) | (3, 1, 10, 10) per_group (groupsize=2) | (3, 3, 10, 2) per_group (groupsize=2) for axis = 3 | (3, 3, 2, 10) Output: quantized tensor with requested dtype )_quantize_affinerrrrrrrs r7rrAs,J     r6ct|||\}}|tvr tj}t |||||||S)a_Quantize tensor using affine quantization with integer zero point domain. Op definition that has compatible signatures with custom op library. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) output_dtype: Target quantized dtype (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None Returns: Quantized tensor with requested dtype Note: zero_point_domain is pre-defined as INT, meaning: quantized_val = (float_val / scale) (integer) + zero_point (integer) )rrSrjuint8_quantize_affine_no_dtype_castrxrs r7rrqsb<4L)YWWIy,,,{ )      br6c|jtjtjtjfvsJd|jt ||ks"Jd|d|t||\}}|j }| |}|} |D]} d| | <| | }|.| dkr| | }nd}tj t|d|z z|z||} | |} | S)aQuantize tensor using affine quantization without dtype casting. Performs quantization with integer zero point domain without casting to target dtype. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_min: Minimum quantized value quant_max: Maximum quantized value Returns: Quantized tensor without dtype casting The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale and zero_point with zero_point_domain = INT 3. Reshape the quantized result to original shape Unsupported input dtype: Got input dim:, block_size: rHNr?rzrjfloat32float16bfloat16rdimrsizeshapeviewnumelclampreapply rrrrrrrroriginal_shapeshape_after_reductionr`quants r7rrs> ;      15;00     z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +E/ %%#$a  JJ, - -E*"2"2"4"4q"8"8__%:;;  K UcEk*++j8)Y  E JJ~ & &E Lr6ct|||\}}|tvr tj}t |||||||S)aQuantize tensor using affine quantization with float zero point domain for tinygemm. Specialized quantization for tinygemm int4mm kernel where zero point is in floating point domain. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) output_dtype: Target quantized dtype (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None Returns: Quantized tensor with requested dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale and zero_point with zero_point_domain = FLOAT 3. Reshape the quantized result to original shape Note: zero_point_domain is pre-defined as FLOAT, meaning: quantized_val = (float_val - (zero_point (float) - scale * mid_point)) / scale )rrSrjr'_quantize_affine_tinygemm_no_dtype_castrxrs r7r!r!scF4L)YWWIy,,,{ 2      br6c|jtjtjtjfvsJd|jt ||ks"Jd|d|t||\}}|j }| |}|} |D]} d| | <| | }|.| dkr| | }nd}||zdzdz } ||| zz } tj t|| z |z ||} | |} | S)aQuantize tensor using affine quantization with float zero point domain without dtype casting. Specialized quantization for tinygemm int4mm kernel where zero point is in floating point domain. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_min: Minimum quantized value quant_max: Maximum quantized value Returns: Quantized tensor without dtype casting The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale and zero_point with zero_point_domain = FLOAT 3. Reshape the quantized result to original shape rrrrHNrrIr)rrrrrrrrrrr` mid_pointmin_valrs r7rrs> ;      15;00     z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +E/ %%#$a  JJ, - -E*"2"2"4"4q"8"8__%:;;  Y&*a/I59,,G K ego%>??I V VE JJ~ & &E Lr6ct|||\}}|tvr tj}t |||||||S)a+Quantize tensor using affine quantization without zero point. Specialized quantization for cases where zero point is not needed (e.g., floatx quantization). Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (ignored, should be None) output_dtype: Target quantized dtype (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None Returns: Quantized tensor with requested dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale with zero_point_domain = NONE 3. Reshape the quantized result to original shape Note: zero_point_domain is pre-defined as NONE, meaning: quantized_val = (float_val / scale) | This is primarily used for floatx quantization where we do not want to round values to nearest integer and instead scale and cast. )rrSrjr,_quantize_affine_no_zero_point_no_dtype_castrxrs r7r r PscH4L)YWWIy,,,{ 7      br6c|jtjtjtjfvsJd|jt ||ks"Jd|d|t||\}}|j }| |}|} |D]} d| | <| | }|.| dkr| | }nd}tj t|d|z z||} | |} | S)aQuantize tensor using affine quantization without zero point and without dtype casting. Specialized quantization for cases where zero point is not needed without casting to target dtype. Args: input: Input tensor to quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (ignored, should be None) quant_min: Minimum quantized value quant_max: Maximum quantized value Returns: Quantized tensor without dtype casting The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Quantize the input based on the quantization parameters scale with zero_point_domain = NONE 3. Reshape the quantized result to original shape rrrrHNrrrrs r7rrsx> ;      15;00     z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +E/ %%#$a  JJ, - -E*"2"2"4"4q"8"8__%:;;  K UcEk%:;;Y R RE JJ~ & &E Lr6r input_dtypec 0t||||||||S)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 Default value for zero_point is in integer domain, zero point is added to the quantized integer value during quantization Output: dequantized Tensor, with requested dtype or fp32 r)_dequantize_affinerrrrrrrrs r7rrs38   !    r6c  |tvr |j|ksJd|d|j|tjtjtjfvs Jd|t |||\}}t|||||||S)aDequantize tensor using affine dequantization with integer zero point domain. Op definition that has compatible signatures with custom op library. Args: input: Quantized tensor to dequantize block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) input_dtype: Expected dtype of input tensor (e.g., torch.uint8, torch.int8) quant_min: Minimum quantized value for input tensor quant_max: Maximum quantized value for input tensor output_dtype: Target output dtype (default: torch.float32) Returns: Dequantized tensor with requested output dtype Expected: , got: Unsupported output dtype: )rSrzrjrrrr!_dequantize_affine_no_dtype_checkrs r7rrs:///{k))) : : :U[ : :*))       3L22     4KIVVIy ,    r6c>t||ks"Jd|d|t||\}}|j} ||}|} |D]} d| | <|| }||| }||d} || ||z } | |z} | | |S)aDequantize tensor using affine dequantization without dtype checking. Converts quantized tensors to their high precision floating point representation. Args: input: Quantized tensor to dequantize block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_min: Minimum quantized value for input tensor quant_max: Maximum quantized value for input tensor output_dtype: Target output dtype (default: torch.float32) Returns: Dequantized tensor with requested output dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Dequantize the input based on the quantization parameters scale and zero_point 3. Reshape the quantized result to original shape and change dtype to the output_dtype rrrHNT)copyrrrrrrrx rrrrrrrrrrrr`dequants r7rrs/> z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +E/ %%#$a  JJ, - -E__%:;; hh|$h//GJMM,777oG << ' ' * *< 8 88r6ct||ks"Jd|d|t||\}}|j} ||}|} |D]} d| | <|| }| Jd||} | |z} | | |S)a9Dequantize tensor using affine dequantization without zero point and without dtype checking. Converts quantized tensors to their high precision floating point representation without zero point. Args: input: Quantized tensor to dequantize block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (ignored, should be None) quant_min: Minimum quantized value for input tensor quant_max: Maximum quantized value for input tensor output_dtype: Target output dtype (default: torch.float32) Returns: Dequantized tensor with requested output dtype The op does the following: 1. Figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. Dequantize the input based on the quantization parameters scale (no zero point) 3. Reshape the quantized result to original shape and change dtype to the output_dtype rrrHNz>zero_point should be None for _dequantize_affine_no_zero_pointrrs r7/_dequantize_affine_no_zero_point_no_dtype_checkrTs> z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +E/ %%#$a  JJ, - -E   H   hh|$$GoG << ' ' * *< 8 88r6c  |tvr |j|ksJd|d|j|tjtjtjfvs Jd|t |||\}}t|||||||S)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization, no zero point is used for this op input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 Default value for zero_point is in integer domain, zero point is added to the quantized integer value during quantization Output: dequantized Tensor, with requested dtype or fp32 rrr)rSrzrjrrrrrrs r7r%r%s:///{k))) : : :U[ : :*))       3L22     4KIVVIy :    r6c4t||ks"Jd|d|t||\}}|j} ||}|} |D]} d| | <|| }||| }||zdzdz } || z } | |} | |z} || |z } | | |S)aThis function converts AQT tensors to their high precision floating point representation The op does the following: 1. figure out the dimension for reduction based on block_size, also reshape the input to align with the shape after reduction 2. dequantize the input based on the quantization parameters scale and zero_point and args like zero_point_domain 3. reshape the quantized result to origianl shape and change dtype to the output_dtype rrrHNrIr)rrrrrrrrrrrr`rrs r7*_dequantize_affine_tinygemm_no_dtype_checkrs>" z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +E/ %%#$a  JJ, - -E__%:;; Y&*a/IiGjj&&G uG: << ' ' * *< 8 88r6c  |tvr |j|ksJd|d|j|tjtjtjfvs Jd|t |||\}}t|||||||S)a Args: input (torch.Tensor): quantized tensor, should match the dtype `dtype` argument block_size: (List[int]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (Tensor): quantization parameter for affine quantization zero_point (Tensor): quantization parameter for affine quantization input_dtype (torch.dtype): requested dtype (e.g. torch.uint8) for output Tensor quant_min (Optional[int]): minimum quantized value for input Tensor quant_max (Optional[int]): maximum quantized value for input Tensor output_dtype (torch.dtype): dtype for output Tensor, default is fp32 Default value for zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) Output: dequantized Tensor, with requested dtype or fp32 rrr)rSrzrjrrrrrrs r7r&r&s:///{k))) : : :U[ : :*))       3L22     4KIVVIy 5    r6 quant_dtypezero_point_domainc |td|tjur|tdt||||||||\}} | S)a General fake quantize op for quantization-aware training (QAT). This is equivalent to calling `quantize_affine` + `dequantize_affine` but without the dtype casts. Args: input (torch.Tensor): original float32, float16 or bfloat16 Tensor block_size: (Tuple[int, ...]): granularity of quantization, this means the size of the tensor elements that's sharing the same qparam e.g. when size is the same as the input tensor dimension, we are using per tensor quantization scale (float): quantization parameter for affine quantization zero_point (int): quantization parameter for affine quantization quant_dtype (torch.dtype): desired quantized dtype for determining and validating quant_min and quant_max values. quant_min (Optional[int]): minimum quantized value for output Tensor, if not specified, it will be derived from dtype quant_max (Optional[int]): maximum quantized value for output Tensor, if not specified, it will be derived from dtype zero_point_domain (ZeroPointDomain): the domain that zero_point is in, should be either integer or float if zero_point is in integer domain, zero point is added to the quantized integer value during quantization if zero_point is in floating point domain, zero point is subtracted from the floating point (unquantized) value during quantization default is ZeroPointDomain.INT N/Please use ZeroPointDomain.NONE instead of None8zero_point should be None when zero_point_domain is NONE)rrr;_do_fake_quantize_affine) rrrrrrrr_fqs r7r+r+sm> JKKK o2 2 2z7MSTTT&    GQ Ir6c |td||tdt||||||||\}} tj||k||k} | | fS)ah General fake quantize op for quantization-aware training (QAT). This is equivalent to calling `quantize_affine` + `dequantize_affine` but without the dtype casts. Note: Compared to :func:`~torchao.quantization.quant_primitives._fake_quantize_affine`, this consumes more memory and returns an additional outlier mask for intermediate quantized values. Args: Same as :func:`~torchao.quantization.quant_primitives._fake_quantize_affine`. Returns: A 2-tuple of ( final fake quantized values, outlier mask for intermediate quantized values ) Nrr)rrrj logical_and) rrrrrrrrqdqmasks r7r,r,Js: JKKK  "z'=STTT&    GQ  a9nY @ @D :r6c b|j}t|||\}}|tjkrt} t } nP|tjkrt} t} n1|tj krt} t} ntd|| ||||||} | | ||||||} | | fS)a*Helper function for fake quantization that returns both intermediate and final values. Performs quantization followed by dequantization without dtype casting, returning both the intermediate quantized values and the final dequantized values. Args: input: Input tensor to fake quantize (float32, float16, or bfloat16) block_size: Granularity of quantization - size of tensor elements sharing same qparam scale: Quantization scale parameter zero_point: Quantization zero point parameter (optional) quant_dtype: Target quantized dtype for determining quant_min/quant_max quant_min: Minimum quantized value, derived from dtype if None quant_max: Maximum quantized value, derived from dtype if None zero_point_domain: Domain of zero point (INT, FLOAT, or NONE) Returns: Tuple of (intermediate quantized values, final dequantized values) Helper function for `_fake_quantize_affine` that returns both the intermediate quantized values and the final dequantized values. z Unrecognized zero point domain: r) rzrrr9rrr:rrr;rrr) rrrrrrrrrrrrrs r7rrys>+K3KIVVIyO///9> o3 3 3BG o2 2 2GLO>>JZI>>J ; 88+el8 ; ;Z]]>K>> r6Trr preserve_zeroc | tdt|||\}}|tjtjtjfvs Jd||| Jd|j|jks Jd||j}|tj|jj }|j } | rOtj |tj |} tj |tj |}n|} |}|tjks|tjkr|tjkr/tj | |}|t||z dz z }nR|tjksJ| t|z }|t|z }||k}tj|||}| std| t jkrtd| t jkrd}n+tj|t)||zd zdz }tj|| }n|tjksJ|| z tjt||z || z }tj|| }| t jkrd}n| t jkrE|t0| |z z }tj|||}| tj} n-| t jks Jd ||zd zdz }| ||zz}||| }|||j |fS)alA variant of :func:`~torchao.quantization.quant_primitives.choose_qparams_affine` operator that pass in min_val and max_val directly instead of deriving these from a single input. This is used for observers in static quantization where min_val and max_val may be obtained through tracking all the data in calibration data set. Args: Mostly same as :func:`~torchao.quantization.quant_primitives.choose_qparams_affine`. with one difference: instead of passing in `input` Tensor and use that to calculate min_val/max_val and then scale/zero_point, we pass in min_val/max_val directly Nrrz@Need to provide `min_val` and `max_val`, got: {min_val, max_val}z]Expecting `min_val` and `max_val` to have the same dtype, got: {min_val.dtype, max_val.dtype}rIzBpreserve_zero == False is not supported for symmetric quantizationzbzero_point_domain should be ZeroPointDomain.INT or ZeroPointDomain.NONE for symmetric quantizationrHrrzGzero_point must be in FLOAT/INT/None domain for asymmetric quantizationr)rrrr2r3r4rzrjr}rrr~ zeros_likerrwhererr:r; full_likeintrtensorr9rerrrx)rrrrrrrrrrrr scale_devicerrrsminsmaxrrrs r7rrts0 JKKK3L)YWWIy -   3L22      7#6#6J$7#6 6 =GM ) ) )g * ) )m  {k'-((,>Li)9')B)BCC i)9')B)BCC     --- ;@ @ @ ;0 0 0)[L+>>K5Y)>#?#?!#CDEE;#HHHHHy!1!11Dy!1!11D$;DKdD11E T   5 5 5t   4 4 4JJY5JQ5NRS4S0T0TUUJ Es+++{55555{*el )i' ( ( L/ / /   Es+++  4 4 4JJ /"5 5 5"V\\+2E%F%FFJZIFFJ'#(; $(====Y>==#Y.2a7I%uy'88J]])9]:: 88+gn8 = =z IIr6c |t|||\}}|tjjtjjtjjfvs Jd|||j}|tj|jj }t|| ks"Jd| d|t|| \} } || }tj|| | } tj|| | } tj| tj| }tj| tj| }|tjjks|tjjkr|tjjkr/tj| |}|t)||z dz z }nW|tjjksJ|t)|z }|t)|z }||k}tj|||}tj|t/||zdzdz }tj||}n|tjjksJ||z t)||z z }tj||}|t2||z z }tj|||}| tj}|||j || fS) aoop definition that has compatible signatures with custom op library The op does the following: 1. figure out the dimension for reduction based on block_size 2. find min_val/max_val based on the dimension for reduction 3. calculate quantization parameters based on min_val/max_val based on args like `preserve_zero` and `zero_point_domain` rNrrrrIrHrrr)rrr2rr3r4rzrjr}rrrrrrrrr~rrrrrrrrerrrxr)rrrrrrrrrrrrrrrrrrrrrs r7rrs)*4L)YWWIy "-2#   3L22     k  {k%+&&* z??eiikk ) ) )@@@J@@ * ) )+@EJJLL++' JJ* + +EjNGDDDGjNGDDDG)GU%5g%>%>??K)GU%5g%>%>??K  -222 ;@E E E ;05 5 5)[L+>>K5Y)>#?#?!#CDEE;#H#MMMMMy!1!11Dy!1!11D$;DKdD11E_UCY1F1Ja0O,P,PQQ  Es+++{5:::::{*eI 4I.J.JJ Es+++kE.A!B!BB [Y BB  #${  88+el8 ; ;Z]]>K>> r6wnum_bits group_sizecv |dks Jd||j\ |dd fvs Jd||j}|dkr }| kr|d|f}d|zdz }|dzdz}tjtj|dd }|d|z z}t ||z }||z }tj |d |}||z |z} fd } | |}| |}tjtj|dd } | d z} || z d d tj } | | z}| dd tj} | d| z tj }nd|dz zdz }tjtj|dd } | |z} t || z }tj || |}| | z}tjgtj |}| dd|z zz} |  d tj} ||| |fS)NrKUnsupported num_bits = rTUnsupported groupsize = rIrHT)rrcZ|f}|Srireshape contiguousrsize_ksize_ns r7 reshape_wz:_choose_qparams_and_quantize_affine_qqq..reshape_w^* 66*++6688AHr6g_@rErFrrrO)rrrrjrabsrerrrhalfrkrxint8rrr)rrr orig_device max_q_val half_q_vals_groupq_ww_refr  s_channelt_int8rr s @@r7rr?sA q===>H>>===WNFF "c6* * * *,Sz,S,S * * *(KR F IIr:& ' 'xK!O !m) *UYq\\2t<<<1y= ll1w;''++-- zk#q),,z!''))G3      inn %  Juy//TBBB U )#**,,224==@@LL )%%b!,,//ek/BB ??62..99;;iGKK*L  (Q,'!+ Juy||R>>> Y ll1y=))--//k# z955 Y&,rKHHHQ1x<(( %%fb11<<>>AA%+NN E ))r6c|j}t||\}}||}t j||d}t j||d}d}d} |t|| z dz z } ||z t|| z z } |} t|dzdksJdt|dzf} t| | \}}| |} | |} | } |D]}d| |<t jt j | |d}t jt j | |d}d}d}|t||z z }|t||z z }|| }|| }t j | |z ||}t j | |z ||}| || || || |fS) a There are two sets of qparams: quantized_block_scale, quantized_block_min and super_block_scale_scale and super_block_min_scale the relationship is the following: block_scale = quantized_block_scale * super_block_sclae block_min = quantized_block_min * super_block_min quantized_val = (float_val - block_min) / block_scale + quant_min first we calculate block_scale and block_min then we calculate super_block_scale_scale and super_block_min_scale after that we can calculate quantized_block_scale and quantized_min_scale the returned values are: super_block_scale_scale, super_block_min_scale, quantized_block_scale and quantized_min_scale FrrXrrIrTrHrZ) rzrrrrjrrr _GGUF_QK_Krr rrx)rrrrzrrrrrr block_scale block_minsuper_block_sizerr`block_scale_absmaxblock_min_absmaxqparam_quant_maxqparam_quant_minsuper_block_scale_scalesuper_block_min_scalesuper_block_scale_scale_viewsuper_block_min_scale_viewquantized_block_scalequantized_block_mins r7rrs" KE+@EJJLL++' JJ* + +EjNEBBBGjNEBBBGIIU9y#899A=>KW$i).C(D(DDKI  2 &! + + + +:B78*?+**,,++'""#677K233I/4466 %%#$a   +NEz ).% 05++44-u++00$;#?#?@U#V#V !6!;!; .2244-8867033166-.22-6::02660*,AAK%(;;I""#EFFKABBI ![0H}}^,,H Or6c<t||\}} |} | D]} d| | <|j} ||}|| }|| }dt |dzdf} t| |\}} |}| D]} d|| <||}||}||}||}||z}||z}|| }|| }||z|z}|| }|||}|Sr')rrrrrrrx)rrrr r!r$r%rr)rr*r`rrr+r,rrrs r7r*r*s1FEJJLL11-~*C)G)G)I)I& 2201*1--[N JJ0 1 1E166*.223UVV:B7; .2244-8867033166-.22-6::02660*,AAK%(;;I""#EFFKABBIk!I-Gll>**G**\** Nr6rrc6 |dks Jd||j\ |dd fvs Jd||dkr }| kr~|d|f}d|zdz }|dzdz}||z}||z |ddz} fd} | |}n"|dd |z zz}||z}| |tj}n||}|S) NrKrrTrrrIrHcZ|f}|Srirrs r7r z)_dequantize_affine_qqq..reshape_wnr r6rO)rrr rxrjr) rrrrrrrrw_dqr rr s @@r7r(r(Use q===>H>>===WNFF "c6* * * *,Sz,S,S * * *R F IIr:& ' 'xK!O !m) INN,,,J$$&&Q)?)??      yq8|!45 vvxx)#wwu}%%ww|$$ Kr6rfbetalp_normc |dkrQtj|tjjtj|d|z z zStj|tjjtj|d|z tjtj||dz zz zS)NrHr)rjsignnn functionalrelur pow)rfr2r3s r7 _shrink_lp_opr:s!||z!}}ux277 ! sTz8QRRRRz!}}ux277 IaLLC$J%)EIaLL'A+*N*NN N    r6gffffff?g$@g)\(?)r3r2kappaiters early_stoprzeromin_maxaxisrzrverbose opt_paramsc |d|d|d|d|df\} } } } } ||jntj|}|#|jdkr tjn tj}|||}|||}|||}d}t | D]}tj||z|z|d |d }||z |z }t||z | | }tj |||z |zz |d }| | z} ttj ||z  }|r3td t|d zzdt|z| r ||kr|}n||j}||j}~~~~tjtj||z|z|d |d }|||fS)Nr3r2r<r=r>cudarg@rrHTrArzIter z | Error: )rrjtyperrrxrrkrr:meanrr printstrrE empty_cache)rrr?r@rArzrrBrCr3r2r<r=r>W_f best_errorr`W_qW_rW_e current_errors r7 optimize_weights_proximal_legacyrRsR& 9677< /+GT5% &~V]]EL4H4HF }"(+"7"7 em ))%) / /C HH5H 0 0E 77v7 . .DJ 5\\k#+,--33GAJ KKTzU"C#ItW55z#se 33$MMM  eic 227799::  K 'CAJJ& s=7I7I(I J J J  z))*    HHV] # #E 776= ! !D S#s J +funt+ , , 2 271:wqz J JC t r6val1val2cXt|tj||z z|kSri)rmathceil)rSrTs r7 _is_divisiblerXs( tdit ,,, - - 55r6rNnbitsrcd}d|zdz }||zdzdz }||z |z|j}|} ||} | | |fS)NrrIrH)rrxrzr) rNrr?rYrrrrzero_aoscale_aoW_q_aos r7"_convert_to_affinequantized_formatr^szI51 IY&*a/IDJJLL(EKKMM9==djIIGH XXe__F 8W $$r6@rEoptimize compute_dtype raw_outputoptimize_weightsc |dvs Jd|Tt||s2Jdt|jzdzt|z||t j} | j} |4|dkr| d|gn| |dg} | |d d } | |d d } td |zdz }d }||g}|| | z z d }| |z}|dvrt |}|r| | ||||||\}}}nj||}||}t | |z|z |d |d}d|z }|durt||||| \}}}n|| }|dkr9|| d d}|| d d}n8|d| d}|d| d}|t j|}|||}|||}~ ~ ~ t j|||| fS)aChoose quantization parameters and quantize tensor using HQQ (Half-Quadratic Quantization). Performs quantization using HQQ method with optional weight optimization via proximal solver. Args: tensor: Input tensor to quantize (float32, float16, or bfloat16) nbits: Number of bits for quantization (default: 4) group_size: Size of quantization groups (default: 64) optimize: Whether to optimize weights using proximal solver (default: True) axis: Axis along which to perform quantization (0 or 1, default: 1) compute_dtype: Target compute dtype (default: torch.float16) device: Target device for computation (default: "cuda") verbose: Whether to print optimization error information (default: False) raw_output: If True, return params in HQQ library format (default: False) optimize_weights: Weight optimization function (default: optimize_weights_proximal_legacy) Returns: Tuple of (quantized_weights, scale, zero_point, original_shape) Note: Uses proximal solver to minimize ||W - dequantize(quantize(W))||_p^p for weight optimization. r\zaxis should be either 0 or 1NzEgroup_size should be divisble by the total tensor dimensions. shape: , group_size: rrzrHrTTrFrrIg@)r)rK)rrr?r@rArrBrFr)rXrrJrrxrjrrr~rrkrrerr^rrErK)rrYrr`rArarrBrbrcWr_min_maxmax_vmin_vr@rr?rNs r7rrs.D 6>>>9>>>V\\^^Z88  S&,  *oo   8  u} 55A GE,0AIIAIIr:& ' ' 'AIIzSUFV)r= stochasticearly_stop_tol hp_tensorqminqmaxr=rlrmcF|jdks Jdt|ttfrt |dks Jd|ddkr |ddks Jd||ks Jdt j}t j|j}|j \} } t|d} | | zdksJd| d | d t j d t j fd } d t j d t j fd } |r| n| }| | }| | z}|| || }tt!|t!|pd}|d|z |}|}t)td|D] }|||dz ||}||zdt j}||zdt j}t j|dk||z |}||}||z ||z }||krn|}|||dz ||}|| |  t j}|j}| |}||fS)a Half-Quadratic Quantization (scale-only, symmetric) for 2D weights with row-wise blocks. - hp_tensor: [out, in] (bf16/fp16/fp32 accepted; promoted to fp32 internally) - block_size: must be [1, group_size]; groups along the last dim - qmin, qmax: integer range (e.g., -8, 7 for signed 4-bit) Returns: qdata: int32, same shape as hp_tensor scale: hp_tensor.dtype, shape [out, in // group_size] (one scale per row-wise block) rIzhp_tensor must be 2D [out, in]z)block_size must be a 2-element list/tuplerrHz7block_size must be [1, group_size] with group_size >= 1zqmin must be < qmaxz in_features=z! must be divisible by group_size=rfrgc*|Sri)rkrfs r7 round_detz>_choose_qparams_and_quantize_scale_only_hqq..round_det}swwyyr6cTtj|tj|zSri)rjfloor rand_likerss r7 round_stochz@_choose_qparams_and_quantize_scale_only_hqq..round_stochs!{1uq111222r6rrT)rrz)ndim isinstancelisttuplerrjrr}rrrrsrxrrrr r clamp_mincloner unsqueezersumrrrz)rnrrorpr=rlrmra compute_epsnkrrtrx_rrgn_groupsWgqabsr prev_scalerQgnumdenrelqdata out_dtypes r7rrUs* >Q    @    j4- 0 0S__5I5I5I36J5I I a=A  *Q-1"4"4"4A#5"4 4 $;;;-;;;MM+m,,0K ?DAqZ]##J z>Q   GqGG:GG   U\el3u|3 3333# 1 B  ]##..00AJH 8Z ( (B s4yy#d)) $ $ )D VVXX]]q] ! !D ( 3 3K @ @EJ3q%== ! !RU__R((( ) ) / /d ; ;Bwmmm77Bwmmm77 C!GS3Y ;;   #%%  "''))J,@,@,M,MMRRTT    E  B$$ $ % % + +D$ 7 7B GGAqMM $ $ & & ) )%+ 6 6EI HHY  E %<r6rWniterc|7t||sJd|jd|||}|j}|d|}t |d|d}t|d }| } tj |jd|j | } tj |jd|j | } t|D]} | d|z } tj| d } | | dz } | | z} | d|z }tj|d }| |dz } | |z} | dd t%|z d }t(| |z ||}||tj}|d| z|dd|}|d|z}| |d|d|}|||fS)a SINQ: Sinkhorn-Normalized Quantization (https://www.arxiv.org/abs/2509.22944) Iteratively normalizes row and column standard deviations to minimize matrix imbalance before quantization with dual scales. Args: tensor: Input weight tensor group_size: Quantization group size (default: 64) niter: Number of Sinkhorn iterations (default: 20) compute_dtype: Target compute dtype (default: torch.float16) Returns: Tuple of (qdata, scale_row, scale_col) Nz/group_size must divide tensor elements. shape: rerrTrryrHg:0yE>rfrTr)rXrrrxrr~stditemrrrjonesrrrrr rrr~rerrrrrepeat)rrorprrrargrq_minW_hatscale_col_sinkhornscale_row_sinkhornrq_colq_rowscale_sQr scale_row num_groups scale_cols r7rrs.V\\^^Z88  ffl f fZd f f  8  &&A GE "j!!A !   ""''))155Q5<<+;+;+=+=+B+B+D+D E EE t  E GGIIEAGAJqx}UUUAGAJqx}UUU 5\\ 8 8 a   5( Et,,,***/%7 a   5( Et,,,***/%7yy{{At44uT{{BMMdSSG UW_%%++D$77A FF5MM $ $ & & ) )%* 5 5E b  . .44U1XrBBEEmTTqZ'J"))*55jajADD]SSI )Y &&r6ebitsmbitscRt|dz }dt||z zt|dzd|zz z}|j}|}|dd|z }||S)a3Choose quantization parameters for floatx quantization. Calculates scale parameter for quantizing to custom floating point format. Args: tensor: Input tensor to quantize (float32, float16, or bfloat16) ebits: Number of exponent bits in target floatx format mbits: Number of mantissa bits in target floatx format Returns: Scale tensor for floatx quantization Note: Uses global lookup table as workaround for torch.compile() compatibility since _n_ones() is not compatible due to << operator. rHrIg-q=r) _ONES_TABLErzrr rrrx)rrrexp_bias max_normalrzrs r7rrs2519%H{5)H45EAI!U(+J LE \\^^F JJLL  a & &5 & 1 1J >E 88E??r6c~|}t||ddz ||}|S)aQuantizes the float32 high precision floating point tensor to low precision floating point number and converts the result to unpacked floating point format with the format of 00SEEEMM (for fp6_e3m2) where S means sign bit, e means exponent bit and m means mantissa bit rTrH)rr r)rrrr tensor_floatxs r7r"r"! s; \\^^F+FUZZA5F5F,FuUUM r6ct|||}||ddz}||}|S)NrTrHr)r rrrx)rrrrrs r7r'r', sO%VUE : :F ekkmm((Q// /F YY\Y * *F Mr6rv hp_value_lb hp_value_ubcHtj|j}tdkrG|}||tj|||}||z }nt |j\} } || } |  | d}||tj|||}||z }fdt|jD} | | }|tj urV|tj us Jdtjttj|}|tj S) a Calculates float8 scaling factor for the given high precision tensor. Args: tensor (torch.Tensor): Input tensor to be quantized. float8_dtype (torch.dtype): Data type of the quantized tensor (e.g., torch.float8_e4m3fn, torch.float8_e5m2). scale_dtype (torch.dtype): Data type of the scaling factor (e.g., torch.float32). block_size (Optional[Tuple[int, ...]]): Block size for block-wise quantization. If None, tensorwise quantization is used. hp_value_lb (Optional[float]): the lower bound for high precision floating point value for calculating scale hp_value_ub (Optional[float]): the upper bound for high precision floating point value for calculating scale rNr~rTrc,g|]\}}||zSr5r5)r_r`rrs r7raz(_choose_scale_float8.._ s3   ,9AzJ*Q- '   r6z!Only float8_e8m0fnuz is supportedr)rjr}rrr rrrrr enumeraterrfloat8_e8m0fnuexp2rerlog2rx) rrrvrrrrmax_absrrrtensor_reshaped output_shapes ` r7rr9 s( L))-I :!**,,""$$  "k&=k'{ LLLG)#.C  / / +^!++&9::!%%'',,,NN  "k&=k'{ LLLG)#    =Fv|=T=T    l++%-''e22224W222 6<< 5(9(9::;; 88%-8 ( ((r6 target_shapec jkrSdkrStdtjDrSt jt kr4t dt jdt t fdtt D}ttj|D]1\}\}}}|||zkrt d|d|d|d ||z d 2}t|D]"\}}|dkr| || }#|S) a Expand a scale tensor to match the target tensor shape for block-wise quantization. If this is rowwise quantization, however, just return the scale as is. Args: scale (torch.Tensor): Scale tensor with shape corresponding to block structure target_shape (torch.Size): Target tensor shape to expand to Returns: torch.Tensor: Scale tensor expanded to match target_shape rHc34K|]\}}||kp|dkVdS)rHNr5)r_abs r7 z6_maybe_expand_scale_to_tensor_shape.. s3 G G116 Q!V G G G G G Gr6zScale tensor has z dimensions but target has c3DK|]}|j|zVdSri)r)r_r`rrs r7rz6_maybe_expand_scale_to_tensor_shape.. sC./ Q5;q>)r6z Dimension z: target size z' is not evenly divisible by scale size z (block size would be )ry) rrallziprrr}rrrepeat_interleave)rr block_sizesr` target_dim scale_dimrexpanded_scales`` r7#_maybe_expand_scale_to_tensor_shaperk s {l""  {{}}  G GEK(F(F G G GGG  5;3|,,,, `EK 0 0 ` `SQ]M^M^ ` `   38\9J9J3K3KK 3< L%+{3333. .J : Z/ / /\Q\\j\\!*\\BLyBX\\\  0N";//QQ : >>+==ja=PPN r6c |tj}t||j}||z }tj|j}|| |}t ||S)p Quantizes the high precision floating point tensor to a float8 tensor, using the given scaling factor. r) rxrjrrrr}rrrur)rrrv tensor_fp32scale_expanded tensor_scaled max_valuetensor_clampeds r7r#r# sv))EM**K9 MMN.0M L))-I"((iZY(GGN    = ==r6c|tj}t||j}||z}||S)A Dequantizes the float8 tensor to high precision tensor. )rxrjrrr)rrr fp8_tensorrrns r7r)r) sF5=))J9 MMN^+I << % %%r6c&t|||S)rrrrv)r#rs r7&_quantize_affine_float8_non_decomposedr s$ #!   r6%quantize_affine_float8_non_decomposedc.tj||SNrrj empty_likers r7_quantize_affine_float8_metar   F, 7 7 77r6c&t|||S)rrrr)r)rs r7(_dequantize_affine_float8_non_decomposedr s$ %!   r6'dequantize_affine_float8_non_decomposedc.tj||Srrrs r7_dequantize_affine_float8_metar rr6)NN)NNNNN)NNNNNFri)rVenumrrtypingrrrrr r rj!torchao.prototype.custom_fp_utilsr r r torchao.utilsrr__all__rrr serializationadd_safe_globals float8_e4m3fn float8_e5m2float8_e4m3fnuzfloat8_e5m2fnuzr|rrint16rrGrzr__annotations__r=r>r?r@rArBrCrRrSr[uint1uint2uint3r(uint5uint6uint7updateint1int2int3int4int5int6int7keysrrrlibraryLibrary quant_libregister_custom_opautogradFunctionrerurrno_gradrsrrboolrrr!rr rrrrrrr%rr&r9r+r,rrrrrrJrrrr$r*r(r:inference_moder|dictr}rRrXSizer^rrrrrr"r'rrr#r)rrrrr5r6r7rs ????????????????      B$0     d        4    $$k?%CDDD         K J  K& K& TeEK$=>c3hOPqqqqqqq K J K K PT% \ 9:E#s(OKL   RTtE%+|";> L> <>+> \ >>>>,!& && L& <&+& \ &&&&"Y&&!& 3   L  < +  \   '& 9EFF!& 388 L8 <8+8 \ 888GF8Y&&!&    L  < +  \   '& 9GHH!& 88 L8 <8+8 \ 888IH888r6