Pim4ddlmZmZmZmZddlZddlmZddl m Z m Z ddl m Z dZejo ejjduZejejejejhZej d!dejdejd efd Zejdde jdfd ejd ed e deedejf dZ ejdde jddfdejdejd ed e deed edejfdZ!d ejdejfdZ"d ejdejdejfdZ#dejdejdeedffdZ$dZ%dededefdZ&dejdeeeefdejfdZ'dejfd Z(dS)")IterableOptionalTupleUnionN)AsyncCollectiveTensor all_reduce)ScalingGranularityg-q=Famax float8_dtyperound_scales_to_power_of_2cF|tj}|tvrUtj|jtj|tz }|tj}ntd||rt|}|S)zConverts the amax value of a tensor to the fp8 scale. Args: amax: The amax value of the tensor. float8_dtype: The float8 dtype. round_scales_to_power_of_2: if true, round scaling factor down to the nearest power of 2. )minUnsupported float8_dtype: ) totorchfloat64 FP8_TYPESfinfomaxclampEPSfloat32 ValueError_round_scale_down_to_power_of_2)r r r ress o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/float8/float8_utils.py amax_to_scalers 775= ! !Dy  k,''+ek$C.H.H.HHffU]##DlDDEEE!3-c22 Jx reduce_amaxscaling_granularity axiswise_dimreturnc:|tjur'tjtj|}nM|tjus Jd| Jdtjtj||d}|rtjr|| nd}|-tttj n|}t|d|}t|tr|}|S)N unsupportedT)dimkeepdimMAX)r TENSORWISErrabsAXISWISEr distis_initialized get_grouplistrangeget_world_sizer isinstancerwait)rr device_meshr!r"r pggroups rtensor_to_amaxr78s 0;;;y1&&"&8&AAAA=AAA'''''z%)A,,L$GGG t*,,(3(?[ " " $ $ $T68jU4.0011222b$u-- d1 2 2 99;;D Kr hp_tensorcLt|||||}t|||S)a Compute scaling factor for the given high precision tensor. Args: hp_tensor: high precision tensor float8_dtype: the float8 dtype to use reduce_amax: whether to reduce the max(abs(hp_tensor)) value across distributed ranks scaling_granularity: Defines the scaling granularity axiswise_dim: if axiswise granularity is used, defines the dim to scale across round_scales_to_power_of_2: if true, round scaling factor down to the nearest power of 2. )r )r7r)r8r r r4r!r"r r s rtensor_to_scaler:UsC*    D  l7Q   rc|tvrFtj|j}|| |}||St d|)aConverts a tensor to a saturated fp8 tensor. Note: The default behavior in PyTorch for casting to `float8_e4m3fn` and `e5m2` is to not saturate. In this context, we should saturate. A common case where we want to saturate is when the history of a tensor has a maximum value of `amax1`, and the current amax value is `amax2`, where `amax1 < amax2`. This is common when using delayed scaling. )rrr)rrrrrrr)rr max_values rto_fp8_saturatedr=vsay  K --1 GG G 2 2ttL!!!DlDDEEEryctj|}tj||z }dtj||z zS)zComputes the error between two tensors in dB. For more details see: https://en.wikipedia.org/wiki/Signal-to-noise_ratio Args: x: The original tensor. y: The tensor to compare to the original tensor. )rlinalg vector_normlog10)rr>PsPns r compute_errorrFsK  ! !! $ $B  ! !!a% ( (B  BG$$ $$rtensor.c|tvrtj|j}nt d||j|j}tj||k }|dk }||fS)zCalculate FP8 tensor stats Args: tensor: The tensor to calculate stats for. float8_dtype: The float8 dtype. Returns: A tuple containing the number of zeros and the number of max values. r)dtyper) rrrrr_datar _orig_dtyper*sumitem)rGr FP8_MAXtensor_orig_typenum_maxnum_zeros rfp8_tensor_statisticsrRsy  +l++/DlDDEEE|V-?@@y)**g5::<<AACCG A%**,,1133H g rcxt|dks Jd|d|dko |ddkS)Nz%is_row_major only supports 2D tensorsr)len)strides r is_row_majorrXsB v;;!   D    !9vay 3VAY!^3rsizealignment_valuecd|dz |zz|zS)a Returns the minimum alignment value that is greater than or equal to the given size. Args: size: The size of the data to be aligned. alignment_value: The alignment value to be used. Returns: int: The minimum alignment value that is greater than or equal to the given size. Usage: ``` >>> _get_min_alignment(10, 8) 16 ``` rU)rYrZs r_get_min_alignmentr]s" $(. /? BBrdimsc8|dksJ|j\}}t|tr|f}d|vrt |dn|}d|vrt |dn|}||z }||z }t jj|d|d|fS)aP Pads a 2D tensor with zeros to ensure that its dimensions are multiples of 16, which is required `torch._scaled_mm` Args: tensor: The tensor to pad. dims: Dimensions to pad. Returns: torch.Tensor: The padded tensor. Usage: ``` >>> pad_tensor_for_matmul(torch.randn((10, 10)), dims=0).shape torch.Size([16, 10]) >>> pad_tensor_for_matmul(torch.randn((10, 10)), dims=1).shape torch.Size([10, 16]) >>> pad_tensor_for_matmul(torch.randn((10, 10)), dims=(0, 1)).shape torch.Size([16, 16]) ``` rTrrU) r&shaper2intr]rnn functionalpad)rGr^dim1dim2 dim1_aligned dim2_alignedpad_dim1pad_dim2s rpad_tensor_for_matmulrls. ::<<1    JD$$w4599%dB///$L3499%dB///$Ld"Hd"H 8  " "6AxH+E F FFrscalec|jtjks Jdtjtjtj|S)Nzscale must be float32 tensor)rIrrexp2floorlog2)rms rrrsD ;%- ' ' ')G ' ' ' :ek%*U"3"344 5 55r)F))typingrrrrrtorch.distributed distributedr,)torch.distributed._functional_collectivesrrtorchao.float8.configr rcuda is_availableversionhipIS_ROCM float8_e4m3fn float8_e5m2float8_e4m3fnuzfloat8_e5m2fnuzrno_gradTensorrIboolrr)rbr7r:r=rFrRrXr]rlrr\rrrsI433333333333 WWWWWWWW444444  * ! ! # # E (9(E       (- ,+!%0.@.K"&  |,  3-   \ 8.@.K"&',|+ ,  3- !% \@F FEKFFFF& %U\ %el %u| % % % % L(-  38_,444 CSC3C3CCCC(%G L%G %c8C=&8 9%G \%G%G%G%GP65<666666r