Pi:'dZddlZddlmZmZmZmZmZddlZddl m Z m Z ddl m Z ddlmZmZmZddlmZmZejZGdd eZd ed ed ed eeeffdZdejdeedffdZ d-d eded ededejdeedeeded efdZdejdeedededed ed ejfd!Zd"ejd efd#Zd"ejd efd$Zd"ejd efd%Z d"ejd efd&Z!d'ee ee e fe"e fd efd(Z#d)eee ee e fe"e fd ee e ffd*Z$d+ee e fd dfd,Z%dS).z; Defines an nn module designed to be used during inference N)List NamedTupleOptionalTupleUnion) is_row_majorpad_tensor_for_matmul)FP8Granularity)PerBlockPerRow PerTensor)is_MI300is_sm_at_least_89c>eZdZUdZdZeed<dZeed<dZeed<dS)Float8MMConfiga Configuration for the scaled_mm in the forward and backward pass. Attributes: emulate (bool): Whether to emulate the matmuls in fp32. use_fast_accum (bool): Whether to use the fast-accumulation option for scaled_mm. pad_inner_dim (bool): Whether to pad the inner dimension of a and b with 0s. This is needed for matmuls not aligned to 16. Femulateuse_fast_accum pad_inner_dimN) __name__ __module__ __qualname____doc__rbool__annotations__rrl/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/float8/inference.pyrrsNGT ND   M4rra_datab_datascaled_mm_configreturnc<|jr|d|dks6Jd|dd|dt|d}t|d}t|s|}t|r8|}||fS)aPreprocess the inner fp8 data tensors for admmm Args: a_data: Input tensor A. b_data: Input tensor B. scaled_mm_config: Configuration for _scaled_mm. Returns: Preprocessed tensors A and B in the format for _scaled_mm. rz"Inner dims must match for mm, got z and )dims)rsizer rstride contiguoust)rrr s rpreprocess_datar).s%7{{1~~Q/// VQ V VfkkRSnn V V0//'vA666&vA666   ( (%""$$FMMOO$$-&&((**,, 6>r input_scale input_shape.c|dkr|ddS|d}|dkr!|d|jd}|S)z:Ensures input tensor is correctly formatted for _scaled_mmr#)numelreshape unsqueezedimshape)r*r+s rpreprocess_scaler4Hsa""1a(((''++K1!))"k.?.CDD rFa_scaleb_scale output_dtype output_scalebiasrc |tjkr"| tj|||||||}||zStj||||||||S)z This is the unwrapped version of addmm_float8, which does not take in Float8TrainingTensors as inputs. This is used to standardize the logic between subclassed and non subclassed versions of the linear module. N)scale_ascale_b scale_result out_dtyper)r;r<r9r=r>r)torchfloat32 _scaled_mm) rr5rr6r7r8r9routputs r addmm_float8_unwrapped_inferencerCZs u}$$)9!  %")   }   !%    rscale data_shaper2startendstepctjj}jkr|j||||St fdttD}|t|krS||}|dkr|j||||S|||znd} | ||zdz |znd} |dkrtd|j|| | dS)z Slice the scale tensor appropriately based on the data tensor slicing. This function calculates how the scale should be sliced when the data tensor is sliced along a given dimension, taking into account the block structure. c3DK|]}|j|zVdS)N)r3).0irErDs r z-_slice_scale_for_dimension..s2XXA 1 Q7XXXXXXrr#Nz;Slicing with step > 1 is not implemented for scale tensors.) r?opsatenr3sliceTensortuplerangelenNotImplementedError) rDrEr2rFrGrHrO block_sizesblock_size_for_dim scale_start scale_ends `` r_slice_scale_for_dimensionrZs> 9>D {j  z  UC>>>XXXXXs:AWAWXXXXXK c+ $S)Qz  UC>>>6;5Fe111D % % ).@ @ @  !88%M z  [)QGGGrxct|ds Jdt|jd|dz z|jdfzkS)~Checks if a quantized tensor is rowwise scaled Args: x: quantized tensor (should have `block_size` attribute) block_size.Expecting input to have `block_size` attribute)r#r#r-)hasattrrRr^r2r3r[s r_is_rowwise_scaledrbsV 1l # #UU%UUU #   $!%%''A+"6!'"+"G GGrctds JdtfdtjDS)r]r^r_c3rK|]1}j|dkpj|j|kV2dS)r-N)r^r3)rKrLr[s rrMz(_is_tensorwise_scaled..sUCD Q2>aAGAJ!>r)r`allrSndimras`r_is_tensorwise_scaledrgsd 1l # #UU%UUU # HMaf   rct|ds Jd|j}t|dko+tj|dddko |ddkS)zChecks if a quantized tensor is scaled with a block size of 1x128 Args: x: quantized tensor (should have `block_size` attribute) r^r_r.Nr-r#)r`r^rTmathprodr[bs r_is_1_128_scaledrnsc 1l # #UU%UUU # A q66Q; B49QssV,,1 BaeslBrct|ds Jd|j}t|dko|ddko |ddkS)zChecks if a quantized tensor is scaled with a block size of 128x128 Args: x: quantized tensor (should have `block_size` attribute) r^r_r.rrir#)r`r^rTrls r_is_128_128_scaledrpsT 1l # #UU%UUU # A q66Q; 61Q43; 61Q43;6rgct|dko5|dtddgko|dtddgkS)Nr.rr#ri)rTr )rqs r!_granularity_is_a_1_128_w_128_128rssI q66Q; V1Q48QH#5#55 V!A$(CQT:BVBV:VVr granularitycd}|ttf}nDt|ttfr||f}n"t|ttfrt |dkrt|dtot|dt}t|dtot|dt}t |}|s|s|std|dt|dt|dstd|dt|}ntd|d|S)Nr.rr#zUnsupported granularity types: .zEDifferent granularities for activation and weight are not supported: z#Invalid granularity specification: ) r isinstancer rRlistrTrs ValueErrortype)rtprocessed_granularity is_per_tensor is_per_rowis_a_1_128_w_128_128s r_normalize_granularityrs!!*ikk : K)V!4 5 5O!,k : K% / /OC 4D4D4I4I";q>9== * NIC C  A77 J NF= =  AMM O O/C OM{MMMNN N+a.${1~*>*>?? fXcfff !&k 2 2M{MMMNNN  r granularitiesct|dtot|dt}t|dtot|dt}t|}|s|r*t st s JddSdS|rt s JddSt d|d)a9 Validate that the hardware supports the requested granularities. Args: granularities: Tuple of (activation_granularity, weight_granularity) Raises: AssertionError: If hardware doesn't support the requested granularity ValueError: If invalid granularity type is provided rr#uNFloat8 dynamic quantization requires CUDA compute capability ≥8.9 or MI300+.u[Float8 1x128 activation and 128x128 weight scaling requires CUDA compute capability ≥8.9.zInvalid granularities rvN)rwr r rsrrry)rr|r}r~s r_check_hardware_supportr s}Q/;; a)AAMM!,f55*a&;;J=]KK D D "" hjj  \  0     D ""  i  "  B-BBBCCCr)NNF)&rrjtypingrrrrrr?torchao.float8.float8_utilsrr torchao.float8.typesr torchao.quantization.granularityr r r torchao.utilsrrrQrr)intr4dtyperrCrZrbrgrnrprxrsrrrrrrs  ;;;;;;;;;;;;;; KKKKKKKK//////        Z      % 66> 4%,U38_0&*! %% % % % % + % 6" % 6 %% %%%%P0H <0HS 0H 0H  0H 0H  0H \0H0H0H0HfH%,H4HHHHU\dC CCCCC7%,747777W  nn,- ^ W  WWWW !  ..0 1   "  ! >> )* ! ! ! !FD78D DDDDDDr