Pi&`ddlZddlmZddlZddlmZejddZedededed ed ed ed ed ededededededej j j gedej j j gededededededdZ dZ ejdZ dd ed!ed"ed#ed$ed%ed&efd'Ze d( dd ed!ed"ed#ed$ed%ed&efd)Z dd-ed.ed/ed0eed1ed2ed3eed4ed5ed6ed7ed8ed9ed:ed;edZe d? dd-ed.ed/ed0eed1ed2ed3eed4ed5ed6ed7ed8ed9ed:ed;ed TensorzMunpack_tensor_core_tiled_layout(Tensor packed_w, int inner_k_tiles) -> Tensorzzdequantize_tensor_core_tiled_layout(Tensor packed_w, Tensor scales_and_zeros, int group_size, int inner_k_tiles) -> Tensorzmarlin_24_gemm(Tensor x, Tensor weight_marlin, Tensor meta, Tensor s, Tensor workspace, int bits, int size_m, int size_n, int size_k) -> Tensorzmarlin_qqq_gemm(Tensor x, Tensor weight_marlin, Tensor s_tok, Tensor s_ch, Tensor s_group, Tensor workspace, int size_m, int size_n, int size_k) -> Tensorzrowwise_scaled_linear_cutlass_s8s4(Tensor input, Tensor input_scale, Tensor weight, Tensor weight_scale, Tensor? bias=None, ScalarType? out_dtype=None) -> Tensorzrowwise_scaled_linear_cutlass_s4s4(Tensor input, Tensor input_scale, Tensor weight, Tensor weight_scale, Tensor? bias=None, ScalarType? out_dtype=None) -> Tensorzrowwise_scaled_linear_sparse_cutlass_f8f8(Tensor input, Tensor input_scale, Tensor weight, Tensor weight_meta, Tensor weight_scale, Tensor? bias=None, ScalarType? out_dtype=None) -> TensorzLto_sparse_semi_structured_cutlass_sm9x_f8(Tensor weight) -> (Tensor, Tensor)zsparse24_sm90_sparsify(Tensor input, str metadata_fmt, str activation, str sp_selection_algo, *, ScalarType? dtype = None, Tensor? scale=None) -> (Tensor, Tensor)zsparse24_fp8_sm90_cutlass_gemm(Tensor a, Tensor a_mdata, Tensor b, *, Tensor? a_scale = None, Tensor? b_scale = None, int swizzle_size=8, str swizzle_axis='n', int sm_count=128) -> Tensorz\swizzle_mm(Tensor mat1, Tensor mat2, bool mat1_is_swizzled, bool mat2_is_swizzled) -> Tensorzswizzle_scaled_mm(Tensor mat1, Tensor mat2, bool mat1_is_swizzled, bool mat2_is_swizzled, Tensor scale_a, Tensor scale_b, Tensor? bias=None, Tensor? scale_result=None, ScalarType? out_dtype=None) -> TensorzImx_fp8_bf16(Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> Tensor)tagszImx_fp4_bf16(Tensor a, Tensor b, Tensor a_scale, Tensor b_scale) -> Tensora6qscaled_dot_product(Tensor query, Tensor key, Tensor value, Tensor? attn_mask=None, float dropout_p=0.0, bool is_causal=False, float? scale=None, float q_scale=1.0, int q_zp=0, float k_scale=1.0, int k_zp=0, float v_scale=1.0, int v_zp=0, float a_scale=1.0, int a_zp=0, float o_scale=1.0, int o_zp=0) -> Tensorzida8w4_linear_prepack_cpu(Tensor weight, Tensor scales, Tensor qzeros) -> (Tensor, Tensor, Tensor, Tensor)zda8w4_linear_cpu(Tensor input, Tensor input_scales, Tensor input_qzeros, Tensor weight, Tensor weight_scales, Tensor weight_qzeros, Tensor compensation, Tensor? bias, ScalarType output_dtype) -> Tensorz_scaled_embedding_bag(Tensor qweight, Tensor indices, Tensor offsets, Tensor weight_scale, float o_scale, int mode, bool include_last_offset, ScalarType output_dtype) -> TensorzKfloat8_linear_prepack_cpu(Tensor weight, Tensor scales) -> (Tensor, Tensor)zfloat8_linear_cpu(Tensor input, Tensor input_scales, Tensor weight, Tensor weight_scales, Tensor? bias, ScalarType output_dtype) -> Tensorcfd}|S)NcTtj|SN)torchlibrary register_fakefuncnames _/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/ops.py decoratorz%register_custom_op..decoratorSs#}**d955d;;;rrs` rregister_custom_oprRs$<<<<< rcfd}|S)NcXtjd|S)Nr) mutates_args)r r custom_oprs rrz*register_custom_op_impl..decoratorZs(}&&$yr&BB4HHHrrrs` rregister_custom_op_implrYs)IIIII rctjtj}|jdz|jz}|S)N )r cudaget_device_propertiescurrent_devicemajorminor) device_propscompute_capabilitys rcached_compute_capabilityr%`sA:33EJ4M4M4O4OPPL%+b0<3EE rEXPONENTMANTISSA _in_feats_weights_scalessplitKreturncttjdkfdtjjj||||||S)as Quant-LLM linear layer A @ W.T. See https://arxiv.org/abs/2401.14112 for more details. Arguments EXPONENT: number of exponent bits MANTISSA: number of mantissa bits _in_feats: input activations in FP16 _weights: packed Floatx weights _scales: scale splitK: split K Returns output of linear layer KcdS)NzLquant_llm_linear requires sm7.5+ GPU architecture, but current device has smr)r$srz"quant_llm_linear..ss_qssr)r%r _checkopsrquant_llm_lineardefault)r'r(r)r*r+r,r$s @rr4r4gsh.344 Lb ssss 9  - 5 5(Ix&  rztorchao::quant_llm_linearcHtjdkfdtjjtjtjfvfdtjdkfdtjjtjufdtjdkfdtjjtjtjfvfdj\}}j\}} d|z|z} tj|d z| zjdkd tj|jd kd ||fS) Nc4ddS)Nz!input should be a 2d tensor, got Ddimr)srr1z_..sFIMMOOFFFrcdjS)Nz!weight must be FP16 or BF16, got dtyper<srr1z_..sEIOEErc4ddS)Nz"weight should be a 2d tensor, got r9r:r*srr1z_..sFX\\^^FFFrcdjS)Nzweight must be UINT8, got r>rAsrr1z_..=X^==rr&c4ddS)Nz!scale should be a 2d tensor, got r9r:r+srr1z_..s$X $X$X$XrcdjS)Nz scale must be FP16 or BF16, got r>rEsrr1z_..sB7=BBrcdSNzDimensions mismatchedrrrrr1z_..s@WrrcdSrIrrrrr1z_..s1Hr) r r2r;r?float16bfloat16uint8shape new_empty) r'r(r)r*r+r,BSICOC_N_BITSs ``` rrSrSs L 1FFFF LEM5>::EEEE L !FFFF L%+%==== L XXXX L %-88BBBB _FB NEB \H $F Lq6!X^A%668W8WXXX Lw}Q'')H)HIII   Bx ( ((rF?querykeyvalue attn_mask dropout_p is_causalscaleq_scaleq_zpk_scalek_zpv_scalev_zpa_scalea_zpo_scaleo_zpcttjjj|||||||||| | | | | |||S)a Quantized SDPA with quantized inputs and outputs. Arguments query: input query tensor, key: input key tensor, value: input value tensor, attn_mask: attention mask tensor, dropout_p: dropout probability, is_causal: causal flag, scale: scaling factor applied prior to softmax, q_scale: scale for query from linear quantization, q_zp: zero point for query from linear quantization, k_scale: scale for key from linear quantization, k_zp: zero point of key from linear quantization, v_scale: zero point for value from linear quantization, v_zp: zero point of value from linear quantization, a_scale: scale for attention from softmax quantization, a_zp: zero point for attention from softmax quantization, o_scale: scale for output from linear quantization, o_zp: zero point for output from linear quantization, Returns output of quantized SDPA )r r3rqscaled_dot_productr5rWrXrYrZr[r\r]r^r_r`rarbrcrdrerfrgs rririsXT 9  0 8 8         #  rztorchao::qscaled_dot_productc|Sr rrjs rrSrSs ( Lrpacked_w inner_k_tilescXtjjj||S)a Unpacks weights that were packed with `torch.ops.aten._convert_weight_to_int4pack` to original tensor of shape `N x K`. Assumes that the packed weights were generated with `torch.ops.aten._convert_weight_to_int4pack` with `inner_k_tiles = 2 | 4 | 8`" Args: packed_w: torch.tensor: 4D tensor with shape (N / 8) x (K / (inner_k_tiles * 16)) x 32 x inner_k_tiles, dtype is torch.int32 inner_k_tiles: int Returns: torch.tensor of shape is N x K, dtype is torch.int32 rlrm)r r3runpack_tensor_core_tiled_layoutr5ros rrprps/ 9  < D D E  rz(torchao::unpack_tensor_core_tiled_layoutcntjdkfdtjjtjufdtj|dkp |dkp|dkdtjddkdtjd |dz kd d dz}d |zd z}tj||ftjjS)Nc4ddS)Nz*packed weight should be a 42d tensor, got r9r:rlsrr1z_..sNX\\^^NNNrcdjSNzweight must be INT32, got r>rtsrr1z_..#rCrr7rGcdSNz inner_k_tiles must be 2, 4, or 8rrrrr1z_..'2r cdSNz#packed weight must have 32 at dim 2rrrrr1z_..)1VrcdSNz0packed weight must have inner_k_tiles/2 at dim 3rrrrr1z_..,Brrr&r?device)r r2r;r?int32sizeemptyr)rlrmNKs` rrSrSsG L !NNNN L%+%==== LFmq0FMQ4F22 Lq!!R')V)VWWW L aMA--BB  a1A a=(2-A ;1vU[ I I IIrscales_and_zeros group_sizecZtjjj||||S)a Dequantizes by: - Unpacking weights that were packed with `torch.ops.aten._convert_weight_to_int4pack` to original tensor of shape `N x K` - Upcasting to bfloat16 - Dequantizing with the scales_and_zeros that were packed with `torchao.quantization.utils.pack_tinygemm_scales_and_zeros` Assumes: - packed weights were generated with `torch.ops.aten._convert_weight_to_int4pack` with `inner_k_tiles = 2 | 4 | 8`" - packed scales_and_zeros were generated with `torchao.quantization.utils.pack_tinygemm_scales_and_zeros` - qGroupSize is 32 | 64 | 128 | 256 Args: packed_w: torch.tensor: 4D tensor with shape `(N / 8) x (K / (inner_k_tiles * 16)) x 32 x inner_k_tiles / 2`, dtype is torch.int32 scales_and_zeros: torch.tensor: 3D tensor with shape `numQGroups x N x 2`, dtype is torch.bfloat16 where numQGroups is K / qGroupSize group_size: int inner_k_tiles: int Returns: torch.tensor of shape is N x K, dtype is torch.bfloat16 )r r3r#dequantize_tensor_core_tiled_layoutr5)rlrrrms rrr4s.0 9  @ H H"J   rz,torchao::dequantize_tensor_core_tiled_layoutctjdkfdtjjtjufdtj|dkp |dkp|dkdtjddkdtjd |dz kd d dz}d |zd z}tj|jtjudtj|d kdtj|dkp|dkp |dkp|dkdtj|d ||zkdtj|d |kdtj|ddkdtj||ftjjS)Nrrc4ddS)Nz)packed weight should be a 4d tensor, got r9r:rtsrr1z_..XsMHLLNNMMMrcdjSrvr>rtsrr1z_..\rCrr7rGcdSrxrrrrr1z_..`ryrrzcdSr|rrrrr1z_..br}rr~cdSrrrrrr1z_..errrr&rcdS)Nz!scales_and_zeros must be bfloat16rrrrr1z_..ms3rcdS)Nz9scales_and_zeros must be 3D, got {scales_and_zeros.dim()}rrrrr1z_..qsKr@cdS)Nz&qGroupSize must be 32, 64, 128, or 256rrrrr1z_..us8rcdS)Nz3scales_and_zeros must have K // qGroupSize at dim 0rrrrr1z_..ysErcdS)Nz%scales_and_zeros must have N at dim 1rrrrr1z_..|/VrcdS)Nz%scales_and_zeros must have 2 at dim 2rrrrr1z_..rrr) r r2r;r?rrrLrr)rlrrrmrrs` rrSrSQs  L !MMMM L%+%==== LFmq0FMQ4F22 Lq!!R')V)VWWW L aMA--BB  a1A a=(2-A L%.033 L!#KK LbVJ",V c0AVZSVEV88 La  AO3EE La  A%'V'V La  A%'V'V ;1vU^HO L L LLrx weight_marlinmetas workspacebitssize_msize_nsize_kc dtjjj||||||||| S)a Sparse Marlin 2:4 matrix multiplication. Reference: https://github.com/IST-DASLab/Sparse-Marlin/tree/main Args: x: input matrix of shape `(n, k/2)` in column-major layout. weight_marlin: weight matrix of original shape `(m, k)` in Marlin format; see `Layer.pack()`. meta: metadata information for 2:4 sparsity. s: scales of shape `(n / groupsize / 2, m)`. workspace: tensor with at least `m / 128 * max_par` entries that are all zero. bits: number of bits for quantization. size_m: number of rows in input matrix. size_n: number of columns in weight matrix. size_k: number of columns in input matrix. Returns: output matrix of shape `(n, m)` in column-major layout. )r r3rmarlin_24_gemmr5) rrrrrrrrrs rrrs74 9  + 3 3 =$9dFFF  rztorchao::marlin_24_gemmc f d d d} tjdkpdkfddz} tjdkfd tjd kfd tj zdk fd tj zd zdk fdtjd kfdtjd  zdk fdd  z| z tj k fdtjddzd zd zkfdtjd d zkfdtjjdtjdtjjdtjdtjjdtjdtjjdtjdddd krPtjdzdkfddzd ztjdkpdkfdtj zdk fd z| ztjkfd tjdd fjj!S)"NrrrrrrGcdS)Nznum_bits must be 4 or 8. Got = r)rsrr1z_..s(P$(P(Prrzrc:dddSNzShape mismatch: x.size(0) = r , size_m = rrrsrr1z_.."MqvvayyMMVMMrr&c:dddSNzShape mismatch: x.size(1) = r& , size_k = rrrsrr1z_..rrcddSN size_k = ! is not divisible by tile_size = r TILE_SIZErsrr1z_..PFPPYPPrr7c@ddddSNz(Shape mismatch: weight_marlin.size(0) = rrz, tile_size = rrrrsrr1z_..0~=;M;Ma;P;P~~]c~~s|~~rc:dddS)Nz s.size(1) = r& , size_n = r)rrsrr1z_..s"%RAFF1II%R%R&%R%Rrc:dddSNzweight_marlin.size(1) = r&rrrrsrr1z_..'n=+=+=a+@+@nnclnnrcddS)N size_n = , actual_size_n = r actual_size_nrsrr1z_..sEFEEmEErcLddddzdzdzS)Nzmeta.size(0) = rz is not size_k / 8 / 2 / 2 = rGr7r)rrsrr1z_..s3d$))A,,ddVWX[\]M]abMbddrc@ddddzS)Nzmeta.size(1) = r&z is not size_n * 2 = r7r)rrsrr1z_..s&Q$))A,,QQVaZQQrcdSNzx is not on GPUrrrrr1z_..$5rcdSNzx is not contiguousrrrrr1z_..,ArcdSNzweight_marlin is not on GPUrrrrr1z_..0MrcdSNzweight_marlin is not contiguousrrrrr1z_../PrcdS)Nzmeta is not on GPUrrrrr1z_..';rcdS)Nzmeta is not contiguousrrrrr1z_../GrcdS)Nzs is not on GPUrrrrr1z_..rrcdS)Nzs is not contiguousrrrrr1z_..rrc:dddS)Nrz! is not divisible by s.size(0) = rr)rrsrr1z_..s#TTTPQTTrcdSNzUnexpected groupsize = r groupsizesrr1z_..s5)55rcddSNrz$ is not divisible by min_thread_n = r MIN_THREAD_Nrsrr1z_.. VFVV VVrc8ddS)Nzworkspace.numel =  is below min_workspace_size = numelmin_workspace_sizersrr1z_..s#kY__%6%6kkWikkrr) r r2ris_cuda is_contiguousrrr?r)rrrrrrrrrMAX_PARALLELISM pack_factorrrrrrs````````` @@@@@rrSrSsILO L  TQY P P P P*K L!&&))MMMMM  L!&&))MMMMM LaPPPPP L 9  !m&8&8&;&;;~~~~~~  L q VRRRRR L1 )Q.nnnnn #''**i7;FM L-EEEEE  L ! ! q(A--ddddd L !  "QQQQQ  L55666 L""$A$ABBB L&(M(MNNN L##%%'P'P  L;;<<< L##%%'G'GHHH L55666 L""$A$ABBBIvvayy1}} QVVAYY ! # T T T T T   affQii' a  LR*9?5555  L"VVVVV!L0OC L//kkkkk ;q 166!99-QWQX N N NNrs_toks_chs_groupc dtjjj||||||||| S)a Marlin for W4A8 mixed precision matrix multiplication. See https://arxiv.org/pdf/2406.09904 for more details. Reference: https://github.com/HandH1998/QQQ/tree/main Args: x: `torch.int8` input matrix of shape `(m, k)` in standard row-major layout. weight_marlin: `torch.int32` weight matrix of original shape `(k, n)` in the specified format. s_tok: `torch.float32` activation per-token quantization scales of shape `(m, 1)`. s_ch: `torch.float32` weight per-channel quantization scales of shape `(1, n)`. s_group: `torch.half` weight per-group quantization scales of shape `(m / groupsize, n)`, it should be empty when group_size != -1. workspace: `torch.int32` tensor with at least `n / 128 * max_par` entries that are all zero. size_m: number of rows in input matrix. size_n: number of columns in weight matrix. size_k: number of columns in input matrix. Returns: `torch.half` out matrix of shape `(m, n)` in standard row-major layout. )r r3rmarlin_qqq_gemmr5) rrrrrrrrrs rrrs78 9  , 4 4 =%w 666  rztorchao::marlin_qqq_gemmc "  d d d} d} tjdkfdtjkfdtjdkfdtj zdk fd tj zdk fd dkrd ndztjd vfd tjkfdtjd zdk fdd krctjdkfdtjdzdkfdd z| z tj k fdtjjdtjdtjjdtjdtjjdtjdtjjtjkdtjjdtjdtjjtjkdtjjdtjdtjjtjkdtj zdk fd  z| ztjkfd!tj ftjj "S)#NrrrGrc:dddSrrrsrr1z_..Jrrc8ddS)Nz Shape mismatch: s_tok.numel() = rr)rrsrr1z_..Ns U5;;==UUVUUrr&c:dddSrrrsrr1z_..TrrcddSrrrsrr1z_..Xrrc@ddddSrrrsrr1z_..\rrr)rrcdSrrrsrr1z_..as1V91V1Vrc8ddS)NzShape mismatch: s_ch.numel() = rr)rrsrr1z_..fs S$**,,SS6SSrc:dddSrrrsrr1z_..jrrc:dddS)Nz"Shape mismatch: s_group.size(1) = r&rr)rrsrr1z_..os#]a]]U[]]rc:dddS)Nrz' is not divisible by s_group.size(0) = rr)rrsrr1z_..ss#```w||\]``rcddS)NzShape mismatch: size_n = rrrsrr1z_..ysUFUUmUUrcdSrrrrrr1z_..}rrcdSrrrrrr1z_..~rrcdSrrrrrr1z_..rrcdSrrrrrr1z_..rrcdS)Nzs_tok is not on GPUrrrrr1z_..s(=rcdS)Nzs_tok is not contiguousrrrrr1z_..s0IrcdS)Nzs_tok's dtype is not float32rrrrr1z_..s7UrcdS)Nzs_ch is not on GPUrrrrr1z_..rrcdS)Nzs_ch is not contiguousrrrrr1z_..rrcdS)Nzs_ch's dtype is not float32rrrrr1z_..s6SrcdS)Nzs_group is not on GPUrrrrr1z_..s*ArcdS)Nzs_group is not contiguousrrrrr1z_..s2Mrzs_group's dtype is not float16cddSrrrsrr1z_..rrc8ddS)Nzworkspace.numel() = rrrsrr1z_..s#my'8'8mmYkmmrr) r r2rrrrr?float32rKrr)rrrrrrrrrr PACK_FACTORrrrrrs````````` @@@@@rrSrS6sILOK L!&&))MMMMM L%++--UUUUU  L!&&))MMMMM LaPPPPP L 9 !3!3A!6!66~~~~~~ mmoo**',,q//0II Li')V)V)V)VWWW L SSSSS L1 )Q.nnnnnB LLOOv % ] ] ] ] ]     W\\!__ $ ) ` ` ` ` `   #''**i7;FM L-UUUUU  L55666 L""$A$ABBB L&(M(MNNN L##%%'P'P  L = =>>> L$$&&(I(IJJJ L -/U/UVVV L;;<<< L##%%'G'GHHH Lu},.S.STTT L"A"ABBB L&&((*M*MNNN L%-/1QRRR L"VVVVV!L0OC L//mmmmm ;'u}QX N N NNrinput input_scaleweight weight_scalebias out_dtypec^tjjj||||||S)a CUTLASS-based row-wise scaled W4A8 linear operator. Args: input: quantized input tensor, in row-major layout. input_scale: scale factors for input tensor, has to be tensor of the same shape as the input tensor, minus the last dimension. weight: quantized weight matrix, in row-major layout. weight_scale: scale factors for weight tensor, one value per row of weight matrix (thus also tensor of the same shape as the weight tensor, minus the last dimension). bias: an optional vector of size equal to number of rows of weight tensor, or None. out_dtype: optional data type for output tensor. Returns: output: result tensor, in row-major layout. )r r3r"rowwise_scaled_linear_cutlass_s8s4r5rrrrrrs rrrs6* 9  ? G G     rz+torchao::rowwise_scaled_linear_cutlass_s8s4c||n|j}|j}tjg|jdd|jdR||SNrrrr?rr rrNrrrrrrr?rs rrSrSS#.IIK4EE \F ;;SbS);6<?;;5QW X X XXrc^tjjj||||||S)a CUTLASS-based row-wise scaled W4A4 linear operator. Args: input: quantized input tensor, in row-major layout. input_scale: scale factors for input tensor, has to be tensor of the same shape as the input tensor, minus the last dimension. weight: quantized weight matrix, in row-major layout. weight_scale: scale factors for weight tensor, one value per row of weight matrix (thus also tensor of the same shape as the weight tensor, minus the last dimension). bias: an optional vector of size equal to number of rows of weight tensor, or None. out_dtype: optional data type for output tensor. Returns: output: result tensor, in row-major layout. )r r3r"rowwise_scaled_linear_cutlass_s4s4r5rs rr$r$s1* 9  ? G G {FL$   rz+torchao::rowwise_scaled_linear_cutlass_s4s4c||n|j}|j}tjg|jdd|jdR||Srr r!s rrSrSr"r weight_metac `tjjj|||||||S)a. CUTLASS-based row-wise scaled F8F8 linear operator, for sparsified weight case. Args: input: quantized input tensor, in row-major layout. input_scale: scale factors for input tensor, has to be tensor of the same shape as the input tensor, minus the last dimension. weight: sparsified quantized weight matrix, in row-major layout. weight_meta: sparsify metadata for weight tensor. weight_scale: scale factors for weight tensor, one value per row of weight matrix (thus also tensor of the same shape as the weight tensor, minus the last dimension). bias: an optional vector of size equal to number of rows of weight tensor, or None. out_dtype: optional data type for output tensor. Returns: output: result tensor, in row-major layout. )r r3r)rowwise_scaled_linear_sparse_cutlass_f8f8r5)rrrr&rrrs rr(r(s3. 9  F N N {FKtY  rz2torchao::rowwise_scaled_linear_sparse_cutlass_f8f8c||n|j}|j}tjg|jdd|jdR||Srr ) rrrr&rrrr?rs rrSrSsS#.IIK4EE \F ;;SbS);6<?;;5QW X X XXrcTtjjj|S)a CUTLASS-based conversion from sparsified input tensor to corresponding compressed tensor, along with corresponding metadata tensor. Args: weight: input tensor, in row-major layout. Returns: weight_compressed: compressed weight tensor, with sparsity eliminated, in row-major layout. weight_meta: metadata tensor, describing the sparsity structure of the input tensor, also in row-major layout. )r r3r)to_sparse_semi_structured_cutlass_sm9x_f8r5rs rr+r+,s  9  F N Nv V VVrz2torchao::to_sparse_semi_structured_cutlass_sm9x_f8c||d|ddz||dt|ddzdtjfS)Nrr&r7rGrr>)rOmaxr charr,s rrSrS;s] F1IN33Cq Q$;$;5:NN r input_tensormetadata_format activation algorithmcVtjj||||||S)N)r?r])r r3rsparse24_sm90_sparsify)r0r1r2r3r?r]s rr5r5Hs4 9  3 3oz9EQV 4  rztorchao::sparse24_sm90_sparsifyc||n|j}tj|jd|jddzf||jtj|jd|jddzftj|jfS)Nrr&r7rrG)r?r rrNrrM)r0r1r2r3r?r]rs rrSrSUs* 0BI   "L$6q$9Q$> ?&      "L$6q$9Q$> ?+&    rrGnrabb_scale swizzle_size swizzle_axissm_countc Ztjj||||||||S)N)rdr:r;r<r=)r r3rsparse24_fp8_sm90_cutlass_gemmr8rr9rdr:r;r<r=s rr?r?ms> 9  ; ;   !! <   rz'torchao::sparse24_fp8_sm90_cutlass_gemmc~tj|jd|jdftj|jS)Nrr&r)r rrNrLrr@s rrSrSs0 ; AGAJ/u~ah W W WWrmat1mat2mat1_is_swizzledmat2_is_swizzledcZtjjj||||SzP Similar to torch.mm but Tensor inputs can be SwizzleTensor instances. )r r3r swizzle_mmr5rBrCrDrEs rrHrHs/ 9  ' / / d$&6  rztorchao::swizzle_mmcZ||jd|jdSNrr&rOrNrIs rrSrSs$ >>$*Q-A 7 77rscale_ascale_b scale_resultc dtjjj||||||||| SrG)r r3rswizzle_scaled_mmr5 rBrCrDrErMrNrrOrs rrQrQs? 9  . 6 6      rztorchao::swizzle_scaled_mmc Z||jd|jdSrKrLrRs rrSrSs$ >>$*Q-A 7 77rcvttdrtjtjfStjfS)zGTODO: when e8m0 is hardened and major release lets remove uint8 supportfloat8_e8m0fnu)hasattrr rMrUrrr _get_dtypesrWs1u&''3 U122 K>rct}tjj|vfdtjj|vfddS)NcdjS)Nz4A_scale tensor must be uint8 or float8_e8m0fnu, got r>)A_scalesrr1z%_check_scale_dtypes..Vw}VVrcdjS)Nz4B_scale tensor must be uint8 or float8_e8m0fnu, got r>)B_scalesrr1z%_check_scale_dtypes..r[r)rWr r2r?)rZr]allowed_dtypess`` r_check_scale_dtypesr_sm ]]N L 'VVVV L 'VVVVrztorchao::mx_fp8_bf16ABrZr]ctj|d|dftj|jS)zMeta impl for mx_fp8_bf16rr&rr rrrLrr`rarZr]s rmeta_mx_fp8_bf16res8 ;q 166!99-U^AH U U UUrczt||tjjj||||S)a=Defines a matmul between two fp4 tensors w/ MX scales in E8MO and returns a bf16 tensor. The expected format is fp4_e2m1 specified: https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final.pdf (Section 5.3.3) Note: The mx scales are E8MO tensors stored in uint8 tensors (for now). The layout of the scales is very particular, see: https://docs.nvidia.com/cuda/cublas/index.html#d-block-scaling-factors-layout Args: A: fp4 tensor (2 fp4 elements are packed into 1 byte -> elem0|elem1) B: fp4 tensor (2 fp4 elements are packed into 1 byte -> elem0|elem1) A_scale: E8M0 scale tensor for A with groupsize=32 in swizzled layout B_scale: E8M0 scale tensor for B with groupsize=32 in swizzled layout Returns: MXN bf16 Tensor )r_r r3r mx_fp4_bf16r5rds rrgrgs6*))) 9  ( 0 0Aw H HHrztorchao::mx_fp4_bf16ctj|d|dftj|jS)zMeta impl for mx_fp4_bf16rr&rrcrds rmeta_mx_fp4_bf16ris8 ;q 166!99-U^AH U U UUrscalesqzeroscXtjjj|||S)z Prepack weights for DA8W4 linear operator on CPU. Args: weight: weight tensor. scales: scales for weight tensor. qzeros: zero points for weight tensor. Returns: packed weight, scales, and zero points. )r r3rda8w4_linear_prepack_cpur5rrjrks rrmrm s$ 9  5 = =fff U UUrz!torchao::da8w4_linear_prepack_cpuc0|||tjfSr )r rrns rrSrSs 665<>> 11r input_scales input_qzeros weight_scales weight_qzeros compensationc dtjjj||||||||| S)a DA8W4 linear operator on CPU. Args: input: input tensor. input_scales: scales for input tensor. input_qzeros: zero points for input tensor. weight: weight tensor. weight_scales: scales for weight tensor. weight_qzeros: zero points for weight tensor. compensation: compensation tensor for weight. bias: optional bias tensor. out_dtype: output data type. Returns: output tensor in out_dtype. )r r3rda8w4_linear_cpur5) rrprqrrrrsrtrrs rrvrv s?4 9  - 5 5     rztorchao::da8w4_linear_cpuc |dksJ|d|dzdz} |jg|jdd| Rd|iS)Nrrrr~r7rr?r;rrOrN) rrprqrrrrsrtrrrs rrSrSGsp ::<<1     AQ'!+A 5? AEK, Aa A A Ay A AArztorchao::_scaled_embedding_bagqweightindicesoffsetsw_scalesmodeinclude_last_offsetcx|dksJ|jddz }|||jd|S)NTrr&r>)rNrO) ryrzr{r|rfr}r~r batch_sizes rrSrSXsH $ & & & &q!A%J   Zq)9  K KKrcVtjjj||S)z Prepack weights for float8 linear operator on CPU. Args: weight: weight tensor. scales: scales for weight tensor. Returns: packed weight, packed scales )r r3rfloat8_linear_prepack_cpur5rrjs rrris" 9  6 > >vv N NNrz"torchao::float8_linear_prepack_cpuc ||fSr rrs rrSrSxs 6>rc^tjjj||||||S)aH float8 linear operator on CPU. Args: input: input tensor. input_scales: scales for input tensor. weight: weight tensor. weight_scales: scales for weight tensor. bias: optional bias tensor. out_dtype: output data type. Returns: output tensor in out_dtype. )r r3rfloat8_linear_cpur5)rrprrrrrs rrr}s6( 9  . 6 6     rztorchao::float8_linear_cpuc |dvsJ|dkr+|d|dzn|d}|jg|jdd|Rd|iS)N)r7rrrrrr~rr?rx)rrprrrrrrs rrSrSs ::<<6 ! ! ! !+1::<<1+<+< AQ''&++a..A 5? AEK, Aa A A Ay A AAr)r&)NrUFNrVrrVrrVrrVrrVr)NN)NNrGr7r)- functoolstypingrr rr Librarylibdefine_CTagneeds_fixed_stride_orderrr lru_cacher%intr4rSfloatboolrirprrrr?rr$r(r+strr5r?rHrQrWr_rergrirmrvrrrrrrs   mIz22 { S A V a h h C R i B b T  O (, / 0  O (, / 0  } o P w Q Q         B/00 &)&)&)&)&) &)  &)  &) &)&)&)10&)Z#'!#<< < < < <  <  < E?<< << << << < !<" #<$ %<<<<~233 #'!#         E?     !" #$ %43,fSV&>??JJsJvJJJ@?J0(.-..mO mOmO mO mO  mO  mO mO mO mO mOmOmO/.mO`            B.//iO iOiO iO  iO  iO  iO iO iO iO iOiOiO0/iOb"'+     6    $  >ABB "'+ Y Y  Y Y  Y Y 6  Y  $ Y  Y Y YCB Y*"'+     6    $  4ABB "'+ Y Y  Y Y  Y Y 6  Y  $ Y  Y Y YCB Y,"'+      6   $ 8HII"'+YY YY Y Y  Y 6  Y $Y YYYJIY" W  W f W W W WHII    f   JI "         f    566    766!% $  f   f     ,=>> !% $ X X  X  X Xf  X f  X  X X X X X?> X    26 JN      )**8 88268JN8 888+*8         6 6" $ 8011 8  8  8 8 8  8  8 6  86" 8 $ 8  8 8 821 8   *++VV6VFVVVVV,+V I6IfIvIIIII2*++VV6VFVVVVV,+V V V V V VVVV"7882f2f2f2222982$ $$$  $  $  $$ 6 ${$$$$N/00 B  B B B  B  B  B B 6  B{ B  B B B10 B 455 L  L  L L L  L  L L{ L  L L L65 L O  O  O  O O O O899ff:9    6   { <011 B  B B  B B 6  B { B  B B B21 B B Br