PiddlZddlmZddlmZddlmZmZmZm Z ddl Z ddl m Z ddl mZddlmZddlmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZdd lm Z ddl!m"Z" ddl#m$Z$n#ej%Z$YnxYwe j&'ddZ(e"e(Z)dgZ*dZ+dZ,dZ-dZ.dZ/e)de j0de1de j0de j0de j0de e j0dee2de j0fd Z3d!Z4de5fd"Z6d#efd$Z7d%Z8d&Z9d'Z:d(Z;d)Zd,Z?d-Z@d.ZAd/ZBd0ZCd1ZDd2ZEd3ZFd4ZGd5e jHjfd6ZIeGd7d8ZJd9edefd:ZKdS);N) dataclass)partial)AnyCallableListOptional) out_dtype)quantized_decomposed_lib) GraphModule) InternalMatch)ReplacedPatternsreplace_pattern_with_filters) WrapperModule)"_get_aten_graph_module_for_pattern,_replace_literals_with_existing_placeholders'_replace_literals_with_new_placeholders"remove_tensor_overload_for_qdq_ops) MappingType)_get_per_token_block_size)_register_custom_op)$_disable_aten_to_metadata_assertionstorchaoFRAGMENT reference_representation_rewritectjj|||||tj}tjj||||| tj}tjjj||| }tjj|| | | |tj}|SN) torchopsquantized_decomposeddequantize_per_tensorint8atenlineardefaultquantize_per_tensor)x_i8x_scale x_zero_point x_quant_min x_quant_max weight_i8 weight_scaleweight_zero_pointweight_quant_minweight_quant_max bias_fp32 out_scaleout_zero_point out_quant_min out_quant_maxx_fp32 weight_fp32out_fp32out_i8s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/quantization/pt2e/reference_representation_rewrite.py_qdq_quantized_linearr:,s"Y + A A g|[+uzF)0FF Ky~$,,V[)LLH Y + ? ?)^]M5:F Mctjj|||}tjj||| }|tj}|tj}t tjjjjtj ||z ||z d}||z}t tjjj j tj | |}||z}t tjjj j tj |||z| z | z}tjj|| |tj }|Sr)rrr"clamptoint16r r#r$int32divTensormulr!)r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4x_i16 weight_i16acc_i32 bias_scalebias_i32r8s r9_reference_quantized_linearrIOs@( 9>  k; ? ?D $$Y0@BRSSI GGEK Eek**J %  && G<'J+2EKJWWH G  IN  % K  l "Y .      Y^ ! !'=- H H K KEJ W WF Mr;c tjj||||tj\} } tjj|| | ||tj} tjj| | | ||tj}tjj|||||tj} tjjj || | }|Sr) rrrchoose_qparamsr!r%r r"r#r$)r5r)r*x_epsr+r,r-r.r/r0r'r(r&r6r7s r9_qdq_dynamic_quantized_linearrMs"I:II [%G\ 9 ) = ={K  DY + A A g|[+uzF)0FF Ky~$,,V[)LLH Or;c &tjj||||tj\} } || z }tj|}|tj} | | z} tj| ||} | tj} tjj |||}| tj }|tj }ttjj j j tj|| z ||z d}| |z}ttjj jjtj| |}||z}|| |zz}|SNdtype)rrrrKr!roundr>r@r=r"r?r r#r$rArB)r5r)r*rLr+r,r-r.r/r0r'r(x_i32r&rDrErFrGrHr7s r9#_reference_dynamic_quantized_linearrTsW"I:II [%G\ g F[ F IIEKI ( (E L E K{K 8 8E 88%*8 % %D $$Y0@BRSSI GGEK Eek**J %  && G<'J+2EKJWWH G'L01H Or;c 4tj}t|}d} d} tjj||j|tj| | |tj tj \} } tjj ||| | tj| | } tjj | || | tj| | tj }|dks Jd|j d|zdks Jd|dks Jdd|f}tjj ||||tjd d }tjjj|||}|S) NrzGroup size must be positivez&Weight must be divisible by group_sizezWeight must be 2D tensor)r ASYMMETRICrrrrchoose_qparams_affinenamer!float32r@quantize_affinedequantize_affineshapedimr"r#r$)r5rL weight_i4r,r-r0 group_sizex_mapping_typeper_token_block_sizer)r*r'r(r& block_sizer6r7s r9,_qdq_dynamic_quantized_linear_4bit_groupwiseris!+N4V<<KK!I-CC      G\ 9  , ,   DY  0 0     F >>>8>>> ?1  *a / / /0 0 / / ==??a   !;   ZJ)#55   Ky~$,,V[)LLH Or;r5rLrdr,r-r0rereturnc |d}tj}t|}d} d} tjj||j|tj| | |tj tj \} } tjj ||| | tj| | } |j \}}||z}| tj tj }| j }| j d}| d|} | j d}| |||}||||}| tj }| tj }tj||tj |j}t%|}||d<|jdkr|d}t+|D]}|dd|ddf}|dd|ddf}|d}|dd|f}t/tjjjjtj ||d}|| tj |ddzz}| tj dd| ddz|ddz} || z }| dd}!||!z}"||"|z}"|"|S) a Reference implementation for dynamically quantized linear 4-bit groupwise operation. This implementation emulates actual numerics of on-device integer compute. Args: x_fp32: Input activation tensor in fp32 x_eps: Epsilon for quantization parameter computation weight_i4: 4-bit quantized weight (stored as int8 with values in [-8, 7]) weight_scale: Groupwise scales for weight dequantization weight_zero_point: Groupwise zero points for weight (unused for symmetric) bias_fp32: Optional bias tensor in fp32 group_size: Size of each group for groupwise quantization Returns: Output tensor in fp32 rXrVrWr)rQdeviceN)rc)rr\rrrrr]r^r!r_r@r`rbr>bfloat16viewzerosrmlistndim unsqueezerangesumr r"r#r$)#r5rLrdr,r-r0rerfrgr)r*r'r(r& out_features in_features num_groups x_orig_shapek_dim batch_size x_i8_groupedweight_i4_grouped x_i32_groupedweight_i32_groupedacc_fp32 out_shape group_idxx_group weight_groupweight_group_col_sumweight_scale_group group_accweights_col_sum_adjustedx_scale_multiplierr7s# r9_reference_dqlinear_int4rsJ6AJ +N4V<<KK!I-CC      G\ 9  , ,   D!*L+ *J ??5>2255emDDL:L JrNE 99R  DAJ99ZZ@@L"|ZLL!OOEK00M*--ek::{L fmH\""I IbMA#--a00 :&&!7!7 9aaa0)!!!Y/: +//B/77*!!!Y,7 IN ! ) K      ill5=99 >A&& ' %%a,, - ! 66 b!,,,,Hi' == # ##r;c Ztjj||||||d|fS)z Reference implementation for dynamically quantized linear 4-bit groupwise operation. This function now delegates to the custom op implementation. rX)rrrreference_dqlinear_int4)r5rLrdr,r-r0res r92_reference_dynamic_quantized_linear_4bit_groupwisers: 9  4 4  J  r;cJd}d}|jD]}t|tjjra|jdkrV|jtjj j j kr2|j }t|dkr|ddko |ddk}t|tjjr_|jdkrT|jtjj jj kr0|j }t|dkr|dtjk}|o|S)NF call_functionr[rZ) nodes_mapvalues isinstancerfxNodeoptargetrrrar$argslenr`r!)matchoriginal_graph pattern_graphweight_is_int4act_quant_is_int8noders r96_filter_fn_for_dynamic_quantized_linear_4bit_groupwisers N&&((:: tUX] + + @?** uy0BJJJ9D4yyA~~!%aB!?47a< tUX] + + :?** uy0@HHH9D4yyA~~$(Guz$9!  ///r;replacement_patterncxddlm}m}d}d}|jD]-\}}|jt jjj j kr|}n.t|j dkr |j d}|=|=|j |_ ||d|jg|j d<dSdSdS)z Port metadata for dynamically quantized linear 4-bit groupwise operation. It custom_op node's metadata with corresponding linear node's metadata. r) NodeSourceNodeSourceActionNrl#ReplaceInt4DynamicQuantWithCustomOp from_node)torch.fx.tracebackrrritemsrrrr"r#r$r replacementsmetacopyREPLACE)rrr linear_nodeint4_custom_op_node_g_ns r9:_port_metadata_for_dynamic_quantized_linear_4bit_groupwisers@???????K%/55773 :.6 6 6K E 7  +,,q001>rB#6#B#.#3#8#8#:#: J5 (  1  ---#B#Br;c ddg}ddg}ddg}d}ddg}d}tjj|||||tj}tjj||||| tj}tjjj||| |||||| }tjj|| | | |tj}|SNrXrF) rrrr r!r" convolutionr$r%)r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4stridepaddingdilation transposedoutput_paddinggroupsr5r6r7r8s r9_qdq_quantized_conv2drs"VF!fG1vHJVN F Y + A A g|[+uzF)0FF Ky~)11  HY + ? ?)^]M5:F Mr;c ddg}ddg}ddg}d}ddg}d}tjj|||}tjj||| }|tj}|tj}t tjjjjtj ||z ||z d|||||| }||z}t tjjj j tj | |}| d}| d}||z}t tjjj j tj |||z| z | z}tjj|| |tj}|S)NrXrFrl)rrr"r=r>r?r rr$r@rArBrsrCr!)r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4rrrrrrrDrErFrGrHr8s r9_reference_quantized_conv2drs"VF!fG1vHJVN F 9>  k; ? ?D $$Y0@BRSSI GGEK Eek**J "*  &&   G<'J+2EKJWWH!!"%%H!!"%%H G  IN  % K  l "Y .      Y^ ! !'=- H H K KEJ W WF Mr;c tjj||||| tj} tjj||||| tj} | | z} tjj| } tjj| |||| tj} | Sr)rrrr r!r"relur%r&r'r(y_i8y_scale y_zero_pointr1r2 quant_min quant_maxr5y_fp32r7r8s r9_qdq_quantized_add_relurksY + A A g|Y 5:FY + A A g|Y 5:FHy~""8,,H Y + ? ?)^Y 5:F Mr;c |tj} |tj} ttjjjjtj| |z ||z } ttjjjjtj| |z ||z } | | z|z} tjj| || tj } | S)z See comments for `_reference_quantized_add` for more information on how to derive the formula for out_i8 based on x_i8 and y_i8 ) r>rr@r rr"rCrBr=r!r&r'r(rrrr1r2rrrSy_i32out_i32r8s r9_reference_quantized_add_relurs GGEK E GGEK E  !   9    E  !   9    E emn,G Y^ ! !'>9 E E H H T TF Mr;c Btjj||||| tj} tjj||||| tj} | | z} tjj| |||| tj} | Sr)rrrr r!r%rs r9_qdq_quantized_addrsY + A A g|Y 5:FY + A A g|Y 5:FH Y + ? ?)^Y 5:F Mr;c |tj} |tj} tj||z | |z ztj} tj||z | |z ztj} | | z|z} d}d} tjj| || tj} | S)a # How to Derive the formula for out_i8 based on x_i8 and y_i8 # (since quantized add takes x_i8, y_i8 and their quantization parameters, and produce an out_i8) # out_i8 is quantized output, we can write down the formula for it first: out_i8 = out_f32 / out_scale + out_zero_point (1) # then out_fp32 is computed from x_f32 + y_f32, and the x_fp32 and y_fp32 are the dequantized x_i8 and y_i8 out_f32 = x_f32 + y_f32 (2) x_fp32 = (x_i8 - x_zero_point) * x_scale (3) y_fp32 = (y_i8 - y_zero_point) * y_scale (4) # applying the above fomula to the out_i8 equation we can get the following: out_i8 = out_fp32 / out_scale + out_zero_point # (1) = (x_f32 + y_f32) / out_scale + out_zero_point # applying (2) to substitute out_fp32 with x_fp32 + y_fp32 = ((x_i8 - x_zero_point) * x_scale + (y_i8 - y_zero_point) * y_scale) / out_scale + out_zero_point # apply (3) and (4) rVrW)r>rr@rRrr"r=r!rs r9_reference_quantized_addrs: GGEK E GGEK E K9,1EF G G J J5; W WE K9,1EF G G J J5; W WEemn,GII Y^ ! !'9i @ @ C CEJ O OF Mr;c Hd} d} d} d} d} tjj|||||tj}tjjj|| | | | | \}}tjj|||||tj}|Sr) rrrr r!r"max_pool2d_with_indicesr$r%)r&r'r(r)r*r1r2r3r4 kernel_sizerrr ceil_moder5r7rr8s r9_qdq_quantized_max_pool2drsK FGHI Y + A A g|[+uzF).8@@ VWh KHaY + ? ?)^]M5:F Mr;c ld} d} d} d} d} tj|||}|tj}tjjj||z | | | | | \}}|||z z|z}tj|||}|tj}|Sr) rr=r>r@rr"rr$r!)r&r'r(r)r*r1r2r3r4rrrrrrSrrr7r8s r9_reference_quantized_max_pool2drsK FGHI ;t[+ 6 6D GGEK E7?? k67HiJGQ'I-.?H{8]MBBH [[ $ $F Mr;cltjj|||||tj}|Sr)rrrr%r!r5scale zero_pointrrxs r9_quantize_per_tensor_int8r$s2 &::z9i  A Hr;c||z }tj|}|tj}||z}tj|||}|tj}|SrO)rrRr>r@r=r!rs r9#_reference_quantize_per_tensor_int8r+sg A  AA 5;A JA Ay),,A 5:A Hr;cltjj|||||tj}|Sr)rrrr r!)r&rrrrr5s r9_dequantize_per_tensor_int8r;s2 Y + A A eZIuzF Mr;ctjj|||}|tj|z |ztjSrO)rrr"r=r>r_)r&rrrrs r9%_reference_dequantize_per_tensor_int8rBsR 9>  i ; ;DWWU] # #j 0E 9 = =EM = R RRr;c ntjj||||||tj}|Sr)rrrquantize_per_channelr!)r5scales zero_pointsch_axisrrr8s r9_quantize_per_channel_int8rOs6Y + @ @ WiEJF Mr;cNtj||d}tjjtj||z tj|z||}tj||d}|tjSNrl) r transposerr"r=rRr>r@r!)r5rrrrrrs r9$_reference_quantize_per_channel_int8rXs_VWb 1 1Fin"" FVO$$'' 44{BIyGogw33G ::ej ! !!r;c ntjj||||||tj}|Sr)rrrdequantize_per_channelr!)r&rrrrrr7s r9_dequantize_per_channel_int8rcs6y-DD fk7Iy%*H Or;c2tjj|||}tj||d}|tj}||z tj|z}tj||d}|Sr)rrr"r=rr>r@float)r&rrrrrrSr7s r9&_reference_dequantize_per_channel_int8rms| 9>  i ; ;D ?4" - -D GGEK E #'' 44v=Hx"55H Or;gmc0t|dgddddS)Nrlrr)rXrVrW)exclude_literalsliteral_to_ph_idx)r)rs r9'_replace_ph_qdq_per_channel_replacementr{s, 7 bT!!5L5L   r;c8eZdZUdZeedfed<eed<eed<dZe ee ge fed<dZ e ee ge fed<dZ e e ed ejjejjgefed <d Zeed <dZe ed gdfed<dS) _RewriteInfozData needed for rewrite, this includes example inputs, pattern and replacement functions and post transformation functions for the exported pattern and replacement GraphModule .example_inputspattern replacementNpattern_post_transreplacement_post_transr filter_fnFignore_literalsr port_metadata_fn)__name__ __module__ __qualname____doc__tupler__annotations__rrrr rrrqrrGraphboolrrr;r9rrs #s(O### IM;-*D!EFMMMMQHX{m[.H%IJQQQ x XGM NO   "OT!!!GKhx);(t.t.t|tt@ttBt|ttDttFt|ttHttJtLtLt| ttNttPtLtLg } tS|tU5| D]} | j+}| j,}| j-}| j.}| j/}ta||}tS|ta||}tS||r ||}|r ||}|1|1te|||| j3| j4}| j5r|D]}| 5| dddn #1swxYwY|S)NrVrW)rYrrPrX)rri)rXrrrr)rX rZr[)rrr)rXrXrrY)r)rXrrT)rrrrl)r) match_filtersr)6rrandintr!randnrrpinttensorfinfor_epsrrrMrTrrrirrrr:rIrrrrrrrrrrrrrrrrrrrrrrrrrr recompilerrrr)r _QUANTIZED_LINEAR_EXAMPLE_INPUTS(_DYNAMIC_QUANTIZED_LINEAR_EXAMPLE_INPUTS _QUANTIZED_CONV2d_EXAMPLE_INPUTS)_QUANTIZED_ADD_OR_ADD_RELU_EXAMPLE_INPUTS$_QUANTIZED_MAX_POOL2D_EXAMPLE_INPUTS(_QUANTIZE_PER_TENSOR_INT8_EXAMPLE_INPUTS*_DEQUANTIZE_PER_TENSOR_INT8_EXAMPLE_INPUTS)_QUANTIZE_PER_CHANNEL_INT8_EXAMPLE_INPUTS+_DEQUANTIZE_PER_CHANNEL_INT8_EXAMPLE_INPUTS9_DYNAMIC_QUANTIZED_LINEAR_4BIT_GROUPWISE_EXAMPLE_INPUTS_19_DYNAMIC_QUANTIZED_LINEAR_4BIT_GROUPWISE_EXAMPLE_INPUTS_2_REWRITE_INFO_LIST rewrite_inforrrrrmatchesms r9rrs  dCuz::: AU[))) AUY''' dV59--- cU%),,, dCuz::: AU[))) AUY''' dV59--- cU%),,, AU[))) AU[))) AUY''' dV59--- cU%),,,($&  F%+...   EM""& dCuz::: AU[))) AUY''' dV59--- cU%),,, AU[))) 0,  dCUZ@@@ AU[))) AUY''' dV59--- cU%),,, dCUZ@@@ AU[))) AUY''' dV59--- cU%),,, AU[))) AU[))) AUY''' dV59--- cU%),,,($&  dCUZ@@@ AU[))) AUY''' dCUZ@@@ AU[))) AUY''' AU[))) AUY''' dV59--- cU%),,, 1-  dCUZ@@@ AU[))) AUY''' dV59--- cU%),,, AU[))) AUY''' dV59--- cU%),,, ,(  Aq!Qek222 AU[))) AUY''' dV59--- cU%),,, 0,  dCUZ@@@ AU[))) AUY''' dV59--- cU%),,, 2.  Aq!Qek222 AU[))) AUY'''   1-  dCUZ@@@ AU[))) AUY'''   3/  G5;/// EM""& b!WEJ777 Aq ,,, q      AU[)))  A=  Jek222 EM""& b!WEJ777 Aq ,,, q      AU[)))  A=  4 7 8 8 = > > <#'CEK 4N4N4RTU"V    <#'CEK 4N4N4RTU"V      E F G G L M M <K ..2AA#    <K ..2AA#   NN W)   ,  E F G G L M M <K ..2AA#    <K ..2AA#   NN W)   ,  , / 0 0 5 6 6 3 3     , / 0 0 5 6 6 ;rd S S S ;rd S S S     5 1 2 2 7 8 8   5 , - - 2 3 3   0 3 4 4 9 : : 3 3     4 3 4 4 = > >   6 5 6 6 ? @ @   5 4 5 5 > ? ? 3 3     7 6 7 7 @ A A 3 3   Uqf'u--- - / /55. 5 5L)8N"*G&2K!-!@ %1%H "8.QQG .w 7 7 7<^K /{ ; ; ;! 6,,W55% B44[AA        ! ! # # #2*4 , < G, 5 55A 11!44447 5555555555555555< LsC(A@ @ A@@A@)L contextlib dataclassesr functoolsrtypingrrrrr!torch._higher_order_ops.out_dtyper $torch.ao.quantization.fx._decomposedr torch.fxr 6torch.fx.passes.utils.matcher_with_name_node_map_utilsr torch.fx.subgraph_rewriterr r&torchao.quantization.pt2e.export_utilsrtorchao.quantization.pt2e.utilsrrrr%torchao.quantization.quant_primitivesrtorchao.quantization.utilsr torchao.utilsrtorch._export.utilsr nullcontextlibraryLibrary quant_libregister_custom_op__all__r:rIrMrTrirBrrrrr rrrrrrrrrrrrrrrrrrrrrrrr;r9r?sd!!!!!!000000000000 777777IIIIII PPPPPPUUUUUUUU@@@@@@ >=====@@@@@@------BHHHHHHHB+5+A((( M ! !)Z 8 8 ((33'    F222jB,,,^>>>B~$ L~$ ~$|~$, ~$ | ~$  % ~$S ~$ \~$~$~$~$B00 00006 )    8000fOOOd4"""J2&&&R88        S S S"""   0D  LLLLLLL L&YKYKYYYYYYs &A-- A8