Pi' dZddlZddlmcmZddlmZmZgdZ ddZ Gdd ejj Z Gd d e ejj Zejj eejjjjeiZ d ddZdddZdZdS)a( Testing out accuracy-only implementation of SmoothQuant (https://arxiv.org/pdf/2211.10438.pdf) Note: this is an application of input-weight equalization, with the addition that the multiplication by scale is fused into the preceding layer, specifically for relevant parts of transformer blocks. N)$_quant_int8_dynamic_per_token_linear dynamically_quantize_per_channel) get_scaleSmoothFakeDynQuantMixin$SmoothFakeDynamicallyQuantizedLinear!swap_linear_with_smooth_fq_linearsmooth_fq_linear_to_inferenceset_smooth_fq_attribute?ctj||}tj|d|z }||z }|dS)a Calculate the scale based on abs(max(X)), abs(max(W)), and alpha. Args: X_absmax (torch.Tensor): Absolute maximum values of the input tensor X. W_absmax (torch.Tensor): Absolute maximum values of the weight tensor W. alpha (float, optional): Scaling factor. Defaults to 0.5. Returns: torch.Tensor: The calculated scale of dimension `k` if X is of dimension `b*n*k` and W is of dimension `k*m`. g?)torchpowreshape)X_absmaxW_absmaxalphaX_powW_powdivs t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/quantization/smoothquant.pyrr!sC Ih & &E Ihe , ,E %-C ;;r??c2eZdZdZdZdZdZdZdZdS)rcxd|_d|_|dd||_d|_d|_dS)NT smooth_scaleF) calibratingx_running_abs_maxregister_bufferrdebug_skip_scalingstore_w_int_repr_t)selfrs rinit_smoothquant_variablesz2SmoothFakeDynQuantMixin.init_smoothquant_variables4sG!% ^T222 "'#(rcttt|jdz }t jt j||}|j ||_dSt j||j|_dS)Nrdim) tuplerangelenshaperamaxabsrmax)r"Xall_dims_except_last cur_abs_maxs rupdate_x_running_abs_maxz0SmoothFakeDynQuantMixin.update_x_running_abs_maxHsv$U3qw<r<r!rr7r9r4)r"r;_W_zpss r fold_weightz#SmoothFakeDynQuantMixin.fold_weightms-1,G,G,I,I) DM6  " H  z/C/CAq/I/I/T/T/V/V W W W W  z/D/D/F/F G G G KKKrct)z Sets `self.x_running_abs_max` to a value which will lead to smooth scale of all ones if `alpha=0.5`, to enable performance benchmarking without calibration. rArCs rset_debug_x_absmaxz*SmoothFakeDynQuantMixin.set_debug_x_absmaxys "###rN) __name__ __module__ __qualname__r#r1r>rDrGrIrrrr3sq((((TTT+++2$$$   $$$$$rrcLeZdZdZfdZdZeddZdZdZ xZ S) rz This is a replacement for `torch.nn.Linear` which implements dynamic per-token activation quantization and dynamic per-channel weight quantization based on Smoothquant scaling. c|d}tj|i|||dS)Nr)popsuper__init__r#)r"argskwargsr __class__s rrRz-SmoothFakeDynamicallyQuantizedLinear.__init__sJ 7##$)&))) ''.....rc4|jr6||tj||j|j}nZ|js ||jz }|jr|j n|j }t|||j |j|j }|Sr@)rr1Flinearr4biasr rr!r;trr<dtype)r"r.rSrTY W_int_repr_ts rforwardz,SmoothFakeDynamicallyQuantizedLinear.forwards     ) )! , , ,DK33AA* *))#'#:S@Q@Q@S@S 5< 17Arr cd\}}||||jdu|}|j|_|j|_|j|_|j|_t |j}|||S)z Converts a `mod` of class `torch.nn.Linear` to the smooth fake quantized version of it. Note: requires calibration. )r`N)rYr)rY in_features out_featuresr4next parametersdeviceto)clsmodrfake_in_featuresfake_out_featuresnew_mod device_to_uses r from_floatz/SmoothFakeDynamicallyQuantizedLinear.from_floats/3++# /chd6JRW   "o"/x S^^--..5  =!!!rc (|j Jdd|_t|jtjtj|jdddj|j |_ | dS)zl Calculates the smoothquant scale based on calibration in preparation for inference Nzno calibration data foundFrrr%r) rrrrr-r,r4r7valuesrrrGrCs rrDz1SmoothFakeDynamicallyQuantizedLinear.to_inferences %113N111 %  " Iei 5 5a ; ;<J"--e5-AAI E4 + + + + -e]GU S S S S T TrFc|D]c\}}t|ttr*|r||ddS)a Prepares the model for inference by calculating the smoothquant scale for each SmoothFakeDynamicallyQuantizedLinear layer. Args: model (torch.nn.Module): The model containing SmoothFakeDynamicallyQuantizedLinear layers. debug_skip_calibration (bool, optional): If True, sets the running maximum of activations to a debug value for performance benchmarking. Defaults to False. Returns: None N) named_modules isinstancer'rrprIrD)rdebug_skip_calibration_rhs rr r s%%''3 c5!9!@!@!B!BCC D D % )&&(((       rc|D]Z\}}t|ttr!t ||rt |||[dSr@)rrr'rrphasattrr)rattribute_namenew_attribute_valrrhs rr r s~%%''@@3 c5!9!@!@!B!BCC D D @sN++ @^->???@@rrs)Nrwr )rxN)F)rtrtorch.nn.functionalnn functionalrWutilsrr__all__rModulerLinearrmodulesrXNonDynamicallyQuantizableLinearrr r r rMrrrsv     $L$L$L$L$L$ehoL$L$L$^B*B*B*B*B*+BEHOB*B*B*T HO9 H;=a25T TTTT@*@@@@@r