Pi[/ JddlZddlmZedrddlmZddlmZddlmZmZddl m Z dd l m Z m Z dd lmZmZmZdd lmZmZmZdd lmZdd lmZddlmZgdZdedeeejjjjjfdefdZdedeeejjjjjfdefdZej j!j"j#ej j!j"j$ej j!j%j#ej j&j'gZ(dede)fdZ*dZ+ ddede)de)defdZ,dS)N)torch_version_at_least2.7.0) constant_fold)Union) GraphModuleNode) PassManager)_fold_conv_bn_qat_fuse_conv_bn_qat)DuplicateDQPassPortNodeMetaForQDQ Quantizer)_disallow_eval_train_fuse_conv_bn__get_node_name_to_scope)#_convert_to_reference_decomposed_fx)prepare) reference_representation_rewrite) prepare_pt2eprepare_qat_pt2e convert_pt2emodel quantizerreturnct|tjjjjjrddlm}|||Stj d|j }t|}t|| |}||||t!||d|j}|j |t'|}|S)aPrepare a model for post training quantization Args: * `model` (torch.fx.GraphModule): a model captured by `torch.export.export` API. * `quantizer`: A backend specific quantizer that conveys how user want the model to be quantized. Tutorial for how to write a quantizer can be found here: https://pytorch.org/tutorials/prototype/pt2e_quantizer.html Return: A GraphModule with observer (based on quantizer annotation), ready for calibration Example:: import torch from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e from torchao.quantization.pt2e.quantizer import ( XNNPACKQuantizer, get_symmetric_quantization_config, ) class M(torch.nn.Module): def __init__(self) -> None: super().__init__() self.linear = torch.nn.Linear(5, 10) def forward(self, x): return self.linear(x) # initialize a floating point model float_model = M().eval() # define calibration function def calibrate(model, data_loader): model.eval() with torch.no_grad(): for image, target in data_loader: model(image) # Step 1. program capture # NOTE: this API will be updated to torch.export API in the future, but the captured # result shoud mostly stay the same m = torch.export.export(m, *example_inputs).module() # we get a model with aten ops # Step 2. quantization # backend developer will write their own Quantizer and expose methods to allow # users to express how they # want the model to be quantized quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config()) m = prepare_pt2e(m, quantizer) # run calibration # calibrate(m, sample_inference_data) r)rz&torchao.quantization.pt2e.prepare_pt2eFis_qatobs_or_fq_callback) isinstancetorchao quantizationrr#torch.ao.quantization.quantize_pt2er_C_log_api_usage_oncemetarrtransform_for_annotationannotatevalidaterprepare_obs_or_fq_callbackupdater)rrtorch_prepare_pt2eoriginal_graph_metanode_name_to_scopes {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/quantization/pt2e/quantize_pt2e.pyrr*sx)UX2<FPQQ4      "!%333 H  !IJJJ*0775  . .u 5 5E u u  $?    E  J)***  ' 'E Lct|tjjjjjrddlm}|||Stj d|j }t|}| |}| |||t|t!||d|j}|j |t'|}|S)a<Prepare a model for quantization aware training Args: * `model` (torch.fx.GraphModule): see :func:`~torchao.quantization.pt2e.quantize_pt2e.prepare_pt2e` * `quantizer`: see :func:`~torchao.quantization.pt2e.quantize_pt2e.prepare_pt2e` Return: A GraphModule with fake quant modules (based on quantizer annotation), ready for quantization aware training Example:: import torch from torchao.quantization.pt2e.quantize_pt2e import prepare_qat_pt2e from torchao.quantization.pt2e.quantizer import ( XNNPACKQuantizer, get_symmetric_quantization_config, ) class M(torch.nn.Module): def __init__(self) -> None: super().__init__() self.linear = torch.nn.Linear(5, 10) def forward(self, x): return self.linear(x) # initialize a floating point model float_model = M().eval() # define the training loop for quantization aware training def train_loop(model, train_data): model.train() for image, target in data_loader: ... # Step 1. program capture # NOTE: this API will be updated to torch.export API in the future, but the captured # result shoud mostly stay the same m = torch.export.export(m, *example_inputs).module() # we get a model with aten ops # Step 2. quantization # backend developer will write their own Quantizer and expose methods to allow # users to express how they # want the model to be quantized quantizer = XNNPACKQuantizer().set_global(get_symmetric_quantization_config()) m = prepare_qat_pt2e(m, quantizer) # run quantization aware training train_loop(prepared_model, train_loop) r)rz*torchao.quantization.pt2e.prepare_qat_pt2eTr)r r!r"r#rrr$rr%r&r'rr(r)r*r rr+r,r)rrtorch_prepare_qat_pt2er.r/s r0rrst)UX2<FPQQ8      &%eY777 H  !MNNN*077  . .u 5 5E u ue  $?    E  J)***  ' 'E Lr1nc4|jdko |jtvS)aTIf there is any pure ops between get_attr and quantize op they will be const propagated e.g. get_attr(weight) -> transpose -> quantize -> dequantize* (Note: dequantize op is not going to be constant propagated) This filter is added because we don't want to constant fold the things that are not related to quantization call_function)optarget _QUANT_OPS)r4s r0_quant_node_constraintr:s 4? " =qx:'==r1cddlm}ddlm}ddlm}d}d}|D]\}}t |tjj j jsRt |tjj j js)t |tjj j jrd}t ||s t ||st ||rd}|r |r Jd|r |r Jd|S)Nr) FakeQuantize)AffineQuantizedObserverBase) ObserverBaseFTz2Cannot be prepared using both torch.ao and torchao) 'torchao.quantization.pt2e.fake_quantizer<"torchao.quantization.pt2e.observerr=r> named_modulesr r!r"r# fake_quantizeobserver)rtorchao_FakeQuantize#torchao_AffineQuantizedObserverBasetorchao_ObserverBaseis_torch_ao_preparedis_torchao_prepared_ms r01_is_torchao_prepared_do_not_use_outside_this_filerKswXWWWWW ##%% ' '1 q%(/=J K K (!UX2;HII (!UX2;WXX ( $( q. / / '!122 '!@AA ' #'  &  @  & '  @  ' r1FTuse_reference_representation fold_quantizect|sddlm}||||Stjdt |tstd|d|j }t|}t|}ttg}||j}ttg}||j}|r$t!drt#|t$|rt'|}|j |t+|}|S)aConvert a calibrated/trained model to a quantized model Args: * `model` (torch.fx.GraphModule): calibrated/trained model * `use_reference_representation` (bool): boolean flag to indicate whether to produce referece representation or not * `fold_quantize` (bool): boolean flag for whether fold the quantize op or not Returns: quantized model, either in q/dq representation or reference representation Example:: # prepared_model: the model produced by `prepare_pt2e`/`prepare_qat_pt2e` and calibration/training # `convert_pt2e` produces a quantized model that represents quantized computation with # quantize dequantize ops and fp32 ops by default. # Please refer to # https://pytorch.org/tutorials/prototype/pt2e_quant_ptq_static.html#convert-the-calibrated-model-to-a-quantized-model # for detailed explanation of output quantized model quantized_model = convert_pt2e(prepared_model) r)rz&torchao.quantization.pt2e.convert_pt2ezjUnexpected argument type for `use_reference_representation`, please make sure you intend to pass argument z to convert_pt2er)rKr$rr!r%r&r bool ValueErrorr'rr r r graph_modulerrrr:rr,r)rrLrMtorch_convert_pt2er.pms r0rrsz8 =U C CV      "!%)E}UUU H  !IJJJ 2D 9 9  krfs 000000'""-,,,,,,&&&&&&&&::::::TTTTTTTT  988888NNNNNN   U UY 5 ? I SSTUUUUUpS SY 5 ? I SSTSSSSSn I"6> I"6= I"7? I%  >d>t>>>>"""N*/;; ;"&;; ;;;;;;r1