PiEddlZddlZddlmZddlmZddlmZmZm Z m Z ddl Z ddl m Z ddlmZmZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZGd deeZeGdde Z ee de j!j"de de j!j"fdZ#eGdde Z$Gdde$Z%ee$de j!j"de$de j!j"fdZ&eGdde Z'Gdde'Z(ee'de j!j"de'de j!j"fdZ)Gd d!eZ*d"e j!j"d#e ed$fddfd%Z+dS)&N) dataclass)Enum)AnyListOptionalTuple) AOBaseConfig)_QUANTIZE_CONFIG_HANDLER register_quantize_module_handler)TwoStepQuantizer)FakeQuantizedEmbedding)FakeQuantizeConfigFakeQuantizeConfigBaseIntxFakeQuantizeConfig_infer_fake_quantize_configs)FakeQuantizedLinear_log_deprecation_warningceZdZdZdZdZdS)QATStepzZ Enum value for the `step` field in :class:`~torchao.quantization.qat.QATConfig`. prepareconvertN)__name__ __module__ __qualname____doc__PREPARECONVERTp/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/quantization/qat/api.pyrr!s$GGGGr!rc eZdZUdZeeed<eeed<eeed<eed< d dddeedeedeedefd Z d Z dS) QATConfigaV Config for applying quantization-aware training (QAT) to a `torch.nn.Module`, to be used with :func:`~torchao.quantization.quant_api.quantize_`. This config has two steps, "prepare" and "convert". The prepare step applies "fake" quantization to the model and should be applied before training, while the convert step converts the model into an actual quantized model. Fake quantization here refers to simulating the quantization numerics (e.g. int4) using high precision arithmetic (e.g. bf16), with the goal of reducing eventual degradation from quantization. There are two ways to use this config. The first involves passing a base post-training quantization (PTQ) config, which we will use to automatically infer the corresponding fake quantization schemes to use in the prepare phase. In the convert phase, we will then apply the base PTQ config to the model. This will be the most common use case. Example usage:: from torchao.quantization import ( quantize_, Int8DynamicActivationInt4WeightConfig, ) from torchao.quantization.qat import QATConfig base_config = Int8DynamicActivationInt4WeightConfig(group_size=32) quantize_(model, QATConfig(base_config, step="prepare")) train_loop(model) quantize_(model, QATConfig(base_config, step="convert")) Currently only the following are supported as base configs: - :class:`~torchao.quantization.Int8DynamicActivationInt4WeightConfig` - :class:`~torchao.quantization.Int4WeightOnlyConfig` The second way to use this config involves specifying the fake quantization schemes directly. Users will pass in :class:`~torchao.quantization.qat.FakeQuantizeConfigBase` for weights and/or activations instead of the base PTQ config. This use case is mostly for experimentation, e.g. when the corresponding PTQ config does not exist yet. Example usage:: from torchao.quantization import quantize_ from torchao.quantization.qat import IntxFakeQuantizeConfig activation_config = IntxFakeQuantizeConfig( torch.int8, "per_token", is_symmetric=False, ) weight_config = IntxFakeQuantizeConfig( torch.int4, group_size=32, is_symmetric=True, ) qat_config = QATConfig( # must specify one of `base_config` or `weight_config` activation_config=act_config, weight_config=weight_config, step="prepare", ) quantize_(model, qat_config) Args: base_config (Optional[AOBaseConfig]): Base PTQ config to infer the fake quantization configs during the prepare phase, and to apply directly during the convert phase. activation_config (Optional[FakeQuantizeConfigBase]): Custom fake quantization config for input activations, always optional. Must be None if `base_config` is used. weight_config (Optional[FakeQuantizeConfigBase]): Custom fake quantization config for weights. Must be None if `base_config` is used. Keyword args: step (str): One of "prepare" or "convert", determines the QAT phase Raises: ValueError: If `base_config` and `activation_config` are both specified ValueError: If `base_config` and `weight_config` are both specified ValueError: If none of `base_config`, `activation_config`, or `weight_config` are specified ValueError: If either `activation_config` or `weight_config` is specified and `step` is "convert" ValueError: If `step` is not one of "prepare" or "convert" ValueError: If the config is applied on a module that is not a `torch.nn.Linear` or `torch.nn.Embedding`, or it is applied on `torch.nn.Embedding` with an activation config base_configactivation_config weight_configstepNr)r(cf||_||_||_||_|dSN)r%r&r'r( __post_init__)selfr%r&r'r(s r"__init__zQATConfig.__init__s;'!2*  r!ctjd|j|_dt D}|j|vrt d||j|jt d|j|j t d|jt j kr0t|j|j|j fst d|jt j kr|j|j t dt|jtr)|jjj}t |d|d |d dS) Nz"torchao.quantization.qat.QATConfigcg|] }|j Sr )value).0ss r" z+QATConfig.__post_init__..s444q17444r!z`step` must be one of z9Cannot specify both `base_config` and `activation_config`z5Cannot specify both `base_config` and `weight_config`zWMust specify `base_config`, `activation_config`, or `weight_config` in the prepare stepzICannot specify `weight_config` or `activation_config` in the convert stepz~ was passed as `base_config`. Did you mean to do the following instead? qat_config = QATConfig( activation_config=z(...), weight_config=z$(...), step="prepare", ))torch_C_log_api_usage_oncer(lowerr ValueErrorr%r&r'ranyr isinstancer __class__r)r,all_step_values config_types r"r+zQATConfig.__post_init__s $$%IJJJIOO%% 44G444 9O + +GoGGHH H   'D,B,NK    'D,>,JTUU U 9 ' '  t5t7I J1 1 'i  9 ' '  " .$2D2P[  d&(> ? ? *4=K-8*5   r!)NNN) rrrrrr __annotations__rrr-r+r r!r"r$r$*sTTl,'''' 6777723333 MMM /3>B:>  "   l+ $$:;  67      """""r!r$moduleconfigreturnc^ddlm}m}|j}|j}|t jkr|t|\}}n|j}|j }t|tj j rQt||r+|t||sJ||||Stj|||St|tj jr&|t#dt%j||St#dt)|z|t jks Jd|z|j Jd|j Jdt|tt$|fs|Si}d } t-|d rPt|jjt2r1|jjjr |jj|d <|jj|d <d } t|t|fr|}n=t|t$r|}nt#d|d|b| r?t-|dr/t?j dtCj"|}d|_#tIt)|||fi|S|S)a| During the prepare step, perform module swap to apply fake quantization. If the base PTQ config is specified, derive the fake quantization configs from it. During the convert step, first perform module swap to revert all fake quantized modules to the corresponding built-in `torch.nn.Module`s, then apply the base config directly to quantize the module. r)NVFP4FakeQuantizeConfigNVFP4FakeQuantizedLinearN;Activation fake quantization is not supported for embedding-Module of type '%s' does not have QAT supportz!unexpected step '%s' in QATConfigzunexpected `activation_config`zunexpected `weight_config`Fweight_fake_quantizer custom_scalecustom_zero_pointTzEncountered unexpected module z, should never happenintx_choose_qparams_algorithmz'Disabling intx_choose_qparams_algorithm)%torchao.prototype.qatrCrDr%r(rrrr&r'r:r4nnLinear from_linearr Embeddingr8rfrom_embeddingtyperhasattrrGr@rrange_learningscale zero_point to_linear to_embeddingloggingdebugcopydeepcopyrJr ) r?r@rCrDr%r( act_configr'kwargshas_custom_scale_and_zero_points r"_qat_config_transformr_sW$K ;D w  "*F{*S*S 'Z1J"0M feho . . -)@AA !)Z 7..))0;;J +6J  2 3 3 % Q*8OO O?$v,,N  w&&&(Kd(R&&&'//1Q///#++-I+++  "8:R S   M*/' F3 4 4 367>@VWW 3,3B 3 &,%A%GF> "*0*F*QF& '.2 + f24LM N N %%''FF  6 7 7 ((**FFNNNN   ". A7<44 A GHHH"mK88 <@ 9+D,=,=> '- Mr!cNeZdZUdZdZeeed<dZeeed<dZ dS)#IntXQuantizationAwareTrainingConfiga (Deprecated) Please use :class:`~torchao.quantization.qat.QATConfig` instead. Config for applying fake quantization to a `torch.nn.Module`. to be used with :func:`~torchao.quantization.quant_api.quantize_`. Example usage:: from torchao.quantization import quantize_ from torchao.quantization.qat import IntxFakeQuantizeConfig activation_config = IntxFakeQuantizeConfig( torch.int8, "per_token", is_symmetric=False, ) weight_config = IntxFakeQuantizeConfig( torch.int4, group_size=32, is_symmetric=True, ) quantize_( model, IntXQuantizationAwareTrainingConfig(activation_config, weight_config), ) Note: If the config is applied on a module that is not `torch.nn.Linear` or `torch.nn.Embedding`, or it is applied on `torch.nn.Embedding` with an activation config, then we will raise ValueError as these are not supported. Nr&r'c$t|dSr*rr,s r"r+z1IntXQuantizationAwareTrainingConfig.__post_init__F &&&&&r!) rrrrr&rrr>r'r+r r!r"rara&s^6;?x 67>>>6:M823:::'''''r!raceZdZdS) intx_quantization_aware_trainingNrrrr r!r"rfrfKDr!rfcT|}|j}|j}t|tjjrt j|||St|tjjr&|tdtj ||Stdt|z)NrErF) r&r'r:r4rLrMrrNrOr8rrPrQ)r?r@modr&r's r"+_intx_quantization_aware_training_transformrkOs C0(M#ux'' V".      C+ , ,V  (M &4S-HHHH4PS99TUUUr!ceZdZdZdZdS)'FromIntXQuantizationAwareTrainingConfiga (Deprecated) Please use :class:`~torchao.quantization.qat.QATConfig` instead. Config for converting a model with fake quantized modules, such as :func:`~torchao.quantization.qat.linear.FakeQuantizedLinear` and :func:`~torchao.quantization.qat.linear.FakeQuantizedEmbedding`, back to model with the original, corresponding modules without fake quantization. This should be used with :func:`~torchao.quantization.quant_api.quantize_`. Example usage:: from torchao.quantization import quantize_ quantize_( model_with_fake_quantized_linears, FromIntXQuantizationAwareTrainingConfig(), ) c$t|dSr*rrcs r"r+z5FromIntXQuantizationAwareTrainingConfig.__post_init__}rdr!N)rrrrr+r r!r"rmrmhs-&'''''r!rmceZdZdS)%from_intx_quantization_aware_trainingNrgr r!r"rprprhr!rprjct|tr|St|tr|S|S)z If the given module is a fake quantized module, return the original corresponding version of the module without fake quantization. )r:rrVrrW)rjr@s r"0_from_intx_quantization_aware_training_transformrrsO#*++}} C/ 0 0!!! r!ceZdZdZdeefdZdejj de de dejj fdZ dejj de de dejj fd Z d S) ComposableQATQuantizerab Composable quantizer that users can use to apply multiple QAT quantizers easily. Quantizers will be applied in the order they are specified in the constructor. Note: the quantizers provided must apply to different modules in the model, e.g. nn.Linear and nn.Embedding, otherwise the behavior will be undefined. Example usage:: my_quantizer = ComposableQATQuantizer([ QATQuantizer1(), QATQuantizer2(), QATQuantizer3(), ]) model = my_quantizer.prepare(model) train(model) model = my_quantizer.convert(model) quantizerscRtjd||_dS)Nz/torchao.quantization.qat.ComposableQATQuantizer)r4r5r6ru)r,rus r"r-zComposableQATQuantizer.__init__s$ $$%VWWW$r!modelargsr]rAcD|jD]}||}|Sr*)rurr,rwrxr] quantizers r"rzComposableQATQuantizer.prepare/ - -I%%e,,EE r!cD|jD]}||}|Sr*)rurrzs r"rzComposableQATQuantizer.convertr|r!N) rrrrrr r-r4rLModulerrrr r!r"rtrts&%4(8#9%%%%X_-0._set_initializeds+ a* + + "!ANNN " "r!N)r4r5r6'torchao.quantization.qat.fake_quantizerrrLr~apply)rwrrrs @r"initialize_fake_quantizersrs H  !VWWWJIIIII"EHO"""""" KK !!! E>r!),rZrX dataclassesrenumrtypingrrrrr4torchao.core.configr %torchao.quantization.transform_moduler r torchao.quantization.unifiedr embeddingrfake_quantize_configrrrrlinearrutilsrstrrr$rLr~r_rarfrkrmrprrrtrr r!r"rs !!!!!!------------ ,,,,,,:99999------ ('''''++++++c4 NNNNN NN Nb"!),,f HOf f X_fff-,fR  ' ' ' ' ', ' '  'H     'J   "!"EFFV HOV /V X_VVVGFV0 '''''l'' '2     ,S   "!"IJJ    3  X_   KJ  $$$$$-$$$N 8?#s(O r!