Pi= ddlmZmZddlmZddlmZmZ ddlm Z n#e $r ddlm Z YnwxYwddl m Z mZmZ ddlmZmZddlmZmZmZmZn#e $rdd lmZmZmZmZmZmZYnwxYwgd Zd Z dd l mZn #e $rd ZYnwxYwiZiZiZGddZdee<dee<eed<eed<GddZ dee <dee<eed<eed<GddeZ!eZ"eZ#dee!<e"ed<e#ed<GddeZ$eZ%eZ&dee$<e%ed<e&ed<deedee'fdZ(de'defd Z)de'defd!Z* d(d#ej+d$ed%d&ed%dd"fd'Z,d"S)))CallableOptional)nn) LoRALinear QATLoRALinear)TensorCoreTiledLayout)TensorCoreTiledLayoutType)int4_weight_only#int8_dynamic_activation_int4_weight quantize_)Int4WeightOnlyQATQuantizer Int8DynActInt4WeightQATQuantizer)disable_4w_fake_quantdisable_8da4w_fake_quantenable_4w_fake_quantenable_8da4w_fake_quant)rrrrr r)get_quantizer_modeInt4WeightOnlyQuantizerr $Int4WeightOnlyQATQuantizerModuleSwapInt8DynActInt4WeightQuantizerr*Int8DynActInt4WeightQATQuantizerModuleSwapT)qatFc&eZdZdZddefdZdZdS)rz Quantizer for applying int8 per token dynamic activation + int4 per group weight quantization to linear layers in the model.  groupsizec||_dSN)r)selfrs s/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/training/quantization.py__init__z&Int8DynActInt4WeightQuantizer.__init__Ss "cNt|j}t|||Sr)r rr )rmodel quantize_fns rquantizez&Int8DynActInt4WeightQuantizer.quantizeVs&9$.II %%%% r!N)r__name__ __module__ __qualname____doc__intr r%r!rrrMsM #######r!r8da4wz 8da4w-qatc*eZdZdZd dedefdZdZdS) rz Quantizer for applying int4 per group weight only quantization to linear layers in the model using the efficient tinygemm kernel. r inner_k_tilesc"||_||_dSr)rr1)rrr1s rr z Int4WeightOnlyQuantizer.__init__ms"*r!cxt|j}t|j|}t |||Sr)rr1r rr )rr# layout_typer$s rr%z Int4WeightOnlyQuantizer.quantizeqs8+D,>?? &t~{CC %%%% r!N)r/r0r&r,r!rrrgsT ++#+C++++r!r4wz4w-qatceZdZdS)rNr'r(r)r,r!rrrDr!rz4w-qat-module-swapceZdZdS)rNr7r,r!rrrr8r!rz8da4w-qat-module-swap quantizerreturncRtt|dS)aGiven a quantizer object, returns a string that specifies the type of quantization. For example, in the case of int4 weight only quantization, we'll return "4w". If the quantizer is not recognized as a known quantizer, we'll return None. Currently supported: - :class:`~torchtune.training.quantization.Int8DynActInt4WeightQuantizer`: "8da4w" - :class:`~torchtune.training.quantization.Int4WeightOnlyQuantizer`: "4w" - :class:`~torchao.quantization.qat.Int8DynActInt4WeightQATQuantizer`: "8da4w-qat" - :class:`~torchao.quantization.qat.Int4WeightOnlyQATQuantizer`: "4w-qat" Args: quantizer (Optional[Callable]): A callable object that implements the `quantize` method. Returns: Optional[str]: The quantization mode. N)_quantizer_to_modegettype)r:s rrrs &  ! !$y//4 8 88r!quantizer_modec8t|dS)zGiven a quantizer mode, return the corresponding function for disabling fake quantize in a model prepared by the quantizer. If the quantizer is not recognized as a known QAT quantizer, return None. N)%_quantizer_mode_to_disable_fake_quantr>r@s r_get_disable_fake_quantrDs 1 4 4^T J JJr!c8t|dS)zGiven a quantizer mode, return the corresponding function for enabling fake quantize in a model prepared by the quantizer. If the quantizer is not recognized as a known QAT quantizer, return None. N)$_quantizer_mode_to_enable_fake_quantr>rCs r_get_enable_fake_quantrGs 0 3 3ND I IIr!Nmoduleactivation_qat_configFakeQuantizeConfigweight_qat_configc|D]S\}}t|tr(tj|||}t |||Bt |||TdS)a` Swap all `LoRALinear` in the model with `QATLoRALinear`. This is used for combining QAT + LoRA during finetuning. The resulting linear layers will apply the following transformation instead: x -> fake_quantize(W_frozen) @ fake_quantize(x) + BAx Fake quantization here refers to simulating the quantization numerics without actual dtype casting, with the goal of providing improved accuracies when the model is ultimately quantized after finetuning. Args: module (nn.Module): The model to swap linear layers on activation_qat_config (Optional[FakeQuantizeConfig]): The config for specifying how to fake quantize input activations in the base linear layer weight_qat_config (Optional[FakeQuantizeConfig]): The config for specifying how to fake quantize base linear weights N)named_children isinstancerrfrom_lora_linearsetattrswap_lora_linear_with_qat)rHrIrKnamechild new_linears rrQrQs4,,..   e eZ ( ( &7%!J FD* - - - - %%!      r!)NN)-typingrrtorchrtorchtune.modules.peft.lorarrtorchao.dtypesr ImportErrorr torchao.quantizationr r r torchao.quantization.qatr rtorchao.quantization.qat.linearrrrr"torchao.quantization.prototype.qat__all___torchao_0_7_supportedrr=rBrFrrr!disable_4w_fake_quant_module_swap enable_4w_fake_quant_module_swapr$disable_8da4w_fake_quant_module_swap#enable_8da4w_fake_quant_module_swapstrrrDrGModulerQr,r!rrfs~&%%%%%%%AAAAAAAAR4444444RRRQQQQQQQQR         #(((((((###"#(*%')$        5<017B345M%k24K$[1"/3*+19-.2G%h/1E$X.     +E   %:!#7 ;O78&& %%      1Q   (@$&=#AX=>)& (% 9(8"49#9999,KCKHKKKKJ3J8JJJJ=A8< '' I'$$89 '  45 '  ''''''s/ --AA('A(2A99BB