Pi8.ddlZddlmZmZmZddlZddlmcmZ ddlmZddl m Z m Z ddl mZddlmZGddejeZGd d eZd ejd dfd Zd ejd dfdZdS)N)ListOptionalUnion)nn) linear_nf4to_nf4)_register_nf4_dispatch_ops) AdapterModuleceZdZdZ ddededededed ed effd Zd d dee e e j efdefdZ dZdee fdZde jde jfdZxZS) LoRALinearaLoRA linear layer as introduced in `LoRA: Low-Rank Adaptation of Large Language Models `_. LoRA perturbs a given layer via a low-rank approximation where only the rank decomposition matrices are trainable. In a linear layer instead of :math:`x \mapsto W_0x` a LoRALinear layer is defined as :math:`x \mapsto W_0x + (\alpha / r)BAx`, where :math:`r` is the rank of the matrices :math:`A` and :math:`B` and :math:`\alpha` is a scaling factor. As in the original implementation, we support dropout before multiplication by the low-rank matrices. Args: in_dim (int): input dimension out_dim (int): output dimension rank (int): rank of the low-rank approximation alpha (float): scaling factor for the low-rank approximation dropout (float): dropout probability. Default: 0.0 use_bias (bool): whether to include bias in the original linear layer. Default: False quantize_base (bool): Whether to quantize base linear weight or not. Default: False **quantization_kwargs: Keyword arguments to pass to `to_nf4` when quantizing the base linear weight. Examples of valid arguments are `block_size` and `scaler_block_size`, which control the granularity of weight quantization and scaler quantization respectively. This is only used if `quantize_base` is True. Default None Raises: ValueError: If ``quantize_base`` is False, but quantization kwargs are provided. Fin_dimout_dimrankalphadropoutuse_bias quantize_basec ~t||_||_||_||_||_||_|js=td| Drtd|tj |||j} |js| j nt| j fi|} |jr| jnd} d|_|dtj| |d| tj| nd|dkrtj|ntj|_tj ||d|_tj ||d|_d|_|dS) Ncg|]}|Sr).0vs o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/peft/lora.py z'LoRALinear.__init__..Ds+T+T+T!A+T+T+TzO``quantize_base`` is False, but received the following quantization arguments: ) in_features out_featuresbiasFweightrr )p)super__init__rrrrr_quantize_baseanyvalues ValueErrorrLinearr rrdisabledregister_parameter ParameterDropoutIdentityrlora_alora_bmergedinitialize_parameters) selfrrrrrrrquantization_kwargslinearr r __class__s rr#zLoRALinear.__init__1s       +" s+T+T7J7Q7Q7S7S+T+T+T'U'U wbuww  vG$-XXX& >FMM ==)<==  #m5v{{   ",v*>*>???  $*:BL&&&   18# rzG,,,,2;== iFERRR iDwUSSS   ""$$$$$rT)recursedevicer6cv|j|||j||dS)N)r7r6)r.to_emptyr/)r2r7r6s rr9zLoRALinear.to_empty`s@ FG<<< FG<<<<"?@=KO==== ))) S    %,rr ceZdZdZ ddededededed ed d ed ffd Zd ej dej fdZ e dde d ed d ed ddfdZ xZS) QATLoRALineara LoRA linear layer with quantization-aware training (QAT) applied to the activations and/or weights before the low rank adapters. QAT leverages fake quantization to simulate the quantization numerics during training without actually casting the data to lower precision. This class combines LoRA with QAT to improve the final quantized accuracy during inference while reducing the memory required during training. Args: in_dim (int): input dimension out_dim (int): output dimension rank (int): rank of the low-rank approximation alpha (float): scaling factor for the low-rank approximation dropout (float): dropout probability. Default: 0.0 activation_qat_config (Optional[FakeQuantizeConfig]): config for specifying how input activations will be fake quantized, defaults to None weight_qat_config (Optional[FakeQuantizeConfig]): config for specifying how weights will be fake quantized, defaults to None Raises: ValueError: If `in_dim` is not divisible by weight `group_size` Example usage:: activation_qat_config = FakeQuantizeConfig( dtype=torch.int8, granularity="per_token", is_symmetric=False, ) weight_qat_config = FakeQuantizeConfig( dtype=torch.int4, group_size=8, is_symmetric=True, ) qat_lora_linear = QATLoRALinear( in_dim=512, out_dim=1024, rank=8, alpha=16, dropout=0.0, activation_qat_config=activation_qat_config, weight_qat_config=weight_qat_config, ) qat_lora_linear(torch.randn(512)) r Nrrrrractivation_qat_configFakeQuantizeConfigweight_qat_configc t|||||dd ddlm}ddlm} n"#t $r} td| d} ~ wwxYw|#t||sJ| ||_ ntj |_ |Lt||sJ|j } | || zdkrtd|d| d | ||_ dStj |_ dS) NF)rrr)rW) FakeQuantizerz2QATLoRALinear is only compatible with torchao 0.7+zin_dim (z#) must be divisible by group_size ())r"r#torchao.quantization.qat.apirW'torchao.quantization.qat.fake_quantizerrZ ImportErrorr' isinstanceactivation_fake_quantizerrr- group_sizeweight_fake_quantizer) r2rrrrrrVrXrWrZerrrar5s rr#zQATLoRALinear.__init__st            G G G G G G M M M M M M M   D   ! ,35GHH H HH-:];P-Q-QD * *-/[]]D *  (/1CDD D DD*5J%&:*=*B*B jvvzzz+*77H)I)ID & & &)+D & & &s 8 AAAr@r=c@||}||j}tj||}|jr|S|||}|j|j z | |z}||zS)rB) r`rbr rDr4r)r.rrrr/)r2r@_xwrErFs rrGzQATLoRALinear.forwards + +A . .  & &t{ 3 3hr1oo = J;;t||A//J*dkk(.C.CCX~r lora_linearc T|jtd|jrtdt|jt jr |jj}nd}||j|j |j |j |||}|j j tj dkr |j |_ |jj j tj dkr|jj |j_ |jj j tj dkr|jj |j_ |S)zv Create a `QATLoRALinear` from an existing `LoRALinear`, preserving the weights and adapters. Nz'Bias is not supported in QAT + LoRA yetz/quantize_base is not compatible with QAT + LoRAr meta)rr'r$r_rrr,r!rrrrr r7rPr.r/)clsrgrVrXr new_linears rfrom_lora_linearzQATLoRALinear.from_lora_linears   ' @ A A A  % J H I I I k)2: 6 6 !)+GGGS          !      $ V(<(< < < + 2J    $ +u|F/C/C C C'2'9'@J  $   $ +u|F/C/C C C'2'9'@J  $r)r NN)NN)rHrIrJrKrLrMrr#rPrQrG classmethodr rlrRrSs@rrUrUs4--jAE<@17171717 17  17  17 ((<=17$$89171717171717f%,$ AE<@ &&& ((<= & $$89 & &&&[&&&&&rrUr@r=cvtj|jt jddS)z6 Initialize LoRA A weight to Kaiming uniform. )aN)rinitkaiming_uniform_r mathsqrtr@s rr;r;-s/GQX166666rcNtj|jdS)z, Initialize LoRA B weight to zeros. N)rrqzeros_r rus rr<r<4s GNN18r)rstypingrrrrPtorch.nn.functionalr functionalrDtorchao.dtypes.nf4tensorrrtorchtune.modules.low_precisionr torchtune.modules.peftr Moduler rUr(r;r<rrrrs\  (((((((((( 77777777FFFFFF000000xxxxxMxxxv\\\\\J\\\~7297777729r