PiddlZddlmZmZmZddlZddlmcmZ ddlmZddl m Z m Z ddl mZddlmZGddejeZd ejd dfd Zd ejd dfd ZdS) N)ListOptionalUnion)nn) linear_nf4to_nf4)_register_nf4_dispatch_ops) AdapterModuleceZdZdZ ddededededed ed effd Zd d dee e e j efdefdZ dZe jdZdZdee fdZde jde jfdZxZS) DoRALinearaQDoRA linear layer as introduced in `DoRA: Weight-Decomposed Low-Rank Adaptation of Large Language Models `_. DoRA (Weight-Decomposed Low-Rank Adaptation) fine-tunes a layer by decomposing the pre-trained weights into two components: magnitude and direction. The magnitude component is a learnable scalar vector that scales each output channel, while the direction component, modified via LoRA, adjusts the orientation of weights. By scaling the LoRA update component :math:`BAx` with the `magnitude` vector, DoRA allows the model to apply distinct scaling adjustments across different output dimensions. Args: in_dim (int): input dimension out_dim (int): output dimension rank (int): rank of the low-rank approximation alpha (float): scaling factor for the low-rank approximation dropout (float): dropout probability. Default: 0.0 use_bias (bool): whether to include bias in the original linear layer. Default: False quantize_base (bool): Whether to quantize base linear weight or not. Default: False **quantization_kwargs: Keyword arguments to pass to `to_nf4` when quantizing the base linear weight. Examples of valid arguments are `block_size` and `scaler_block_size`, which control the granularity of weight quantization and scaler quantization respectively. This is only used if `quantize_base` is True. Default None Raises: ValueError: If ``quantize_base`` is False, but quantization kwargs are provided. Fin_dimout_dimrankalphadropoutuse_bias quantize_basec t||_||_||z |_||_||_|js=td|Drtd|tj |||j} |js| j nt| j fi|} |jr| jnd} d|_|dtj| |d| tj| nd|dkrtj|ntj|_tj ||d|_tj ||d|_tjt/j||_|dS) Ncg|]}|Sr).0vs o/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/peft/dora.py z'DoRALinear.__init__..Ds+T+T+T!A+T+T+TzO``quantize_base`` is False, but received the following quantization arguments: ) in_features out_featuresbiasFweightrr )p)super__init__rrscalingr_quantize_baseanyvalues ValueErrorrLinearr rrdisabledregister_parameter ParameterDropoutIdentityrlora_alora_btorchempty magnitudeinitialize_parameters) selfrrrrrrrquantization_kwargslinearr r __class__s rr#zDoRALinear.__init__2s   t|   +" s+T+T7J7Q7Q7S7S+T+T+T'U'U wbuww  vG$-XXX& >FMM ==)<==  #m5v{{   ",v*>*>???  $*:BL&&&   18# rzG,,,,2;== iFERRR iDwUSSS ek'&:&:;; ""$$$$$rT)recursedevicer9c2|j|||j||tjt j|j||jj}t j |j||S)N)r:r9)r:) requires_grad) r/to_emptyr0rr,r1 empty_liker3r<utils swap_tensors)r5r:r9r3s rr=zDoRALinear.to_empty`s FG<<< FG<<<L  T^F ; ; ;.6       ;;; rcVt|jt|jdS)N)_lora_a_init_paramsr/_lora_b_init_paramsr0)r5s rr4z DoRALinear.initialize_parametersms* DK(((DK(((((rcxt|jj|jjj|jjjgrt d|j|jjj}|jj|jjz}|j | ||dS)a= DoRA initializes the magnitude vector such that its outputs are initially identical to standard LoRA's outputs. This must be called after loading/initializing base model and LoRA params. Raises: RuntimeError: If base or LoRA parameters are still on meta device. zUCannot initialize DoRA magnitude if base or LoRA parameters are still on meta device.N) r&r is_metar/r0 RuntimeErrortodtyper3copy__get_weight_norm)r5 base_weight lora_weights rinitialize_dora_magnitudez$DoRALinear.initialize_dora_magnitudess  # "* "*   g knnT[%7%=>> k(4;+==  T22; LLMMMMMrc||j|zz}tj|d|j}|S)N)dim)r$r1linalgnormrGrH)r5r rL weight_norms rrJzDoRALinear._get_weight_normsC$,44l''A'6699&,GG rreturncgd}|S)z Return a list of strings corresponding to the names of the ``nn.Parameter`` s in the model coming from the adapter. For DoRA this means lora_a.weight, lora_b.weight, and magnitude. )z lora_a.weightz lora_b.weightr3r)r5adapter_paramss rrVzDoRALinear.adapter_paramssIHHrxc2|jr(t||j}|jr ||jz}n t j||j|j}|jr|S||}| | |}tj |j jj d|j jj|j}| | |j}|j}|j|j}|||}|}||z dd} | dz |z| |z|jzz} | |zS)z Args: x (torch.Tensor): input tensor with shape ``(..., in_dim)`` Returns: Tensor: output tensor with shape ``(..., out_dim)`` )inputr rO)r:rH)r%rr rrFr7r*rr0r/r1eyeshaper:rHTr3rGrJdetachviewr$) r5rWbase_outlora_outx_eyerLr3r rSmag_norm_scaledora_outs rforwardzDoRALinear.forwards   ;!$+>>>H} 0#di/x4; ::H = O LLOO;;t{{1~~.. K  $Q ' 0B0IQRQX   kk$++e"4"4557 N ((++FK4F4F4H4HII !((** #k1772>> Q  %04<?@(""r)r FF)__name__ __module__ __qualname____doc__intfloatboolr#rrstrr1r:r=r4no_gradrMrJrrVTensorrf __classcell__)r8s@rr r snF#,%,%,%,% ,%  ,%  ,%,%,%,%,%,%,%,%^SW   !%U\3(>"?@ KO    ))) U]__NN_N0 S $#$#%,$#$#$#$#$#$#$#$#rr rWrTcvtj|jt jddS)z6 Initialize LoRA A weight to Kaiming uniform. )aN)rinitkaiming_uniform_r mathsqrtrWs rrBrBs/GQX166666rcNtj|jdS)z, Initialize LoRA B weight to zeros. N)rruzeros_r rys rrCrCs GNN18r)rwtypingrrrr1torch.nn.functionalr functionalr[torchao.dtypes.nf4tensorrrtorchtune.modules.low_precisionr torchtune.modules.peftr Moduler r)rBrCrrrrs) (((((((((( 77777777FFFFFF000000k#k#k#k#k#Mk#k#k#\7297777729r