PiddlZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl Z ddl mZmZddl mZddlmZmZddlmZdd lmZmZdd lmZmZmZmZmZdd lm Z m!Z!dd l"m#Z#dd l$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m0Z0ej1dZ2Gdde/Z3ej4deddfdZ5e6dkrej7e5dSdS)N)partial)AnyDictListOptionalTuple)warn) DictConfig ListConfig)nn)destroy_process_groupinit_process_group) Optimizer) DataLoaderDistributedSampler)configmodulesrlhftrainingutils)CROSS_ENTROPY_IGNORE_IDXpadded_collate_dpo) ConcatDataset) disable_adapter DoRALinearget_adapter_paramsget_adapter_state_dictget_lora_module_namesget_merged_lora_ckpt LoRALinearset_trainable_params(validate_missing_and_unexpected_for_lora)FTRecipeInterface)tqdmDEBUGceZdZdZdeddfdZdedeeeffdZ deeefddfd Z deddfd Z d&d ed e d e de de deeefde eede eeefdejfdZ d'dede eeefdefdZdedededefdZdede dedeeeffdZdeddfd Zd!ejd"eejejfdeejejejejffd#Zd(d$Zd(d%ZdS))LoRADPORecipeDistributeda Distributed LoRA DPO recipe for dense transformer-based LLMs such as Llama2. This recipe supports distributed training and can be run on a single node (1 to 8 GPUs). This is based on HF's DPOTrainer in the TRL library: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L65 Features: - FSDP. Supported using PyTorch's FSDP APIs. CPU offload of parameters, gradients, and optimizer states is supported via ``fsdp_cpu_offload``. Resharding of parameters after the forward pass is done by default (corresponding to FULL_SHARD sharding strategy), but can be disabled by setting the config ``fsdp_reshard_after_forward`` to False (this corresponds to SHARD_GRAD_OP sharding strategy). DDP is currently not supported. Training on CPU is not supported. - Activation Checkpointing. This can be controlled using the ``activation_checkpointing`` flag. Activation checkpointing helps reduce the memory footprint since we no longer keep activations in memory and instead recompute them during the backward pass. This is especially helpful for larger batch sizes when you're memory constrained. But these savings in memory come at the cost of training performance. In most cases training can slow-down quite a bit as a result of this activation recomputation. - Activation Offloading. This can be controlled using the ``enable_activation_offloading`` flag. Activation offloading is a technique similar to activations checkpointing that helps reduce the memory footprint to prevent OOMs on CUDA and enable bigger batches. Where activations checkpointing drops the activation in the forward to recompute it later in the backward, activations offloading will drop the activation in the forward to the CPU and bring it back during the backward pass. As always, there is a tradeoff--these savings in memory can come at the cost of training performance and CPU resources. To recover some runtime cost, we've added an option to enable offloading on a different stream to permit overlapping with the computation. This option is currently only available on PyTorch 2.5 or later and will be enabled by default if an acceptable torch version is found. Activation offloading can be used in conjunction with activation checkpointing. - Precision. Full fp32 and bf16 training are supported. Precision is controlled using the ``dtype`` flag. When ``dtype=bf16``, all activations, gradients and optimizer states are in bfloat16. In most cases this should halve the memory footprint of full precision (fp32) training, without loss in model quality (will depend on the model, training data and other settings). For GPUs which do not support bfloat16, we fall back to fp32. Mixed precision training and fp16 precision are currently not supported. - Gradient Accumulation. You can simulate larger batch sizes by accumulating gradients. This is controlled using the ``gradient_accumulation_steps`` flag. Total Batch Size = batch_size * number of GPUs * gradient accumulation steps. For example: with batch_size=1, nproc_per_node=2 and gradient_accumulation_steps=32 we get a total batch size of 64. Gradient accumulation is especially useful when you are memory constrained. In this case, accumulating gradients might give you better training speed than enabling activation checkpointing. - Checkpointing. Model weights are checkpointed both at the end of each epoch and at the end of training. Currently we checkpoint both the adapter weights (trainable params only) and the complete merged weights (adapter weights added back to the base model). For more details please take a look at our LoRA tutorial (https://pytorch.org/torchtune/main/tutorials/lora_finetune.html). Optimizer State and recipe state (seed, total_epochs, number of epochs run etc) are only saved at the end of a given epoch and used in case of resuming training. Resuming training is controlled by the ``resume_from_checkpoint`` flag. Mid-epoch checkpointing is currently not supported. For more details on the checkpointer, please take a look at our checkpointer deepdive (https://pytorch.org/torchtune/main/tutorials/checkpointer.html). - Logging. Terminal, Disk, WandB and TensorBoard are all supported. The following losses are supported in this recipe: - :class:`~torchtune.rlhf.loss.DPOLoss`: Direct Preference Optimization (DPO). - :class:`~torchtune.rlhf.loss.RSOPLoss`: Rejection Sampling Optimization (RSO). For a full list of example configs for this recipe, run ``tune ls`` on the command line. Each config has example commands for how to kick-off training. Args: cfg (DictConfig): OmegaConf object parsed from yaml file Raises: ValueError: If ``dtype`` is set to fp16. ValueError: If world_size is 1 RuntimeError: If ``dtype`` is set to bf16 and the hardware does not support bf16. RuntimeError: If ``enable_activation_offloading`` is True and device is not CUDA. RuntimeError: If ``enable_activation_offloading`` is True and ``enable_activation_checkpointing`` is False. cfgreturnNcltj|j|_t j|j|j|_|jtj krtdtj \|_ |_ |j dk|_|j|_|dd|_|dd|_|jr1|jjdkr!t*d d|_|d d|_|d d|_|jr6|jjdkrt3d |jst3d n!|jrtjt*dt j|j|dd|_d|_|j|_|j |_ d|_!|j"|_#|dd|_$|j%|_&dS)NdevicezVfull fp16 training is not supported with this recipe. Please use bf16 or fp32 instead.rlog_every_n_stepslog_peak_memory_statsFcudazplog_peak_memory_stats was set to True, however, training does not use cuda. Setting log_peak_memory_stats=False.enable_activation_checkpointingenable_activation_offloadingzFenable_activation_offloading should only be True when training on CUDAz]enable_activation_offloading should only be True when enable_activation_checkpointing is TruezHint: enable_activation_checkpointing is True, but enable_activation_offloading isn't. Enabling activation offloading should reduce memory further.cudnn_deterministic_mode)seed debug_modesave_adapter_weights_only)'r get_devicer,_devicer get_dtypedtype_dtypetorchfloat16 ValueErrorget_world_size_and_rank world_sizerank _is_rank_zero output_dir _output_dirget_log_every_n_steps_log_peak_memory_statstypeloginfo _enable_activation_checkpointing_enable_activation_offloading RuntimeError log_rank_zeroset_seedr4 epochs_runepochs total_epochsmax_steps_per_epoch global_stepresume_from_checkpoint_resume_from_checkpoint_save_adapter_weights_onlygradient_accumulation_steps_gradient_accumulation_steps)selfr(s p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/recipes/lora_dpo_distributed.py__init__z!LoRADPORecipeDistributed.__init__~sA'sz::: (4<HHH ;%- ' 'h &+%B%D%D"!Y!^>"%''*=q"A"A&)gg.Eu&M&M#  & 04<+<+F+F HHC   +0D '14 -u1 1 -.1WW *E. . *  - | F**"\8 "s  2   O   %cgg.H$&O&O    J#&#: '*'A$*-''2Mu*U*U',/,K)))cfg_checkpointerctj||j|_|j}|jr2t j|vrtd|||S)z Extract the checkpoint state from file and validate. This includes the base model weights. If resume_from_checkpoint is True, this also includes the adapter weights and recipe state )should_load_recipe_statezPAdapter weights not found. Please ensure a valid adapter checkpoint is provided.) r instantiaterV _checkpointerload_checkpointr ADAPTER_KEYr>_update_recipe_state)rZr^checkpoint_dicts r[rcz(LoRADPORecipeDistributed.load_checkpoints $/ %)%A   ,<<>>  ' 7#?:: f  % %o 6 6 6r] ckpt_dictc8 |tj|_|j|tjkr:t d|tj|tj|_|j|tjkr:t d|tj|tj|_|j|tj krt d|jdSdS#t$r}td|d}~wwxYw)z; Updates the recipe state from checkpoint. zWConfig value for seed does not match the checkpoint value, using the checkpoint value: )messagezfConfig value for max_steps_per_epoch does not match the checkpoint value, using the checkpoint value: z[Config value for total_epochs does not match the checkpoint value, using the config value: zCheckpoint does not contain the required keys needed for updating recipe state. Are you sure you passed in the right recipe checkpoint?N) r EPOCHS_KEYrPr4SEED_KEYr rS MAX_STEPS_KEYrRTOTAL_EPOCHS_KEYKeyError)rZrges r[rez-LoRADPORecipeDistributed._update_recipe_statesf" '(;z8LoRADPORecipeDistributed._setup_data..s<&"#5QQQr])datasetsrr) num_replicasrAr{r4T) padding_idx ignore_idx)rr|sampler drop_last collate_fnz$Dataset and Sampler are initialized.)rr rrrarrr@rArrrpad_idrrrNrI)rZrzr{r|rdsr dataloaders` r[rz$LoRADPORecipeDistributed._setup_datas k: . . L*5H111BB#K4?KKKB$ T_49gTU    !" O23     C!GHHH ""r]epochc i}|dz|jk}tj|j|j|j|j}|r-tj|j|j|j|j}nd}|jrS|jr|}nLt|}t||j |j }| tj|i| tj|i|re| tj|tj|jtj|jtj|jtj|ji|j |j t1|j|j|jdd}| tj|i|j||||jdSdS) a Checkpoint the state of the recipe. The constructed checkpoint state dict contains the following information: - Merged weights with key MODEL_KEY - Adapter weights with key ADAPTER_KEY - Relevant recipe state if training is not complete - If the `self._save_adapter_weights_only` option is True, the checkpointer will save only the adapter weights Checkpointer will save the merged weights, adapter weights and recipe state in different checkpoint files. To correctly resume from training, the adapter weights and recipe state must be provided along with the base model weights.r.)r,adapter_weights_onlyr+N)rAalphaLORA)rrtarget_modules peft_type)rintermediate_checkpoint adapter_only)rRrgather_cpu_state_dictrrBr8rWget_full_optimizer_state_dictrrrrrupdaterrdrrkr4rjrPrmrlrSrrrrADAPTER_CONFIGrbsave_checkpoint) rZrrfrcpu_state_dictryadapter_state_dictmerged_state_dictadapter_configs r[rz(LoRADPORecipeDistributed.save_checkpoints "'!)d.?"?"7 K  |j"|jd|j#"|xj$dz c_$|%}|&d|'|dzd|j$d||j$|j(zdkrk|j)rct j|z }||jj*dd|||j+zz |d,|d,|d,|d|dz ,|d ,|d ,|d ,|d ,d }|j-r-|&tj.|j/|j01||j$d}d|D}d}t j}|xjdz c_|2|dS)z) The core training loop. r)rewards/chosenrewards/rejectedrewards/accuracieslog_probs/chosenlog_probs/rejected logits/chosenlogits/rejected)totaldisableNr.rrrr r r r )opT) set_to_none|z|Loss: lr) rrtokens_per_second_per_gpurrrzrewards/marginsr r r r r+)stepci|]}|dS)r)rkeys r[ z2LoRADPORecipeDistributed.train..4s&I&I&I#sA&I&I&Ir])r)3rcleanup_before_trainingr zero_gradrrrangerPrRr set_epochr$rrA enumeraterrSrYr<tensornumelrrdetachmeanno_gradrrfloatbackwardr all_reduceReduceOpAVGrrrTitemrset_descriptionrFrB param_groupsr@rrGrr8rlog_dictr)rZt0 running_lossrunning_metrics num_tokens curr_epochpbaridxrpolicy_chosen_log_probspolicy_rejected_log_probspolicy_chosen_logitspolicy_rejected_logitspolicy_chosen_logits_meanpolicy_rejected_logits_meanreference_chosen_log_probsreference_rejected_log_probs_rchosen_rewardsrejected_rewardsreward_accuraciesscaling_factorr loss_to_log time_per_stepr+s r[trainzLoRADPORecipeDistributed.trains (*** !!###    !"# !"#     1BCCW 3W 3J M # #J / / /d3$)q.=QRRRD'(899M -M - U,8 AA/00Eel58>>+;+;<<< --dk5AA +-(*-A,G,G,I,I,N,N,P,P).D.K.K.M.M.R.R.T.T+)*@]__FFodk&B&BFF 11$+uEE 24 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:>+-.0 ::6n&6 &46F%F$M$M$O$O!yy{{d??$ 99 0111"^%8%8%:%::111 2333"%5%:%:%<%<<333 4555"%6%;%;%=%==555 2333"%<%C%C%E%E%J%J%L%LL333 4555"%>%E%E%G%G%L%L%N%NN555 000"%>>000 1222"%@@222 !Gt@@AEE%00>>>%00<<<.)44+C0U5F5O5S5O((***O--$-???&++---$$)$$"."3"3"5"5KKKNNN((%>RRD,<RR[RR (4+BBaGG .H)-(9(;(;b(@ $/"&/">q"A$"G9C,t>:@.=>N.O.S.S.U.U0? 21!cee2A 43!cee /0@ A"12D"E!F!cee0? 21!cee2A 43!cee-<_-M-Q-Q-S-S/>?P/Q/U/U/W/W/$$2 6$OO ( 9 N N N+44$!%!15 $%L&I&I&I&I&IO!"J*,,B OOq OO  z 2 2 2 2oW 3W 3s6 G%!!G  G%G G%G G%%G) ,G) cb|jr|jtdSN)rBrcloser )rZs r[cleanupz LoRADPORecipeDistributed.cleanup<s4   (   % % ' ' 'r]rrD)r)N) __name__ __module__ __qualname____doc__r r\rstrrrcrerboolrrr Modulerrrintrrrrrrr<TensorrrBrFrr]r[r'r')sRRh:LJ:L4:L:L:L:Lx tCH~2&d38n&&&&&PK K K K K K j6:<@}}}*.}'+ }  } $ } $CH~} (S 2}"*$sCx.!9} }}}}@UY  ' 9A$sCx.9Q      $     (#(#(# (# !:- . (#(#(#(#TUU UUUUnVYV',U\5<-G'HV u|U\5<E FVVVVBr3r3r3r3h      r]r'r(r)ctjstdtd|ddrtjt jd|t|}| || | dS) z Entry point for the recipe. Configurable parameters are read in the following order: - Parameters specified in config (see available configs through ``tune ls``) - Overwritten by arguments from the command-line zDistributed finetune recipe should be run via a distributed launcher.If using tune CLI, please specify --nnodes 1 and --nproc_per_node [num_gpus]zcuda:nccl,cpu:gloorrFr') recipe_namer()r(N) ris_distributedrMrrEset_torch_num_threadsrrr'rrBrF)r(recipes r[ recipe_mainrUBs  " $ $  [   +,,, ww!5))) &((( "<#FFFF %# . . .F LLSL LLNNN NNr]__main__)8sysr functoolsrtypingrrrrrwarningsr r< omegaconfr r r torch.distributedr r torch.optimrtorch.utils.datarr torchtunerrrrrtorchtune.datarrtorchtune.datasetsrtorchtune.modules.peftrrrrrrr r!r"torchtune.recipe_interfacesr#r$ get_loggerrIr'parserUrGexitrr]r[rgsm 33333333333333 ,,,,,,,,GGGGGGGG!!!!!!;;;;;;;;<<<<<<<<<<<<<<GGGGGGGG,,,,,,                      :99999ewV  V  V  V  V  0V  V  V  rZD6 z CH[[]]r]