PijgddlZddlZddlmZddlmZmZmZmZddl m Z ddl Z ddl m Z mZddl mZddlmZddlmZmZdd lmZmZmZmZmZdd lmZmZdd lmZdd l m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)ddl*m*Z*ej+dZ,Gdde)Z-ej.de ddfdZ/e0dkrej1e/dSdS)N)partial)AnyDictOptionalTuple)warn) DictConfig ListConfig)nn) Optimizer) DataLoaderDistributedSampler)configmodulesrlhftrainingutils)CROSS_ENTROPY_IGNORE_IDXpadded_collate_dpo) ConcatDataset)disable_adapterget_adapter_paramsget_adapter_state_dictget_lora_module_namesget_merged_lora_ckptset_trainable_params(validate_missing_and_unexpected_for_lora)FTRecipeInterface)tqdmDEBUGceZdZdZdeddfdZdedeeeffdZ deeefddfd Z deddfd Z d$d ed e d e de deeefde eeefdejfdZ d$dede eeefdefdZde ededede efdZdede dedeeeffdZdeddfdZdejd eejejfdeejejejejffd!Zd%d"Zd%d#ZdS)&LoRADPORecipeSingleDevicead LoRA DPO recipe for dense transformer-based LLMs such as Llama2 for single device training. This is based on HF's DPOTrainer in the TRL library: https://github.com/huggingface/trl/blob/main/trl/trainer/dpo_trainer.py#L65 This recipe supports: - Activation checkpointing. This is enabled by default but is configurable. - Activation offloading - this is enabled by default and should only be used alongside activation checkpointing. - Full bf16 training for supported HW architectures. We currently check bf16 support via the `torch.cuda.is_bf16_supported` API. This is disabled by default but can be enabled via setting `dtype=bf16` in configuration. - Checkpointing: of LoRA adapter parameters and their optimizer states. When resuming from a checkpoint, the adapter parameters are loaded from the checkpoint along with the base model weights. Note that intra-epoch resumption is not supported. - Logging to terminal, WandB, or TensorBoard. The following losses are supported in this recipe: - :class:`~torchtune.rlhf.loss.DPOLoss`: Direct Preference Optimization (DPO). - :class:`~torchtune.rlhf.loss.RSOPLoss`: Rejection Sampling Optimization (RSO). Assumptions: - Checkpoints are ONLY saved at epoch boundaries. In case of failure, work done in ongoing epoch is lost. - Datasets are Map-style and data fits in memory (not streamed). The following configs can be used to run this recipe: >>> tune ls RECIPE CONFIG lora_dpo_single_device llama2/7B_lora_dpo_single_device Args: cfg (DictConfig): OmegaConf object parsed from yaml file Raises: ValueError: If ``dtype`` is set to fp16. RuntimeError: If ``dtype`` is set to bf16 and the hardware does not support bf16. RuntimeError: If ``enable_activation_offloading`` is True and device is not CUDA. RuntimeError: If ``enable_activation_offloading`` is True and ``enable_activation_checkpointing`` is False. cfgreturnNc tj|j|_t j|j|j|_|jtj krtd|j |_ | dd|_| dd|_|jr1|jjdkr!t"dd|_| d d|_| d d|_|jr6|jjd krt+d |jst+d n!|jrtjt"dt j|j| dd|_d|_|j|_|j|_d|_|j|_| dd|_ |j!|_"dS)NdevicezHfp16 precision is not supported in this recipe. Please use fp32 or bf16.log_every_n_stepslog_peak_memory_statsFcpuzglog_peak_memory_stats was set to True, however, training uses cpu. Setting log_peak_memory_stats=False.enable_activation_checkpointingenable_activation_offloadingcudazFenable_activation_offloading should only be True when training on CUDAz]enable_activation_offloading should only be True when enable_activation_checkpointing is TruezHint: enable_activation_checkpointing is True, but enable_activation_offloading isn't. Enabling activation offloading should reduce memory further.cudnn_deterministic_mode)seed debug_modersave_adapter_weights_only)#r get_devicer'_devicer get_dtypedtype_dtypetorchfloat16 ValueError output_dir _output_dirget_log_every_n_steps_log_peak_memory_statstypeloginfo _enable_activation_checkpointing_enable_activation_offloading RuntimeError log_rank_zeroset_seedr0 epochs_runepochs total_epochsmax_steps_per_epoch global_stepresume_from_checkpoint_resume_from_checkpoint_save_adapter_weights_onlygradient_accumulation_steps_gradient_accumulation_steps)selfr#s r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/recipes/lora_dpo_single_device.py__init__z"LoRADPORecipeSingleDevice.__init__Ss'sz::: (4<HHH  ;%- ' 'Z  >"%''*=q"A"A&)gg.Eu&M&M#  & 04<+<+E+E HHy   +0D '14 -u1 1 -.1WW *E. . *  - | F**"\8 "s  2   O   %cgg.H$&O&O    J#&#: '*'A$*-''2Mu*U*U',/,K)))cfg_checkpointerctj||j|_|j}|jr2t j|vrtd|||S)z Extract the checkpoint state from file and validate. This includes the base model weights. If resume_from_checkpoint is True, this also includes the adapter weights and recipe state )should_load_recipe_statezPAdapter weights not found. Please ensure a valid adapter checkpoint is provided.) r instantiaterN _checkpointerload_checkpointr ADAPTER_KEYr:_update_recipe_state)rRrVcheckpoint_dicts rSr[z)LoRADPORecipeSingleDevice.load_checkpoints $/ %)%A   ,<<>>  ' 7#?:: f  % %o 6 6 6rU ckpt_dictc8 |tj|_|j|tjkr:t d|tj|tj|_|j|tjkr:t d|tj|tj|_|j|tj krt d|jdSdS#t$r}td|d}~wwxYw)z; Updates the recipe state from checkpoint. zWConfig value for seed does not match the checkpoint value, using the checkpoint value: )messagezfConfig value for max_steps_per_epoch does not match the checkpoint value, using the checkpoint value: z[Config value for total_epochs does not match the checkpoint value, using the config value: zCheckpoint does not contain the required keys needed for updating recipe state. Are you sure you passed in the right recipe checkpoint?N) r EPOCHS_KEYrHr0SEED_KEYrrK MAX_STEPS_KEYrJTOTAL_EPOCHS_KEYKeyError)rRr_es rSr]z.LoRADPORecipeSingleDevice._update_recipe_statesf" '(;) ) % o # 7,1,A,A'-B-- )L//-7 )L/0"5"5!%!;%+%+    )1(O /) ) % F FFFGGG  *  "5 ) ) ) <  % %#4DLIIIL  %l 3 3 3 s3AA AA A A AA!Armrnctj||j}|r||t d|S)Nz#Optimizer and loss are initialized.)rrYr parametersrrArB)rRrmrnrs rSrz*LoRADPORecipeSingleDevice._setup_optimizerQsY&}dk6L6L6N6NOO  6  % %n 5 5 5 6777rUrsrtruc|tddStj||j||}td|S)NzDNo learning rate scheduler configured. Using constant learning rate.)rtruz'Learning rate scheduler is initialized.)rArBrrYr)rRrsrtrurrs rSrz-LoRADPORecipeSingleDevice._setup_lr_scheduler[sj  # HHV   4)  O1!     :;;;rUrorprqc vt|trfd|D}t|}ntj|j}t |dd|d}t|||dttjj t }t d ||fS) z All data related setup happens here. Currently this recipe only supports Map-style Datasets which fit into memory and an option for random shuffling. Samplers, iterable datasets, and streaming datasets are not supported. cFg|]}tj|jS)r)rrYr).0single_cfg_datasetrRs rS z9LoRADPORecipeSingleDevice._setup_data..}s<&"#5QQQrU)datasetsrr)r) num_replicasrankrpr0T) padding_idx ignore_idx)rsamplerrq drop_last collate_fnz$Dataset and Sampler are initialized.) isinstancer rrrYrrr rrpad_idrrArB)rRrorprqrdsr dataloaders` rSrz%LoRADPORecipeSingleDevice._setup_dataqs k: . . L*5H111BB#K4?KKKB$      !" O23     7888 ""rUepochc i}|dz|jk}|r||tj|jtj|jtj|j tj |jtj |j it|j}|tj|i|jsrd|jD}t%||j|j}|tj|i|j|jt-|j|j|jdd}|tj|i|j||||jdS)a Checkpoint the state of the recipe. The constructed checkpoint state dict contains the following information: - Merged weights with key MODEL_KEY - Adapter weights with key ADAPTER_KEY - Relevant recipe state if training is not complete - If the `self._save_adapter_weights_only` option is True, the checkpointer will save only the adapter weights To correctly resume from training, the adapter weights and recipe state must be provided along with the base model weights. r)c>i|]\}}||S)r+)rkvs rS z=LoRADPORecipeSingleDevice.save_checkpoint..s&RRRA!QUUWWRRRrU)ralphaLORA)rrtarget_modules peft_type)rintermediate_checkpoint adapter_onlyN)rJupdaterrr state_dictrcr0rbrHrerdrKrrr\rOitemsrrrr~rrrrADAPTER_CONFIGrZsave_checkpoint)rRrr_radapter_state_dictrmerged_state_dictadapter_configs rSrz)LoRADPORecipeSingleDevice.save_checkpoints "'!)d.?"? "    $do&@&@&B&B%ty'-t/@*D,D    4DK4J4J4L4LMM(.0BCDDD. FSR1G1G1I1I1O1O1Q1QRRRJ 4_&!!!    h02CD E E E*3''*      (1>BCCC ** $;8 +     rUr}batchcn|\}}||j}||j}|jddz}|j5||}dddn #1swxYwYt j||}|d|}||d} |d|} ||d} || | | fS)az Run forward pass of the model with chosen and rejected samples concatenated. Args: model (nn.Module): The model to be used for the forward pass. batch (Tuple[torch.Tensor, torch.Tensor]): Tuple of input_ids and labels. Returns: Tuple of chosen log probs, rejected log probs, chosen logits, rejected logits. rN)tor4shaperrget_batch_log_probs) rRr}rconcatenated_input_idsconcatenated_labels len_chosen all_logits all_log_probschosen_log_probsrejected_log_probs chosen_logitsrejected_logitss rSconcatenated_forwardz.LoRADPORecipeSingleDevice.concatenated_forwards7<3 3!7!:!:4 param_groupsr+r?rrr4rwlog_dictr)rRt0 running_loss num_tokens curr_epochpbaridxrpolicy_chosen_log_probspolicy_rejected_log_probspolicy_chosen_logitspolicy_rejected_logitspolicy_chosen_logits_meanpolicy_rejected_logits_meanreference_chosen_log_probsreference_rejected_log_probs_rchosen_rewardsrejected_rewardsreward_accuracies loss_to_log time_per_steprs rStrainzLoRADPORecipeSingleDevice.trains    HHC         1BCCf 3f 3J M # #J / / /d3444D'(899^ -^ - U,8 AA/00EeAhnn... --dk5AA +-(*-A,G,G,I,I,N,N,P,P).D.K.K.M.M.R.R.T.T+)*@]__FFodk&B&BFF 11$+uEE 24 FFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:>+-.0 ::6n&6yy{{%36F%F$M$M$O$O!d??$  !Gt@@AEEO((***O--$-???)5*//111$$)$$"."3"3"5"5KKKNNN((%>RRD,<RR[RR '$*AAQFF(,(9(;(;b(@ $/"&/">q"A$"G9Cm9S.<.A.A.C.C.G.G.I.I0@0E0E0G0G0K0K0M0M2C2H2H2J2J2N2N2P2P0>AQ0Q!TVV SUU2K2R2R2T2T!TVV SUU0G0N0N0P0P!TVV SUU/J/N/N/P/P-F-J-J-L-L#$$& 6$OO ( 9 N N N+44$!%!15 $%L!"J*,,B OOq OO  z 2 2 2 2Mf 3f 3s6F3/!F  F3F F3#F $F33F7 :F7 c8|jdSN)rwclose)rRs rScleanupz!LoRADPORecipeSingleDevice.cleanupqs !!#####rUr)r$N)__name__ __module__ __qualname____doc__r rTrstrrr[r]rboolrr Moduler|r rintrrrr rrr8TensorrrrrrUrSr"r"'s))V:LJ:L4:L:L:L:Lx tCH~,&d38n&&&&&PH H H H H H b=A888*.8'+ 8  8 $CH~ 8"*$sCx.!98 8888vUY'9A$sCx.9Q ":.   )  ,)#)#)# )# !:- . )#)#)#)#V; S; T; ; ; ; zVYV',U\5<-G'HV u|U\5<E FVVVVBu3u3u3u3n$$$$$$rUr"r#r$ctjd|t|}||||dS)z Entry point for the recipe. Configurable parameters are read in the following order: - Parameters specified in config (see available configs through ``tune ls``) - Overwritten by arguments from the command-line r") recipe_namer#)r#N)rrxr"rrr)r#recipes rS recipe_mainr&usb "=3GGGG &3 / / /F LLSL LLNNN NNrU__main__)2sysr functoolsrtypingrrrrwarningsrr8 omegaconfr r r torch.optimr torch.utils.datar r torchtunerrrrrtorchtune.datarrtorchtune.datasetsrtorchtune.modules.peftrrrrrrrtorchtune.recipe_interfacesrr get_loggerrAr"parser&rexitrrUrSr7s; ------------ ,,,,,,,,!!!!!!;;;;;;;;<<<<<<<<<<<<<<GGGGGGGG,,,,,,:99999ewK $K $K $K $K $ 1K $K $K $\ Z D     z CH[[]]rU