PiddlZddlZddlmZddlmZmZmZmZddl m Z ddl Z ddl m Z mZddl mZddlmZddlmZdd lmZdd lmZmZmZmZdd lmZdd lmZdd lm Z ddl!m"Z"ddl#m$Z$m%Z%ddl&m'Z'ddl(m(Z(ej)dZ*Gdde"Z+ej,de ddfdZ-e.dkrej/e-dSdS)N)partial)AnyDictOptionalUnion)warn) DictConfig ListConfig)nn) Optimizer)StatefulDataLoader)StatefulDistributedSampler)configmodulestrainingutils)_get_component_from_path)padded_collate_packed) ConcatDataset)FTRecipeInterface) DummyProfiler PROFILER_KEY)get_lr)tqdmDEBUGceZdZdZdeddfdZdedeeeffdZ deeefddfd Z deddfd Z d(d e ede ejjeffd Zd ededededeeefdejf dZ d)dedede eeefde efdZde ededede efdZ d(dedededed e eeefdef d!Zd"eddfd#Zd$eeejfdejfd%Zd*d&Z d*d'Z!dS)+FullFinetuneRecipeSingleDeviceaG Full finetuning recipe for dense transformer-based LLMs such as Llama2. This recipe is optimized for single GPU training. Training on CPU is not supported. Features: - Activation Checkpointing. This can be controlled using the ``enable_activation_checkpointing`` flag. Activation checkpointing helps reduce the memory footprint since we no longer keep activations in memory and instead recompute them during the backward pass. This is especially helpful for larger batch sizes when you're memory constrained. But these savings in memory come at the cost of training performance. In most cases training can slow-down quite a bit as a result of this activation recomputation. - Activation Offloading. This can be controlled using the ``enable_activation_offloading`` flag. Activation offloading is a technique similar to activations checkpointing that helps reduce the memory footprint to prevent OOMs on CUDA and enable bigger batches. Where activations checkpointing drops the activation in the forward to recompute it later in the backward, activations offloading will drop the activation in the forward to the CPU and bring it back during the backward pass. As always, there is a tradeoff--these savings in memory can come at the cost of training performance and CPU resources. To recover some runtime cost, we've added an option to enable offloading on a different stream to permit overlapping with the computation. This option is currently only available on PyTorch 2.5 or later and will be enabled by default if an acceptable torch version is found. Activation offloading can be used in conjunction with activation checkpointing. - Precision. Full fp32 and bf16 training are supported. Precision is controlled using the ``dtype`` flag. When ``dtype=bf16``, all activations, gradients and optimizer states are in bfloat16. In most cases this should halve the memory footprint of full precision (fp32) training, without loss in model quality (will depend on the model, training data and other settings). For GPUs which do not support bfloat16, we fall back to fp32. Mixed precision training and fp16 precision are currently not supported. - Gradient Accumulation. You can simulate larger batch sizes by accumulating gradients. This is controlled using the ``gradient_accumulation_steps`` flag. Total Batch Size = batch_size * gradient accumulation steps. For example: with batch_size=1 and gradient_accumulation_steps=32 we get a total batch size of 32. Gradient accumulation is especially useful when you are memory constrained. In this case, accumulating gradients might give you better training speed than enabling activation checkpointing. - Optimizer in Backward. Fusing the optimizer step into the backward pass helps reduce the memory footprint associated with gradients. This can be especially helpful when you are memory constrained. Note that users can only use ONE of gradient accumulation or optimizer in backward. These features currently do not work together. For more details on optimizer in backward, please see this tutorial: https://pytorch.org/tutorials/intermediate/optimizer_step_in_backward_tutorial.html - Lower precision optimizers. This recipe supports lower-precision optimizers from the bitsandbytes library (https://huggingface.co/docs/bitsandbytes/main/en/index). We've tested the recipe with 8-bit AdamW and Paged AdamW. These optimizers are especially helpful when you are memory constrained since they help reduce the memory footprint associated with the optimizer states. - Checkpointing. Model weights are checkpointed both at the end of each epoch and at the end of training. Optimizer State and recipe state (seed, total_epochs, number of epochs run etc) are only saved at the end of a given epoch and used in case of resuming training. Resuming training is controlled by the ``resume_from_checkpoint`` flag. Mid-epoch checkpointing is currently not supported. For more details on the checkpointer, please take a look at our checkpointer deepdive (https://pytorch.org/torchtune/main/deep_dives/checkpointer.html). - Logging. Terminal, Disk, WandB and TensorBoard are all supported. - Gradient Clipping. Gradient clipping is supported using the ``clip_grad_norm`` flag. By default, ``clip_grad_norm`` is set to ``None``. If you only want to log the grad norm, you can set ``clip_grad_norm='inf'``. For a full list of example configs for this recipe, run ``tune ls`` on the command line. Each config has example commands for how to kick-off training. Args: cfg (DictConfig): OmegaConf object parsed from yaml file Raises: ValueError: If ``dtype`` is set to fp16. RuntimeError: If ``dtype`` is set to bf16 and the hardware does not support bf16. RuntimeError: If ``gradient_accumulation_steps > 1`` and ``optimizer_in_bwd`` is `True`. RuntimeError: If ``left_pad_sequence`` is set as the data collator. RuntimeError: If ``enable_activation_offloading`` is True and device is not CUDA. RuntimeError: If ``enable_activation_offloading`` is True and ``enable_activation_checkpointing`` is False. cfgreturnNctj|j|_t j|j|j|_|jtj krtd|j |_ | dd|_| dd|_|jr1|jjdkr!t"dd|_|j|_|j|_|j|_| d d|_|jr0|jt5d |jdkrt5d | d d|_| d d|_|jr6|jjdkrt5d|jst5dn1|jr*|jjdkrtjt"dt j |j!| dd|_!d|_"|j#|_$|j%|_%d|_&dS)NdevicezVfull fp16 training is not supported with this recipe. Please use bf16 or fp32 instead.log_every_n_stepslog_peak_memory_statsFcpuzglog_peak_memory_stats was set to True, however, training uses cpu. Setting log_peak_memory_stats=False.clip_grad_normzsGradient clipping is not supported with optimizer in bwd.Please set clip_grad_norm=None, or optimizer_in_bwd=False.zGradient accumulation is not supported with optimizer in bwd.Please set gradient_accumulation_steps=1, or optimizer_in_bwd=False.enable_activation_checkpointingenable_activation_offloadingcudazFenable_activation_offloading should only be True when training on CUDAz]enable_activation_offloading should only be True when enable_activation_checkpointing is True LLAMA3_VISIONzHint: enable_activation_checkpointing is True, but enable_activation_offloading isn't. Enabling activation offloading should reduce memory further.cudnn_deterministic_mode)seed debug_moder)'r get_devicer"_devicer get_dtypedtype_dtypetorchfloat16 ValueError output_dir _output_dirget_log_every_n_steps_log_peak_memory_statstypeloginforesume_from_checkpoint_resume_from_checkpointgradient_accumulation_steps_gradient_accumulation_stepsoptimizer_in_bwd_optimizer_in_bwd_clip_grad_norm RuntimeError _enable_activation_checkpointing_enable_activation_offloading checkpointer model_type log_rank_zeroset_seedr- epochs_runepochs total_epochsmax_steps_per_epoch global_step)selfrs w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/recipes/full_finetune_single_device.py__init__z'FullFinetuneRecipeSingleDevice.__init__ws'sz::: (4<HHH  ;%- ' 'h  >"%''*=q"A"A&)gg.Eu&M&M#  & 04<+<+E+E HHy   +0D '(+'A$,/,K)!$!5"ww'7>>  ! #/"Q0144"[ 14 -u1 1 -.1WW *E. . *  - | F**"\8 "s   1  +>>  O   %cgg.H$&O&O    J#&#: cfg_checkpointerctj||j|_|j}|jr|||S)z Extract the checkpoint state from file and validate. If resume_from_checkpoint is True, this also includes the recipe state. )should_load_recipe_state)r instantiater@ _checkpointerload_checkpoint_update_recipe_state)rRrVcheckpoint_dicts rSr[z.FullFinetuneRecipeSingleDevice.load_checkpointsb $/ %)%A   ,<<>>  ' 7  % %o 6 6 6rU ckpt_dictc8 |tj|_|j|tjkr:t d|tj|tj|_|j|tjkr:t d|tj|tj|_|j|tj krt d|jdSdS#t$r}td|d}~wwxYw)z; Updates the recipe state from checkpoint. zWConfig value for seed does not match the checkpoint value, using the checkpoint value: )messagezfConfig value for max_steps_per_epoch does not match the checkpoint value, using the checkpoint value: z[Config value for total_epochs does not match the checkpoint value, using the config value: zCheckpoint does not contain the required keys needed for updating recipe state. Are you sure you passed in the right recipe checkpoint?N) r EPOCHS_KEYrMr-SEED_KEYrrP MAX_STEPS_KEYrOTOTAL_EPOCHS_KEYKeyError)rRr^es rSr\z3FullFinetuneRecipeSingleDevice._update_recipe_statesf" '(;_setup_optimizer optimizerrCr@OPT_KEY _optimizerloss_loss_fn compile_loss __class____name__set_num_output_chunksnum_output_chunksr9 _setup_datadatasetrqrrDATALOADER_KEY _dataloaderlenrB_steps_per_epochrPrMrQ_setup_lr_schedulerrO _lr_scheduler_setup_profilerr _profilerr4full ignore_indexr0ignore_labels_cache)rRrr^ collate_names rSsetupz$FullFinetuneRecipeSingleDevice.setups %01BCC &&s+++(()9::   :  3; d ''i,0,Q)-)K-&x'9: (   !,S];; 6777//- 1/3/KU (*++QU 0  *3844 = 1  !$- 0 0 0 = " +/H H H K - -dm.M N N N '(((ww|-PQQ ++ K~#/ (122,   (  ! !T%F F   $ 0(4+@@@$($ wait, warmup_steps -> warmup, active_steps -> active, num_cycles -> repeat wait_steps: int warmup_steps: int active_steps: int num_cycles: int NenabledF _component_z'torchtune.training.setup_torch_profilerzdOnly torch profiler supported currently: component must be `torchtune.training.setup_torch_profiler`z& Profiler config after instantiation: profile_memory wait_steps warmup_steps active_steps) r r9rrYr=r>profiler_profile_memoryprofiler_wait_stepsprofiler_warmup_stepsprofiler_active_steps)rRrprofiler profiler_cfgs rSrz.FullFinetuneRecipeSingleDevice._setup_profiler^sV  %y%&899L   M4 0 0 8*SL ' '  //<===u==="(!3L!A!A, H,HHIII'3'7'78H%'P'P$  " F'3L'AD $)5n)ED &)5n)ED &rUrir(r)rjrkctj|j5|j5t j|}dddn #1swxYwYdddn #1swxYwY|rtj||r!tj|tj h| |tj | |jtj |||_td|jd|jjdkr.tj|j}tj||S)zO Set up the model including enabling activation checkpointing. N)auto_wrap_policy)r2z$Model is initialized with precision .r&r!)rset_default_dtyper3r0rrYrjset_activation_checkpointingrTransformerSelfAttentionLayerload_state_dictvalidate_expected_param_dtypenamed_parametersget_act_offloading_ctx_manageractivations_handling_ctxr=r>r<get_memory_statslog_memory_stats)rRrir(r)rjrkr~ memory_statss rSr}z+FullFinetuneRecipeSingleDevice._setup_models ' 4 4 2 2dl 2 2&y11E 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2  *  "5 ) ) ) *   1)N(O     ./// .  " " $ $DK    )1(O /) ) % F FFFGGG <  % %#4DLIIIL  %l 3 3 3 s3AA AA A A AA!AFrlrCrmc>|rfd|jD}tj|j|tj|j||_|> |j|n"#t$r}td|d}~wwxYwt ddStj |j}|r||t d|S)zp Set up the optimizer. This method also handles loading the optimizer state_dict, if specified. c>i|]}|tj|gS)rrY).0prls rS zCFullFinetuneRecipeSingleDevice._setup_optimizer..s:6%maS99rU)r~ optim_dictNzzFailed loading in-backward optimizer checkpoints.Please make sure run being restored from was using in-backward optimizer.z"In-backward optimizers are set up.zOptimizer is initialized.) r parametersrregister_optim_in_bwd_hookscreate_optim_in_bwd_wrapper_optim_ckpt_wrapperr BaseExceptionrFr=r>rrY)rRrlrCrmrrfrs ` rSrz/FullFinetuneRecipeSingleDevice._setup_optimizers[ ! //11J  0kj    (0'Kkj(((D $ ),<<^LLLL$&d HH9 : : :4*=$+:P:P:R:RSSI :)).999 HH0 1 1 1 s'B B! BB!rurvrwcv|tddS|jr9tt |jj}n|j}tj ||||}|jr|j |td|S)aH Set up the learning rate scheduler based on the provided configuration. It handles both standard optimization and optimizer-in-backward cases, and supports schedulers from both torchtune.modules and torch.optim. Args: cfg_lr_scheduler (Optional[DictConfig]): The learning rate scheduler configuration. num_training_steps (int): The total number of training steps. last_epoch (int): The index of the last epoch. Returns: lr_scheduler (Optional[Optimizer]): The learning rate scheduler. NzDNo learning rate scheduler configured. Using constant learning rate.)rvrwz'Learning rate scheduler is initialized.) r=r>rDnextiterr optim_mapvaluesrrrYset_lr_scheduler)rRrurvrwrrts rSrz2FullFinetuneRecipeSingleDevice._setup_lr_schedulers&  # HHV   4  ! (T$":"D"K"K"M"MNNOOIII)  1!      ! D  $ 5 5l C C C :;;;rUrprqrrrorsc t|tr0fd|D}t|}t|dd}n0t j|j}|dd}d|vrtdt|}t|dd|d } t||| |s&t|jj jj nt d } | S) z All data related setup happens here. This recipe currently supports only map-style datasets. If a state_dict is provided (meaning we are resuming a training run), it is loaded into the dataloader. cDg|]}tj|jSr)rrYr)rsingle_cfg_datasetrRs rS z>FullFinetuneRecipeSingleDevice._setup_data..4s9&"#5tGGrU)datasetspackedFleft_pad_sequencez1left_pad_sequence collator is only for inference.r$r) num_replicasrankrqr-) padding_idx ignore_idxT)rrrsamplerro drop_last) isinstancer rgetattrrrYrr9rFrrr rpad_idrrr) rRrprqrrrorsrdsrr dataloaders ` rSrz*FullFinetuneRecipeSingleDevice._setup_data&s; k: . . 6*5H111BR511FF#KAAB __Xu55F * , ,RSS S-j99 ,     (! + $ 6#}9 +   "rUepochc ^tj|ji}|dz|jkr|tj|jtj|j tj |jtj |j tj |ji|js'|j|tj<n&|j|tj<|j|||dz|jkdS)z Save state dict to file. The recipe save_checkpoint method is responsible for correctly creating the checkpoint dict and passing to the checkpointer. r$)rintermediate_checkpointN)rrr state_dictrOupdaterbr-rarMrdrcrPrrrDrrrrZsave_checkpoint)rRrr^s rSrz.FullFinetuneRecipeSingleDevice.save_checkpoint]s ')?)?)A)AB 19t( ( (   %ty'-t/@*D,D+T-=-H-H-J-J    ) T.2o.H.H.J.J (*++.2.F.Q.Q.S.S (*+ ** %*QY1B%B +     rUbatchc|d}|j5|jdi|}dddn #1swxYwYtj|dddf|jd|jdf}t|ts>| d}| d| d}| ||}~|S)Nlabels.r$rr) poprrr4hstackrshaperlistreshapesizer)rRrrlogitsrs rS _loss_stepz)FullFinetuneRecipeSingleDevice._loss_stepxs"8$$  * * * T[))5))F * * * * * * * * * * * * * * *  CG_d67Ha7HI J  &$'' 9^^B''F^^B B88F}}VV,,  s 7;;c P |jrtd|js|jt j}d}d}|j t|j |j D]}t|j}|jj|t%|jD]\}}|dkrM|jrF||j|jzkr3|jjdkr#t0jjt9j||j|d|jjk }||z }|!||z} || z }| "|dz|j#zdkr|jstIj%|j&d|z |j'Ot0j(j)|j&*tW|j'} |j,|jd |j-|j-,|xj.dz c_.|/|z } |0d|1|dzd |j.d | |j.|j2zdkrt j|z } | tg|js|jn|j4|| z d } |jjdkr4|j5r-| 0tIj6|j|j'| 0d| i|j78| |j.d}d}t j}|dkrW|jrP||j|jz|j9zkr5|jjdkr%t0jjd|j,|dz|j#z|j:krn|xj dz c_ |;||j<dS)z The core training loop. Supports training on subsets of the dataset using the ``max_steps_per_epoch``. zpNOTE: torch.compile is enabled and model is compiled in first forward. Expect a relatively slow first iteration.r)totalr*rr$N)max_normT) set_to_none|z|Loss: )rlrtokens_per_second_per_gpur&r! grad_norm)step)r)r)=r|r=r>rDr zero_gradtime perf_counterrstartrangerMrOrrrr set_epoch enumeraterrrr0r<r4r*memory_record_memory_historyrbatch_to_devicerrsumrbackwardrBr scale_gradsrrEr clip_grad_norm_rfloatrrrQitemrset_descriptionr:rrr;rrylog_dictrrPrstop)rRt0 running_loss num_tokens curr_epochpbaridxrcurrent_num_tokens current_lossr loss_to_log time_per_stepr s rStrainz$FullFinetuneRecipeSingleDevice.trains =  HHC   % ( O % % ' ' '     1BCCj 3j 3Jd3444D   $ . .z : : :'(899d d  U!OO4$t7$:TTTT )V33J%<<>>>%eT\::: (Ot}'AA#%%#00  $u558JJ  , %%'''!Gt@@AEE1D ,T[!j.III/;(-(F(F $ 6 6 8 8).t/C)D)D)G))I,,...11d1CCC)5*//111$$)$$"."3"3"5"5 "BKKKNNN((%>RRD,<RR[RR '$*AAQFF(,(9(;(;b(@ $/#),0+A%BDOO)-)A ##:Dm9S $ $ <,55$:U5$OO ( 9 N N N /;$OO[),DEEE+44$!%!15 $%L!"J*,,B!OO4$/010111 )V33J%<r:s ------------ ,,,,,,,,!!!!!!<<<<<<LLLLLL666666666666<<<<<<000000,,,,,,999999::::::::333333ewq $q $q $q $q $%6q $q $q $h Z D     z CH[[]]rU