PiddlZddlZddlmZddlmZmZmZmZm Z m Z ddl m Z ddl Z ddlmZmZddl mZddlmZmZddlmZdd lmZmZdd lmZmZmZmZdd lm Z dd l!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-ddl.m.Z.ej/dZ0Gdde&Z1ej2deddfdZ3e4dkrej5e3dSdS)N)partial)AnyDictListOptionalTupleUnion)warn) DictConfig ListConfig)nn)destroy_process_groupinit_process_group) Optimizer) DataLoaderDistributedSampler)configmodulestrainingutils)_get_component_from_path)padded_collate_packed) ConcatDataset)FTRecipeInterface) DummyProfiler PROFILER_KEY)(apply_selective_activation_checkpointing)get_lr)tqdmDEBUGceZdZdZdeddfdZdedeeeffdZ deeefddfd Z deddfd Z d&d e ede ejjeffd Z d'd edededededeeefde eede ede ede edejfdZ d(dedede eeefde efdZdededed edeeeff d!Zd"eddfd#Zd)d$Z d)d%Z!dS)*QATRecipeDistributeda Quantization-aware training (QAT) recipe for dense transformer-based LLMs such as Llama2. This recipe supports distributed training and can be run on a single node (1 to 8 GPUs). Only compatible with PyTorch 2.4+. Features: - Quantization-aware training (QAT). Perform fake quantization on weights and/or activations during finetuning, with the goal of ultimately producing a quantized model with minimal accuracy degradation. This recipe produces an unquantized model in the original dtype, which can then be quantized separately. - Delayed fake quantization. Optionally specify the step after which fake quantization occurs. Empirically, allowing the model to finetune without fake quantization initially allows the weight and activation values to stabilize before fake quantizing them, potentially leading to improved quantized accuracy. This can be specified through ``fake_quant_after_n_steps``. - FSDP. Supported using PyTorch's FSDP APIs. CPU offload of parameters, gradients, and optimizer states is supported via ``fsdp_cpu_offload``. Resharding of parameters after the forward pass is done by default (corresponding to FULL_SHARD sharding strategy), but can be disabled by setting the config ``fsdp_reshard_after_forward`` to False (this corresponds to SHARD_GRAD_OP sharding strategy). DDP is currently not supported. Training on CPU is not supported. - Activation Checkpointing. This can be controlled using the ``enable_activation_checkpointing`` flag. Activation checkpointing helps reduce the memory footprint since we no longer keep activations in memory and instead recompute them during the backward pass. This is especially helpful for larger batch sizes when you're memory constrained. But these savings in memory come at the cost of training performance. In most cases training can slow-down quite a bit as a result of this activation recomputation. - Activation Offloading. This can be controlled using the ``enable_activation_offloading`` flag. Activation offloading is a technique similar to activations checkpointing that helps reduce the memory footprint to prevent OOMs on CUDA and enable bigger batches. Where activations checkpointing drops the activation in the forward to recompute it later in the backward, activations offloading will drop the activation in the forward to the CPU and bring it back during the backward pass. As always, there is a tradeoff--these savings in memory can come at the cost of training performance and CPU resources. To recover some runtime cost, we've added an option to enable offloading on a different stream to permit overlapping with the computation. This option is currently only available on PyTorch 2.5 or later and will be enabled by default if an acceptable torch version is found. Activation offloading can be used in conjunction with activation checkpointing. - Precision. Full fp32 and bf16 training are supported. Precision is controlled using the ``dtype`` flag. When ``dtype=bf16``, all activations, gradients and optimizer states are in bfloat16. In most cases this should halve the memory footprint of full precision (fp32) training, without loss in model quality (will depend on the model, training data and other settings). For GPUs which do not support bfloat16, we fall back to fp32. Mixed precision training and fp16 precision are currently not supported. - Gradient Accumulation. You can simulate larger batch sizes by accumulating gradients. This is controlled using the ``gradient_accumulation_steps`` flag. Total Batch Size = batch_size * number of GPUs * gradient accumulation steps. For example: with batch_size=1, nproc_per_node=2 and gradient_accumulation_steps=32 we get a total batch size of 64. Gradient accumulation is especially useful when you are memory constrained. In this case, accumulating gradients might give you better training speed than enabling activation checkpointing. - Checkpointing. Model weights are checkpointed both at the end of each epoch and at the end of training. Optimizer state and recipe state (seed, total_epochs, number of epochs run etc) are only saved at the end of a given epoch and used in case of resuming training. Resuming training is controlled by the ``resume_from_checkpoint`` flag. Mid-epoch checkpointing is currently not supported. For more details on the checkpointer, please take a look at our checkpointer deepdive (https://pytorch.org/torchtune/main/deep_dives/checkpointer.html). - Logging. Terminal, Disk, WandB and TensorBoard are all supported. - Gradient Clipping. Gradient clipping is supported using the ``clip_grad_norm`` flag. By default, ``clip_grad_norm`` is set to ``None``. If you only want to log the grad norm, you can set ``clip_grad_norm='inf'``. For a full list of example configs for this recipe, run ``tune ls`` on the command line. Each config has example commands for how to kick-off training. Args: cfg (DictConfig): OmegaConf object parsed from yaml file Raises: ValueError: If ``dtype`` is set to fp16. ValueError: If ``compile`` is set to True. RuntimeError: If ``dtype`` is set to bf16 and the hardware does not support bf16. RuntimeError: If ``left_pad_sequence`` is set as the data collator. RuntimeError: If ``enable_activation_offloading`` is True and device is not CUDA. RuntimeError: If ``enable_activation_offloading`` is True and ``enable_activation_checkpointing`` is False. cfgreturnNctj|j|_t j|j|j|_|jtj krtd| ddrtd|j |_ | dd|_| dd|_|jr1|jjd kr!t"d d|_tj\|_|_|jd k|_|j|_|j|_| d d|_| d d|_| dd|_d|_|jr0|jt?d|jdkrt?d| dd|_ | dd|_!|j!r6|jjd krt?d|j st?dn1|j r*|j"j#dkrtj$t"dt j%|j&| dd|_&d |_'|j(|_)|j*|_*d |_+dS)NdevicezVfull fp16 training is not supported with this recipe. Please use bf16 or fp32 instead.compileFz?Compile is not yet supported for QAT. Please set compile=False.log_every_n_stepslog_peak_memory_statscudazplog_peak_memory_stats was set to True, however, training does not use cuda. Setting log_peak_memory_stats=False.roptimizer_in_bwdclip_grad_normfake_quant_after_n_stepszsGradient clipping is not supported with optimizer in bwd.Please set clip_grad_norm=None, or optimizer_in_bwd=False.zGradient accumulation is not supported with optimizer in bwd.Please set gradient_accumulation_steps=1, or optimizer_in_bwd=False.enable_activation_checkpointingenable_activation_offloadingzFenable_activation_offloading should only be True when training on CUDAz]enable_activation_offloading should only be True when enable_activation_checkpointing is True LLAMA3_VISIONzHint: enable_activation_checkpointing is True, but enable_activation_offloading isn't. Enabling activation offloading should reduce memory further.cudnn_deterministic_mode)seed debug_mode),r get_devicer'_devicer get_dtypedtype_dtypetorchfloat16 ValueErrorget output_dir _output_dir_log_every_n_steps_log_peak_memory_statstypeloginfoget_world_size_and_rank world_sizerank _is_rank_zeroresume_from_checkpoint_resume_from_checkpointgradient_accumulation_steps_gradient_accumulation_steps_optimizer_in_bwd_clip_grad_norm_fake_quant_after_n_steps_quantizer_mode RuntimeError _enable_activation_checkpointing_enable_activation_offloading checkpointer model_type log_rank_zeroset_seedr4 epochs_runepochs total_epochsmax_steps_per_epoch global_step)selfr#s k/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/recipes/qat_distributed.py__init__zQATRecipeDistributed.__init__s 'sz::: (4<HHH ;%- ' 'h  779e $ $ Q  >"%''*=q"A"A&)gg.Eu&M&M#  & 04<+<+F+F HHC   +0D '%*%B%D%D"!Y!^(+'A$,/,K)!$);U!C!C"ww'7>>),1KT)R)R&#  ! #/"Q0144"[ 14 -u1 1 -.1WW *E. . *  - | F**"\8 "s   1  +>>  O   %cgg.H$&O&O    J#&#: cfg_checkpointerctj||j|_|j}|jr|||S)z Extract the checkpoint state from file and validate. If resume_from_checkpoint is True, this also includes the recipe state. )should_load_recipe_state)r instantiaterK _checkpointerload_checkpoint_update_recipe_state)r^rbcheckpoint_dicts r_rgz$QATRecipeDistributed.load_checkpointsb $/ %)%A   ,<<>>  ' 7  % %o 6 6 6ra ckpt_dictc8 |tj|_|j|tjkr:t d|tj|tj|_|j|tjkr:t d|tj|tj|_|j|tj krt d|jdSdS#t$r}td|d}~wwxYw)z; Updates the recipe state from checkpoint. zWConfig value for seed does not match the checkpoint value, using the checkpoint value: )messagezfConfig value for max_steps_per_epoch does not match the checkpoint value, using the checkpoint value: z[Config value for total_epochs does not match the checkpoint value, using the config value: zCheckpoint does not contain the required keys needed for updating recipe state. Are you sure you passed in the right recipe checkpoint?N) r EPOCHS_KEYrYr4SEED_KEYr r\ MAX_STEPS_KEYr[TOTAL_EPOCHS_KEYKeyError)r^rjes r_rhz)QATRecipeDistributed._update_recipe_statesf" '(;_compile _setup_modelmodelrSrTr MODEL_KEY_model tokenizer _tokenizer_setup_optimizer optimizerrNrKOPT_KEY _optimizerloss_loss_fn compile_loss __class____name__set_num_output_chunksnum_output_chunksrrWrD _setup_datadatasetrr_sampler _dataloaderlenrM_steps_per_epochr\rYr]_setup_profilerr _profilerr;full ignore_indexr7ignore_labels_cache)r^r#ri collate_names r_setupzQATRecipeDistributed.setup s   0"("4S5F"G"GD    * *3 / / /..@P.QQ 511 ''i,0,Q)-)K"%''*A4"H"H WW%7??"%''*F"M"M,X-?@GGIt,,ggk400''+t44(    !,S];;//-!3/ 011 0  *3844 = M  !$-9K L L L L = " +/H H H K - -dm.M N N N C!7888ww|-PQQ *.*:*: K~# +;+ + ' t'  ! !T%F F   $ 0(4+@@@$($ wait, warmup_steps -> warmup, active_steps -> active, num_cycles -> repeat wait_steps: int warmup_steps: int active_steps: int num_cycles: int NenabledF _component_z'torchtune.training.setup_torch_profilerzdOnly torch profiler supported currently: component must be `torchtune.training.setup_torch_profiler`z& Profiler config after instantiation: profile_memory wait_steps warmup_steps active_steps) r r>rrerrWrDrIprofiler_profile_memoryprofiler_wait_stepsprofiler_warmup_stepsprofiler_active_steps)r^rprofiler profiler_cfgs r_rz$QATRecipeDistributed._setup_profilerbsT  %y%&899L   M4 0 0 8*SL ' '  //<===u==="(!3L!A!A,  H,HH      J+7+;+;i|]}|tj|gS)rre).0paramr~s r_ z9QATRecipeDistributed._setup_optimizer.."s:v)-%AAra)r optim_dictzzFailed loading in-backward optimizer checkpoints.Please make sure run being restored from was using in-backward optimizer.z"In-backward optimizers are set up.zOptimizer is initialized.)r parametersrregister_optim_in_bwd_hookscreate_optim_in_bwd_wrapper_optim_ckpt_wrapperkeys#load_from_full_optimizer_state_dict optim_mapr7 BaseExceptionrRrrWrDrre)r^r~r-rrrrrrs ` r_rz%QATRecipeDistributed._setup_optimizers - ![3355J  0kj    (0'Kkj(((D $ )+0022 ! !E ! D K 4>uE*51 L  )!!!*h !!!  %I J J J4*=$+:P:P:R:RSSI <K"L   %@ A A A s=7B55 C?CCrrrrc ,t|tr0fd|D}t|}t|dd}n0t j|j}|dd}d|vrtdt|}t|j j |d}t|||d |s&t|jjjj nt$ } t'jt*d || fS) z All data related setup happens here. Currently this recipe only supports the DistributedSamplers with Map-style Datasets which fit into memory. Other samplers, iterable datasets and streaming datasets are not supported. cDg|]}tj|jSr)rrer)rsingle_cfg_datasetr^s r_ z4QATRecipeDistributed._setup_data..]s9&"#5tGGra)datasetspackedFleft_pad_sequencez1left_pad_sequence collator is only for inference.r) num_replicasrHrr4T) padding_idx ignore_idx)rrsampler drop_lastrz$Dataset and Sampler are initialized.) isinstancer rgetattrrrerr>rRrrrGrHrrpad_idrrrrrWrD) r^rrrrrdsrr dataloaders ` r_rz QATRecipeDistributed._setup_dataOsS k: . . 6*5H111BR511FF#KAAB __Xu55F * , ,RSS S-j99 $ T_49gTU    ! + $ 6#}9 +   " C!GHHH ""raepochc $i}|dz|jk}tjtdt j}t j|j|j |j }tjtdt j|z dd|rt j}tjtd|j s-t j |j|j |j |j }nPi}|jjD]/\}}t j |j||j |j ||<0tjtdt j|z ddnd }|j rt j}|t j|i|re|t j|t j|jt j|jt j|jt j|ji|j||| td t j|z ddt<j d S) a Checkpoint the state of the recipe. The constructed checkpoint state dict contains the following information: - Model weights with key training.MODEL_KEY - Relevant recipe state if training is not complete Checkpointer will save the model weights and recipe state in different checkpoint files. To correctly resume training from an intermediate checkpoint, the model weights and recipe state must be provided. r*zOSaving checkpoint. This may take some time. Retrieving full model state dict...r&z#Getting full model state dict took rrzGetting optimizer state dict...z"Getting optimizer state dict took N)rintermediate_checkpointzSaving checkpoint took )!r[rrWrDrrrgather_cpu_state_dictrrIr7rNget_full_optimizer_state_dictrrritemsupdaterrrnr4rmrYrpror\rfsave_checkpointrEr;rr) r^rrirstartcpu_state_dictrropts r_rz$QATRecipeDistributed.save_checkpoints"'!)d.?"?   ]   !##"7 K  <      X$2C2E2E2M X X X X   # "%''E  %F G G G) !)!GKO&< """"$"&":"D"J"J"L"LJE3,4,R S$*| 8d } | 8d | 9d } |.| | | z}~ ||z }|jrKt<j:;|t<j:;|||j<|z z}|=|d z|jzdkr|jst<j:;|t<j:;|tj>|j)|j<|z |j?at<j@j,A|j)Bt|j? D}|jE|jd |xj#d z c_#|F|z }|Gd |H|d zd|j#d||j#|jIzdkr|jrtj |z }|t|js|jn|j|||j<zz d}|jKr-|GtjL|j|j?|Gd|i|jMN||j#d}d}tj }|jr]|dkrW|jrP||j|jz|jOzkr5|jjdkr%t<jj !d|j E|xj d z c_ |P|Q|j QdS)z) The core training loop. r)totaldisableNr,z7Step 0: Disabling fake quant, will re-enable in step %szStep %s: Enabling fake quantlabels.r*)max_normT) set_to_none|z|Loss: )rlrtokens_per_second_per_gpur& grad_norm)step)r)rr)Rrcleanup_before_trainingrNr zero_gradrrvaluesrrrrrangerYr[r set_epochrrrH enumeraterr\rMrIrrrr7rCr;r,memory_record_memory_historyrPr]rDrEr_get_disable_fake_quantrQrapply_get_enable_fake_quantrbatch_to_devicerrsumpoprhstackrshaperlistreshapesizer all_reducerGbackward scale_gradsrOr clip_grad_norm_rfloat full_tensorritemrset_descriptionrArrBrrlog_dictrrstop)r^rt0 running_loss num_tokens curr_epochpbaridxbatch disable_fq enable_fqcurrent_num_tokensr logits current_lossr loss_to_log time_per_stepr.s r_trainzQATRecipeDistributed.trains (***% O % % ' ' ' '/9@@BB        1BCCk 3k 3J M # #J / / /d3$)q.=QRRRD'(899b *b * U,8 AA/00E&?"a4(t7$:TTTT )V33J%<<>>>1='1,,U"<=&.%:%R%R 0&&  ))*5555)T-KKK:"<=%-$9$P$P 0%%  )))444%eT\::: (Ot}'AA#%%#00 8,,222(T[11511F222222222222222 CG_d&>?Pa?P&QR"&$//A#^^B//F#^^B B@@F $}}VV<>>$04?Z3O#PL%%'''!Gt@@AEE1D)44Z@@@)44\BBB!,T[$/J:VWWW/;(-(F(F $ 6 6 8 8).t/C)D)D)G))*kmm&,,...11d1CCC$$)$$"."3"3"5"5 "BKKKNNN((%>RRD,<RR[RR (4+BBaGG .H)-(9(;(;b(@ $/"(,0+A%BDOO)-)A ##:D,t>:@ $ $ 6$OO ( 9 N N N /;$OO[),DEEE+44$!%!15 $%L!"J*,,B* O&!OO 8,3454555!L-77 )@@@NNN N''))) OOq OO  z 2 2 2 2 sK""K& )K& cb|jr|jtdSN)rIrcloser)r^s r_cleanupzQATRecipeDistributed.cleanups4   (   % % ' ' 'rar@)NNNN)FN)r$N)"r __module__ __qualname____doc__r r`rstrrrgrhrrr r;rprofilerrboolrintr Modulerrrrrrrrr>rBrrar_r"r"$sYYvSJS4SSSSj  tCH~    &d38n&&&&&PT T T T T T n48BB$Z0B u~%}4 5BBBBX6:!%#'.2rrr*.r'+ r  r $ rsCx.r (S 2r#rC=r +r rrrrn"'37 33!33!c3h0 3 )  3333j3#3#3# 3#  3# !:- . 3#3#3#3#jX$X$ X$X$X$X$tBBBBH      rar"r#r$ctjstdtd|ddrtjt jd|t|}| || | dS) z Entry point for the recipe. Configurable parameters are read in the following order: - Parameters specified in config (see available configs through ``tune ls``) - Overwritten by arguments from the command-line zDistributed finetune recipe should be run via a distributed launcher.If using tune CLI, please specify --nnodes 1 and --nproc_per_node [num_gpus]zcuda:nccl,cpu:glooruFr") recipe_namer#)r#N) ris_distributedrRrr>set_torch_num_threadsrrr"rr>rB)r#recipes r_ recipe_mainrPs  " $ $  [   +,,, ww!5))) &((( "8cBBBB !c * * *F LLSL LLNNN NNra__main__)6sysr functoolsrtypingrrrrrr warningsr r; omegaconfr r r torch.distributedrr torch.optimrtorch.utils.datarr torchtunerrrrtorchtune.config._utilsrtorchtune.datartorchtune.datasetsrtorchtune.recipe_interfacesrtorchtune.trainingrrtorchtune.training.activationsr torchtune.training.lr_schedulersrr get_loggerrDr"parserPrexitrrar_resE :::::::::::::::: ,,,,,,,,GGGGGGGG!!!!!!;;;;;;;;666666666666<<<<<<000000,,,,,,999999::::::::SSSSSS333333ewA A A A A ,A A A HZD6 z CH[[]]ra