PiddlZddlZddlZddlZddlmZddlmZddlm Z m Z m Z m Z m Z mZddlmZddlZddlmZmZddlmZddlmZdd lmZmZdd lmZmZmZmZm Z m!Z!dd l"m#Z#dd l$m%Z%dd l&m'Z'ddl(m)Z)ddl*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0ddl1m1Z1e!j2dZ3dej4j_5Gdde)Z6ej7deddfdZ8e9dkrej:e8dSdS)N)partial)chain)AnyDictListOptionalTupleUnion)warn) DictConfig ListConfig)nn) Optimizer) DataLoaderDistributedSampler)config generationmodulesrlhftrainingutils)padded_collate) ConcatDataset)local_kv_cache)FTRecipeInterface)PPOStats Trajectory)disable_dropout DummyProfiler PROFILER_KEY)tqdmDEBUGc(eZdZdZdeddfdZdeddfdZ d8deedee j j e ffdZ d9d Zdeddfd Zd ed ed ededeejejejejff dZdededededeeefdeeefdeeefdeeefdeejejejffdZ d:dededeeeefdefdZdeded edeeeffd!Z d;d"ed#eddfd$Z!d%eeefddfd&Z"d'e j#de$fd(Z%d'e j#de$fd)Z&d9d*Z'd+e$d,e j#d-e j#d.ede(f d/Z)d+e$d0e(d1e j#d2e j#d3e j#d4e j#ddfd5Z*d+e$d0e(d,e j#d-e j#d1e j#d2e j#ddfd6Z+d9d7Z,dS)). Features: - Activation Checkpointing. This can be controlled using the ``activation_checkpointing`` flag. Activation checkpointing helps reduce the memory footprint since we no longer keep activations in memory and instead recompute them during the backward pass. This is especially helpful for larger batch sizes when you're memory constrained. But these savings in memory come at the cost of training performance. In most cases training can slow-down quite a bit as a result of this activation recomputation. - Precision. Full fp32 and bf16 training are supported. Precision is controlled using the ``dtype`` flag. When ``dtype=bf16``, all activations, gradients and optimizer states are in bfloat16. In most cases this should halve the memory footprint of full precision (fp32) training, without loss in model quality (will depend on the model, training data and other settings). For GPUs which do not support bfloat16, we fall back to fp32. Mixed precision training and fp16 precision are currently not supported. - Adjusting batch sizes when memory constrained. This recipe uses three different batch sizes: - ``batch_size`` controls the total number of samples which are sampled from the dataset for a single trajectory. - ``forward_batch_size`` controls the mini-batch size for trajectory generation. Since gradients are disabled during trajectory generation, memory consumption is lower and this can be higher than ``ppo_batch_size``. - ``ppo_batch_size`` controls the number of samples used for a single optimization step during PPO optimization. Since we're optimizing two models at once, adjusting this parameter can have a big impact during training. - Gradient Accumulation. You can simulate larger ``ppo_batch_size`` sizes by accumulating gradients. This is controlled using the ``gradient_accumulation_steps`` flag. For example: with ``ppo_batch_size``=32 and ``gradient_accumulation_steps``=16, each backward pass during PPO optimization uses a 'micro batch size' of 2. Gradient accumulation is especially useful when you are memory constrained. In this case, accumulating gradients might give you better training speed than enabling activation checkpointing. - Optimizer in Backward. Fusing the optimizer step into the backward pass helps reduce the memory footprint associated with gradients. This can be especially helpful when you are memory constrained. Note that users can only use ONE of gradient accumulation or optimizer in backward. These features currently do not work together. For more details on optimizer in backward, please see this tutorial: https://pytorch.org/tutorials/intermediate/optimizer_step_in_backward_tutorial.html This paramater can provide significant performance gains, since there the number of optimization steps scales with ``ppo_epochs`` and ``batch_size``. Depending on the maximum sequence length sampled from the dataset, we've found that setting ``ppo_batch_size`` to the highest you can fit in memory, and `optimizer_in_bwd=True` to provide significant memory savings. - Lower precision optimizers. This recipe supports lower-precision optimizers from the bitsandbytes library (https://huggingface.co/docs/bitsandbytes/main/en/index). We've tested the recipe with 8-bit AdamW and Paged AdamW. These optimizers are especially helpful when you are memory constrained since they help reduce the memory footprint associated with the optimizer states. - Checkpointing. Model weights are checkpointed both at the end of each epoch, and at the end of training. Optimizer State and recipe state (seed, total_epochs, number of epochs run etc) are only saved at the end of a given epoch and used in case of resuming training. Resuming training is controlled by the ``resume_from_checkpoint`` flag. Mid-epoch checkpointing is currently not supported. For more details on the checkpointer, please take a look at our checkpointer deepdive (https://pytorch.org/torchtune/main/deep_dives/checkpointer.html). - Logging. Terminal, Disk, WandB and TensorBoard are all supported. Args: cfg (DictConfig): OmegaConf object parsed from yaml file Raises: RuntimeError: If ``dtype`` is set to fp16. cfgreturnNctj|j|_t j|j|j|_|jtj krtd|j |_ | dd|_| dd|_|jr1|jjdkr!t"dd|_t j|j| d d |_tj|j|j|_d |_d |_d |_d |_d |_|j|_|j|_ dS) NdevicezVfull fp16 training is not supported with this recipe. Please use bf16 or fp32 instead.log_every_n_stepslog_peak_memory_statsFcudazplog_peak_memory_stats was set to True, however, training does not use cuda. Setting log_peak_memory_stats=False.cudnn_deterministic_mode)seed debug_moder)!r get_devicer*_devicer get_dtypedtype_dtypetorchfloat16 RuntimeError output_dir _output_dirget_log_every_n_steps_log_peak_memory_statstypeloginfoset_seedr0 Generator manual_seed_rng _total_steps _steps_run _total_epochs _epochs_run global_stepresume_from_checkpoint_resume_from_checkpointgradient_accumulation_steps_gradient_accumulation_steps)selfr&s {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/recipes/ppo_full_finetune_single_device.py__init__z*PPOFullFinetuneRecipeSingleDevice.__init__oso'sz::: (4<HHH  ;%- ' 'h  >"%''*=q"A"A&)gg.Eu&M&M#  & 04<+<+F+F HHC   +0D '%cgg.H$&O&O    ODL11==diHH (+'A$,/,K)))c tj|j_j||j|j|j|j \_ }_ }j }| }j }| }|j _ |j_|j|j|jj |t(j|t(j|t(j|t(j\____tj|j_t8d|j|jj r|t(j!nd_"tj|j#_$t8d%|j&|j'|j(\_)_*+|,|fd_-j r.|j/j0zj(j1zz_23|4tjd_6dS)z Sets up the recipe state correctly. This includes setting recipe attributes based on the ``resume_from_checkpoint`` flag. ) cfg_modelcfg_reward_value_modelenable_activation_checkpointing compile_modelpolicy_state_dictref_policy_state_dictvalue_model_state_dictreward_model_state_dictz#Tokenizer is initialized from file.N) cfg_optimizeroptimizer_in_bwdopt_state_dictzLoss is initialized.) cfg_datasetshuffle batch_sizec~|r(tjjjj|nt jS)N)rar5r*decoder_max_seq_len)r _policy_model_forward_batch_sizer6r3 contextlib nullcontext)enable_kv_cachercrOs rPz9PPOFullFinetuneRecipeSingleDevice.setup..sO *N"3k|$7     '))rR)7r instantiate metric_logger_metric_logger log_config_setup_checkpointers checkpointerref_policy_checkpointervalue_checkpointerreward_checkpointer_policy_checkpointer_value_checkpointerload_checkpointcompiler]_optimizer_in_bwd _setup_models policy_modelreward_and_value_modelrVr MODEL_KEYrd _value_model _reward_model_ref_policy_model tokenizer _tokenizerr@rA_setup_optimizer optimizerrLOPT_KEY _optimizerloss_loss_fn _setup_datadatasetr`ra_sampler _dataloader_setup_training_parameters_setup_training_hyperparameterscache_ctx_manager_update_recipe_staterG _ppo_epochs_ppo_batch_sizerJ_setup_profilerr<r _profiler)rOr&rprrpolicy_model_checkpoint_dictrYvalue_model_checkpoint_dictr[s` rPsetupz'PPOFullFinetuneRecipeSingleDevice.setups %01BCC &&s+++  % %    '  "  #     % #  $ (,'@'P'P'R'R$ 7 G G I I'+&>&N&N&P&P#"5"E"E"G"G { !$!5   &#&#=,/,O,:8;MN"78J"K#>x?Q#R$;H> 0  *3844  '(((+/*:*: K~+;+ + ' t' '',,, ,,S111 " " " "   ' D  % %&B C C C O $"66 8 --cgglD.I.IJJrR cfg_profilerc|tddi}|ddd|d<n#|ddks Jdtj|\}}td||dd|_|dr'|d |_|d |_|d |_ |S) a Parses the `profiler` section of top-level `cfg` and sets up profiler Args: cfg_profiler (Optional[DictConfig]): ``profiler`` section of the top-level ``cfg`` (the main config passed to `recipe.main`). Default None. Returns: profiler: Union[torch.profiler.profile, DummyProfiler] - DummyProfiler is a nullcontext with no-op methods for `start`, `stop`, and `step` that can be used in place of `torch.profiler.profile` if profiler is not enabled such that the instrumented training loop does not need to be changed profiling is disabled. The profiler config can be provided in configs under the `profiler` key with the following layout: .. code-block:: yaml profiler: enabled: bool #Output directory of trace artifacts output_dir: str #`torch.profiler.ProfilerActivity` types to trace cpu: bool cuda: bool #Trace options profile_memory: bool with_stack: bool record_shapes: bool with_flops: bool # `torch.profiler.schedule` options: # wait_steps -> wait, warmup_steps -> warmup, active_steps -> active, num_cycles -> repeat wait_steps: int warmup_steps: int active_steps: int num_cycles: int NenabledF _component_z'torchtune.training.setup_torch_profilerzdOnly torch profiler supported currently: component must be `torchtune.training.setup_torch_profiler`z& Profiler config after instantiation: profile_memory wait_steps warmup_steps active_steps) r r<rrjr@rAprofiler_profile_memoryprofiler_wait_stepsprofiler_warmup_stepsprofiler_active_steps)rOrprofiler profiler_cfgs rPrz1PPOFullFinetuneRecipeSingleDevice._setup_profilersV  %y%&899L   M4 0 0 8*SL ' '  //<===u==="(!3L!A!A, H,HHIII'3'7'78H%'P'P$  " F'3L'AD $)5n)ED &)5n)ED &rRc4|j|_|j|_|j|_|j|_|j|_ |j |_ |j |_ |j|_|j|_|j|_|ddr6|j}|jj|vr t1d|jjd|dn3t3|jdst1dg}n |jj}t7j||j|_d S) z Sets up the training hyperparameters for the recipe. This includes the GAE hyperparameters, generation hyperparameters, reward masking hyperparameters, and stop token ids. stop_token_idsFztokenizer eos_id (z) is not in stop_token_ids (z().This may lead to unexpected behaviour. stop_tokenszkNo stop tokens defined in tokenizer, and no stop_token_ids provided. This may lead to unexpected behaviour.r)N)kl_coeff _kl_coeffgamma_gammalmbda_lmbdawhiten_rewards_whiten_rewards temperature _temperaturetop_k_top_kmax_generated_tokens_max_generated_tokensmin_response_length_min_response_lengthpenalise_no_eos_penalise_no_eosreward_penalty_reward_penaltyr<rreos_idr hasattrrr7tensorr3_stop_token_ids)rOr&rs rPrzAPPOFullFinetuneRecipeSingleDevice._setup_training_hyperparametersJs9 i i "1 Oi %(%="%($;! # 3"1 77#U + + = /N%^;;=)?==]k=== 4?M:: =B"$!%!<$|N4<PPPrRc |j|_|j|_|j|_|j|_|j|_|j|jz|_ |j |_ |j|jzdkr td|jd|jd|j|jzdkr td|jd|jd|j|jzdkr td|jd|jd|jdkr|j rtd |j|jz|_t!dt#|j}t'j|j|z |_|jdkr td |jd |jd |jt#|jkr5t-d |jdt#|jd|jd|j|kr,|j|zdkrt-d|jd|d|dt.d|jd|jdS)a Validates and sets up parameters for used during training and for tracking training state, batch sizes for model forward passes during trajectory generation, PPO minibatches, and PPO microbatches for gradient accumulation. Raises - ValueError if: - batch_size is not divisible by forward_batch_size - batch_size is not divisible by ppo_batch_size - ppo_batch_size is not divisible by gradient_accumulation_steps - num_steps is less than batch_size - gradient_accumulation_steps > 1 and optimizer_in_bwd is True rz batch_size (z3) must be exactly divisible by forward_batch_size (z).z/) must be exactly divisible by ppo_batch_size (zppo_batch_size (z<) must be exactly divisible by gradient_accumulation_steps (r,zGradient accumulation is not supported with optimizer in bwd.Please set gradient_accumulation_steps=1, or optimizer_in_bwd=False.z num_steps z% must be greater than the batch size .zThere are fewer total steps (z2, (num_steps//batch_size) than there are batches (z,) in the dataset. Training will stop after (z/) steps without saving intermediate checkpointsz num_steps (zD) is not exactly divisible by the number of batches in the dataset (z5). Intermediate checkpoints will only be saved every z steps.zTotal steps to run: z, Total epochs to run: N)raforward_batch_sizere ppo_epochsrppo_batch_sizerrMrN_ppo_backward_batch_sizerh ValueErrorrwr9 num_stepsrFmaxlenrmathceilrHr r@rA)rOr&batches_per_epochs rPrz"1,/,K)  $"C C % #2 ?T5 5 : :DtDD'+'?DDD  ?T1 1Q 6 6ppp      1 1 1   1 1Q 6 6 `cm``9J``EV```     a4#4 a aTM_ a a     rR policy_cfgref_policy_cfg value_cfg reward_cfgc~|jsT|j|jksJd|jd|j|j|jksJd|jd|jtj||j}tj|d}tj|d}tj|d}||||fS)z Sets up checkpointers for policy, reference policy, value, and reward models. Only the policy checkpoint handles recipe state for resuming from checkpoints. zrPolicy and reference policy should be loaded from the same checkpoint directoriesat the start of training. Found: z andzlPolicy and reference policy should be loaded from the same checkpoint filesat the start of training. Found: )should_load_recipe_stateF)rLcheckpoint_dircheckpoint_filesrrj) rOrrrrpolicy_checkpointerrprqrrs rPrnz6PPOFullFinetuneRecipeSingleDevice._setup_checkpointerss6"+ ,0MMMM34>4M33!033NMM ..2QQQQ54>4O55!255RQQ %0 %)%A    #)"4 %*# # #  $/ %*    %0 %*    #     rRrTrUrVrWrXrYrZr[c tj|j5|j5t j|} t j|} t j|} t j|} dddn #1swxYwYdddn #1swxYwY|rPtj| tj| tj| tj| |rBtj| tj htj| tj h| || |tj || | |tj || | |tj | |jtj | |jtj | |jtj | |jtd|jdt!| t!| | | | D] } d| _ | D] } d| _ |jjdkr.tj|j}tj|| | | | fS) zb Sets up the policy model, reference policy model, reward model, and value model. N)auto_wrap_policy)r5z&Models are initialized with precision rFr.r))rset_default_dtyper6r3rrjrWset_activation_checkpointingrTransformerSelfAttentionLayerload_state_dict update_state_dict_for_classifiernamed_parametersvalidate_expected_param_dtyper@rAreval parameters requires_gradr?get_memory_statslog_memory_stats)rOrTrUrVrWrXrYrZr[ryref_policy_model reward_model value_modelp memory_statss rPrxz/PPOFullFinetuneRecipeSingleDevice._setup_modelss* ' 4 4 E Edl E E!-i88L%1)<< !-.DEEL ,-CDDK  E E E E E E E E E E E E E E E E E E E E E E E E E E E E E E  1  "< 0 0 0  "#3 4 4 4  "; / / /  "< 0 0 0 *   10U/V      1w/T.U     $$%6777(()>??? 1 #\%B%B%D%D    $$%<=== 1 "K$@$@$B$B    ##$:;;; .  ( ( * *$+     .  ) ) + +4;     .  ( ( * *$+     .  - - / /t{     H$+HHHIII  %%% $$$ ((** $ $A#AOO!,,.. $ $A#AOO <  & &#4DLIIIL  %l 3 3 3[,8HHHs5BAA?3 B?B BB BBBFr\r]r^cJ|rfdt|j|jD}t j|j|t j|j|t j|j||_t j|j||_|> |j|n"#t$r}td|d}~wwxYwt ddStjt|j|j}|r||t d|S)Nc>i|]}|tj|gS)rrj).0rr\s rP zFPPOFullFinetuneRecipeSingleDevice._setup_optimizer..Xs:6%maS99rR)model optim_dictzzFailed loading in-backward optimizer checkpoints.Please make sure run being restored from was using in-backward optimizer.z"In-backward optimizers are set up.zOptimizer is initialized.)rrdrr|rregister_optim_in_bwd_hookscreate_optim_in_bwd_wrapper_optim_ckpt_wrapperr BaseExceptionr9r@rArrj)rOr\r]r^rers ` rPrz2PPOFullFinetuneRecipeSingleDevice._setup_optimizerOs , &1133T5F5Q5Q5S5SJ  0(Z      0'J    (0'K(Z(((D $(0'K'J(((D $ ),<<^LLLL$&d HH9 : : :4*d(3355t7H7S7S7U7UVVI :)).999 HH0 1 1 1 sC## D-C==Dr_r`rac >t|trfd|D}t|}ntj|j}t |dd|d}t|||dttdd d gjj  }||fS) z6 All data related setup happens here. cFg|]}tj|jS)r)rrjr)rsingle_cfg_datasetrOs rP zAPPOFullFinetuneRecipeSingleDevice._setup_data..s<&"#5QQQrR)datasetsrr,r) num_replicasrankr`r0Tlefttokenslabels) pad_direction keys_to_pad padding_idx)rsamplerra drop_last collate_fn) isinstancer rrrjrrrrrpad_id)rOr_r`rardsr dataloaders` rPrz-PPOFullFinetuneRecipeSingleDevice._setup_datas k: . . L*5H111BB#K4?KKKB$      !$%x0 O2      ""rRepochis_intermediate_checkpointctj|ji}tj|ji}|r|tj|jtj|j tj |j tj |j tj|jtj|ji|js'|j|tj<n&|j|tj<|j||||j||ddS)z Save state dict to file. The recipe save_checkpoint method is responsible for correctly creating the checkpoint dict and passing to the checkpointer. )r intermediate_checkpointFN)rr{rd state_dictr|updateSEED_KEYr0 EPOCHS_KEYrITOTAL_EPOCHS_KEYrH MAX_STEPS_KEYrF STEPS_KEYrGRNG_KEYrE get_staterwrrrrssave_checkpointrt)rOr r policy_ckpt_dictvalue_ckpt_dicts rPrz1PPOFullFinetuneRecipeSingleDevice.save_checkpoints_%.0B0M0M0O0OP#-t/@/K/K/M/MN & :  # #%ty')9-t/A*D,=&$di&9&9&;&;    ) :59_5O5O5Q5Q !122,7799!$ !11 $> 2    00 $) 1     rR ckpt_dictcp |j|tjks6|j|tjks|j|tjkrtdtj|tj|_|j |tj |tj |_ |tj|_|tj|_|tj|_dS#t $r}t |dd}~wwxYw)z; Updates the recipe state from checkpoint. zzConfigured value for seed, total_steps, or total_epochs does not match the value stored in checkpoint.)message)r0zCheckpoint does not contain the required keys needed for updating recipe state.Are you sure you passed in the right recipe checkpoint?N)r0rrrFrrHrr rBrE set_staterrrGrrIKeyError)rOrrs rPrz6PPOFullFinetuneRecipeSingleDevice._update_recipe_states"   Yx'8999$ (2H(III%83L)MMMF!)y9J/KLLLDI I   (*: ; < < <'(:;DO )(*@ AD !*8+D!ED ()<=D      J!!  sDD D5D00D5 input_idsc |j\}}||j||jz5t j|j||j|j|j|j j |j \}}dddn #1swxYwY|dd|df }||j j k}t j |}t j|} ~tj|||j} ~||| |} tj| |} tj| ||j} ~ ||| |} tj| |d} tj||j|j j \}}|t1j||gd| |}~t5j|}|d||zddddfd}|js|jr+tj|||j|j}|j ||<d | |<d | |<t1j!|d k||jdz kz|dz|}| }|"d|#dd }d | |<tI|| | | || ||||| S)a4 Generates a trajectory given the current policy and value models, the reference policy model, the reward model, and batch of inputs. This is done over the following steps: 1: Generate responses, and logits corresponding to the responses using the current policy, generating (query, response) pairs. 2. Estimate logprobs of the generated responses using the current policy. 3. Estimate values from the generated responses using the current value function. 4. Replace any tokens in the response after the first stop token (usually EOS token) with padding, producting truncated responses. 5. Run the reward model on the (query, truncated-response) pairs. 6. Mask out all the invalid values in the trajectory due to padding tokens. Args: input_ids (torch.Tensor): tensor of input token IDs with shape [b, seq_length] Returns: Trajectory: An instance of :class:`~torchtune.rlhf.Trajectory` comprising the current trajectory. )rc)rpromptrrrrrngN input_posmaskr,)dim)r%?rF) query_responseslogprobs ref_logprobsvaluesmasks position_idsresponse_padding_masksvalue_padding_masksvalue_seq_idxsscoresseq_lens)%shaperrhrrgeneraterdrrrrrEclone!get_causal_mask_from_padding_mask"get_position_ids_from_padding_maskrlogits_to_logprobsr~truncate_sequence_for_logprobsr|squeeze%truncate_sequence_at_first_stop_tokenrr}r7catrget_unmasked_sequence_lengthsgatherrrget_reward_penalty_maskrwherescatter_ unsqueezer)rOr_context_lengthr*logits responsesquery_response_padding_masksr.r/r+ ref_logitsr,r-r0r3r4reward_penalty_maskr2r1s rPgenerate_trajectoryz5PPOFullFinetuneRecipeSingleDevice.generate_trajectorys*&O>  # #  .1K K$   '1&9( %)%? -k-I''' #OV               $AAA~$67==?? '6$/:P'P$< (  "D (   )*69d>OPP ++ |%,  8^TT .z9dFWXX ""?lQV"WW4V^LLTTUWXX-1,V t+T_-C- - ) ## Iy),! 4 4 4"$   9:PQQq8n#&%) ##  +/*>F& ',/'(/2 +, \h)Ca)GG H qL    5::<<1:: ~''++U  '*"#+%%#9 3)    sAA==BBc^g}tj5td|j|jD]<}||||jz}|||= dddn #1swxYwYtttj t|S)a Generates a self.batch_size batch of trajectories using self._forward_batch_size batch sizes. See generate_trajectory for more details. Args: input_ids (torch.Tensor): tensor of input token IDs with shape [b, seq_length] Returns: Trajectory: An instance of :class:`~torchtune.rlhf.Trajectory`, comprising the current trajectory. rN) r7no_gradrangerareappendrLrmapr>zip)rOr trajectories batch_startbatch_input_idss rPgenerate_trajectory_batchedz=PPOFullFinetuneRecipeSingleDevice.generate_trajectory_batchedqs*, ]__ O O$Q9QRR O O "++0H"HH###D$<$<_$M$MNNNN  O O O O O O O O O O O O O O O O3uy#|*<==>>sAA<<BBcn |jrtd|js|jd}|jt|j |j }t|j |j D]}|j|t!|jD]\}}|dkrM|jrF||j|jzkr3|jjdkr#t.jj|d|j}|j\}}|}t=j} | |} t=j| z } tCj"| j#| j$| j%|j&| j'\} } }tCj(| j)| |j*|j+| j,\}}t=j}g}t|j-D]q}t/j.|j/|j}td|j/|j0D]1}||||j0z}g}td|j0|j1D]}}||||j1z}tetgtit.j5d| | }|6|7||||||~~|6tqtgtrtu||js4|j;|jd |xj<d z c_<3st=j|z }|xj d z c_ |j |j=zdkrF|>| tqtgt.j?tu|| ||| z ||z |@| |||| ||Ad |dkrW|jrP||j|jz|jBzkr5|jjdkr%t.jjd |j;|j |j krd }n|xj d z c_ |C|| |r|jDd S|jDd S)z The core training loop.zoNOTE: torch.compile is enabled and model is compiled in first forward.Expect a relatively slow first iteration.F)totalinitialrr.r)r.r))r&indexT) set_to_noner,N)r)r )Ervr@rArwr zero_gradrstartr!rFrGrOrIrHr set_epoch enumeraterrrrr3r?r7r.memory_record_memory_historytor5numeltime perf_counterrVrget_rewards_ppor3r+r,rr2estimate_advantagesr-rrr0rrandpermrarrrrQr index_selectrPppo_steprsumrRsteprJr= log_metricsstackcleanup_after_steprrrstop)rOtraining_completedpbar curr_epochidxbatchrErF num_tokenst0_traj trajectory traj_timerewardskl kl_rewards advantagesreturnst0_ppo ppo_stats batch_idxsimini_batch_idxsbatch_ppo_statsjbackward_batch_idxsbatch_trajectoryppo_times rPtrainz'PPOFullFinetuneRecipeSingleDevice.trains <  HH<    % ( O % % ' ' '" $+T_EEE 0$2DEEC C J M # #J / / /'(899t t  U!OO4$t7$:TTTT )V33J%<<>>>h**4<88$)K!>"[[]] +--!==eDD  -//'9 +/*>%'+N- ++'Z'+&>%KK%<< '''# G*,,,. t/00'.'.A!& !U!U!UJ"1dot7KLL%.%.*4QT=Q9Q5Q*R:<!&t3T5R""11A3B !A(E$E E3/0:!$$+(-(:,-.A%&%&%& %/ """" 0,,22 $ $4$./B$C$+,?$@$2 !"!"!1 0!((3sCRR - z1!!!^__2DEtGX  := J56 &&  & -!'  8   '"++ 69 :12;?--        %<rrrrllog_dictrJ)rOrxrr{r|rrrs rPrmz-PPOFullFinetuneRecipeSingleDevice.log_metricsos!',,..)@DDRHHLLNN%,1133jnnQ6G6G6L6L6N6NN&&))..""#**//11N''))$05577#.3355!*//11&++-- ) ; @ @ B B * 3 9 9 ; ; @ @ B B4P-C   <  & &4+F & OOH5T\JJJ K K K $$XD4D$EEEEErRc*|D]}~~|D]}~~~~~~dS)zH Cleanup tensors after each PPO step to free up memory. Nr)rOrxrr}r~r{r|vs rProz4PPOFullFinetuneRecipeSingleDevice.cleanup_after_stepsB  A   A     JJrRc 8|jdSN)rlclose)rOkwargss rPcleanupz)PPOFullFinetuneRecipeSingleDevice.cleanups !!#####rRr)r'N)FN)F)-__name__ __module__ __qualname____doc__r rQrrr r7rprofilerrrrr r CheckpointerrnboolrstrrrModulerxrrintrrrrrTensorrrLrVrrrjrmrorrrRrPr%r%$sHHT&LJ&L4&L&L&L&LPnKnKnKnKnKnKb48AA$Z0A u~%}4 5AAAAF&Q&Q&Q&QPH jH TH H H H T6 6 #6  6  6    6 6 6 6 pYIYI!+YI*. YI  YI  S> YI $CH~YI!%S#XYI"&c3hYI ry")RY. /YIYIYIYI|"'37 33!33!c3h0 3  3333j##%##04##BE## !:- .########L>C' ' ' 6:' ' ' ' ' Rd38n:@ U\@ j@ @ @ @ D?U\?j????,UUUUnO O LO  O  O  O O O O bFFF L F L F ',l F!& F FFFFBL    L L 0$$$$$$rRr%r&r'ctjd|t|}||||dS)z Entry point for the recipe. Configurable parameters are read in the following order: - Parameters specified in config (see available configs through ``tune ls``) - Overwritten by arguments from the command-line r%) recipe_namer&)r&N)rrmr%rrr)r&recipes rP recipe_mainrsb "E3OOOO .3 7 7 7F LLSL LLNNN NNrR__main__);rfrsysrd functoolsr itertoolsrtypingrrrrr r warningsr r7 omegaconfr r r torch.optimrtorch.utils.datarr torchtunerrrrrrtorchtune.datartorchtune.datasetsrtorchtune.modulesrtorchtune.recipe_interfacesrtorchtune.rlhfrrtorchtune.trainingrrr r! get_loggerr@_dynamocache_size_limitr%parserrexitrrRrPrsi :::::::::::::::: ,,,,,,,,!!!!!!;;;;;;;;HHHHHHHHHHHHHHHH)))))),,,,,,,,,,,,999999////////KKKKKKKKKKew)+ %E$E$E$E$E$(9E$E$E$P$ Z D     z CH[[]]rR