Pi`ddlZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl Z ddl mZmZddl mZddlmZmZddlmZdd lmZdd lmZdd lmZmZmZmZmZm Z dd l!m"Z"dd l#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*m+Z+ddl,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3ddl4m5Z5ddl6m6Z6e j7dZ8Gdde/Z9ej:deddfdZ;e"%''*=q"A"A&)gg.Eu&M&M#  & 04<+<+F+F HHC   +0D ' #(:E B B#+#C D,A$ $ $   43444 8:: D $!QY(+'A$"ww'7>>14 -u1 1 - %38444 JODL11==diHH cfg_checkpointerctj||j|_|j}|jr|||S)z Extract the checkpoint state from file and validate. If resume_from_checkpoint is True, this also includes the recipe state. rL)r instantiaterM _checkpointerload_checkpoint_update_recipe_state)rZr_checkpoint_dicts r\rdz1FullGRPOFinetuneRecipeDistributed.load_checkpointsb $/ #'#?   ,<<>>  ' 7  % %o 6 6 6r^cfg_ref_checkpointercntj|d|_|j}|S)zP Extract the reference checkpoint state from file and validate. Fra)rrb_ref_checkpointerrd)rZrgref_checkpoint_dicts r\load_ref_checkpointz5FullGRPOFinetuneRecipeDistributed.load_ref_checkpoints@"(!3 " " " #4DDFF""r^ ckpt_dictc |tj|_|j|tj|j|tjkr:td|tj|tj|_|j |tj krtd|j dSdS#t$r}td|d}~wwxYw)z; Updates the recipe state from checkpoint. zWConfig value for seed does not match the checkpoint value, using the checkpoint value: )messagez[Config value for total_epochs does not match the checkpoint value, using the config value: zCheckpoint does not contain the required keys needed for updating recipe state. Are you sure you passed in the right recipe checkpoint?N) r EPOCHS_KEYrVrY set_stateRNG_KEYr5SEED_KEYr rRTOTAL_EPOCHS_KEYKeyError)rZrles r\rez6FullGRPOFinetuneRecipeDistributed._update_recipe_states) ()<=D  I   (*: ; < < <yIh&7888V7@AR7SVV &h&78  Ih.G$HHHG373DGGIH   J  sC C C.C))C.c |jrtj|jr8t j|j|_|j|| |j }| |j }| dd|_||j|j| dd|j|tj|tj\|_|_t j|j|_||j|jr|tjnd|_t j|j|_|jr tj|j|j t?j tBd | d d }|"|j#|j$|j%||jr|tj&nd |_'tQ|j'|_)|j*|j)z|_+|,| dd|j-|j)z|j+dz |_.|/| t`d|_1|j2|_2|j3|_4|j5|_6|j7|_8|j%|_%|j9|_:|j;|_<|j=|_>|j?|_@| ddr6|jA}|jjB|vr td|jjBd|dn3t|jdstdg}n |jjE}tjG||jH|_IdS)z Setup the recipe. This includes training state (if resume_from_checkpoint is True), model, tokenizer, loss, optimizer, lr scheduler, sampler, and dataloader. )r_)rgcompileFcustom_sharded_layersN) cfg_modelr4rxr1model_state_dictref_model_state_dict) cfg_optimizeropt_state_dictverbosezLoss is initialized. collate_fnz)torchtune.dev.grpo.data.padded_collate_rl) cfg_datasetshuffle batch_sizerdataloader_state_dict lr_schedulerr.)cfg_lr_schedulernum_training_steps last_epochstop_token_idsztokenizer eos_id (z) is not in stop_token_ids (z().This may lead to unexpected behaviour. stop_tokenszkNo stop tokens defined in tokenizer, and no stop_token_ids provided. This may lead to unexpected behaviour.r+)Jr1rset_torch_num_threadsrKrrb metric_logger_metric_logger log_configrd checkpointerrkref_checkpointerr@_compile _setup_modelmodelrO MODEL_KEY_model _ref_model tokenizer _tokenizer_setup_optimizer optimizerrMOPT_KEY _optimizerloss_loss_fn compile_lossr log_rank_zerorD _setup_datadatasetrrDATALOADER_KEY _dataloaderlen_steps_per_epochrVrS_setup_lr_schedulerrR _lr_scheduler_setup_profilerr" _profiler grpo_samples temperature _temperaturetop_k_top_kmax_generated_tokens_max_generated_tokensforward_batch_size_forward_batch_size ppo_epochs _ppo_epochssave_every_n_epochs_save_every_n_epochs num_stepsrUreos_idr hasattrrr;tensorr7_stop_token_ids)rZr(rfrj collate_namers r\setupz'FullGRPOFinetuneRecipeDistributed.setups   -  * , , ,   0"("4S5F"G"GD    * *3 / / /..@P.QQ"66!$!57   511 '+'8'8i,0,Q"%''*A4"H"H!2,X-?@!4X5G!H (9( ( $ T_!,S];;//-/ 011 0  *3844 = M  !$-9K L L L L C!7888ww E   ++ K~#/ 788,   &!$D$4 5 5+d.CC"55 WW^T::#043HH'!+6  --cgglD.I.IJJ ,Oi %(%=".#  >$'$;!M 77#U + + = /N%^;;=)?==]k=== 4?M:: =B"$!%!<$|N4<PPPr^rrrc|#|jrtddS|j}t j||||}|jrtd|S)a Set up the learning rate scheduler based on the provided configuration. It supports both standard optimization and optimizer-in-backward cases. Args: cfg_lr_scheduler (Optional[DictConfig]): The learning rate scheduler configuration. num_training_steps (int): The total number of training steps. last_epoch (int): The index of the last epoch. Returns: lr_scheduler (Optional[Optimizer]): The learning rate scheduler. NzDNo learning rate scheduler configured. Using constant learning rate.)rrz'Learning rate scheduler is initialized.)rKrDrErrrb)rZrrrrrs r\rz5FullGRPOFinetuneRecipeDistributed._setup_lr_scheduler;s$  #! Z4O )  1!       @ HH> ? ? ?r^ cfg_profilerc|tddi}|ddd|d<n#|ddks Jdtj|\}}t jt d||jrJ|dd|_|dr'|d |_ |d |_ |d |_ |S) a Parses the `profiler` section of top-level `cfg` and sets up profiler Args: cfg_profiler (Optional[DictConfig]): ``profiler`` section of the top-level ``cfg`` (the main config passed to `recipe.main`). Default None. Returns: profiler: Union[torch.profiler.profile, DummyProfiler] - DummyProfiler is a nullcontext with no-op methods for `start`, `stop`, and `step` that can be used in place of `torch.profiler.profile` if profiler is not enabled such that the instrumented training loop does not need to be changed profiling is disabled. The profiler config can be provided in configs under the `profiler` key with the following layout: .. code-block:: yaml profiler: enabled: bool #Output directory of trace artifacts output_dir: str #`torch.profiler.ProfilerActivity` types to trace cpu: bool cuda: bool #Trace options profile_memory: bool with_stack: bool record_shapes: bool with_flops: bool # `torch.profiler.schedule` options: # wait_steps -> wait, warmup_steps -> warmup, active_steps -> active, num_cycles -> repeat wait_steps: int warmup_steps: int active_steps: int num_cycles: int NenabledF _component_z'torchtune.training.setup_torch_profilerzdOnly torch profiler supported currently: component must be `torchtune.training.setup_torch_profiler`z& Profiler config after instantiation: profile_memory wait_steps warmup_steps active_steps) r r@rrbrrrDrKprofiler_profile_memoryprofiler_wait_stepsprofiler_warmup_stepsprofiler_active_steps)rZrprofiler profiler_cfgs r\rz1FullGRPOFinetuneRecipeDistributed._setup_profilercsT  %y%&899L   M4 0 0 8*SL ' '  //<===u==="(!3L!A!A,  H,HH      J+7+;+;I  IIr|r}ctj||j}|r!t j|j|||jtjtd|S)NzOptimizer is initialized.) rrbrrr#load_from_full_optimizer_state_dictr7rrrD)rZr|r}rs r\rz2FullGRPOFinetuneRecipeDistributed._setup_optimizersm &}dk6L6L6N6NOO    8      C!<===r^rrrrrc t|trfd|D}t|}ntj|j}t |}t|jj |j }t|||t|jj d} |$| |t| | S)z All data related setup happens here. Currently this recipe only supports the DistributedSamplers with Map-style Datasets which fit into memory. Other samplers, iterable datasets and streaming datasets are not supported. cDg|]}tj|jS)rrbr).0single_cfg_datasetrZs r\ zAFullGRPOFinetuneRecipeDistributed._setup_data..9s9&"#5tGGr^)datasets) num_replicasrIrr5) padding_idxT)rrsamplerr drop_last) isinstancer rrrbrrrrJrIr5rrpad_idload_state_dictlist) rZrrrrrrdsr dataloaders ` r\rz-FullGRPOFinetuneRecipeDistributed._setup_data*s k: . . B*5H111BB#KAAB.j99 ,     (! $ 6      ! ,  & &'< = = =    r^epochci}|dz|jk}tjtdt j}t j|j|j |j }tjtdt j|z dd|rt j}tjtdt j |j|j |j |j }tjtdt j|z ddnd }|j r t j}| t j|i|r| t j|t j|jt j|jt j|jt j|jt j|ji|j||| td t j|z ddt<j d S) a Checkpoint the state of the recipe. The constructed checkpoint state dict contains the following information: - Model weights with key training.MODEL_KEY - Relevant recipe state if training is not complete Checkpointer will save the model weights and recipe state in different checkpoint files. To correctly resume training from an intermediate checkpoint, the model weights and recipe state must be provided. r.zOSaving checkpoint. This may take some time. Retrieving full model state dict...r+z#Getting full model state dict took rrzGetting optimizer state dict...z"Getting optimizer state dict took N)rintermediate_checkpointzSaving checkpoint took )!rRrrrDrrrgather_cpu_state_dictrrKr7get_full_optimizer_state_dictrupdaterrrrr5rorVrsrqrY get_staterr state_dictrcsave_checkpointrEr;rr)rZrrfrstartcpu_state_dictr}s r\r z1FullGRPOFinetuneRecipeDistributed.save_checkpoint_sp"'!)d.?"?   ]   !##"7 K  <      X$2C2E2E2M X X X X   # "%''E  %F G G G%C "| N  [T5F5H5H55P[[[[     "N   W%''E  " "H$6#G H H H' && (. )49 +T-= 143D ($)*=*=*?*? /1A1L1L1N1N       . .(? /    HHUt/@/B/BU/JUUUU V V V !!#####r^ input_idsanswersc v|j\}}|j}|dddddfd|d}|||zd}t |j||z|j|j||jz5t|j||j|j |j |j j |j|j jd \}}dddn #1swxYwYt jt&j|jt j|dd|df} ||j j k} t3j| } t3j| } ~ ||| | } | dd|dz df} t9j| | |j }~ t j||| | }t9j||}t9j|| |j }~t jt9j | |j!|j j \}} | ||d} tE|j | |\}}|#|j}|#|j}||$dd z |%dd d zz }|||z}~ t jt'j&|}d ||<d ||<tO||||||z|||z|| | || S) a Generates a trajectory given the current policy model, the reference policy model, the reward function, and batch of inputs. This is done over the following steps: 1: Generate responses, and logits corresponding to the responses using the current policy, generating (query, response) pairs. 2. Estimate logprobs of the generated responses using the current policy. 3. Compute rewards and successes for the generated responses. 4. Estimate advantages using GRPO. 5. Replace any tokens in the response after the first stop token (usually EOS token) with padding, producing truncated responses. Args: input_ids (torch.Tensor): tensor of input token IDs with shape [b, seq_length] answers (List[str]): list of answers corresponding to the input_ids Returns: Trajectory: An instance of :class:`~torchtune.rlhf.GRPOTrajectory` comprising the current trajectory. N)rrr,r9decoder_max_seq_lenF) rpromptrrrrrngr return_logits input_posmaskr.T)keepdimg-C6??) query_responseslogprobs ref_logprobsrewards successes advantagesmasks position_idsresponse_padding_masksseq_lens)(shaperexpandreshaperrr7r:rrrrrrrYrr;rrr _distributedrecursive_reshardr0 empty_cachecloner!get_causal_mask_from_padding_mask"get_position_ids_from_padding_maskrbatched_logits_to_logprobsrtruncate_sequence_for_logprobs%truncate_sequence_at_first_stop_tokenrrtomeanstdget_unmasked_sequence_lengthsr)rZrrrcontext_length grpo_sizebatch_input_idsr_ responsesquery_response_padding_masksr!r"logitsr ref_logitsrr#rrr r$s r\generate_trajectoryz5FullGRPOFinetuneRecipeDistributed.generate_trajectorys#.&/_" N% #AAAtQQQJ/66r9bII)11*y2H"MM+!I-<+ .1K K      "*k&%)%? -k-I O7# " " " OQ               & !!###// <<<    #AAA~$67==?? '6$/:P'P$< (  "D (   )_ 5QQ>A-///0269dFWXX     __ |%%  8^TT 6  4#4         6 t+T_-C   "  %%j)R@@ < OY  **T\**LL..  Q = == KK4K ( (4 /  '' Y(>??     9:PQQ,/'(/2 +,+%OOJ$:;;'' Y(>??!%#9    s=A CCCcg}tj5td|j|jD]}||||jz}||||jz}tj||||tj dddn #1swxYwYtttj t|S)a Generates a ``self.batch_size`` batch of trajectories using `self._forward_batch_size` batch sizes. See ``generate_trajectory`` for more details. Args: input_ids (torch.Tensor): tensor of input token IDs with shape [b, seq_length] answers: (List[str]): list of answers corresponding to the input_ids Returns: Trajectory: An instance of :class:`~torchtune.rlhf.Trajectory`, comprising the current trajectory. rN) r;no_gradrangerrr0r*appendr=rmapcatzip)rZrr trajectories batch_startr7 batch_answerss r\generate_trajectory_batchedz=FullGRPOFinetuneRecipeDistributed.generate_trajectory_batched9s=.0 ]__ ) )$Q9QRR ) ) "++0H"HH#!(+0H"HH!  &&(((##,,_mLL &&(((( ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) )s59c<.@AABBsB(C  CC trajectoryr5ctj||j|j|j}tj||}tj ||jdd|df|j d}d||j <~tj| |j ||j|j|j \}}}}} tj|tj5d||j z dz} dddn #1swxYwYt)||||| | S) a Perform a single GRPO optimization step over a batch of trajectories and corresponding advantages and returns. Args: trajectory (Trajectory): a batch of trajectories context_length (int): input ids sequence length Returns: GRPOStats: An instance of :class:`~torchtune.rlhf.PPOStats`, a NamedTuple containing: - loss (torch.Tensor): The total PPO loss. - ratios (torch.Tensor): The ratio between the current and old policy probabilities. - clipfrac (torch.Tensor): The fraction of ratios that were clipped. - approx_policy_kls: Average estimated KL divergence between the policy before and after the optimisation step. rNr.) chunk_sizer) padding_masksg?)r;r0r*rrr"r!rr/r.rr#rrrr backwardr?powr2r) rZrIr5 pi_logits pi_logprobsr policy_losskl_lossratiosclipfracapprox_policy_klss r\ grpo_stepz+FullGRPOFinetuneRecipeDistributed.grpo_stepXs,     KK  & -!   7 >RR 5   &qqq.//'9 :      := J56     8<}}     #  !%<< 8E8 8 4k7FH       ]__  {Z%88==a@@@dff                           s,3E++E/2E/c ztj|jd}d}|jt |j|jD]<}t|j |j }|j j |t|j D]\}}|j rC|dkr=|jr6||j|jzkr#t&jj|d}|d}||j}|j\} } |||} t&jg} t |jD]$} || | } | | |j Ot&j!j"#|j$%tM|j }t&j|j'|jd t&j|xj(d z c_(|j)|j)'&|xj*d z c_*|j*|j+zdkrUi}tY|j|d <|||d <|j-| t]t_t&j0tc| fi||2| | |3d |j*|j4krd}n|xjd z c_|j|j5zdkr|6||rdS>|j7dS) z) The core training loop. NF)totaldisablertokensr)max_normT) set_to_noner.lr grad_norm)8rcleanup_before_trainingr zero_gradrr r@rVrRr$rrKrr set_epoch enumeraterrrr;r0memory_record_memory_historyr1r7r%rHrrrrWrArNr rclip_grad_norm_rrfloatsteprSrrTrAr# log_metricsrrBstackrDcleanup_after_steprrUrr stop)rZr_training_completed curr_epochpbaridxbatchr[rr8r5rI grpo_stats step_stats extra_metricss r\trainz'FullGRPOFinetuneRecipeDistributed.trains (*** !!### "  0$2CDDE E Jd3AS=STTTD   $ . .z : : :'(899< <  U&?"a4(t7$:TTTTJ%<<>>>x *4<00$*L!>!==fgNN !))+++.0 t/0022A!% N!K!KJ%%j111+7$)HN$B$B K2244%*4+?%@%@%C%% %--///O((***O--$-???%--///$$)$$)5*//1111$?T%<<AA$&M*0*A*AM$' ,5> k2$D$"!3u{C4D#E#EF( '' J??? A?d&777)-&E8    !  $";;q@@$$Z000!   r^rrc |j}tj|dtjjj|j}tj|dtjjj|||j d |j |j |j |j|j|j|jd |}|jjdkr4|jr-|t/j|j|jr#|j||jdSdS) zW Log metrics and statistics for the current step to the metric logger. r)dstopr) rrnum_stop_tokensrrRrSrUrTapprox_policy_klresponse_lengthsr0r+)rhN)rr2r;rreduceReduceOpAVGrr#anysumrrRrSrUrTrVr$rgr7rCrBrrrrKrlog_dictrS)rZrIrrextrasrrrs r\riz-FullGRPOFinetuneRecipeDistributed.log_metricss $))++   aE4E4N4R SSS(--//    e6G6P6T UUU")@DDRHHLLNNO((**%16688!)..00"+0022 ',,.. * < A A C C * 3 9 9 ; ; @ @ B B      <  & &4+F & OOH5T\JJJ K K K   J   ( (8H ( I I I I I J Jr^cb|jr|jtdSN)rKrcloser )rZs r\cleanupz)FullGRPOFinetuneRecipeDistributed.cleanups4   (   % % ' ' 'r^ l_grpo_statsc.|D]}~~|D] }|D]}~~ ~dSrr)rZrIrvgs r\rkz4FullGRPOFinetuneRecipeDistributed.cleanup_after_stepsJ   A   A  A LLr^r)r)N),__name__ __module__ __qualname____doc__r r]rstrrrdrkrerrintrrrr;rprofiler!rboolrtupler Modulerrrrr Tensorrr=rHrrWrurirrrkrr^r\r'r'#s:22h2IJ2I42I2I2I2Ih  tCH~     # #tCQTH~ # # # #d38nBqQqQqQqQqQqQf&":.& & & )  &&&&R48BB$Z0B u~%}4 5BBBBV6:p p p *.p  p sCx. p #38n p  (S 2p  ry")# $p p p p j48!!c3h0 )  .;? 3333 3  3 (S#X7 3 3333jS$S$ S$S$S$S$jC C 04S C C C C C JCC04S C CCCC>C "C C   C C C C JWWWWrJ(J6?J JJJJ>    " 9o        r^r'r(r)ct|}tjd|||||dS)z Entry point for the recipe. Configurable parameters are read in the following order: - Parameters specified in config (see available configs through ``tune ls``) - Overwritten by arguments from the command-line )r(r') recipe_namer(N)r'rrrrur)r(recipes r\ recipe_mainr)sb/3 7 7 7F "E3OOOO LLSL LLNNN NNr^__main__)>sysr functoolsrtypingrrrrrwarningsr r; omegaconfr r r torch.distributedr r torch.optimrtorchdata.stateful_dataloaderr%torchdata.stateful_dataloader.samplerr torchtunerrrrrrtorchtune.config._utilsrtorchtune.datasetsrtorchtune.dev.grpo.generationrtorchtune.dev.grpo.rewardsrtorchtune.dev.grpo.typesrrtorchtune.modulesrtorchtune.recipe_interfacesrtorchtune.trainingr r!r" torchtune.training.lr_schedulersr#r$ get_loggerrDr'parserrexitrr^r\rs 33333333333333 ,,,,,,,,GGGGGGGG!!!!!!<<<<<<LLLLLLHHHHHHHHHHHHHHHH<<<<<<,,,,,,222222FFFFFF>>>>>>>>,,,,,,999999KKKKKKKKKK333333ewCCCCC(9CCCL  Z D      z CH[[]]r^