PiD\ddlmZmZddlZddlmZddlmZGddejZ dS))OptionalTupleN)rlhfceZdZdZ ddededeffd Z dd ejd ejd ejd ejd ejdejdeejdeejde ejejejejejffdZ xZ S)PPOLossa Proximal Policy Optimization (PPO) Loss module. This implementation uses the following references: https://arxiv.org/abs/1707.06347 eqn. 7 https://github.com/vwxyzjn/lm-human-preference-details/blob/ccc19538e817e98a60d3253242ac15e2a562cb49/lm_human_preference_details/train_policy_accelerate.py#L719 https://github.com/openai/baselines/blob/ea25b9e8b234e6ee1bca43083f8f3cf974143998/baselines/ppo2/model.py#L68-L75 Args: epsilon (float): clipping range for PPO update. value_clip_range (float): clipping range for value function update. value_coeff (float): coefficient for the value function loss contribution. 皙?皙?epsilonvalue_clip_range value_coeffcrt||_||_||_dS)N)super__init__r r r )selfr r r __class__s k/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/rlhf/loss/ppo.pyrzPPOLoss.__init__ s8  0&Npi_old_logprobs pi_logprobs advantagesphi_old_values phi_valuesreturns padding_masksvalue_padding_masksreturnc rtj||z } tj| d|jz d|jz} | | z} | | z} | | k|j} || ntj| |} tj | | }||ntj||}tj|||j z ||j z}tj ||z dz||z dz}|d|zndtj||z}|||j zz}|| | |  | fS)a Forward pass of the PPO loss module. Args: pi_old_logprobs (torch.Tensor): Log probabilities of the old policy. pi_logprobs (torch.Tensor): Log probabilities of the current policy. advantages (torch.Tensor): Advantage values. phi_old_values (torch.Tensor): Value predictions of the old value function. phi_values (torch.Tensor): Value predictions of the current value function. returns (torch.Tensor): Return values. padding_masks (Optional[torch.Tensor]): Padding token masks of the same shape as ``pi_logprobs``, where True indicates the corresponding loss values should participage in policy loss calculation. value_padding_masks (Optional[torch.Tensor]): Padding token masks of the same shape as ``pi_logprobs``, where True indicates the corresponding loss values should participage in value loss calculation. Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]: A tuple of five tensors: - loss: The total PPO loss. - policy_loss: The policy function loss. - value_loss: The value function loss. - ratios: The ratio between the current and old policy probabilities. - clipfrac: The fraction of ratios that were clipped. g?Ng?) torchexpclampr todtypemeanr masked_meanmaximumr r detach)rrrrrrrrrratiosclipped_ratiospolicy_losses_clippedpolicy_losses_unclippedclipfrac policy_lossvalues_clipped value_losslosss rforwardzPPOLoss.forward+sH;899VS4<-?t|ASTT!+ n <#-+"6),CCGG     $ MMOOO!(M::  m$9;RSS $      !+}==    T2 2 T2 2   ] ' !a '.7*Bq)H  #* *//## # #t' 4GHHH  j4+;;<          KKMM " " OO     r)rr r)NN) __name__ __module__ __qualname____doc__floatrrTensorrrr1 __classcell__)rs@rrrs2&"% ' ' '  ' ' ' ' ' ' '&156:O O O \O L O  O L O O   -O &el3O  u|U\5<u|S TO O O O O O O O rr) typingrrrtorch.nnnn torchtunerModulerrrr?s#""""""" l l l l l bil l l l l r