PiddlmZddlZddlmZddlmcmZGddejZ GddejZ dS))TupleNc eZdZdZ d dedeffd Zdejdejd ejd ejd eejejejff d Z xZ S)DPOLossaO Direct Preference Optimization (DPO) Loss module: https://arxiv.org/abs/2305.18290 Simply stated from the paper: Intuitively, the DPO update increases the relative log probability of preferred to dispreferred responses, but it incorporates a dynamic, per-example importance weight that prevents the model degeneration that we find occurs with a naive probability ratio objective. Based on the implementation in HF's TRL library: https://github.com/huggingface/trl/blob/5d1deb1445828cfd0e947cb3a7925b1c03a283fc/trl/trainer/dpo_trainer.py#L844 DPO retains similarities to PPO (https://arxiv.org/abs/2009.01325), where it optimizes a policy (language) model to align with human preferences, and regularizes the loss function using a baseline reference (the frozen, initial language model) to prevent over-fitting to the preference dataset. It differs from PPO by optimizing the policy model directly using labelled preference data, rather than using an additional reward model to provide feedback. This significantly simplifies training and reduces compute overhead. Args: beta (float): Temperature parameter for the DPO loss, typically in the range of 0.1 to 0.5. Default is 0.1. label_smoothing (float): Parameter encoding uncertainty about the labels. Default is 0. 皙?betalabel_smoothingcdt||_||_dSN)super__init__rr )selfrr __class__s k/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/rlhf/loss/dpo.pyr zDPOLoss.__init__&s0  .policy_chosen_logpspolicy_rejected_logpsreference_chosen_logpsreference_rejected_logpsreturncB||z }||z }||z }tj|j|z d|jz ztj|j |z|jzz }|j||z z} |j||z z} || | fS)a Compute the DPO loss for a batch of policy and reference model log probabilities. Args: policy_chosen_logps (torch.Tensor): Log probabilities of the policy model for the chosen responses. Shape: (batch_size) policy_rejected_logps (torch.Tensor): Log probabilities of the policy model for the rejected responses. Shape: (batch_size) reference_chosen_logps (torch.Tensor): Log probabilities of the reference model for the chosen responses. Shape: (batch_size) reference_rejected_logps (torch.Tensor): Log probabilities of the reference model for the rejected responses. Shape: (batch_size) Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: A tuple of three tensors: - losses: The DPO loss for each example in the batch. - chosen_rewards: Rewards for the chosen responses. - rejected_rewards: Rewards for the rejected responses. )F logsigmoidrr detach rrrrr pi_logratios ref_logratioslogitslosseschosen_rewardsrejected_rewardss rforwardzDPOLoss.forward/s6+-BB .1II  - \$)f, - - -T5I1I JlDI:.//$2FF G  I,/EEMMOO O  I.1IIQQSS S ~'777r)rr __name__ __module__ __qualname____doc__floatr torchTensorrr# __classcell__rs@rrrs2!$//////////8"\/8 %|/8!& /8 #(, /8 u|U\5<7 8 /8/8/8/8/8/8/8/8rrc eZdZdZ d deffd Zdejdejdejdejd eejejejff d Z xZ S) RSOLossa Statistical Rejection Sampling Optimization (RSO) or "hinge" loss module: https://arxiv.org/abs/2309.06657. Intuition from the paper: DPO is a logistic regression on human preference data, and SLiC (https://arxiv.org/abs/2305.10425) is almost equivalent to a support vector machine (SVM) with hinge loss. [RSO] improve[s] SLiC as the SVM counter part of DPO. Based on the implementation in HF's TRL library: https://github.com/huggingface/trl/blob/4dce042a3863db1d375358e8c8092b874b02934b/trl/trainer/dpo_trainer.py#L1141 Args: gamma (float): Equivalent temperature parameter (from DPO) for the RSO loss. rgammacVt||_dSr )r r r0)rr0rs rr zRSOLoss.__init__ps&  rrrrrrc||z }||z }||z }tjd|j|zz }|j||z z} |j||z z} || | fS)a Compute the RSO loss for a batch of policy and reference model log probabilities. Args: policy_chosen_logps (torch.Tensor): Log probabilities of the policy model for the chosen responses. Shape: (batch_size) policy_rejected_logps (torch.Tensor): Log probabilities of the policy model for the rejected responses. Shape: (batch_size) reference_chosen_logps (torch.Tensor): Log probabilities of the reference model for the chosen responses. Shape: (batch_size) reference_rejected_logps (torch.Tensor): Log probabilities of the reference model for the rejected responses. Shape: (batch_size) Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: A tuple of three tensors: - losses: The RSO loss for each example in the batch. - chosen_rewards: Rewards for the chosen responses. - rejected_rewards: Rewards for the rejected responses. r)r*relur0rrs rr#zRSOLoss.forwardws6+-BB .1II  -A V 3344 J-0FFNNPP P  J/2JJRRTT T ~'777r)rr$r-s@rr/r/as   )8"\)8 %|)8!& )8 #(, )8 u|U\5<7 8 )8)8)8)8)8)8)8)8rr/) typingrr*torch.nnnntorch.nn.functional functionalrModulerr/rrr;s P8P8P8P8P8biP8P8P8f?8?8?8?8?8bi?8?8?8?8?8r