Pi#ddlmZmZddlZ ddejdejdededejf d Z d d ejd ejd ejd edeejdeejejejff dZ d dejdejdeedejfdZ d!dejdejdedejfdZ d"dejdeejdedejfdZ d dejdejdededeejdeejejff dZ dS)#)OptionalTupleNT padding_masksseq_lenspenalise_no_eosmin_response_lengthreturnctj|t}|r|d}| |||kz}|S)a0 Calculates a mask to penalise scores corresponding to sequences generated during PPO, where True indicates the score at the corresponding position should be penalised. This function assumes sequences have already been truncated at an EOS, if present, and padded to length, e.g. by :func:`torchtune.rlhf.sequence_processing.truncate_sequence_at_first_stop_token`. Scores are penalised such that: - If ``min_response_length`` is set, scores for sequences with ``length < min_response_length`` are penalised. - If ``penalise_no_eos`` is True, scores for sequences with no EOS token are penalised. Args: padding_masks (torch.Tensor): torch.Tensor where True indicates a padding token in the generated sequence, and False otherwise. Shape: ``(b, response_len)`` seq_lens (torch.Tensor): The length of each generated sequence. Shape: ``(b,)`` penalise_no_eos (bool, optional): Whether to penalise sequences with no EOS token. Defaults to True. min_response_length (int, optional): The minimum length of the response. If set, any responses is shorter than this length will be penalised. Defaults to None. Returns: torch.Tensor: A mask tensor with shape ``(b,)`` where True indicates the corresponding score should be penalised. )torch zeros_liketoboolany)rrrrreward_penalty_masks j/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/rlhf/rewards.pyget_reward_penalty_maskr sd4 *84477==5,00444&-@!@AA scoreslogprobs ref_logprobskl_coeffvalid_score_idxsc||z }| |z}|}|>|d|d|dn|dddfxx|z cc<|||fS)a Calculates PPO rewards for the given scores, logprobs, and reference logprobs. Args: scores (torch.Tensor): Reward model scores, shape ``(b,)``. logprobs (torch.Tensor): Policy logprobs, shape ``(b, response_len)``. ref_logprobs (torch.Tensor): Reference base model logprobs, shape ``(b, response_len)``. kl_coeff (float): KL reward contribution coefficient. valid_score_idxs (Optional[torch.Tensor]): A tensor of indexes for valid (non-padded) token predictions. This is useful when calculating rewards for padded sequences, as scores and value estimates are defined for the last valid predicted token. Shape: ``(b,)``. Default None. Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: A tuple of tensors with shape ``(b, response_len)`` each: - total_reward: total reward combining per-token kl rewards and reward model score. - kl: kl divergence between policy and reference policy logprobs. - kl_reward: kl divergence scaled by ``kl_coeff``. Notation used for tensor shapes: - b: batch size - response_len: model response length Nr )clone scatter_add_ unsqueeze)rrrrrkl kl_reward total_rewards rget_rewards_ppor"1s< L B BI??$$L #!! ))"--v/?/?/C/C     QQQUv% Y &&rxmaskdimch||z|||dzz S)a Compute mean of tensor with masked values. Taken from https://github.com/huggingface/trl/blob/main/trl/core.py Args: x (torch.Tensor): The input tensor. mask (torch.Tensor): The bool mask tensor, where True indicates the corresponding value in ``x`` should participate in the mean calculation. dim (Optional[int]): The axis to calculate the mean over. Default None. Returns: torch.Tensor: The mean tensor. )r%:0yE>)sum)r#r$r%s r masked_meanr)as4 H>>c> " "dhh3h&7&7$&> ??rcentered_valuesunbiasedct|d|}|r$|dz}||dz z }||z}|S)aW Compute variance of mean-centered tensor with masked values. Taken from https://github.com/huggingface/trl/blob/main/trl/core.py We use ``centered_values`` to avoid repeated calls to ``masked_mean``. Args: centered_values (torch.Tensor): The mean-centered tensor e.g. ``x - masked_mean(x)``. mask (torch.Tensor): The bool mask tensor, where True indicates the corresponding value in ``x`` should participate in the mean calculation. unbiased (bool): Whether to use the unbiased variance. Returns: torch.Tensor: The variance tensor. r'r)r)powr()r*r$r+varmask_sumbessel_corrections r masked_varr2ssZ o))!,,d 3 3C&88::$$1 5%% Jr shift_meanc|$t||}t||z |}n(||}}||z t j|dzz}|r||z }|S)a* Whiten (normalises) values, optionally with masked values. Taken from https://github.com/huggingface/trl/blob/main/trl/core.py Args: x (torch.Tensor): The input tensor. mask (Optional[torch.Tensor]): The bool mask tensor with the same shape as ``x``, and where True indicates the corresponding value in ``x`` should participate in the mean calculation. Default None. shift_mean (bool): Whether to shift normalised values by the mean. Default True. Returns: torch.Tensor: The whitened tensor. Nr')r)r2meanr/r rsqrt)r#r$r3r5r/whiteneds rwhitenr8sz 1d##T4((FFHHaeeggcDEKd 333HD Orvaluesrewardsgammalmbdamaskscd}g}|jd}tt|D][}||dz kr|dd|dzfnd} |dd|f|| zz|dd|fz } | ||z|zz}||\t j|dddd} | |z} t | |} |d| |<| | fS)a Estimates the advantages and returns for the PPO algorithm using Generalized Advantage Estimation https://arxiv.org/pdf/1506.02438.pdf Args: values (torch.Tensor): The predicted values for each state. Shape: ``(b, response_len)`` rewards (torch.Tensor): The rewards received at each time step. Shape: ``(b, response_len)`` gamma (float): The discount factor. lmbda (float): The GAE-Lambda parameter. masks (Optional[torch.Tensor]): A bool mask tensor, where True indicates the corresponding value in ``values`` should participate in the mean calculation. Default None. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple containing the estimated advantages and returns. - advantages (torch.Tensor): The estimated advantages. Shape: ``(b, response_len)`` - returns (torch.Tensor): The estimated returns. Shape: ``(b, response_len)`` Notation: - b: batch size - response_len: model response length rr rNg)axis)r$)shapereversedrangeappendr stackr8) r9r:r;r<r= last_gae_lamadvantages_reversedresponse_lengtht next_valuesdelta advantagesreturnss restimate_advantagesrMs6Ll2&OeO,, - - 1 1*+o.A*A*AfQQQAX&&s 1  33fQQQTlBuu}|;; ""<0000026Q???J 6!G ///J  E6 w r)TN)N)T)NT)typingrrr Tensorrintrfloatr"r)r2r8rMrrrSsP#""""""" !# ""<"l"" "  \ """"T04 -'-' L-'l-',-' -' u|, -'  5<u| 34 -'-'-'-'b?C@@ |@ <@.6sm@ \@@@@&IM\).AE \2NR |#EL1FJ \<%) 88 L8 \8 8  8 EL ! 8  5< %& 888888r