§ ÇPƒiÄã ó:—ddlmZddlZddlmcmZddlmZddl m Z  ddej dej de deej ej ffd „Z dd ej dej d edej fd „Ze dfd ejdejde dedejf d„Zdej de dej fd„ZdS)é)ÚTupleN)Úrlhf)ÚCROSS_ENTROPY_IGNORE_IDXÚ sequencesÚ stop_tokensÚ fill_valueÚreturncóŒ—tj||¦«}tj|d¬¦«}|dk|dk|zz}|||<||fS)aq Truncates sequence(s) after the first stop token and pads with ``fill_value``. Args: sequences (torch.Tensor): tensor of shape [batch_size, sequence_length] or [sequence_length]. stop_tokens (torch.Tensor): tensor containing stop tokens. fill_value (int): value to pad the sequence with after the first stop token, usually ``pad_id``. Returns: Tuple[torch.Tensor, torch.Tensor]: A tuple of two tensors with the same shape as ``sequences``: - padding_mask (torch.Tensor): a bool tensor where True indicates the token has been truncated. - sequences (torch.Tensor) a tensor of truncated and padded sequences. Example: >>> stop_token_ids = torch.tensor([2, 869]) >>> fill_value = 0 >>> sequences = torch.tensor( >>> [ >>> [869, 30, 869], >>> [2, 30, 869], >>> [869, 30, 2], >>> [50, 30, 869], >>> [13, 30, 2], >>> [13, 30, 5], >>> [13, 2, 20], >>> [13, 2, 2], >>> [2, 2, 2], >>> ] >>> ) >>> eos_mask, truncated_sequences = rlhf.truncate_sequence_at_first_stop_token( >>> sequences, stop_token_ids, fill_value >>> ) >>> eos_mask >>> torch.tensor([ >>> [False, True, True], >>> [False, True, True], >>> [False, True, True], >>> [False, False, False], >>> [False, False, False], >>> [False, False, False], >>> [False, False, True], >>> [False, False, True], >>> [False, True, True], >>> ] >>> ) >>> truncated_sequences >>> torch.tensor([ >>> [869, 0, 0], >>> [2, 0, 0], >>> [869, 0, 0], >>> [50, 30, 869], >>> [13, 30, 2], >>> [13, 30, 5], >>> [13, 2, 0], >>> [13, 2, 0], >>> [2, 0, 0], >>> ] >>> ) é©Údim)ÚtorchÚisinÚcumsum)rrrÚeos_maskÚseq_lensÚ padding_masks úv/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/rlhf/sequence_processing.pyÚ%truncate_sequence_at_first_stop_tokenrsX€õ|Œz˜) [Ñ1Ô1€HÝŒ|˜H¨!Ð,Ñ,Ô,€Hؘq’L h°!¢m¸°yÑ%@ÑA€LØ(€IˆlÑØ ˜Ð "Ð"óçð?ÚlogitsÚ temperaturecó¨—tjtj||z d¬¦«d| d¦«¦« d¦«S)aø Converts logits corresponding to a generated sequence to logprobs over the generated tokens. Args: logits (torch.Tensor): The logits tensor of shape [b, response_length, vocab_size]. sequences (torch.Tensor): The corresponding tokens of shape [b, response_length]. temperature (float): The temperature to scale the logits. Default 1.0 Returns: torch.Tensor: The log probabilities corresponding to each token in ``sequences``. Shape [b, response_length]. éÿÿÿÿr é)rÚgatherÚFÚ log_softmaxÚ unsqueezeÚsqueeze)rrrs rÚlogits_to_logprobsr"TsP€õ Œ<Ý Œ f˜{Ñ*°Ð3Ñ3Ô3Ø Ø×Ò˜BÑÔñ ô ÷‚gˆbk„kð rFÚlabelsÚlabel_pad_token_idÚreturn_average_logprobscóV—|jdd…|jkrtd¦«‚|dd…dd…f ¦«}|dd…dd…dd…f}||k}d|||k<t||d¬¦«}|rt j||d¬¦«S||z d¦«S) a( Calculate log probabilities based on provided logits and labels. Args: logits (torch.FloatTensor): direct logits output of the model of shape (b, s, v) labels (torch.LongTensor): ground-truth labels to compute log probs with, shape (b, s). Label tokens with a value of label_pad_token_id are ignored. label_pad_token_id (int): token id to ignore in labels. return_average_logprobs (bool): If True, return the average log probs across the sequence. Default is False. See https://github.com/eric-mitchell/direct-preference-optimization/blob/f8b8c0f49dc92a430bae41585f9d467d3618fe2f/trainers.py#L96 # noqa Returns: Calculated log probs of shape (b, ) Raises: ValueError: If logits and labels have different shapes. NrzKLogits (batch and sequence length dim) and labels must have the same shape.r rr)rr )ÚshapeÚ ValueErrorÚcloner"rÚ masked_meanÚsum)rr#r$r%Ú loss_maskÚper_token_log_probss rÚget_batch_log_probsr.hsÚ€ð0„|CRCÔ˜FœLÒ(Ð(ÝØ Yñ ô ð ðAAAqrrEŒ]× Ò Ñ "Ô "€FØ AAAs˜s˜A˜A˜AIÔ €FØÐ,Ò,€Ià+,€Fˆ6Ð'Ò 'Ñ(å,¨V°VÈÐMÑMÔMÐàð9ÝÔÐ 3°YÀBÐGÑGÔGÐGà# iÑ/×4Ò4°RÑ8Ô8Ð8rÚquery_response_logitsÚcontext_lengthcó$—|dd…|dz d…fS)a Truncates logits generated over a sequence for estimating logprobs over the tokens in the sequence. This assumes the sequence is of the (query, response) format with length (context_length + response_length) Args: query_response_logits (torch.Tensor): The logits tensor of shape [b, context_length + response_length, vocab_size]. context_length (int): The length of the context. Returns: torch.Tensor: The truncated logits for the response with shape [b, response_length, vocab_size].Nr r©)r/r0s rÚtruncate_sequence_for_logprobsr3“s"€ð !    N°QÑ$6¸Ð$;Ð!;Ô <Ðr@s–ððÐÐÐÐÐà € € € ØÐÐÐÐÐÐÐÐØÐÐÐÐÐØ3Ð3Ð3Ð3Ð3Ð3ðKLðB#ðB#ØŒ|ðB#Ø*/¬,ðB#ØDGðB#à ˆ5Œ<˜œÐ %Ô&ðB#ðB#ðB#ðB#ðLILððØ ŒLðØ%*¤\ðØ@Eðà „\ððððð.7Ø$)ð (9ð(9Ø Ô ð(9à Ô ð(9ðð(9ð"ð (9ð  Ôð (9ð(9ð(9ð(9ðV =Ø œ<ð =Ø9<ð =à „\ð =ð =ð =ð =ð =ð =r