Pi7ddlmZmZmZmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZddlmZGdd e Zdd ddd d d ededeeeefdedeedeededeeefdefdZdS))AnyCallableDictListMappingOptionalN) load_dataset)Dataset)ChosenRejectedToMessagesCROSS_ENTROPY_IGNORE_IDX) Transform)ModelTokenizerceZdZdZddddedededeed e d e ee fd dfd Z d Z ded e eeeffdZdeee fd e eeeffdZdS)PreferenceDataseta Primary class for fine-tuning via preference modelling techniques (e.g. training a preference model for RLHF, or directly optimizing a model through DPO) on a preference dataset sourced from Hugging Face Hub, local files, or remote files. This class requires the dataset to have "chosen" and "rejected" model responses. These are typically either full conversations between user and assistant in separate columns:: | chosen | rejected | |----------------------------------------|----------------------------------------| | [{"role": "user", "content": Q1}, | [{"role": "user", "content": Q1}, | | {"role": "assistant", "content": A1}] | {"role": "assistant", "content": A2}] | or a user prompt column with separate chosen and rejected assistant reponses:: | prompt | chosen | rejected | |----------|----------|------------| | Q1 | A1 | A2 | In the above case when the format is prompt-chosen-rejected, only single-turn interactions are supported. At a high level, this class will load the data from source and apply the following pre-processing steps when a sample is retrieved: 1. Dataset-specific transform. This is typically unique to each dataset and extracts the necessary prompt and chosen/rejected columns into torchtune's :class:`~torchtune.data.Message` format, a standardized API for all model tokenizers. 2. Tokenization with optional prompt template if configured All datasets are formatted into a list of :class:`~torchtune.data.Message` because preference datasets can be considered as chosen and rejected "conversations" with the model, or AI assistant. Thus, we can standardize all text content as messages in a conversation assigned to a role: - ``"user"`` messages contain the input prompt into the model - ``"assistant"`` messages are the response of the model and what you actually want to train for and compute loss directly against The :class:`~torchtune.data.Message` forms the core data unit that all tokenizer APIs expect. The key component of this class that ensures any dataset is transformed into this format is the ``message_transform``. This is a callable class that takes in a sample dictionary - typically a single row from the source dataset - that processes the sample in any configurable way to output a list of messages:: [ Message( role=, content=, ), ... ] For any custom dataset, use the ``message_transform`` to contain all pre-processing to return the list of messages. Args: source (str): path to dataset repository on Hugging Face. For local datasets, define source as the data file type (e.g. "json", "csv", "text") and pass in the filepath in ``data_files``. See `Hugging Face's `_ ``load_dataset`` for more details. message_transform (Transform): callable that keys into the desired fields in the sample and converts text content to a list of :class:`~torchtune.data.Message`. It is expected that the final list of messages are stored in the ``"chosen"`` and ``"rejected"`` keys. tokenizer (ModelTokenizer): Tokenizer used by the model that implements the ``tokenize_messages`` method. Since PreferenceDataset only supports text data, it requires a :class:`~torchtune.modules.transforms.tokenizers.ModelTokenizer` instead of the ``model_transform`` in :class:`~torchtune.datasets.SFTDataset`. filter_fn (Optional[Callable]): callable used to filter the dataset prior to any pre-processing. See the Hugging Face `docs `_ for more details. packed (bool): Whether or not to pack the dataset to ``max_seq_len`` prior to training. Default is False. Packed is currently not supported for ``PreferenceDataset`` and a ``ValueError`` will be raised if this is set to True. **load_dataset_kwargs (Dict[str, Any]): additional keyword arguments to pass to ``load_dataset``. See Hugging Face's `API ref `_ for more details. Raises: ValueError: If ``packed`` is True, this feature is not supported for ``PreferenceDataset``. NF) filter_fnpackedsourcemessage_transform tokenizerrrload_dataset_kwargsreturnc |rtd||_||_t|fi||_|!|j||_dSdS)Nz:Packed is currently not supported for preference datasets.) ValueError _tokenizer_message_transformr _datafilter)selfrrrrrrs r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/datasets/_preference.py__init__zPreferenceDataset.__init__fss  L $"3!&@@,?@@  **955DJJJ ! c*t|jSN)lenr)rs r__len__zPreferenceDataset.__len__|s4:r!indexcF|j|}||Sr#)r_prepare_sample)rr&samples r __getitem__zPreferenceDataset.__getitem__s"E"##F+++r!r)c ||}|j|d\}}tt j|t |}|j|d\}}tt j|t |}t|t|ksJt|t|ksJt||||} | S)Nchosenrejected)chosen_input_ids chosen_labelsrejected_input_idsrejected_labels) rrtokenize_messageslistnpwherer r$dict) rr)transformed_sampler. chosen_masksr/r0rejected_masksr1tokenized_dicts rr(z!PreferenceDataset._prepare_samples!44V<<*.)J)J x (* * &, H\#;=M N N  .2_-N-N z *. . *N H^%=?Q R R  #$$M(:(:::::%&&#o*>*>>>>>-'1+    r!)__name__ __module__ __qualname____doc__strr rrrboolrrr r%intrr*rr(r!rrrsPPp)-6666% 6 " 6 H% 66 $CH~6 6666,,,c49n)=,,,,gc3h&7Dd3i`_ ``load_dataset`` for more details. column_map (Optional[Dict[str, str]]): a mapping from the expected columns "chosen" and "rejected" in the message transform :class:`~torchtune.data.ChosenRejectedToMessages` to the new column names in the dataset. Keys should be "chosen" and "rejected" and values should be the actual column names. If None, keep the default columns "chosen" and "rejected". train_on_input (bool): Whether the model is trained on the prompt or not. Default is False. new_system_prompt (Optional[str]): if specified, prepend a system message to every sample for both chosen and rejected. This can serve as instructions to guide the model response. Setting this will OVERRIDE any system messages already present in the dataset. Default is None. filter_fn (Optional[Callable]): callable used to filter the dataset prior to any pre-processing. See the Hugging Face `docs `_ for more details. split (str): ``split`` argument for ``datasets.load_dataset``. You can use this argument to load a subset of a given split, e.g. ``split="train[:10%]"``. Default is "train". **load_dataset_kwargs (Dict[str, Any]): additional keyword arguments to pass to ``load_dataset``. Examples: :: my_preference_dataset.json [ { "chosen_conversations": [ { "content": "What do I do when I have a hole in my trousers?", "role": "user" }, { "content": "Fix the hole.", "role": "assistant" } ], "rejected_conversations": [ { "content": "What do I do when I have a hole in my trousers?", "role": "user" }, { "content": "Take them off.", "role": "assistant" } ] } ] :: >>> from torchtune.datasets import preference_dataset >>> column_map = { ... "chosen": "chosen_conversations", ... "rejected": "rejected_conversations" >>> } >>> dataset = preference_dataset( ... tokenizer=tokenizer, ... source="json", ... column_map=column_map, ... data_files="my_preference_dataset.json", ... train_on_input=False, ... split="train", >>> ) >>> tokenizer.decode(dataset[0]["chosen_input_ids"], skip_special_tokens=True) What do I do when I have a hole in my trousers?Fix the hole. >>> tokenizer.decode(dataset[0]["rejected_input_ids"], skip_special_tokens=True) What do I do when I have a hole in my trousers?Take them off. This can also be accomplished via the yaml config: .. code-block:: yaml dataset: _component_: torchtune.datasets.preference_dataset source: json data_files: my_preference_dataset.json column_map: chosen: chosen_conversations rejected: rejected_conversations train_on_input: False split: train Returns: PreferenceDataset: The preference dataset built from source paired data. )rErDrF)rrrrrGrB)r r) rrrDrErFrrGrrs rpreference_datasetrIsZ^1%+  +       r!)typingrrrrrrnumpyr4datasetsr torch.utils.datar torchtune.datar r torchtune.modules.transformsr 'torchtune.modules.transforms.tokenizersrrr?r@rIrBr!rrQs@???????????????!!!!!!$$$$$$MMMMMMMM222222BBBBBBMMMMMMMMh,0 '+$(\\\\ \c3h( \  \  } \!\ \ S>\\\\\\\r!