PiddlmZmZmZmZmZddlmZddlm Z ddl m Z ddl m Z ddddddd d e d ed eeeefd edeededeededeeefdee e ffdZdS))AnyCallableDictOptionalUnion)InputOutputToMessages) PackedDataset) SFTDataset)ModelTokenizerNFtrain) column_maptrain_on_inputnew_system_promptpacked filter_fnsplit tokenizersourcer rrrrrload_dataset_kwargsreturnc t|||} td|| |||d|} |r,|jtdt | |jS| S)a Configure a custom dataset with user instruction prompts and model responses. This builder function can be used to configure a custom instruct dataset directly from the yaml config as an alternative to :class:`~torchtune.datasets.SFTDataset`, as it is made to be config friendly. The dataset should follow this format: .. code-block:: text | input | output | |-----------------|------------------| | "user prompt" | "model response" | If your column names are different, you can use the ``column_map`` parameter to change the expected column names. For example, if your dataset has columns ``"question"`` and ``"answer"`` you can use:: column_map = {"input": "question", "output": "answer"} Masking of the prompt during training is controlled by the ``train_on_input`` flag, which is set to ``False`` by default - If ``train_on_input`` is True, the prompt is used during training and contributes to the loss. - If ``train_on_input`` is False, the prompt is masked out (tokens replaced with -100) Args: tokenizer (ModelTokenizer): Tokenizer used by the model that implements the ``tokenize_messages`` method. source (str): path to dataset repository on Hugging Face. For local datasets, define source as the data file type (e.g. "json", "csv", "text"), pass in the filepath in ``data_files``, and set ``split="train"``. See `Hugging Face's `_ ``load_dataset`` for more details. column_map (Optional[Dict[str, str]]): a mapping to change the expected "input" and "output" column names to the actual column names in the dataset. Keys should be "input" and "output" and values should be the actual column names. Default is None, keeping the default "input" and "output" column names. train_on_input (bool): Whether the model is trained on the user prompt or not. Default is False. new_system_prompt (Optional[str]): if specified, prepend a system message. This can serve as instructions to guide the model response. Default is None. packed (bool): Whether or not to pack the dataset to tokenizer's ``max_seq_len`` prior to training. Default is False. filter_fn (Optional[Callable]): callable used to filter the dataset prior to any pre-processing. See the Hugging Face `docs `_ for more details. split (str): ``split`` argument for ``datasets.load_dataset``. You can use this argument to load a subset of a given split, e.g. ``split="train[:10%]"``. Default is "train". **load_dataset_kwargs (Dict[str, Any]): additional keyword arguments to pass to ``load_dataset``, such as ``data_files`` or ``split``. Examples: :: my_dataset.json [ { "question": "What time is it in London?", "answer": "It is 10:00 AM in London.", }, { ... }, ..., ] :: >>> from torchtune.datasets import instruct_dataset >>> dataset = instruct_dataset( ... tokenizer=tokenizer, ... source="json", ... data_files="my_dataset.json", ... column_map={ ... "input": "question", ... "output": "answer", ... }, ... train_on_input=False, ... packed=False, ... split="train", ... ) >>> tokens = dataset[0]["tokens"] >>> tokenizer.decode(tokens) "What time is it in London?It is 10:00 AM in London." This can also be accomplished via the yaml config: .. code-block:: yaml dataset: _component_: torchtune.datasets.instruct_dataset source: json data_files: my_dataset.json column_map: input: question output: answer train_on_input: False packed: False split: train Returns: Union[SFTDataset, PackedDataset]: the configured :class:`~torchtune.datasets.SFTDataset` or :class:`~torchtune.datasets.PackedDataset` if ``packed=True`` Raises: ValueError: If ``packed=True`` and ``tokenizer.max_seq_len`` is not set. )rr r)rmessage_transformmodel_transformrrNz@PackedDataset requires a max_seq_len to be set on the tokenizer.) max_seq_len)rr r ValueErrorr ) rrr rrrrrrrdss p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/datasets/_instruct.pyinstruct_datasetrsn.%+  +!       BD  (R RY-BCCCC I)typingrrrrrtorchtune.datartorchtune.datasets._packedr torchtune.datasets._sftr 'torchtune.modules.transforms.tokenizersr strboolrrr rr(sQ87777777777777000000444444......BBBBBB,0 '+$(LLLL Lc3h( L  L  } L L!L L S>L :} $%LLLLLLr