Pi ddlmZddlmZmZmZmZmZddlm Z ddl m Z ddl m Z ddlmZddd d dd d d ededeeeefdededeededeeefdee e ffdZeedZde_dS))partial)AnyCallableDictOptionalUnion)AlpacaToMessages) PackedDataset) SFTDataset)ModelTokenizerztatsu-lab/alpacaNTFtrain)source column_maptrain_on_inputpacked filter_fnsplit tokenizerrrrrrrload_dataset_kwargsreturnc t||}td|||||d|} |r,|jtdt | |jS| S)a Support for family of Alpaca-style datasets from Hugging Face Datasets using the `data input format `_ and `prompt template `_ from the original alpaca codebase, where ``instruction``, ``input``, and ``output`` are fields from the dataset. This template is automatically applied independent of any prompt template configured in the tokenizer. Masking of the prompt during training is controlled by the ``train_on_input`` flag, which is set to ``True`` by `default `_ - If ``train_on_input`` is True, the prompt is used during training and contributes to the loss. - If ``train_on_input`` is False, the prompt is masked out (tokens replaced with -100) Args: tokenizer (ModelTokenizer): Tokenizer used by the model that implements the ``tokenize_messages`` method. source (str): path to dataset repository on Hugging Face. For local datasets, define source as the data file type (e.g. "json", "csv", "text") and pass in the filepath in ``data_files``. See `Hugging Face's `_ ``load_dataset`` for more details. Default is ``tatsu-lab/alpaca``. column_map (Optional[Dict[str, str]]): a mapping from the expected columns in the message transform :class:`~torchtune.data.AlpacaToMessages` to the new column names in the dataset. Keys should be "instruction", "input", and "output" and values should be the actual column names. If None, uses the default column names ``"instruction``, ``"input"``, and ``"output"`` in ``tatsu-lab/alpaca``. train_on_input (bool): Whether the model is trained on the prompt or not. Default is True. packed (bool): Whether or not to pack the dataset to ``max_seq_len`` prior to training. Default is False. filter_fn (Optional[Callable]): callable used to filter the dataset prior to any pre-processing. See the Hugging Face `docs `_ for more details. split (str): ``split`` argument for ``datasets.load_dataset``. You can use this argument to load a subset of a given split, e.g. ``split="train[:10%]"``. Default is "train". **load_dataset_kwargs (Dict[str, Any]): additional keyword arguments to pass to ``load_dataset``. See Hugging Face's `API ref `_ for more details. Returns: Union[SFTDataset, PackedDataset]: dataset configured with source data and transform Raises: ValueError: If ``packed`` is True and ``max_seq_len`` is not set on the tokenizer. Example: >>> alpaca_ds = alpaca_dataset(tokenizer=tokenizer) >>> for batch in Dataloader(alpaca_ds, batch_size=8): >>> print(f"Batch size: {len(batch)}") >>> Batch size: 8 )rr)rmessage_transformmodel_transformrrNz@PackedDataset requires a max_seq_len to be set on the tokenizer.) max_seq_len)r r r ValueErrorr ) rrrrrrrrrdss n/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/datasets/_alpaca.pyalpaca_datasetrsx)%*  +!       BD  (R RY-BCCCC Izyahma/alpaca-cleaned)ra Builder for a variant of Alpaca-style datasets with the cleaned version of the original Alpaca dataset, `yahma/alpaca-cleaned `_. See the dataset page and :func:`~torchtune.datasets.alpaca_dataset` for more details. ) functoolsrtypingrrrrrtorchtune.data._messagesr torchtune.datasets._packedr torchtune.datasets._sftr 'torchtune.modules.transforms.tokenizersr strboolralpaca_cleaned_dataset__doc__rr rr+sr77777777777777555555444444......BBBBBB %+/$(MMMM Mc3h( M  M  M!M M S>M :} $%MMMM`!8NOOO"r