§ ÇPƒi« ãóŒ—ddlmZmZmZmZddlmZddlmZ ddede d ee d eed e d ee efd efd„Z dS)é)ÚAnyÚCallableÚDictÚOptional©ÚTextCompletionDataset)ÚModelTokenizerúccdv/cnn_dailymailNÚtrainÚ tokenizerÚsourceÚ max_seq_lenÚ filter_fnÚsplitÚload_dataset_kwargsÚreturnc ó*—td||d|||ddœ|¤ŽS)a Support for family of datasets similar to `CNN / DailyMail `_, a corpus of news articles. This builder only extracts the articles and not the highlights for general text completion tasks. Args: tokenizer (ModelTokenizer): Tokenizer used by the model that implements the ``tokenize_messages`` method. source (str): path string of dataset, anything supported by Hugging Face's ``load_dataset`` (https://huggingface.co/docs/datasets/en/package_reference/loading_methods#datasets.load_dataset.path) max_seq_len (Optional[int]): Maximum number of tokens in the returned input and label token id lists. Default is None, disabling truncation. We recommend setting this to the highest you can fit in memory and is supported by the model. For example, llama2-7B supports up to 4096 for sequence length. filter_fn (Optional[Callable]): callable used to filter the dataset prior to any pre-processing. See the Hugging Face `docs `_ for more details. split (str): ``split`` argument for ``datasets.load_dataset``. You can use this argument to load a subset of a given split, e.g. ``split="train[:10%]"``. Default is "train". **load_dataset_kwargs (Dict[str, Any]): additional keyword arguments to pass to ``load_dataset``. Returns: TextCompletionDataset: the configured TextCompletionDataset Úarticlez3.0.0)r r ÚcolumnrrrÚname©r)r r rrrrs úu/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/datasets/_cnn_dailymail.pyÚcnn_dailymail_articles_datasetrsA€õ> !ð ØØØØØØðð ð ð ð ð ð ó)r NNr ) ÚtypingrrrrÚ#torchtune.datasets._text_completionrÚ'torchtune.modules.transforms.tokenizersr ÚstrÚintrrrrúr sÓðð1Ð0Ð0Ð0Ð0Ð0Ð0Ð0Ð0Ð0Ð0Ð0àEÐEÐEÐEÐEÐEàBÐBÐBÐBÐBÐBð 'Ø!%Ø$(Øð +ð+Øð+à ð+ð˜#”ð+ð˜Ô!ð +ð ð +ð    S œ>ð +ðð+ð+ð+ð+ð+ð+r