PivddlmZmZddlmZddlmZddlmZej dZ GddeZ dS) )ListTuple)Dataset)utils) PackedDatasetDEBUGcneZdZdZdeefdZdedeeeeeffdZ defdZ dS) ConcatDatasetaR A dataset class for concatenating multiple sub-datasets into a single dataset. This class enables the unified handling of different datasets as if they were a single dataset, simplifying tasks such as training models on multiple sources of data simultaneously. The class internally manages the aggregation of different datasets and allows transparent indexing across them. However, it requires all constituent datasets to be fully loaded into memory, which might not be optimal for very large datasets. Upon initialization, this class computes the cumulative length of all datasets and maintains an internal mapping of indices to the respective datasets. This approach allows the :class:`~torchtune.datasets.ConcatDataset` to delegate data retrieval to the appropriate sub-dataset transparently when a particular index is accessed. Note: Using this class with very large datasets can lead to high memory consumption, as it requires all datasets to be loaded into memory. For large-scale scenarios, consider other strategies that might stream data on demand. Args: datasets (List[Dataset]): A list of datasets to concatenate. Each dataset must be an instance of a class derived from :class:`~torch.utils.data.Dataset`. Raises: ValueError: if instanse of `PackedDataset` is in `datasets` Examples: >>> dataset1 = MyCustomDataset(params1) >>> dataset2 = MyCustomDataset(params2) >>> concat_dataset = ConcatDataset([dataset1, dataset2]) >>> print(len(concat_dataset)) # Total length of both datasets >>> data_point = concat_dataset[1500] # Accesses an element from the appropriate dataset This can also be accomplished by passing in a list of datasets to the YAML config:: dataset: - _component_: torchtune.datasets.instruct_dataset source: vicgalle/alpaca-gpt4 split: train train_on_input: True packed: True - _component_: torchtune.datasets.grammar_dataset split: train[:1%] train_on_input: False packed: True This class primarily focuses on providing a unified interface to access elements from multiple datasets, enhancing the flexibility in handling diverse data sources for training machine learning models. datasetsc||_d|D}t|rt|stdt||_t d|D|_g|_d}t|D]6\}}|t|z}|j |||f|}7dS)Nc8g|]}t|tS) isinstancer.0datasets n/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/datasets/_concat.py z*ConcatDataset.__init__..Fs"PPPGZ77PPPzDConcatDataset can't process a mix of packed and non-packed datasets.c34K|]}t|VdSN)lenrs r z)ConcatDataset.__init__..Ls(BBgS\\BBBBBBrr) _datasetsanyall ValueErrorpackedsum_len_indexes enumeraterappend)selfr is_packedcumulative_indexidxrnext_cumulative_indexs r__init__zConcatDataset.__init__Cs(0PPxPPP y>> #i.. V )nn BBBBBBB 46 %h// 5 5LC$4s7||$C ! M "24I3!O P P P4   5 5rindexreturncv|jD]0\}}}||cxkr|krn|j|}|||z cS1dSr)r!r)r$r*startstop dataset_indexrs r __getitem__zConcatDataset.__getitem__Vsl*.- . . &E4$$$$$$$$$.7uu}----% . .rc|jSr)r )r$s r__len__zConcatDataset.__len__\s yrN) __name__ __module__ __qualname____doc__rrr)intrr0r2rrrr r s..`5g5555&..tCy$s)/C)D.... rr N) typingrrtorch.utils.datar torchtunertorchtune.datasets._packedr get_loggerlogr rrrr>s$$$$$$444444ewKKKKKGKKKKKr