Pi./ddlmZddlmZmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZmZddlmZmZmZddlmZdd lmZe d e Z d7d eededeedeefdZde ee fddfdZ!de de deddeee effdZ"deedefdZ#e d8de ded eed!e$d"ed#ed$ed%d&e$d'ee efdefd(Z%e d9d*ee efd+ee e&fd,e d"edef d-Z'e d:d/ed0ed1ed2eeegefd3e$d#ed$ed%d4eed5e$defd6Z(d S);)Path)AnyCallableDictListLiteralOptionalTypeVarUnion)request) load_dataset)split_dataset_by_node)default_collateDistributedSampler) DatasetTypeLoaderrequires_torchdata) Transform)get_world_size_and_rankT)boundNtokens max_seq_leneos_idreturnc@|d|}||d|kr||d<|S)a Truncate a list of tokens to a maximum length. If eos_id is provided, the last token will be replaced with eos_id. Args: tokens (List[Any]): list of tokens to truncate max_seq_len (int): maximum length of the list eos_id (Optional[Any]): token to replace the last token with. If None, the last token will not be replaced. Default is None. Returns: List[Any]: truncated list of tokens N)rrrtokens_truncateds i/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/data/_utils.pytruncater!s:$l{l+ .r2f<<%  image_loczPIL.Image.ImagecTddlm}t|trP|dr; t j|}n%#t$r}td||d}~wwxYw | |}n%#t$r}td||d}~wwxYw|S)a Convenience method to load an image in PIL format from a local file path or remote source. Args: image_loc (Union[Path, str]): Local file path or remote source pointing to the image which will be loaded in PIL format. Note: If loading an image from a remote source, the function expects the URL provided in ``image_loc`` to start with "http" or "https" e.g. "https://www.wikipedia.org/en/bird.jpg". Raises: ValueError: If the image cannot be loaded from remote source, **or** if the image cannot be opened as a :class:`~PIL.Image.Image`. Examples: >>> # Load from remote source >>> image = load_image("https://www.wikipedia.org/en/bird.jpg") >>> # Load from local file path >>> image = load_image(Path("/home/user/bird.jpg")) Returns: PIL.Image.Image: The loaded image. r)ImagehttpzFailed to load image from Nz'Failed to open image as PIL Image from ) PILr% isinstancestr startswithr urlopen Exception ValueErroropen)r#r%eimages r load_imager1/s:)S!!Ni&:&:6&B&BN N 22II N N NE)EEFFA M NW 9%% WWWN9NNOOUVVW Ls/A A)A$$A)-B B% B  B%content image_tagimagesc||}t||kr%tdt|d|d|||}g}t |D]q\}}t|dkr|d|d|t|dz kr+|d|ddr|S) a Given a raw text string, split by the specified ``image_tag`` and form into list of dictionaries to be used in the :class:`~torchtune.data.Message` content field:: [ { "role": "system" | "user" | "assistant", "content": [ {"type": "image", "content": }, {"type": "text", "content": "This is a sample image."}, ], }, ... ] Args: content (str): raw message text image_tag (str): string to split the text by images (List["PIL.Image.Image"]): list of images to be used in the content Raises: ValueError: If the number of images does not match the number of image tags in the content Examples: >>> content = format_content_with_images( ... "<|image|>hello <|image|>world", ... image_tag="<|image|>", ... images=[, ] ... ) >>> print(content) [ {"type": "image", "content": }, {"type": "text", "content": "hello "}, {"type": "image", "content": }, {"type": "text", "content": "world"} ] Returns: List[Dict[str, Any]]: list of dictionaries to be used in the :class:`~torchtune.data.Message` content field zNumber of images (z') does not match number of image tags (z) in content: rtext)typer2r0)countlenr-split enumerateappendpop)r2r3r4num_image_tags_in_content split_contentfinal_content_listisubstrs r format_content_with_imagesrD^sZ!( i 8 8 6{{/// CV C C) C C9@ C C   MM),,M}--SS 6 v;;??  % %v&&I&I J J J s=!!A% % %  % %w6::a==&Q&Q R R R r"funcscfd}|S)z Chain a list of functions together into a single function. Args: *funcs (List[Callable]): list of functions to chain together Returns: Callable: chained function c(D] }||}|SNr)xfnrEs r chained_fnzchain..chained_fns&  B1AAr"r)rErKs` r chainrLs$ r"TthreadFsource transform filter_fnshuffleseed num_workersparallel_method)processrM streamingload_dataset_kwargsc ddlm} m} m} d|vr)d|vs Jd||d|d<t |fi|} || |} t\} }|r6t| || } |r| |} | | }n4t| | ||| }| |}t| j |}| |||| }|S) a Load a HuggingFace dataset (Map or Streaming) and apply a Transform to it. Args: source (str): HuggingFace dataset source. transform (Transform): Transform to apply to the samples of the dataset. filter_fn (Optional[Callable]): Filter function to pass to HuggingFace dataset. shuffle (bool): Whether to shuffle the dataset. Default is True. For streaming datasets, this is passed to HuggingFace dataset as .shuffle(). For map datasets, a DistributedSampler is used. seed (int): Seed for the random number generator in the case of Map style dataset shuffling. Default is 0. num_workers (int): Number of workers to use for loading the dataset. Default is 0 (no parallelism). Setting this greater than 0 will create `parallel_method` workers to perform transforms to the dataset. parallel_method (Literal["process", "thread"]): Method to use for parallelism. Default is "thread". No effect if num_workers is 0. streaming (bool): whether to load a streaming vs map-style dataset. Default False. **load_dataset_kwargs (Dict[str, Any]): Additional Keyword arguments to pass to HuggingFace dataset. See Hugging Face's documentation. Returns: A ``torchdata.nodes`` iterator that can be passed directly to a Loader, or combined with other-datasets in a multi-dataset sampler. r)IterableWrapperParallelMapperSamplerWrappersubsetnamezTfound both 'subset' and 'name' found, you may only specify one, load_dataset_kwargs=N)rank world_size)rR) num_replicasr^rQrRmap_fnrSmethod) torchdata.nodesrYrZr[r>r filterrrrQrrL __getitem__)rNrOrPrQrRrSrTrVrWrYrZr[datasetr_r^nodesamplers r load_hf_datasetrjs`DPOOOOOOOOO&&& - - - - dNa d d . - -&9&=&=h&G&GF#699%899G..++.00J:'dzRRR  1oo4o00Gw''$ #    ~g&&'-y99 > YK   D Kr"!CYCLE_UNTIL_ALL_DATASETS_EXHASTEDdatasetsweights stop_criteriac,ddlm}|||||S)a Given a dictionary of datasets and their corresponding weights, return a dataset that samples from the given datasets according to the specified weights. Args: datasets (Dict[str, DatasetType]): dictionary of datasets weights (Dict[str, float]): dictionary of weights for each dataset. If not stop_criteria (str): stop criteria for the sampler. Default "CYCLE_UNTIL_ALL_DATASETS_EXHASTED". See also: torchdata.nodes.StopCriteria seed (int): seed for the random number generator. Default 0. Returns: A ``torchdata.nodes`` iterator which can be passed to Loader, or further composed with other Nodes. r)MultiNodeWeightedSampler) source_nodesrmrnrR)rdrp)rlrmrnrRrps r get_multi_datasetrrs<*988888 # ##     r"rgmodel_transform batch_size collate_fn drop_lastprefetch_factor pin_memoryc ddlm} m} m} m} |t }| ||||} | | ||} | | |||} |r | | } | | | |} t | S)ax This will configure TorchData Nodes to approximate torch.utils.data.DataLoader. Given a dataset, apply model_transform (eg tokenization), batching, collation, memory pinning, and pre-fetching. Args: dataset (DatasetType): dataset to load. May be a MultiNodeWeightedSampler model_transform (Transform): model transform to apply to the samples of the dataset batch_size (int): batch size collate_fn (Optional[Callable[[Any], Any]]): collate function to apply to the samples of the dataset. If None, use torch.utils.data.default_collate. Default None. drop_last (bool): whether to drop the last batch. Default is True. num_workers (int): number of workers to use for loading the dataset. Default is 0 (no parallelism parallel_method (Literal["process", "thread"]): method to use for parallelism. Default is "thread". prefetch_factor (Optional[int]): number of batches to prefetch. Default is 4. pin_memory (bool): whether to pin memory. Default is False. Returns: A ``torchdata.nodes`` Loader, an Iterable that returns batches. r)BatcherrZ PinMemory PrefetcherNra)rw)rdr{rZr|r}rr)rgrtrurvrwrSrTrxryr{rZr|r}rhs r get_dataloaderr~sBONNNNNNNNNNN$ >[   D 74y 9 9 9D > Z[   Dy"z$00 $<<r"rH)NTrrrMF)rkr)NTrrMrsF))pathlibrtypingrrrrrr r r urllibr rlr datasets.distributedrtorch.utils.datarrtorchtune.data._torchdatarrrtorchtune.modules.transformsrtorchtune.utilsrr7rintr!r)r1rDrLboolrjfloatrrr~rr"r rsOOOOOOOOOOOOOOOOOOOO!!!!!!666666@@@@@@@@MMMMMMMMMM222222333333 GCt ! I SM #Y 0,%c *,/@,,,,^< < #<-12C-D< $sCx.<<<<~$x.X&%)4<CC CC!C C  C  C01CC S>CCCCCL= 3 #$ #u*     < 264<%&11 1113%*-. 1  1  1011c]11 111111r"