&`iZddlZddlmZddlmZmZmZmZmZddl Z ddl m Z ddl m Z mZmZmZddlmZddlmZmZed Gd d ZdS) N) defaultdict)DictListLiteralOptionalUnion) ActorHandle) DataIteratorDatasetExecutionOptions NodeIdStr)ExecutionResources) DeveloperAPI PublicAPIstable) stabilitycFeZdZdZ ddeedeefdeee e ee ffde fdZ d e d e fd Zd ed e fdZede eefdedeeedeeed ee eeff dZed e fdZdS) DataConfigzClass responsible for configuring Train dataset preprocessing. For advanced use cases, this class can be subclassed and the `configure()` method overriden for custom data preprocessing. allNTdatasets_to_splitexecution_optionsenable_shard_localityct|ts|dkr||_n(tdt |jd|dt t|tr|tfd|_ t|tr|j |||_ d|_d|_dS)aConstruct a DataConfig. Args: datasets_to_split: Specifies which datasets should be split among workers. Can be set to "all" or a list of dataset names. Defaults to "all", i.e. split all datasets. execution_options: The execution options to pass to Ray Data. Can be either: 1. A single ExecutionOptions object that is applied to all datasets. 2. A dict mapping dataset names to ExecutionOptions. If a dataset name is not in the dict, it defaults to ``DataConfig.default_ingest_options()``. By default, the options are optimized for data ingest. When overriding, base your options off ``DataConfig.default_ingest_options()``. enable_shard_locality: If true, dataset sharding across Train workers will consider locality to minimize cross-node data transfer. Enabled by default. rzV`datasets_to_split` should be a 'all' or a list of strings of dataset names. Received z with value .c,tjS)N)copydeepcopy)default_execution_optionsss/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/train/_internal/data_config.pyz%DataConfig.__init__..9sDM";<<gN) isinstancelist_datasets_to_split TypeErrortype__name__rdefault_ingest_optionsr r_execution_optionsdictupdate_enable_shard_locality_num_train_cpus_num_train_gpus)selfrrrrs @r__init__zDataConfig.__init__s. ' . . 2Cu2L2L&7D # #V)**3VVARVVV  %/$E$E$G$G! ')9 : : :(9 %?J < < < <@ @  ' . . >  # * *+< = = =&;#""r!num_train_cpusnum_train_gpusc"||_||_dS)zSet the total number of CPUs and GPUs used by training. If CPU or GPU resource limits are not set, they will be set to the total cluster resources minus the resources used by training. N)r-r.)r/r1r2s rset_train_total_resourcesz$DataConfig.set_train_total_resourcesCs .-r! dataset_namereturnc@tj|j|S)zKReturn a copy of the configured execution options for a given dataset name.)rrr))r/r5s r_get_execution_optionsz!DataConfig._get_execution_optionsMs}T4\BCCCr!datasets world_sizeworker_handlesworker_node_idsc Hdt|D}|D]!\}}|j||"|jdkr"t |} nt |j} |jr|nd} |D]\} } || } | r8| j t|j |j| _ | | } | | j_| | vr7t%| |d| D]\}}|||| <t|D]}| ||| < |S)aConfigure how Train datasets should be assigned to workers. Args: datasets: The datasets dict passed to Train by the user. world_size: The number of Train workers in total. worker_handles: The actor handles of the Train workers. worker_node_ids: The node ids of the Train workers. kwargs: Forwards compatibility placeholder. Returns: A list of dataset splits for each worker. The size of the list must be equal to `world_size`. Each element of the list contains the assigned `DataIterator` instances by name for the worker. cg|]}iSr?).0_s r z(DataConfig.configure..hs000"000r!Nr)cpugpuT)equallocality_hints)rangeitemsnameset_namer$setkeysr,r8is_resource_limits_defaultexclude_resourcesaddrr-r.rcontextr enumeratestreaming_splititerator)r/r9r:r;r<kwargsoutputr5datasetrrFrIdsrisplits r configurezDataConfig.configureQs.10eJ//000%-^^%5%5 / / !L'|#  ...  "e + + #HMMOO 4 4   #D$; < < ,0,GQT (( 4 4HD" $ ; ;D A A  ;;== &7;;* $ 4$:N"3B+)>$?? @ -# $ -#-#-#-#^..u....D3D;KDDDD>sG|$>>!k!23 > "$y/2 > d3 $% &>>>\>@ $4   \   r!r)r collectionsrtypingrrrrrr` ray.actorr ray.datar r r r 9ray.data._internal.execution.interfaces.execution_optionsrray.util.annotationsrrrr?r!rrss  ######77777777777777 !!!!!!GGGGGGGGGGGGXXXXXX88888888 XU U U U U U U U U U r!