PiddlZddlmZmZddlmZddlmZmZej e Z GddZ edZ d Zd Zeejd efd ZdS) N)PoolRLock)tqdm) experimentalloggingceZdZdZdS)ParallelBackendConfigN)__name__ __module__ __qualname__ backend_namen/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/datasets/parallel/parallel.pyr r sLLLrr c vtjt||||||||| St||||||||| S)a **Experimental.** Apply a function to iterable elements in parallel, where the implementation uses either multiprocessing.Pool or joblib for parallelization. Args: function (`Callable[[Any], Any]`): Function to be applied to `iterable`. iterable (`list`, `tuple` or `np.ndarray`): Iterable elements to apply function to. num_proc (`int`): Number of processes (if no backend specified) or jobs (using joblib). types (`tuple`): Additional types (besides `dict` values) to apply `function` recursively to their elements. disable_tqdm (`bool`): Whether to disable the tqdm progressbar. desc (`str`): Prefix for the tqdm progressbar. single_map_nested_func (`Callable`): Map function that applies `function` to an element from `iterable`. Takes a tuple of function, data_struct, types, rank, disable_tqdm, desc as input, where data_struct is an element of `iterable`, and `rank` is used for progress bar. )r r_map_with_multiprocessing_pool_map_with_joblib) functioniterablenum_procbatched batch_sizetypes disable_tqdmdescsingle_map_nested_funcs r parallel_maprs]")1- h':ulTXZp    (Hgz5,PTVl  rc |t|kr|nt|}g} t|D]q} t||z} t||z} | | zt| | z} | | z| | krdndz}| ||| ||||| ||frt|t d| Dkr9t dt|dt d| Dt d|dt|d d | Dd \}}|stftj }}t||| 5}| || }dddn #1swxYwYt d |dd|D}t dt|d|S)Nrc3@K|]}t|dVdSr Nlen.0is r z1_map_with_multiprocessing_pool..7s,::!C!II::::::rzHError dividing inputs iterable among processes. Total number of objects z , length: c3@K|]}t|dVdSr"r#r%s rr(z1_map_with_multiprocessing_pool..;s,993qt99999999rz Spawning z processes for z objects in slices of c8g|]}t|dS)r r#r%s r z2_map_with_multiprocessing_pool..?s)RqRqRqabSVWXYZW[S\S\RqRqRqr)NN)initargs initializerz Finished z processescg|] }|D]}| Srr)r&proc_resobjs rr+z2_map_with_multiprocessing_pool..Gs% = = =hH = =Sc = = = =rz Unpacked z objects) r$rangeminappendsum ValueErrorloggerinforrset_lockrmap)rrrrrrrrr split_kwdsindexdivmodstartendr,r-poolmappeds rrr+s|$s8}}44xx#h--HJxrr(mmx'(mmh&e c%oo-ck%#++QQ158XeCi%8':uV[]ikopqqqq 8}}::z::::::: <'*8}} < <99j99999 < <    KKsHssS]]ssRqRqfpRqRqRqss'Hk :!& DM+ h{ C C C>t0*==>>>>>>>>>>>>>>> KK0H000111 = =& = = =F KK1CKK111222 Ms9FF #F c  ddl tj|5  fd|DcdddS#1swxYwYdS)Nr)n_jobsc 3dK|]*}|dddfV+dS)NT)delayed)r&r0rrrjoblibrrs rr(z#_map_with_joblib..Usc! !  3FNN1 2 2Hc7JX]_ceiko3p q q! ! ! ! ! ! r)rFparallel_backendr rParallel) rrrrrrrrrrFs ` ``` `@rrrMs MMM  !6!CH U U   v  ! ! ! ! ! ! ! ! ! ! ! !                     s.A''A+.A+rc#K|t_|dkrddlm}| dVdt_dS#dt_wxYw)a **Experimental.** Configures the parallel backend for parallelized dataset loading, which uses the parallelization implemented by joblib. Args: backend_name (str): Name of backend for parallelization implementation, has to be supported by joblib. Example usage: ```py with parallel_backend('spark'): dataset = load_dataset(..., num_proc=2) ``` sparkr)register_sparkN)r r joblibsparkrK)rrKs rrGrG[sn *6&w...... 2 -1***T*1111s 8A) contextlibmultiprocessingrr tqdm.autorutilsrr get_loggerr r6r rrrcontextmanagerstrrGrrrrTs''''''''))))))))  H % %4D     23222222r