Pi} ddlZddlZddlmZddlmZddlmZmZddl m Z m Z m Z ddl Z ddlmZddl mZddlmZdd lmZd d lmZd d lmZd d lmZd dlmZd dlmZd dlmZ d dl!m"Z"m#Z#m$Z$m%Z%m&Z&d dl'm(Z(e e)e*e*fe)e*e)dfZ+e*ej,Z-ej.e/Z0Gdde*Z1Gdde2Z3dZ4ej,ddgej5gdej6gdiZ7dZ8ej9ej:dkr dd gZ;gd!Zd)e=DZ?ej,d*giZ@d+d,giZAe4gZBeAe?e>e@gZCd-ZDgd.ZEd/e*d0eFfd1ZGd2e eHeIe*fd0eHe*e eIe*d3fffd4ZJd5e*d/e*d0eFfd6ZKd5e*d/e*d0eFfd7ZLd8e e*geIe*fd0eHe*eIe*ffd9ZM dKd/e*d:e*d;e eIe*dZO dLd?e*di|]}|dt|DS)cZg|](}tD]}||t)S)keywordsep)"KEYWORDS_IN_FILENAME_BASE_PATTERNSformatNON_WORDS_CHARS.0r4patterns r# z.LO    9    wO<<    r"SPLIT_KEYWORDSr:splits r# rBKK&&&     %e,   &&&r"c>i|]}|dt|DS)cZg|](}tD]}||t)Sr3)"KEYWORDS_IN_DIR_NAME_BASE_PATTERNSr7r8r9s r#r<z.Tr=r"r>r@s r#rBrBSrCr"z**logsz **/*.evalz*[])z README.mdz config.jsonzdataset_info.jsonzdataset_infos.jsonzdummy_data.zipzdataset_dict.jsonr;returncDtfdtDS)Nc3 K|]}|vV dSNr)r:wildcard_characterr;s r# z%contains_wildcards..vs*[[1C!W,[[[[[[r")anyWILDCARD_CHARACTERS)r;s`r#contains_wildcardsrPus' [[[[GZ[[[ [ [[r"patterns DataFilesListct|trd|DSt|tr t|giSt|t rt d|Dr|D]o}t|trFt|dkr3d|vr/t|dtt fstd|pd|D}tt|t|krtd|d |DSt|iStt |S) a/ Take the data_files patterns from the user, and format them into a dictionary. Each key is the name of the split, and each value is a list of data files patterns (paths or urls). The default split is "train". Returns: patterns: dictionary of split_name -> list of patterns cdi|]-\}}t|t|tr|n|g.Srstr isinstancelist)r:keyvalues r#rBz%sanitize_patterns..s:kkkJCQVC:eT#:#:G%%kkkr"c3@K|]}t|tVdSrK)rWdictr:r;s r#rMz$sanitize_patterns..s,AAWz'4((AAAAAAr"rApathz]Expected each split to have a 'path' key which can be a string or a list of strings, but got cg|] }|d SrArr]s r#r<z%sanitize_patterns..s???7gg&???r"z*Some splits are duplicated in data_files: ci|]B}t|dt|dtr|dn|dgCS)rAr_rUr]s r#rBz%sanitize_patterns..s]GG$%%*WV_VZ:[:['rwvbijpbqarr") rWr\itemsrVSANITIZED_DEFAULT_SPLITrXrNlenget ValueErrorsetsanitize_patterns)rQr;splitss r#ririys(D!!1kkZbZhZhZjZjkkkk Hc " "1'(44 Hd # #1 AAAAA A A 7#  w--G ))7**"7;;v#6#6d DD+%BxBB+ @?h???F3v;;3v;;.. !Vf!V!VWWW'  ,X6 6 h000r"matched_rel_pathcdt|jjD}dt|jjD}t|t|kS)u When a path matches a pattern, we additionally check if it's inside a special directory we ignore by default (if it starts with a double underscore). Users can still explicitly request a filepath inside such a directory if "__pycache__" is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── __pycache__ └── b.txt >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "**") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "*/b.txt") True >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__pycache__/*") False >>> _is_inside_unrequested_special_dir("__pycache__/b.txt", "__*/*") False c<g|]}|d|S__ startswithr:parts r#r<z6_is_inside_unrequested_special_dir..s-"u"u"uD_c_n_nos_t_t"u4"u"u"ur"c<g|]}|d|Srnrprrs r#r<z6_is_inside_unrequested_special_dir..s-%o%o%otY]YhYhimYnYn%od%o%o%or")rparentpartsre)rkr;data_dirs_to_ignore_in_pathdata_dirs_to_ignore_in_patterns r#"_is_inside_unrequested_special_dirrysg8#v"uH=M4N4N4U4["u"u"u%o%ox7H7H7O7U%o%o%o" * + +s3Q/R/R RRr"cdt|jD}dt|jD}t|t|kS)u9 When a path matches a pattern, we additionally check if it's a hidden file or if it's inside a hidden directory we ignore by default, i.e. if the file name or a parent directory name starts with a dot. Users can still explicitly request a filepath that is hidden or is inside a hidden directory if the hidden part is mentioned explicitly in the requested pattern. Some examples: base directory: ./ └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_file.txt", ".*") False base directory: ./ └── .hidden_dir └── a.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".*/*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/a.txt", ".hidden_dir/*") False base directory: ./ └── .hidden_dir └── .hidden_file.txt >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", "**") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".*/.*") False >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/*") True >>> _is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir(".hidden_dir/.hidden_file.txt", ".hidden_dir/.*") False cdg|]-}|drt|dhk+|.S.rqrhrrs r#r<zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..sJ"""T__S=Q=Q"Z]^bZcZchkglZlZlZlZlZlr"cdg|]-}|drt|dhk+|.Sr|r~rrs r#r<zS_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dir..sJ%%%DOOC4H4H%QTUYQZQZ_b^cQcQcQcQcQcr")rrvre)rkr;hidden_directories_in_pathhidden_directories_in_patterns r#?_is_unrequested_hidden_file_or_is_inside_unrequested_hidden_dirrswj""!"2339"""%%!'**0%%%! ) * *c2O.P.P PPr"pattern_resolverc tD] dd} ||}n#t$rY2wxYwt|dkrt |D]K}t t |t }|J |dLtd Drtdtd d  fd tDt d tDz z} fd |DcS tD]| g} D]Q\}}|D]I} ||}n#t$rYwxYwt|dkr||nJR|r fd |DcS}td|d|)a+ Get the default pattern from a directory or repository by testing all the supported patterns. The first patterns to return a non-empty list of data files is returned. In order, it first tests if SPLIT_PATTERN_SHARDED works, otherwise it tests the patterns in ALL_DEFAULT_PATTERNS. z{split}*rNrAc3LK|]}tjt| V dSrK)rematchrr@s r#rMz+_get_data_files_patterns..s1FFerx 5111FFFFFFr"zSplit name should match 'z '' but got 'z'.c6g|]}|vt|SrrV)r:rArjs r#r<z,_get_data_files_patterns..s%WWWEuPVSZZr"c,h|]}t|Srrr@s r# z+_get_data_files_patterns..sAAA#e**AAAr"c@i|]}||gS)ra)r7)r:rA split_patterns r#rBz,_get_data_files_patterns..s/ZZZ5EM00u0==>ZZZr"c"i|] }|| Srr)r:rA patterns_dicts r#rBz,_get_data_files_patterns..&s NNNEE=/NNNr"zCouldn't resolve pattern z with resolver )ALL_SPLIT_PATTERNSreplaceFileNotFoundErrorrerhrraddrNrgrDEFAULT_SPLITSsortedALL_DEFAULT_PATTERNSrcappend) rr; data_filespp_parts sorted_splitsnon_empty_splitsrArQrrrjs @@@r#_get_data_files_patternsrs,[[ '' 377 ))'22JJ     H  z??Q  "uuF - -(1y7O7OPP*** 77+,,,,FFvFFFFF ` !^Y!^!^TZ!^!^!^___WWWW^WWWZ`AA.AAAA[[M[ZZZMZZZ Z Z Z . O O ,2244  OE8#  !1!1'!:!:JJ(Hz??Q&&$++E222E'  ONNNN=MNNN N N N O bbbP`bb c ccs 1 >> E  E- ,E- base_pathallowed_extensionsdownload_configc t|rt||}nDt|r3tj|dtjz}nd}t||\}}t|fi|\} ttt|hz d|vr| ddn-t|jtr|jn |jd}|dkr|dzndi}|dkrd|d< fd |j|fd d i|D}{fd |D} t%| t%|krLt't|t| z } t(d|d| n|} | s,d|d} | dt'z } t-| | S)a Resolve the paths and URLs of the data files from the pattern passed by the user. You can use patterns to resolve multiple local files. Here are a few examples: - *.csv to match all the CSV files at the first level - **.csv to match all the CSV files at any level - data/* to match all the files inside "data" - data/** to match all the files inside "data" and its subdirectories The patterns are resolved using the fsspec glob. In fsspec>=2023.12.0 this is equivalent to Python's glob.glob, Path.glob, Path.match and fnmatch where ** is unsupported with a prefix/suffix other than a forward slash /. More generally: - '*' matches any character except a forward-slash (to match just the file or directory name) - '**' matches any character including a forward-slash / Hidden files and directories (i.e. whose names start with a dot) are ignored, unless they are explicitly requested. The same applies to special directories that start with a double underscore like "__pycache__". You can still include one if the pattern explicitly mentions it: - to include a hidden file: "*/.hidden.txt" or "*/.*" - to include a hidden directory: ".hidden/*" or ".*/*" - to include a special directory: "__special__/*" or "__*/*" Example:: >>> from datasets.data_files import resolve_pattern >>> base_path = "." >>> resolve_pattern("docs/**/*.py", base_path) [/Users/mariosasko/Desktop/projects/datasets/docs/source/_config.py'] Args: pattern (str): Unix pattern or paths or URLs of the data files to resolve. The paths can be absolute or relative to base_path. Remote filesystems using fsspec are supported, e.g. with the hf:// protocol. base_path (str): Base path to use when resolving relative paths. allowed_extensions (Optional[list], optional): White-list of file extensions to use. Defaults to None (all extensions). For example: allowed_extensions=[".csv", ".json", ".txt", ".parquet"] download_config ([`DownloadConfig`], *optional*): Specific download configuration parameters. Returns: List[str]: List of paths or URLs to the local or remote files that match the patterns. rr://filehfF expand_infocHg|]\}}|ddksQ|drxtjtj|r.ms Hd LF " "txx'9'9 "bgnnRWM]M]^fMgMg>h>h " x  7 728ZHH8OPXZdee8X%%?X+E8 7 7r"detailTNcg|]G}tfdt|dddDE|HS)c3&K|] }d|zvV dS)r}Nr)r:suffixrs r#rMz-resolve_pattern...ys-gg&3<#55ggggggr"r}rN)rNrrA)r:rrs r#r<z#resolve_pattern..vso   ggggIhDWDWD]D]^aDbDbcdceceDfggggg    r"z Some files matched the pattern 'z-' but don't have valid data file extensions: zUnable to find ''z with any supported extension )rrrrr_ splitdriver5rr rhFILES_TO_IGNORErrArWprotocolrVglobrcrerXloggerrr)r;rrrstorage_optionsfsr glob_kwargs matched_pathsoutinvalid_matched_files error_msgrrrs ` @@@r#resolve_patternr*s`   7++ w  G&&w//2RV;  @ZijjjG_w::/::NB /**i.@.@-AAO G    eQ' S99Mbkkr{1~ +3f*<*(?(?!(C(E(EFNN{\_abccc )))44 $m&<<< 779  D(%% $;;S NN$ $ $ $  2r"r max_workersc :||n tj}td|Dr1fdt|dt |dkpdDSt t t||tdt |dkpdS)Nc3K|]}d|vV dS)rNr)r:rs r#rMz'_get_origin_metadata..s' < .s6    ( ? S S S   r"zResolving data files)descdisabler)r tqdm_classrr)r&HF_DATASETS_MULTITHREADING_MAX_WORKERSallhf_tqdmrer rr)rrrs ` r#_get_origin_metadatars "-!8++f>kK < < < < <<<      $+J2-5      +_MMM #J2%-   r"ceZdZdZdeedeeddffd ZddZe dd eed e j j d e ed e eed e eddf dZe dd eed e ed e eed e eddf dZe dd eed e ed e eed e eddf dZdddde eede eeddfdZxZS)rRa List of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns: - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover, DataFilesList has an additional attribute ``origin_metadata``. It can store: - the last modified time of local files - ETag of remote files - commit sha of a dataset repository Thanks to this additional attribute, it is possible to hash the list and get a different hash if and only if at least one file changed. This is useful for caching Dataset objects that are obtained from a list of data files. rorigin_metadatarHNcXt|||_dSrK)super__init__r)selfrr __class__s r#rzDataFilesList.__init__*s) $$$.r"othercDtg|||j|jzSrK)rRrrrs r#__add__zDataFilesList.__add__.s%_t_e_d.BUEZ.Z[[[r"rQ dataset_inforrrcd|jd|jd|pdd}|||||S)Nzhf://datasets/r/rrrr)idsharstrip from_patterns)clsrQrrrrs r# from_hf_repozDataFilesList.from_hf_repo1sc\\_[[|7G[[)/WY[[bbcfgg    >Pbq!   r"c||n1t}|||||SNr)rresolveas_posixr)rrQrrrs r#from_local_or_remotez"DataFilesList.from_local_or_remote?sT"+!6IIDFFNNPbq!   r"c D||n1t}g}|D]I} |t ||||*#t $rt |sYFwxYwt||}|||SNrr)rrrextendrrrr)rrQrrrrr;rs r#rzDataFilesList.from_patternsLs"+!6IIDFFNNK|]}tj|VdSrKrescape)r:exts r#rMz'DataFilesList.filter..ks*"H"Hc29S>>"H"H"H"H"H"Hr"z.*(z )(\..+)?$c3>K|]}tj|VdSrKr )r:fns r#rMz'DataFilesList.filter..ns*!E!EB")B--!E!E!E!E!E!Er"z.*[\/]?(z)$cLg|]tfdD S)c3BK|]}|VdSrK)r)r:r;rs r#rMz2DataFilesList.filter...rs07i7iU\ i8P8P7i7i7i7i7i7ir")rN)r:rrQs @r#r<z(DataFilesList.filter..rs=jjjyC7i7i7i7i`h7i7i7i4i4ijjjjr")r)joinrrcompilerRrrX)rrr ext_pattern fn_patternrQs @r#filterzDataFilesList.filterfs  G(("H"HZ"H"H"HHHK OOBJ'D['D'D'DEE F F F  D!E!E*!E!E!EEEJ OOBJ'A:'A'A'ABB C C C  S jjjjDjjj $ 4  !dT=QRRR Rr")rrRrHrRNNN)rr r!__doc__rXrVSingleOriginMetadatarr classmethodhuggingface_hubhf_api DatasetInforrrrrr __classcell__rs@r#rRrRsO"/49/tDX?Y/^b//////\\\\ $(2648   s)  &,8  C=  %T#Y/  ".1       [  $(2648   s)  C=  %T#Y/  ".1      [  $(2648 00s)0C=0%T#Y/ 0 ".1 0  000[0448[_SSS%d3i0SEMdSViEXS SSSSSSSSr"c&eZdZdZe ddeeeeee ffde ede eede e ddf dZ e ddeeeeee ffd e jjde ede eede e ddf d Ze ddeeeeee ffde ede eede e ddf d Zddd d e eede eeddfdZdS) DataFilesDicta Dict of split_name -> list of data files (absolute local paths or URLs). It has two construction methods given the user's data files patterns : - ``from_hf_repo``: resolve patterns inside a dataset repository - ``from_local_or_remote``: resolve patterns from a local path Moreover, each list is a DataFilesList. It is possible to hash the dictionary and get a different hash if and only if at least one file changed. For more info, see [`DataFilesList`]. This is useful for caching Dataset objects that are obtained from a list of data files. Changing the order of the keys of this dictionary also doesn't change its hash. NrQrrrrHc|}|D]=\}}t|tr|nt||||||<>|Sr)rcrWrRrrrQrrrrrYpatterns_for_keys r#rz"DataFilesDict.from_local_or_remotescee%-^^%5%5   !C!. >>  "77$''9$3 8 HH r"rc|}|D]>\}}t|tr|nt|||||||<?|S)N)rrrr)rcrWrRr) rrQrrrrrrYr"s r#rzDataFilesDict.from_hf_reposcee%-^^%5%5   !C!. >>  "//$!-''9$3 0 HH r"c|}|D]=\}}t|tr|nt||||||<>|Sr)rcrWrRrr!s r#rzDataFilesDict.from_patternsscee%-^^%5%5   !C!. >>  "00$''9$3 1 HH r"rrrct|}|D]\}}|||||< |S)Nr)rrcr)rrrrrYdata_files_lists r#rzDataFilesDict.filtersWd4jjll$(JJLL \ \ C&--PZ-[[CHH r"r)rr r!rrr\rVr rXrRrrrrrrrrrrr"r#rrys  $(2648 sE$s)]":;;<C=%T#Y/  ".1   [* $(2648 sE$s)]":;;<&,8C=  %T#Y/  ".1  [.$(2648 sE$s)]":;;<C=%T#Y/  ".1   [,48[_%d3i0EMdSViEX r"rceZdZdZdeedeeeeffd ZdZe ddeedeeeddfdZ dd ed ee dd fd Z d eeddfdZ xZS)DataFilesPatternsListz List of data files patterns (absolute local paths or URLs). For each pattern there should also be a list of allowed extensions to keep, or a None ot keep all the files for the pattern. rQrcXt|||_dSrK)rrr)rrQrrs r#rzDataFilesPatternsList.__init__s+ """"4r"cDtg|||j|jzSrK)rRrrs r#rzDataFilesPatternsList.__add__s%_t_e_d.EH`.`aaar"NrHc<|||gt|zSrK)re)rrQrs r#rz#DataFilesPatternsList.from_patternss%s801CMMABBBr"rrrRc x||n1t}g}t||jD]L\}} |t ||||-#t$rt|sYIwxYwt||}t||Sr) rrrziprrrrrrrR)rrrrr;rrs r#rzDataFilesPatternsList.resolves "+!6IIDFFNN.s eee7I% 2eeer")r(r)rrs `r#filter_extensionsz'DataFilesPatternsList.filter_extensionss2$ eeeeTMdeee   r"rK)rr r!rrXrVrrrrrrrr0rrs@r#r(r(s8 5s)5!$s)!45555555bbbLPCCCyC6>tCy6IC CCC[C59:::".1:  ::::. DI :Q        r"r(ceZdZdZe d deeeefdeeeddfdZ d dedee dd fd Z d eeddfd Z dS)DataFilesPatternsDictz[ Dict of split_name -> list of data files patterns (absolute local paths or URLs). NrQrrHc|}|D];\}}t|tr|nt||||<<|S)N)r)rcrWr(r)rrQrrrYr"s r#rz#DataFilesPatternsDict.from_patternsszcee%-^^%5%5   !C!.0EFF  *88$'99 HH r"rrrct}|D]\}}|||||<|SrK)rrcr)rrrrrYdata_files_patterns_lists r#rzDataFilesPatternsDict.resolve sL oo-1ZZ\\ T T )C)/77 ?SSCHH r"rct|}|D]\}}||||<|SrK)rrcr0)rrrrYr5s r#r0z'DataFilesPatternsDict.filter_extensions*sPd4jjll-1ZZ\\ N N )C)/AA*MMCHH r"rK) rr r!rrr\rVrXrrrrr0rr"r#r2r2 sW[  CcN+ AI$s)AT     [ $59".1  DI:Qr"r2)NNrK)Wrr functoolsrrrpathlibrrtypingrrr r fsspec.corer r packagingr tqdm.contrib.concurrentr rrdownloadrnamingrrjrutilsrrrutils.file_utilsrrrrrutils.py_utilsrtuplerVrTRAINrd get_loggerrrrrr&SPLIT_PATTERN_SHARDED VALIDATIONTESTr?r8FSSPEC_VERSIONparser6rFr"DEFAULT_PATTERNS_SPLIT_IN_FILENAME"DEFAULT_PATTERNS_SPLIT_IN_DIR_NAMEDEFAULT_PATTERNS_ALLDEFAULT_PATTERNS_LOGSrrrOrboolrPr\rXriryrrrrrintrrRrr(r2rr"r#rPs """""""",,,,,,,,,,!!!!!!((((((......$$$$$$""""""rrrrrrrrrrrrrr******U38_eCj%)CD#ek**  H % %     #        )   a K':& ;;; J999  =7=4444*GI\)]&***&&  ]W];7777*IK^)_&***&&+AB`)a&***&+u/<&&  &&&"&&  &&&" K$ +/+,&&  \\\\\\#1dD#o 6#14U4PS9VeKeEf@f;g#1#1#1#1LSSsStSSSSB;QVY;Qdg;Qlp;Q;Q;Q;Q|)dxtCy8H/I)ddSVX\]`XaSaNb)d)d)d)d^/304 ]] ]]!c+]n- ]  #Y ]]]]@XjXjXjx7OXj[_`ceijmen`n[oXjXjXjXjz15n-015!%S n-#  :^S^S^S^S^SDI^S^S^SBZZZZZDm+,ZZZz2 2 2 2 2 DI2 2 2 j#####D&;!;<#####r"