PiLdZddlZddlZddlZddlZddlZddlmZddlmZddl m Z m Z m Z ddl Z ddlmZddlmZmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZee Z!eGddZ"eGddZ#Gdde$Z%Gdde$Z&eGddZ'eGddZ(Gdde)e*e(fZ+dS)awDatasetInfo record information we know about a dataset. This includes things that we know about the dataset statically, i.e.: - description - canonical location - does it have validation and tests splits - size - etc. This also includes the things that can and should be computed once we've processed the dataset as well: - number of examples (in each split) - etc. N) dataclass)Path)ClassVarOptionalUnion) url_to_fs) DatasetCardDatasetCardData)config)Features) SplitDict)Version) get_logger)asdict unique_valuesc,eZdZUdZeed<dZeed<dS)SupervisedKeysDatainputoutputN)__name__ __module__ __qualname__rstr__annotations__ra/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/datasets/info.pyrr7s/E3OOOFCrrc,eZdZUdZeed<dZeed<dS)DownloadChecksumsEntryDatarkeyvalueN)rrrr"rrr#rrrr!r!=s*CMMME3OOOOOrr!ceZdZdZdS)MissingCachedSizesConfigErrorz;The expected cached sizes of the download file are missing.Nrrr__doc__rrrr%r%CsEEEErr%ceZdZdZdS)NonMatchingCachedSizesErrorz/The prepared split doesn't have expected sizes.Nr&rrrr)r)Gs9999rr)cjeZdZUdZeeed<dZeeed<dZ e deddfdZ dS)PostProcessedInfoNfeaturesresources_checksumsc|j:t|jts"tj|j|_dSdSdSN)r, isinstancer from_dictselfs r __post_init__zPostProcessedInfo.__post_init__PsA = $Z x-P-P $$.t}==DMMM % $ $ $rpost_processed_info_dictreturncdtj|D|difd|DS)Nch|] }|j Srname.0fs r z.PostProcessedInfo.from_dict..W???!qv???rc$i|] \}}|v || Srrr<kv field_namess r z/PostProcessedInfo.from_dict..Xs*\\\tq!1P[K[K[aK[K[K[rr dataclassesfieldsitems)clsr5rDs @rr1zPostProcessedInfo.from_dictUsY??{'9#'>'>??? s]]\\\\'?'E'E'G'G\\\]]]r) rrrr,rr rr-dictr4 classmethodr1rrrr+r+Ks#'Hhx '''*.$...>>> ^^:M^^^[^^^rr+c6eZdZUdZejeZeed<ejeZ eed<ejeZ eed<ejeZ eed<dZ e eed<dZe eed <dZe eed <dZe eed <dZe eed <dZe eed <dZe eeefed<dZe eed<dZe eed<dZe eed<dZe eed<dZ e eed<dZ!e eed<gdZ"e#e$eed<dZ%d+de efdZ&d,dZ'dZ(e)de$dfdZ*e)d-dede ed dfd!Z+e)d"ed dfd#Z,d.d/d&Z-d0d'Z.d efd(Z/e)d)ed dfd*Z0dS)1 DatasetInfoa Information about a dataset. `DatasetInfo` documents datasets, including its name, version, and features. See the constructor arguments and properties for a full list. Not all fields are known on construction and may be updated later. Attributes: description (`str`): A description of the dataset. citation (`str`): A BibTeX citation of the dataset. homepage (`str`): A URL to the official homepage for the dataset. license (`str`): The dataset's license. It can be the name of the license or a paragraph containing the terms of the license. features ([`Features`], *optional*): The features used to specify the dataset's column types. post_processed (`PostProcessedInfo`, *optional*): Information regarding the resources of a possible post-processing of a dataset. For example, it can contain the information of an index. supervised_keys (`SupervisedKeysData`, *optional*): Specifies the input feature and the label for supervised learning if applicable for the dataset (legacy from TFDS). builder_name (`str`, *optional*): The name of the `GeneratorBasedBuilder` subclass used to create the dataset. It is also the snake_case version of the dataset builder class name. config_name (`str`, *optional*): The name of the configuration derived from [`BuilderConfig`]. version (`str` or [`Version`], *optional*): The version of the dataset. splits (`dict`, *optional*): The mapping between split name and metadata. download_checksums (`dict`, *optional*): The mapping between the URL to download the dataset's checksums and corresponding metadata. download_size (`int`, *optional*): The size of the files to download to generate the dataset, in bytes. post_processing_size (`int`, *optional*): Size of the dataset in bytes after post-processing, if any. dataset_size (`int`, *optional*): The combined size in bytes of the Arrow tables for all splits. size_in_bytes (`int`, *optional*): The combined size in bytes of all files associated with the dataset (downloaded files + Arrow files). **config_kwargs (additional keyword arguments): Keyword arguments to be passed to the [`BuilderConfig`] and used in the [`DatasetBuilder`]. )default_factory descriptioncitationhomepagelicenseNr,post_processedsupervised_keys builder_name dataset_name config_nameversionsplitsdownload_checksums download_sizepost_processing_size dataset_size size_in_bytes)rXr\r^r,rZ_INCLUDED_INFO_IN_YAMLcT|j8t|jtstj|j|_|j>t|jt s$t |j|_|jlt|jtsRt|jtrt|j|_ntj|j|_|j 8t|j tstj |j |_ |j ht|j tsPt|j ttfrt|j |_ dStdi|j |_ dSdSdS)Nr)r,r0r r1rTr+rYrrrZrfrom_split_dictrUrtuplelistr2s rr4zDatasetInfo.__post_init__s^ = $Z x-P-P $$.t}==DM   *:d>QSd3e3e *"3"="=d>Q"R"RD  < #Jt|W,M,M #$,,, ?&t|44 &0>> ; ":dk9+M+M "#3DK@@DK   +Jt?SUg4h4h +$. >> R'94;O'P$$$'9'Q'QD Example: ```py >>> from datasets import load_dataset >>> ds = load_dataset("cornell-movie-review-data/rotten_tomatoes", split="validation") >>> ds.info.write_to_directory("/path/to/directory/") ``` wb) pretty_printN) ropen posixpathjoinr DATASET_INFO_FILENAME _dump_inforSLICENSE_FILENAME _dump_license)r3dataset_info_dirrhrefs_r=s rwrite_to_directoryzDatasetInfo.write_to_directorys`,+GG0E2GGQ WWY^$4f6RSSUY Z Z :^_ OOALO 9 9 9 : : : : : : : : : : : : : : : < &(8&:QRRTXYY &]^""1%%% & & & & & & & & & & & & & & & & & & & &s$A))A-0A-.CCCc|tjt||rdndddS)zQDump info in `file` file-like object open in bytes mode (to support remote files)Nindentutf-8)writejsondumpsrencode)r3filerhs rrmzDatasetInfo._dump_infosG 4:fTll 3N11$OOOVVW^__`````rc`||jddS)zTDump license in `file` file-like object open in bytes mode (to support remote files)rxN)ryrSr|)r3r}s rrozDatasetInfo._dump_licenses* 4<&&w//00000r dataset_infoscdDtdkr#tfdDrdSdtdD}dtdD}dtdD}dtdD}d}d}||||||| S) Nc:g|]}||Sr/)copy)r< dset_infos r z*DatasetInfo.from_merge..s'bbbiILa))LaLaLarrc30K|]}d|kVdS)rNr)r<rrs r z)DatasetInfo.from_merge..s-)g)gI-*:i*G)g)g)g)g)g)grz c3$K|] }|jV dSr/)rPr<infos rrz)DatasetInfo.from_merge..s%/[/[T0@/[/[/[/[/[/[rc3$K|] }|jV dSr/)rQrs rrz)DatasetInfo.from_merge..$,U,UtT],U,U,U,U,U,Urc3$K|] }|jV dSr/)rRrs rrz)DatasetInfo.from_merge..rrc3$K|] }|jV dSr/)rSrs rrz)DatasetInfo.from_merge..s$+S+STDL+S+S+S+S+S+Sr)rPrQrRrSr,rU)lenallrkrstrip)rJrrPrQrRrSr,rUs ` r from_mergezDatasetInfo.from_merges\bb=bbb }   ! !c)g)g)g)gYf)g)g)g&g&g ! # #kk-/[/[]/[/[/["["[\\bbdd ;;},U,U},U,U,UUUVV\\^^;;},U,U},U,U,UUUVV\\^^++m+S+S]+S+S+SSSTTZZ\\s#+     rrpr6clt|fi|pi^}}td||std|t j|tjdd5}tj |}dddn #1swxYwY| |S)a Create [`DatasetInfo`] from the JSON file in `dataset_info_dir`. This function updates all the dynamically generated fields (num_examples, hash, time of creation,...) of the [`DatasetInfo`]. This will overwrite all previous metadata. Args: dataset_info_dir (`str`): The directory containing the metadata file. This should be the root directory of a specific dataset version. storage_options (`dict`, *optional*): Key/value pairs to be passed on to the file-system backend, if any. Example: ```py >>> from datasets import DatasetInfo >>> ds_info = DatasetInfo.from_directory("/path/to/directory/") ``` zLoading Dataset info from zECalling DatasetInfo.from_directory() with undefined dataset_info_dir.rrxencodingN) rloggerdebug ValueErrorrirjrkr rlrzloadr1)rJrprerqrrr=dataset_info_dicts rfrom_directoryzDatasetInfo.from_directorys4+GG0E2GGQ D2BDDEEE fdee e WWY^$4f6RSSUXcjW k k -op $ !  - - - - - - - - - - - - - - -}}.///s5BBBrcdtj|D|difd|DS)Nch|] }|j Srr9r;s rr>z(DatasetInfo.from_dict..r?rc$i|] \}}|v || SrrrAs rrEz)DatasetInfo.from_dict..s)UUUtq!ADTDTaDTDTDTrrrF)rJrrDs @rr1zDatasetInfo.from_dictsY??{'9#'>'>??? sVVUUUU'8'>'>'@'@UUUVVVrTother_dataset_infocv|j}|jdifd|jDdS)NcHi|]\}}| |tj|Sr/rdeepcopy)r<rBrC ignore_nones rrEz&DatasetInfo.update..!s:AqMM4=##!MMrr)__dict__updaterI)r3rr self_dicts ` rrzDatasetInfo.updatesjM    .7==??     rc^|jdid|jDS)Nc>i|]\}}|tj|Srr)r<rBrCs rrEz$DatasetInfo.copy..)s( W W WADM!$4$4 W W Wrr) __class__rrIr2s rrzDatasetInfo.copy(s7t~XX W WATATAVAV W W WXXXrci}t|}|D]p}||jvret||}t|dr|||<Ct|dr|||<k|||<q|S)N _to_yaml_list_to_yaml_string)rr`getattrhasattrrr)r3 yaml_dictrr"r#s r _to_yaml_dictzDatasetInfo._to_yaml_dict+s "4LL$ + +Cd111c**5/22+%*%8%8%:%:IcNNU$566+%*%:%:%<%z.DatasetInfo._from_yaml_dict..@r?rc$i|] \}}|v || SrrrAs rrEz/DatasetInfo._from_yaml_dict..As)MMMtq!A'>??? sNNMMMMy'8'8MMMNNNr)FN)Fr/)T)rrN)r6rN)1rrrr'rGfieldrrPrrQrRrSr,rr rTr+rUrrVrWrXrYrrrZrr[rKr\intr]r^r_r`rrdr4rsrmrorLrrr1rrrrrrrrNrN[si**Z){(===K===%K%c:::Hc:::%K%c:::Hc:::$;$S999GS999#'Hhx '''26NH./66648OX01888#'L(3-&&&"&L(3-&&&!%K#%%%-1GXeCL) *111"&FHY &&&)----#'M8C='''*.(3-..."&L(3-&&&#'M8C='''333HT#Y/RRR&&&X`aeXf&&&&:aaaa111 tM':   [ .00c0HTN0^k000[0BW$W=WWW[W     YYYY t    OOOOO[OOOrrNc`eZdZd d dZed dZededdfdZdeddfdZdS) DatasetInfosDictFr6Ncfi}tj|tj}tj|tj}|s||}||tj|rct|dd5}d| D}tj |||rdnddddn #1swxYwYtj|rtj|} | j} nd} t!} |r_|| | "tdt%| zdzn| } | t)|dSdS) Nwrxrc4i|]\}}|t|Sr)rr<rXrs rrEz7DatasetInfosDict.write_to_directory..Os3&&&7M{IK !2!2&&&rrurvz--- z --- )ospathrkr DATASETDICT_INFOS_FILENAMEREPOCARD_FILENAMErrexistsrirIrzdumpr rdatar to_dataset_card_datarsaver) r3dataset_infos_dir overwriterhtotal_dataset_infosdataset_infos_pathdataset_readme_pathr=dataset_infos_dict dataset_carddataset_card_datas rrsz#DatasetInfosDict.write_to_directoryEs  W\\*;V=^__ gll+VWW I"&"5"56G"H"H ""4((( 7>>, - - U(#@@@ UA&&QdQjQjQlQl&&&" ,a\8StTTTT  U U U U U U U U U U U U U U U 7>>- . . 2&+,?@@L , 1  L / 1 1   9  4 45F G G GMYMa Gc*;&<&<C>ctd|tjtj|t jrLtj t|t jz j }d|vr| |Stjtj|t j rttj|t j d5}|dtj |DcdddS#1swxYwYdS|S)NzLoading Dataset Infos from dataset_inforxrcJi|] \}}|t|!Sr)rNr1)r<rXrs rrEz3DatasetInfosDict.from_directory..ms>:K):$[%:%:;L%M%Mr)rrrrrrkr rr rrrfrom_dataset_card_datarrirzrI)rJrrr=s rrzDatasetInfosDict.from_directoryas F3DFFGGG 7>>"',,'8&:RSS T T E + 06G1H1H6Kc1c d d i !222112CDDD 7>>"',,'8&:[\\ ] ] bgll#4f6WXXcjkkk ops>Bill>P>P>R>R                  355Ls19E77E;>E;rct|dttfrt|dtr|d|dDSt|d}|ddd|_||j|iS|S)Nrcli|]1}|ddt|2S)rXdefault)rrNr)r<dataset_info_yaml_dicts rrEz;DatasetInfosDict.from_dataset_card_data..zsQ3/22=)LLkNiNi2OOrrXr)r0rrdrKrNrrX)rJrrs rrz'DatasetInfosDict.from_dataset_card_dataus '++N;;dD\ J J +N;TBB Es7H6W  +::;L^;\]] +<^+L+P+PQ^`i+j+j (sL4lCDDD355Lrc:|rd|vr@t|dtr%|ddd|di}n4d|vr.t|dtrd|dD}ni}i|d|D}|D] \}}||d< t |dkrft t||d<|d dd}|dkrd|i|d|d<dSdSg|d<t|D]>\}}| ddd|i|}|d |=dSdS)NrrXrc i|] }|d| S)rXr)r<config_metadatas rrEz9DatasetInfosDict.to_dataset_card_data..s/***'$M2O***rc>i|]\}}||Sr)rrs rrEz9DatasetInfosDict.to_dataset_card_data..s+eee>Tk9; 7 7 9 9eeerr ) r0rKrrdrIrnextitervaluespopsortedappend)r3rdataset_metadata_infosrrXdset_info_yaml_dictrs rrz%DatasetInfosDict.to_dataset_card_datas= % U!222zBSTbBcei7j7j2%n599-SSUfguUv*&& #444DUVdDegk9l9l4**+<^+L***&& *,&#(#eeX\XbXbXdXdeee#  5H4M4M4O4O A A0 05@#M22&''1,,48>Q>X>X>Z>Z9[9[4\4\!.1/?CCMSWXX )++&{9+N;9%n555,+57!.1;ABUB[B[B]B];^;^UU7K!7*..}dCCC.;[-cLb-c*%n5<<=STTTTK% U% UBUUr)FF)r6N)r6r) rrrrsrLrr rrrrrrrDs999998[&K][$&Uo&U$&U&U&U&U&U&Urr),r'rrGrzrrjrpathlibrtypingrrrfsspec fsspec.corerhuggingface_hubr r rr r,r rZrutilsr utils.loggingrutils.py_utilsrrrrrr! Exceptionr%r)r+rNrKrrrrrrs    !!!!!!,,,,,,,,,, !!!!!!88888888%%%%%%11111111 H        FFFFFIFFF:::::):::  ^ ^ ^ ^ ^ ^ ^  ^ eOeOeOeOeOeOeO eOPjUjUjUjUjUtC,-jUjUjUjUjUr