&`i_: *ddlZddlZddlZddlmZmZmZmZmZddl m Z m Z m Z ddl mZmZejeZerddlZddlZd"dZdeed ded fd Zd eded ded effd Zd edeeedefdZddddddddddddd Zdd dedefdZ d#d eded ded effdZ d#deeeefded deeed ffdZdd defdZ dZ!defdZ"defd Z#d edefd!Z$dS)$N) TYPE_CHECKINGListOptionalTupleUnion)quoteunquoteurlparse)RetryingPyFileSystem_resolve_custom_schemereturnpyarrow.fs.PyFileSystemc ddl}ddlm}n#t$rt ddwxYwddlm}m}|||S)zGet fsspec HTTPFileSystem wrapped in PyArrow PyFileSystem. Returns: PyFileSystem wrapping fsspec HTTPFileSystem. Raises: ImportError: If fsspec is not installed. rNHTTPFileSystemz.Please install fsspec to read files from HTTP. FSSpecHandler PyFileSystem)fsspecfsspec.implementations.httprModuleNotFoundError ImportError pyarrow.fsrr)rrrrs q/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/ray/data/datasource/path_util.py_get_fsspec_http_filesystemrsV >>>>>>> VVVJKKQUUV76666666 < nn&6&677 8 88s ( filesystem)pyarrow.fs.FileSystemzfsspec.spec.AbstractFileSystemrc6|dSddlm}t||r|S ddl}n#t$rt ddwxYwt||jjs$t dt|j ddlm }m }|||S)aValidate filesystem and wrap fsspec filesystems in PyArrow. Args: filesystem: Filesystem to validate and potentially wrap. Can be None, a pyarrow.fs.FileSystem, or an fsspec.spec.AbstractFileSystem. Returns: None if filesystem is None, otherwise a pyarrow.fs.FileSystem (either the original if already PyArrow, or wrapped if fsspec). Raises: TypeError: If filesystem is not None and not a valid pyarrow or fsspec filesystem. Nr) FileSystemzfsspec is not installedzYFilesystem must conform to pyarrow.fs.FileSystem or fsspec.spec.AbstractFileSystem, got: r) rr isinstancerr TypeErrorspecAbstractFileSystemtype__name__rr)rrrrrs r_validate_and_wrap_filesystemr&$s$t%%%%%%*j))= ===122<= j&+"@ A A  P484D4D4M P P   76666666 < j11 2 22s#>pathcvddlm}t|dd}|||\}}|t|dfS)a]Try resolving a path with URL encoding for special characters. This handles paths with special characters like ';', '?', '#' that may cause URI parsing errors. Args: path: The path to resolve. filesystem: Optional filesystem to validate against. Returns: Tuple of (resolved_filesystem, resolved_path). r_resolve_filesystem_and_pathz/:ignore)safeerrors)r-)rr*rr )r'rr* encoded_pathresolved_filesystem resolved_paths r_try_resolve_with_encodingr1Nsa 877777D:::L)E)Ej**&  h G G G GG extensionsc|,t|tsJt||dSd|D}tfd|DS)a=Check if a path has a file extension in the provided list. Examples: >>> _has_file_extension("foo.csv", ["csv"]) True >>> _has_file_extension("foo.CSV", ["csv"]) True >>> _has_file_extension("foo.CSV", [".csv"]) True >>> _has_file_extension("foo.csv", ["json", "jsonl"]) False >>> _has_file_extension("foo.csv", None) True Args: path: The path to check. extensions: A list of extensions to check against. If `None`, any extension is considered valid. NTcg|]B}|dsd|n|CS).) startswithlower).0exts r z'_has_file_extension..sW "%!4!4ECIIKK#))++r2c3fK|]+}|V,dSN)r8endswith)r9r:r's r z&_has_file_extension..s9@@ctzz||$$S))@@@@@@r2)r listr$any)r'r3s` r_has_file_extensionrBgs~(  J!=!=  tJ?O?O   =tJ @@@@Z@@@ @ @@r2)local)s3)gcs)hdfs)abfs)py) filerDs3agsrErFviewfsrGabfsshttphttpsschemect|}|dS|j}|dkr|dvrt |S||vS)aCheck if a filesystem is compatible with a URI scheme. Uses PyArrow's `type_name` property for reliable filesystem type detection. This prevents silently using the wrong filesystem for a URI, which can result in malformed paths or incorrect behavior. Args: filesystem: The PyArrow filesystem to check. scheme: The URI scheme (e.g., 's3', 'gs', 'http', 'file', ''). Returns: True if the filesystem can handle the scheme, False otherwise. NTrHrOrP)_SCHEME_TO_FS_TYPE_NAMESgetr8 type_name_is_http_filesystem)rrQexpected_typesfs_types r%_is_filesystem_compatible_with_schemerZse$.11&,,..AANt"G$6%666":... n $$r2c zddl}ddlm}t|} t |}n%#t $r}t d||d}~wwxYwt|d}|jr|j nd}|dvrS|t||r||fS t}|}||fS#t$r} td |d | | d} ~ wwxYw|4t||r$ |||\} }||fS#t$rYnwxYw ||d\}}n#|jjt f$r\} t!|d\}}n=#|jjt t f$r} t d |d | d | | d} ~ wwxYwYd} ~ n+d} ~ wt $r}t d|d||d}~wwxYw||fS)aResolve a single path with filesystem, with fallback to re-resolution on error. This is a helper for lazy filesystem resolution. If a filesystem is provided, it first validates that the filesystem type is compatible with the URI scheme, then attempts to resolve the path. If the filesystem is incompatible or resolution fails, it re-resolves without the cached filesystem. Args: path: A single file/directory path. filesystem: Optional cached filesystem from previous resolution. Returns: Tuple of (resolved_filesystem, resolved_path). Raises: ValueError: If path resolution fails. ImportError: If required dependencies are missing. rNr)zInvalid filesystem provided: Fallow_fragmentsrIrSzCannot resolve HTTP path '': Failed to resolve path 'z'. Initial error: z%. URL encoding fallback also failed: z The path: 'z' has an invalid type )pyarrowrr*r r&r! ValueErrorr rQr8rZrr Exceptionlib ArrowInvalidr1) r'rpar*eparsedrQr/r0 import_error_original_errorencoding_errors r"_resolve_single_path_with_fallbackrls,777777 !$ ' 'DE2:>> EEE<<<==1DEdE 2 2 2F&,m ;V] " " "F"""  !&K ' ' !t# # "="?"?  M& 5 5   DTDDlDD  "GF## ;;D*MM A}}, ,    D  O-I-I$PT-U-U*]] F  , " " " "1KDRV1W1W . #Z; " " "G4GG>GG6DGG" " ""/ OOOFtFF1FFGGQNO  --s+ A AA B22 C<CC-D D  D D!!F68E  F F(FFF F6F11F6pathsc t|tr|g}t|tjrt|g}nct|trt d|Drt d|dt|dkrt dt|}g}|D]} t||\}}n?#t tf$r+}t d|d|dYd }~Nd }~wwxYw||}t|st|}||}||||fS) at Resolves and normalizes all provided paths, infers a filesystem from the paths and assumes that all paths use the same filesystem. Args: paths: A single file/directory path or a list of file/directory paths. A list of paths can contain both files and directories. filesystem: The filesystem implementation that should be used for reading these files. If None, a filesystem will be inferred. If not None, the provided filesystem will still be validated against all filesystems inferred from the provided paths to ensure compatibility. c3BK|]}t|t VdSr=)r str)r9ps rr?z0_resolve_paths_and_filesystem..(s/+R+Rq 1c0B0B,B+R+R+R+R+R+Rr2zIExpected `paths` to be a `str`, `pathlib.Path`, or `list[str]`, but got ``rzMust provide at least one path.r_r^z , skippingN)r rppathlibPathr@rAralenr&rlrloggerwarningrW_unwrap_protocolnormalize_pathappend)rmrresolved_pathsr'r/r0rfs r_resolve_paths_and_filesystemr|s"%%&&<U  t $ $<+R+RE+R+R+R(R(R<        Uq:;;;/z::JN-- 1Sj22 . K(    NNLdLLqLLL M M M HHHH   ,J ##677 <,];;M+::=II m,,,, : %%s=CD "!DD fscddlm}m} ddlm}n#t $rYdSwxYwt |tr|}t ||sdSt |j |ot |j j |S)zKReturn whether ``fs`` is a PyFileSystem handled by a fsspec HTTPFileSystem.rrrF) rrrrrrr r unwraphandlerr})r}rrrs rrWrWMs66666666>>>>>>> uu"*++ YY[[ b, ' 'u bj- 0 0 Z ~66s  cVtjdkr5t|r&tj|St |d}|jr d|jznd}|jr d|jznd}|j }|j dkr)d|j vr |j dd }|j }tjdkrQ|sOt|d kr<|d d kr0|d r|dddvr |d d}||z|z|zS)z2 Slice off any protocol prefixes on path. win32Fr\;rI?rD@r/):z:/N)sysplatform_is_local_windows_pathrsrtas_posixr paramsquerynetlocrQsplitr'ruisalpha)r'rgrrr parsed_paths rrxrxbsI |w#9$#?#?|D!!**,,, dE 2 2 2F$*M 9S6= rF"(, 6C&,  BE ]F } !5!5$$S))"-+K     ! ! Nc ! ! N " " $ $ " !  + +!!""o K & (5 00r2c2t|jdkS)NrIr rQr's r_is_urlrs D>> B &&r2c2t|}|jdvS)NrSr)r'rgs r _is_http_urlrs d^^F =- --r2ctjdkrdSt|dkr|ddkrdSt|dkr@|ddkr4|d d ks |d dkr|drdSdS) z5Determines if path is a Windows file-system location.rFrr\Trrrr)rrrurrs rrrs |wu 4yyA~~$q'T//t D Q GsNN !W^^tAw$ GOO   /t 5r2)r rr=)%loggingrsrtypingrrrrr urllib.parserr r ray.data._internal.utilr r getLoggerr%rv fsspec.specrr`rr&rpr1boolrBrTrZrlr|rWrxrrrr2rrs" >>>>>>>>>>>>>>1111111111PPPPPPPP  8 $ $NNN9999('3 GH'3%& '3'3'3'3TH H01H "C '(HHHH2AcAxS /BAtAAAAL              %'% % %%%%H59R.R. R.01R. "C '(R.R.R.R.n597&7& d3i 7&017& 49- -.7&7&7&7&t3*111D'T''''.$.... r2