*`i^UddlZddlZddlmZddlmZddlZddlmZddl m Z e e d< ddl Z ddlZ dZn #e$rdZYnwxYwejeZdd Zd eed ed eeefd Zdededzd eefdZdeed eeeeffdZdeed efdZ ddedeezdzde ezdzdedzde de d efdZdS)N)Path)Iterator)TokenizerVersion)MultiModalVersion_hub_installedTFreturnc2tstddS)NzxPlease install the `huggingface_hub` package to use this method. Run `pip install mistral-common[hf-hub]` to install it.)r ImportErrorz/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/utils.py_assert_hub_installedrs,   F     r lst chunk_sizec#jKtdt||D]}||||zVdS)aChunk a list into smaller lists of a given size. Args: lst: The list to chunk. chunk_size: The size of each chunk. Returns: An iterator over the chunks. Examples: >>> all_chunks = list(chunks([1, 2, 3, 4, 5], 2)) rN)rangelen)rris r chunksrsP1c#hh + +&&!a*n$%%%%%&&r repo_idrevisionctttjjtjjdg|dz }|l|dz tjjz }| rA| d5}| }dddn #1swxYwY|r0|dz |z }| rtj|SgS)zlist the files of a local Hugging Face repo. Args: repo_id: The Hugging Face repo ID. revision: The revision of the model to use. If `None`, the latest revision will be used. models/Nrefsr snapshots)rrhuggingface_hub constants HF_HUB_CACHEREPO_ID_SEPARATORjoinsplitDEFAULT_REVISIONis_fileopenreadis_diroslistdir)rr repo_cache revision_filefile revision_dirs r list_local_hf_repo_filesr/0s?o/<==@Y@k@p@p 'GMM#&&'AAJ"V+o.G.XX  " " '##C(( 'D99;; ' ' ' ' ' ' ' ' ' ' ' ' ' ' ',!K/(:     ,:l++ + Is*C  CCfilescxg}ttj}ttjdgzfd|Ddgz}|D]k}t |}|j}d|j}|dkr|||fP||vr|||fl|S)zFilter the valid tokenizer files from a list of files. Args: files: The list of files to filter. Returns: The list of tuples of file names and paths to the valid tokenizer files. c(g|]}D] }d|| S)z.model.r ).0vm mm_versionss r z1_filter_valid_tokenizer_files..Xs5___1S^__a..1..____r z.model tekken.json) listr __members__rrnamer"suffixesappend) r0valid_tokenizer_filesinstruct_versionssentencepiece_suffixesr- pathlib_file file_namesuffixr7s @r _filter_valid_tokenizer_filesrEKs-9::(455z.get_one_valid_tokenizer_file..xs TUVWTXr )keyz,Multiple valid tokenizer files found. Using .)rEr ValueErrorsortedloggerwarning)r0$valid_tokenizer_file_names_and_filesrCtokenizer_files r get_one_valid_tokenizer_filerTfs,I+O+O( /00A553444 /00144)M & & %I~ ))%%%%* D..YYYZ\]^_`WnWWWXXXX=a@C r cache_dirtokenforce_downloadlocal_files_onlyc t|r|rtd|s tj}||||}d}n#t jt jt jf$re}|r|t||}d}t dt|dkrtd|d |d |Yd }~nBd }~wwxYwt||}t|dkrtd |d |d  t|} n!#t$rtd|dwxYwtj||| ||||} | S)aDownload the tokenizer file of a Mistral model from the Hugging Face Hub. See [here](https://huggingface.co/mistralai/models) for a list of our OSS models. Note: You need to install the `huggingface_hub` package to use this method. Please run `pip install mistral-common[hf-hub]` to install it. Args: repo_id: The Hugging Face repo ID. cache_dir: The directory where the tokenizer will be cached. token: The Hugging Face token to use to download the tokenizer. revision: The revision of the model to use. If `None`, the latest revision will be used. force_download: Whether to force the download of the tokenizer. If `True`, the tokenizer will be downloaded even if it is already cached. local_files_only: Whether to only use local files. If `True`, the tokenizer will be downloaded only if it is already cached. Returns: The downloaded tokenizer local path for the given model ID. zSYou cannot force the download of the tokenizer if you only want to use local files.)rrVF)rrTzBCould not connect to the Hugging Face Hub. Using local files only.rzXCould not connect to the Hugging Face Hub and no local files were found for the repo ID z and revision z6. Please check your internet connection and try again.Nz%No local files found for the repo ID zz. Please check the repo ID and the revision or try to download the tokenizer without setting `local_files_only` to `True`.)r0z*No valid tokenizer file found in the repo rM)rrUfilenamerVrrXrW)rrNrHfApilist_repo_filesrequestsConnectionError HTTPErrorTimeoutr/rPinforFileNotFoundErrorrThf_hub_download) rrUrVrrWrXhf_api repo_fileserStokenizer_paths r download_tokenizer_from_hf_hubrhs <p*pnooo  $*,,F//(RW/XXJ$  ((*NO    1'HUUUJ#  KK\ ] ] ]:!##'fovff%-fff$#### .gQQQ z??a  #oooxooo  R5JGGG RRRPgPPPQQQR%4)%N s$-A&C9ACCD--E )rN)NNNFF)loggingr)pathlibrtypingrr]%mistral_common.tokens.tokenizers.baser&mistral_common.tokens.tokenizers.imagerbool__annotations__rhuggingface_hub.constantsrr getLogger__name__rPrr:strintrr/tuplerErTrhr r r rvs@ BBBBBBDDDDDD$$$$NNNNN  8 $ $    &S &s&xS /B&&&&"cS4ZDI6!c!tE#s(O7L!!!!6S c8$(# " LL LTzD L #: LDj L  L  L LLLLLLs 7AA