`in*dZddlZddlZddlZddlZddlmZddlmZm Z m Z ddl m Z ddl mZddlmZmZdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZd dlm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(erddl)m*Z*ndZ*ej+e,Z-ee.e/e e.e e.ffgdderdndffderdnderdndffdderdndffdderdndffdderdndffd derd!ndffd"derdndffd#d$erd%nderd&ndffd'd(derdndffd)erd*nddffd+d,d-erd.nderd/ndffd0d1erd2ndffd3d4derd5ndffd6d7d8derdndffd9d:erd;ndffdderd5ndffd?d@erdAndffdBderdndffdCdDerdEnderdFndffdGdHerdnderdndffdIderdndffdJd@erdAndffdKderdndffdLderdndffdMdNerdOnderdPndffdQdRerdSndffdTderd!ndffdUderd!ndffdVderdndffdWdXerdYndffdZd[erd\ndffd]erd^nderd_ndffd`daderd5ndffdbdcddd@erdAndffded:erd;ndffdfdgerdhndffdierdjnderdkndffdlerdnderdndffdmerdnderdndffdnerdnderdndffdoerdnderdndffdpdqerdnderdndffdrdserdtndffdudverdwndffdxdyerdzndffd{d:erd;ndffd|derdndffd}derdndffd~derdndffderdnddffdderd:nderd;ndffdderd5ndffdderdndffderdnddffddderd;ndffdderdndffddderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffdderdndffdderd5ndffdderd5ndffdderd5ndffdderd5ndffdderd5ndffderdnddffdd:erd;ndffdd:erd;ndffdd:erd;ndffdderdndffddderd5ndffdd:erd;ndffddddddderdndffdderdndffdderd5ndffdderdndffddd@erdAndffdderdndffdderdndffdderdndffdd:erd;ndffdd:erd;ndffddXerdYndffderdnderdndffdderdndffderdnderdndffdderdnderdndffdderd5ndffdderdndffdderdndffdderdndffdderdndffdderdndffdderdndffderdnderdndffderdnderdndffderdnderdndffdderdndffdderdndffdderdndffdderdndffdderdndffderdnderdndffddderdndffderdnddffdderdndffdderdndffderdnddffderdnderdndffderdnderdndffdd@erdAndffdderdndffdderdndffdderd:nderd;ndffde rdn erdnder e sdndffde rdn erdnder e sdndffde rdn erdnder e sdndffde rdn erdnder e sdndffdderdndffderdnddffdderdndffdderdndffdderd5ndffdderd5ndffdderd5ndffdderdndffdderdndffdd@erdAndffd erd nderd ndffd derdndffd derdndffdderdndffddderd5ndffdderdndffderdnderdndffderdnderdndffderdnderdndffdderdndffdderdndffdderd;ndffdderdndffdderdndffdderdndffdd erd!ndffd"d:erd;ndffd#derdndffd$derdndffd%derdndffd&d'erd1nderd2ndffd(erd1nderd2ndffd)d*erdnderdndffd+dRerdSndffd,derdndffd-derdndffd.d/derdndffd0de rdn erd5ndffd1erd2nddffd3d4derdndffd5dXerdYndffd6dXerdYndffd7dXerdYndffd8dXerdYndffd9dXerdYndffd:dXerdYndffd;dXerdYndffddXerdYndffd?dXerdYndffd@dXerdYndffdAdBdCerdDndffdEerdnderdndffdFerdGnderdHndffdIerdJnderdKndffdLdMerdNndffdOd@erdAndffdPd@erdAndffdQdRdSerdTndffdUderdndffdVerdWnderdXndffdYerdWnderdXndffdZerdnderdndffd[erd\nddffd]erdnderdndffd^derd5ndffd_erd`nddffdadberdcnddffdddedferdgndffdhderdndffdid:erd;ndffdjerdnderdndffdkerdnderdndffdlerdnderdndffdmdndodpderdndffdqerdrnderdsndffdterdnderdndffduderdndffdvderdndffdwderdndffdxderdndffdydze rdnder e sd5ndffd{d|d}d~dderdndffdderdndffderdnderdndffdderdnddffderdnderdndffderdnderdndffderdnderdndffdderdndffderdnderdndffderdnderdndffderdnderdndffderdnderdndffZ0e"e$e0Z1de$j2DZ3de.de e4edffdZ5 dde e.ej6e.fde e e.ej6e.fde7de e7de e8e.e.fde e e7e.fde e.de7de.de8e.effdZ9GddZ:ddgZ;dS(zAuto Tokenizer class.N) OrderedDict)AnyOptionalUnion)is_mistral_common_available)PretrainedConfig)get_class_from_dynamic_moduleresolve_trust_remote_code)load_gguf_checkpoint)PreTrainedTokenizer)TOKENIZER_CONFIG_FILE) cached_fileextract_commit_hashis_g2p_en_availableis_sentencepiece_availableis_tokenizers_availablelogging)EncoderDecoderConfig)_LazyAutoMapping)CONFIG_MAPPING_NAMES AutoConfigconfig_class_to_model_typemodel_type_to_module_name!replace_list_option_in_docstrings)PreTrainedTokenizerFastaimv2 CLIPTokenizerCLIPTokenizerFastalbertAlbertTokenizerAlbertTokenizerFastalign BertTokenizerBertTokenizerFastarceeLlamaTokenizerLlamaTokenizerFastaria aya_visionCohereTokenizerFastbark)bart) BartTokenizerBartTokenizerFastbarthezBarthezTokenizerBarthezTokenizerFast)bartpho)BartphoTokenizerNbertzbert-generationBertGenerationTokenizer)z bert-japanese)BertJapaneseTokenizerN)bertweet)BertweetTokenizerNbig_birdBigBirdTokenizerBigBirdTokenizerFastbigbird_pegasusPegasusTokenizerPegasusTokenizerFast)biogpt)BioGptTokenizerNbitnetr) blenderbot)BlenderbotTokenizerBlenderbotTokenizerFast)zblenderbot-small)BlenderbotSmallTokenizerNblipzblip-2 GPT2TokenizerGPT2TokenizerFastbloomBloomTokenizerFastblt bridgetowerRobertaTokenizerRobertaTokenizerFastbros)byt5) ByT5TokenizerN camembertCamembertTokenizerCamembertTokenizerFast)canine)CanineTokenizerN chameleon chinese_clipclapclipclipseg)clvp) ClvpTokenizerN code_llamaCodeLlamaTokenizerCodeLlamaTokenizerFastcodegenCodeGenTokenizerCodeGenTokenizerFastcoherecohere2colpalicolqwen2Qwen2TokenizerQwen2TokenizerFastconvbertConvBertTokenizerConvBertTokenizerFastcpm CpmTokenizerCpmTokenizerFast)cpmant)CpmAntTokenizerNcsm)ctrl) CTRLTokenizerN)zdata2vec-audioWav2Vec2CTCTokenizerNz data2vec-textdbrxdebertaDebertaTokenizerDebertaTokenizerFastz deberta-v2DebertaV2TokenizerDebertaV2TokenizerFast deepseek_v2 deepseek_v3 deepseek_vldeepseek_vl_hybrid)dia) DiaTokenizerN diffllama distilbertDistilBertTokenizerDistilBertTokenizerFastdprDPRQuestionEncoderTokenizerDPRQuestionEncoderTokenizerFastelectraElectraTokenizerElectraTokenizerFastemu3ernieernie4_5 ernie4_5_moeernie_mErnieMTokenizer)esm) EsmTokenizerNexaone4falcon falcon_mambaGPTNeoXTokenizerFastfastspeech2_conformerFastSpeech2ConformerTokenizer)flaubert)FlaubertTokenizerN flex_olmofnet FNetTokenizerFNetTokenizerFast)fsmt) FSMTTokenizerNfunnelFunnelTokenizerFunnelTokenizerFastgemmaGemmaTokenizerGemmaTokenizerFastgemma2gemma3 gemma3_textgemma3n gemma3n_textgitglmglm4glm4_moeglm4v glm4v_moezgpt-sw3GPTSw3Tokenizergpt2 gpt_bigcodegpt_neogpt_neox)gpt_neox_japanese)GPTNeoXJapaneseTokenizerNgpt_ossgptj)zgptsan-japanese)GPTSanJapaneseTokenizerN)graniterJN) granitemoer)granitemoehybridr)granitemoesharedrzgrounding-dinogroupvitheliumherbertHerbertTokenizerHerbertTokenizerFast)hubertrxibertideficsidefics2idefics3 instructblipinstructblipvideointernvljambajanusjetmoe)jukebox)JukeboxTokenizerNzkosmos-2XLMRobertaTokenizerXLMRobertaTokenizerFastz kosmos-2.5layoutlmLayoutLMTokenizerLayoutLMTokenizerFast layoutlmv2LayoutLMv2TokenizerLayoutLMv2TokenizerFast layoutlmv3LayoutLMv3TokenizerLayoutLMv3TokenizerFast layoutxlmLayoutXLMTokenizerLayoutXLMTokenizerFastled LEDTokenizerLEDTokenizerFastliltllamallama4 llama4_textllava llava_nextllava_next_videollava_onevision longformerLongformerTokenizerLongformerTokenizerFastlongt5 T5TokenizerT5TokenizerFast)luke) LukeTokenizerNlxmertLxmertTokenizerLxmertTokenizerFastm2m_100M2M100Tokenizermambamamba2marianMarianTokenizermbartMBartTokenizerMBartTokenizerFastmbart50MBart50TokenizerMBart50TokenizerFastmegaz megatron-bert metaclip_2)zmgp-str)MgpstrTokenizerNminimax ministralMistralCommonTokenizermistralmistral3mixtralmllamamlukeMLukeTokenizerzmm-grounding-dino mobilebertMobileBertTokenizerMobileBertTokenizerFast modernbert moonshinemoshimpnetMPNetTokenizerMPNetTokenizerFastmptmramt5 MT5TokenizerMT5TokenizerFastmusicgenmusicgen_melodymvp MvpTokenizerMvpTokenizerFast)myt5) MyT5TokenizerNnemotronnezhanllb NllbTokenizerNllbTokenizerFastznllb-moe nystromformerolmoolmo2olmo3olmoez omdet-turbo oneformerz openai-gptOpenAIGPTTokenizerOpenAIGPTTokenizerFastoptowlv2owlvit paligemma)parakeet)ParakeetCTCTokenizerNpegasus pegasus_x) perceiver)PerceiverTokenizerN persimmonphiphi3phimoe)phobert)PhobertTokenizerN pix2structpixtralplbartPLBartTokenizer) prophetnet)ProphetNetTokenizerNqdqbertqwen2 qwen2_5_omni qwen2_5_vl qwen2_audio qwen2_moeqwen2_vlqwen3 qwen3_moe qwen3_nextqwen3_omni_moeqwen3_vl qwen3_vl_moe)rag) RagTokenizerNrealmRealmTokenizerRealmTokenizerFastrecurrent_gemmareformerReformerTokenizerReformerTokenizerFastrembertRemBertTokenizerRemBertTokenizerFast retribertRetriBertTokenizerRetriBertTokenizerFastrobertazroberta-prelayernorm)roc_bert)RoCBertTokenizerNroformerRoFormerTokenizerRoFormerTokenizerFastrwkv seamless_m4tSeamlessM4TTokenizerSeamlessM4TTokenizerFastseamless_m4t_v2 shieldgemma2siglipSiglipTokenizersiglip2smollm3speech_to_textSpeech2TextTokenizer)speech_to_text_2)Speech2Text2TokenizerNspeecht5SpeechT5Tokenizer)splinter)SplinterTokenizerSplinterTokenizerFast squeezebertSqueezeBertTokenizerSqueezeBertTokenizerFaststablelm starcoder2switch_transformerst5t5gemma)tapas)TapasTokenizerN)tapex)TapexTokenizerN)z transfo-xl)TransfoXLTokenizerNtvpudop UdopTokenizerUdopTokenizerFastumt5 video_llavaviltvipllava visual_bert)vits) VitsTokenizerNvoxtral)wav2vec2rx)z wav2vec2-bertrx)zwav2vec2-conformerrx)wav2vec2_phoneme)Wav2Vec2PhonemeCTCTokenizerNwhisperWhisperTokenizerWhisperTokenizerFastxclipxglm XGLMTokenizerXGLMTokenizerFast)xlm) XLMTokenizerNzxlm-prophetnetXLMProphetNetTokenizerz xlm-robertazxlm-roberta-xlxlnetXLNetTokenizerXLNetTokenizerFastxlstmxmodyosozambazamba2ci|]\}}|| Sr).0kvs ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py r$s@@@41a!Q@@@ class_namereturnc$|dkrtStD]s\}}||vrjt|}|dvr|dkrt jdd}nt jd|d} t ||cS#t$rYowxYwttj D]"}|D]}t |dd|kr|ccS#t jd}t||rt ||SdS) Nr)rrr r z.tokenization_mistral_common transformers.ztransformers.models__name__) rTOKENIZER_MAPPING_NAMESitemsr importlib import_modulegetattrAttributeErrorTOKENIZER_MAPPING_extra_contentvalueshasattr)r module_name tokenizersmodule tokenizer main_modules rtokenizer_class_from_namer'sq...&&#:#@#@#B#B   Z  # #3K@@KAAAjTlFlFl"01OQ_``"01B[1B1BDYZZ vz22222!     $(6==??!! # ! !Iy*d33zAA      B ! ).99K{J''0{J/// 4s8B  BBFpretrained_model_name_or_path cache_dirforce_downloadresume_downloadproxiestokenrevisionlocal_files_only subfolderc  | dd} | -tjdt|t d| }| d} t |t||||||||ddd| } | t diSt| | } t| d 5} tj | }dddn #1swxYwY| |d<|S) a Loads the tokenizer configuration from a pretrained model tokenizer configuration. Args: pretrained_model_name_or_path (`str` or `os.PathLike`): This can be either: - a string, the *model id* of a pretrained model configuration hosted inside a model repo on huggingface.co. - a path to a *directory* containing a configuration file saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force to (re-)download the configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}.` The proxies are used on each request. token (`str` or *bool*, *optional*): The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated when running `hf auth login` (stored in `~/.huggingface`). revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, will only try to load the tokenizer configuration from local files. subfolder (`str`, *optional*, defaults to `""`): In case the tokenizer config is located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Passing `token=True` is required when you want to use a private model. Returns: `dict`: The configuration of the tokenizer. Examples: ```python # Download configuration from huggingface.co and cache. tokenizer_config = get_tokenizer_config("google-bert/bert-base-uncased") # This model does not have a tokenizer config so the result will be an empty dict. tokenizer_config = get_tokenizer_config("FacebookAI/xlm-roberta-base") # Save a pretrained tokenizer locally and you can reload its config from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased") tokenizer.save_pretrained("tokenizer-test") tokenizer_config = get_tokenizer_config("tokenizer-test") ```use_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`. _commit_hashF) rrrrrrrr _raise_exceptions_for_gated_repo%_raise_exceptions_for_missing_entries'_raise_exceptions_for_connection_errorsrz\Could not locate the tokenizer configuration file, will try to use the model config instead.zutf-8)encoding)popwarningswarn FutureWarning ValueErrorgetrrloggerinforopenjsonload)rrrrrrrrrkwargsr commit_hashresolved_config_filereaderresults rget_tokenizer_configrEs\RZZ 0$77N!  A      uvv v**^,,K&%%'))..305  # rsss %&:KHHK "W 5 5 5#6""###############(F> Ms<CC!$C!cheZdZdZdZeeedZe ddZ dS) AutoTokenizera This is a generic tokenizer class that will be instantiated as one of the tokenizer classes of the library when created with the [`AutoTokenizer.from_pretrained`] class method. This class cannot be instantiated directly using `__init__()` (throws an error). c td)Nz}AutoTokenizer is designed to be instantiated using the `AutoTokenizer.from_pretrained(pretrained_model_name_or_path)` method.)OSError)selfs r__init__zAutoTokenizer.__init__s _   rc 6 |dd}|Ctjdt|dt d||d<|dd}d|d<|d d}|d d}|d d}|d } |d} t |d} | 8t d |dddt Dd| \} } |r,| t| } nt d| t| } | t d| d| j |g|Ri|St|fi|}d|vr |d|d<|d}d}d|vrGt|dttfr |d}n|ddd}|t|t sM| r7t#|| fi|}t%|dd}t'jd(i|}nt'j |fd |i|}|j}t-|drd|jvr |jd}|du}t1|t2vp(|duo$t|dupt|dzdu}|rP|r|d |d}n|d}d|vr|dd}nd}t7|||||}|rM|rKt9||fi|} |dd}| | j |g|Rd |i|S|fd} |r)|ds|d}t|} | |}t|} | t d|d | j |g|Ri|St|t>rdt1|j t1|j!ur5t d!|j!j"d"|j j"d#|j!}tGt1|j$}|Vt2t1|\}}|r|s||j |g|Ri|S||j |g|Ri|St d$t d%|j"d&dd't2Dd))a] Instantiate one of the tokenizer classes of the library from a pretrained model vocabulary. The tokenizer class to instantiate is selected based on the `model_type` property of the config object (either passed as an argument or loaded from `pretrained_model_name_or_path` if possible), or when it's missing, by falling back to using pattern matching on `pretrained_model_name_or_path`: List options Params: pretrained_model_name_or_path (`str` or `os.PathLike`): Can be either: - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co. - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`. - A path or url to a single saved vocabulary file if and only if the tokenizer only requires a single vocabulary file (like Bert or XLNet), e.g.: `./my_model_directory/vocab.txt`. (Not applicable to all derived classes) inputs (additional positional arguments, *optional*): Will be passed along to the Tokenizer `__init__()` method. config ([`PretrainedConfig`], *optional*) The configuration object used to determine the tokenizer class to instantiate. cache_dir (`str` or `os.PathLike`, *optional*): Path to a directory in which a downloaded pretrained model configuration should be cached if the standard cache should not be used. force_download (`bool`, *optional*, defaults to `False`): Whether or not to force the (re-)download the model weights and configuration files and override the cached versions if they exist. resume_download: Deprecated and ignored. All downloads are now resumed by default when possible. Will be removed in v5 of Transformers. proxies (`dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request. revision (`str`, *optional*, defaults to `"main"`): The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any identifier allowed by git. subfolder (`str`, *optional*): In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for facebook/rag-token-base), specify it here. use_fast (`bool`, *optional*, defaults to `True`): Use a [fast Rust-based tokenizer](https://huggingface.co/docs/tokenizers/index) if it is supported for a given model. If a fast tokenizer is not available for a given model, a normal Python-based tokenizer is returned instead. tokenizer_type (`str`, *optional*): Tokenizer type to be loaded. trust_remote_code (`bool`, *optional*, defaults to `False`): Whether or not to allow for custom models defined on the Hub in their own modeling files. This option should only be set to `True` for repositories you trust and in which you have read the code, as it will execute code present on the Hub on your local machine. kwargs (additional keyword arguments, *optional*): Will be passed to the Tokenizer `__init__()` method. Can be used to set special tokens like `bos_token`, `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`. See parameters in the `__init__()` for more details. Examples: ```python >>> from transformers import AutoTokenizer >>> # Download vocabulary from huggingface.co and cache. >>> tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased") >>> # Download vocabulary from huggingface.co (user-uploaded) and cache. >>> tokenizer = AutoTokenizer.from_pretrained("dbmdz/bert-base-german-cased") >>> # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*) >>> # tokenizer = AutoTokenizer.from_pretrained("./test/bert_saved_model/") >>> # Download vocabulary from huggingface.co and define model-specific arguments >>> tokenizer = AutoTokenizer.from_pretrained("FacebookAI/roberta-base", add_prefix_space=True) ```rNrrrconfigT _from_autouse_fasttokenizer_typetrust_remote_code gguf_filezPassed `tokenizer_type` z3 does not exist. `tokenizer_type` should be one of z, c3K|]}|VdSNrrcs r z0AutoTokenizer.from_pretrained..+s" D Dq D D D D D Drrzt`use_fast` is set to `True` but the tokenizer class does not have a fast version. Falling back to the slow version.zTokenizer class z is not currently imported.rtokenizer_classauto_maprF)return_tensorsFastrrz-- code_revisionz- does not exist or is not currently imported.z The encoder model config class: z3 is different from the decoder model config class: z. It is not recommended to use the `AutoTokenizer.from_pretrained()` method in this case. Please use the encoder and decoder specific tokenizer classes.zzThis tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.z!Unrecognized configuration class z8 to build an AutoTokenizer. Model type should be one of c3$K|] }|jV dSr)rrs rrz0AutoTokenizer.from_pretrained..s$4[4[AQZ4[4[4[4[4[4[rr)%rrrrrrrjoinrrwarningfrom_pretrainedr isinstancetuplelistr rr r for_modelrrrtypersplitr r register_for_auto_classendswithrdecoderencoder __class__rr)clsrinputsrrrrrrrrtokenizer_class_tupletokenizer_class_nametokenizer_fast_class_nametokenizer_configconfig_tokenizer_classtokenizer_auto_map gguf_path config_dicthas_remote_codehas_local_code class_ref upstream_repo_tokenizer_class_candidate model_typetokenizer_class_pytokenizer_class_fasts rr zAutoTokenizer.from_pretrainedsZ $4d;;  % ME   zz'"". l-F7OHd++#|::j$//$4d;;"JJ':DAAJJ{++   %"O$;$?$?PT$U$U !$, H~HHyy D D,C D D DDDHHH ?T ; "; ,8&?@Y&Z&ZOONN=&"; "!1!5!56G!H!H! ) ) )*:6 FF ]%5j%A""%5j%A%E%EoW[%\%\" " )f&677  +,I9 _ _X^ _ _I"6yQV"W"W"WX`"aK'1@@K@@FF'75IZ^dF&,%; "vz** F&//Q/Q%+__%E",D8f):: "$ . )*@AAMZ,-Cf-LMMUYY    2.q1=.q1 .q1 y  ) 5 5a 8 $ 9!#@.Racp!!   e0 e;IGdoohnooO ?D11A  3 3 5 5 52?2-06J[_e $ /"O W 6 ? ? G G W/E,K,K,K)";[o^dooohnooo$:  _0@ _ _+/994[4[IZ4[4[4[+[+[ _ _ _   rNFc||td|$t|trtd|$t|trtd|=|;t|tr&|j|krtd|jd|d|t jvrt |\}}||}||}t |||f|dS) a Register a new tokenizer in this mapping. Args: config_class ([`PretrainedConfig`]): The configuration corresponding to the model to register. slow_tokenizer_class ([`PretrainedTokenizer`], *optional*): The slow tokenizer to register. fast_tokenizer_class ([`PretrainedTokenizerFast`], *optional*): The fast tokenizer to register. NzKYou need to pass either a `slow_tokenizer_class` or a `fast_tokenizer_classz:You passed a fast tokenizer in the `slow_tokenizer_class`.z:You passed a slow tokenizer in the `fast_tokenizer_class`.zThe fast tokenizer class you are passing has a `slow_tokenizer_class` attribute that is not consistent with the slow tokenizer class you passed (fast tokenizer has z and you passed z!. Fix one of those so they match!)exist_ok)r issubclassrr slow_tokenizer_classrrregister) config_classr-fast_tokenizer_classr+ existing_slow existing_fasts rr.zAutoTokenizer.registers6 ',@,Hjkk k  + ;OQh0i0i +YZZ Z  + ;OQd0e0e +YZZ Z ! ,$0/1HII1$9=QQQ!'<!!Nb!!!  ,; ; ;+<\+J (M=#+'4$#+'4$""<2FH\1]hp"qqqqqr)NNF) r __module__ __qualname____doc__r classmethodrrr  staticmethodr.rrrrrs   &&'>??` ` @?[` D)r)r)r\)r)r)rrrr)NFNNNNFr)rMs43 ######''''''''''GGGGGG333333\\\\\\\\??????555555<<<<<<322222******#BBBBBBB"  H % %P+c5# 1M+N&NOc  '>'>'@'@J##d  c  %?%?%A%AK!!t)@)@)B)BL%%  c  ?;R;R;T;T$^$7$7Z^_`c  #=T=T=V=V%`%9%9\`ab!c " " ;?c @ 2Ac D &@&@&B&BL""*A*A*C*CM&&  Cc P /KbKbKdKd1n1G1GjnopQc R .Sc T D7N7N7P7PZ33VZ[\Uc V KWc X AYc Z /:Q:Q:S:S#]#6#6Y]^_[c \ O$>$@$@J  d(?(?(A(AK$$t  yc F /BYBYB[B[+e+>+>aefgGc J "*A*A*C*CM&&  Ic X '>'>'@'@J##d  Wc f '>'>'@'@J##d  ec r *sc v (B(B(D(DN$$$,C,C,E,EO((4  uc B 'CZCZC\C\)f)?)?bfghCc D D3J3J3L3LV//RVWXEc F T4K4K4M4MW00SWXYGc H %?V?V?X?X'b';';^bcdIc J &@W@W@Y@Y(c(<(<_cdeKc L )F]F]F_F_+i+B+BeijkMc P "<"<">">HD&=&=&?&?I""T  Oc \ .]c ^ 4K4K4M4MW00SWXY_c ` *ac b ;cc d -I`I`IbIb/l/E/Ehlmnec f /:Q:Q:S:S#]#6#6Y]^_gc h 'CZCZC\C\)f)?)?bfghic l (B(B(D(DN$$$,C,C,E,EO((4  kc z $>$>$@$@J  d(?(?(A(AK$$t  yc H $>$>$@$@J  d(?(?(A(AK$$t  Gc V $>$>$@$@J  d(?(?(A(AK$$t  Uc d !$>$>$@$@J  d(?(?(A(AK$$t  cc p (qc t $>$>$@$@J  d(?(?(A(AK$$t  sc @ -LcLcLeLe/o/H/HkopqAc D -5L5L5N5NX11TX  Cc P 'CZCZC\C\)f)?)?bfghQc R /:Q:Q:S:S#]#6#6Y]^_Sc T ?;R;R;T;T$^$7$7Z^_`Uc V d4K4K4M4MW00SWXYWc X $8O8O8Q8Q [ 4 4W[\]Yc Z *D*D*F*FP&&DRVWX[c \ (]c ` #:#:#<#<F$'>'>'@'@J##d  _c l D7N7N7P7PZ33VZ[\mc n $:Q:Q:S:S ] 6 6Y]^_oc r $0C0C0E0E O , ,4QU V qc x 2yc z t4K4K4M4MW00SWXY{c | /:Q:Q:S:S#]#6#6Y]^_}c ~ *c @ %@W@W@Y@Y'c'<'<_cdeAc D $>$>$@$@J  d(?(?(A(AK$$t  Cc R $>$>$@$@J  d(?(?(A(AK$$t  Qc ` $>$>$@$@J  d(?(?(A(AK$$t  _c n $>$>$@$@J  d(?(?(A(AK$$t  mc | $>$>$@$@J  d(?(?(A(AK$$t  {c J $>$>$@$@J  d(?(?(A(AK$$t  Ic V 9P9P9R9R"\"5"5X\]^Wc X 4K4K4M4MW00SWXYYc Z $5L5L5N5NX11TXYZ[c \ d9P9P9R9R\55X\]^]c ^ 46M6M6O6OY22UYZ[_c ` t:Q:Q:S:S]66Y]^_ac b *D*D*F*FP&&DRVWXcc d /:Q:Q:S:S#]#6#6Y]^_ec f AXAXAZAZ*d*=*=`defgc h _=T=T=V=V&`&9&9\`abic j d6M6M6O6OY22UYZ[kc l Bmc n T8O8O8Q8Q[44W[\]oc p /:Q:Q:S:S#]#6#6Y]^_qc r ?sc t -uc v 0wc x 6yc z 6{c | OD[D[D]D]-g-@-@cghi}c ~ o>U>U>W>W'a':':]abcc @ D7N7N7P7PZ33VZ[\Ac B 'CZCZC\C\)f)?)?bfghCc D 3Ec F %AXAXAZAZ'd'='=`defGc H T3J3J3L3LV//RVWXIc J &@W@W@Y@Y(c(<(<_cdeKc L &@W@W@Y@Y(c(<(<_cdeMc N /BYBYB[B[+e+>+>aefgOc P G^G^G`G`0j0C0CfjklQc R &@W@W@Y@Y(c(<(<_cdeSc V $>$>$@$@J  d(?(?(A(AK$$t  Uc b 41H1H1J1JT--PTUVcc f $>$>$@$@J  d(?(?(A(AK$$t  ec r 0sc v )C)C)E)EO%%4-D-D-F-FP))D  uc B ;R;R;T;T^77Z^_`C c D )F]F]F_F_+i+B+BeijkE c F -LcLcLeLe/o/H/HkopqG c H -LcLcLeLe/o/H/HkopqI c J +I`I`IbIb-l-E-EhlmnK c L 7N7N7P7P!Z!3!3VZ[\M c N 'F]F]F_F_)i)B)BeijkO c R $>$>$@$@J  d(?(?(A(AK$$t  Q c ` $>$>$@$@J  d(?(?(A(AK$$t  _ c n $>$>$@$@J  d(?(?(A(AK$$t  m c z #=T=T=V=V%`%9%9\`ab{ c | (BYBYB[B[*e*>*>aefg} c ~ .H_H_HaHa0k0D0Dgklm c @ -G^G^G`G`/j/C/CfjklA c B -LcLcLeLe/o/H/HkopqC c F !;!;!=!=G 4%<%<%>%>H!!D  E c R *S c T %@W@W@Y@Y'c'<'<_cdeU c V *D*D*F*FP&&DRVWXW c X 43J3J3L3LV//RVWXY c Z D4K4K4M4MW00SWXY[ c \ )C)C)E)EO%%4QUVW] c ` $>$>$@$@J  d(?(?(A(AK$$t  _ c n &@&@&B&BL""*A*A*C*CM&&  m c z $@W@W@Y@Y&c&<&<_cde{ c | ?CZCZC\C\,f,?,?bfgh} c @ %-D-D-F-FP))D   c L /M c P #=#=#?#?IT'>'>'@'@J##d  O c ^ /.00R((*D*D*F*FP&&D(?(?(A(AqJeJeJgJgq$$mq   ] c p /.00R((*D*D*F*FP&&D(?(?(A(AqJeJeJgJgq$$mq   o c B /.00R((*D*D*F*FP&&D(?(?(A(AqJeJeJgJgq$$mq   A c T /.00R((*D*D*F*FP&&D(?(?(A(AqJeJeJgJgq$$mq   S c d $>U>U>W>W&a&:&:]abce c f 'A'A'C'CM##tTUg c h G^G^G`G`0j0C0Cfjkli c j -LcLcLeLe/o/H/Hkopqk c l ;R;R;T;T^77Z^_`m c n t:Q:Q:S:S]66Y]^_o c p 46M6M6O6OY22UYZ[q c r #=T=T=V=V%`%9%9\`abs c t 1H1H1J1JT--PTUVu c v #?V?V?X?X%b%;%;^bcdw c z "<"<">">HD&=&=&?&?I""T  y c F m:Q:Q:S:S%]%6%6Y]^_G c H ]AXAXAZAZ,d,=,=`defI c J 7N7N7P7P!Z!3!3VZ[\K c L *M c N d9P9P9R9R\55X\]^O c P ?;R;R;T;T$^$7$7Z^_`Q c T #=#=#?#?IT'>'>'@'@J##d  S c b #=#=#?#?IT'>'>'@'@J##d  a c p %?%?%A%AK!!t)@)@)B)BL%%  o c | $2I2I2K2KU..QUVW} c ~ 43J3J3L3LV//RVWX c @ 40G0G0I0IS,,tTUAc B 43J3J3L3LV//RVWXCc F  5L5L5N5NX11TX Y Ec L ?V?V?X?X(b(;(;^bcdMc P  !?V?V?X?X#b#;#;^b c Oc V 9P9P9R9R"\"5"5X\]^Wc X ?;R;R;T;T$^$7$7Z^_`Yc Z O$>$@$@J  d(?(?(A(AK$$t  Kc X #?V?V?X?X%b%;%;^bcdYc Z "U>U>W>W&a&:&:]abc]c ^ 0_c `  *>aefgKc L )CZCZC\C\+f+?+?bfghMc P  (?(?(A(AK$$t  Oc \ &@W@W@Y@Y(c(<(<_cde]c `  (?(?(A(AK$$t  _c n  (?(?(A(AK$$t  mc |  (?(?(A(AK$$t  {c H ,F]F]F_F_.i.B.BeijkIc J &@W@W@Y@Y(c(<(<_cdeKc L *D[D[D]D],g,@,@cghiMc N (Oc P #=T=T=V=V%`%9%9\`abQc T $>$>$@$@J  d(?(?(A(AK$$t  Sc b 'A'A'C'CM##+B+B+D+DN''$  ac p &@&@&B&BL""*A*A*C*CM&&  oc | +I`I`IbIb-l-E-Ehlmn}c ~ 'CZCZC\C\)f)?)?bfghc B # ;R;R;T;T!^!7!7Z^ _ Ac H 1Ic J )F]F]F_F_+i+B+BeijkKc L $2I2I2K2KU..QUVWMc P *D*D*F*FP&&D.E.E.G.GQ**T  Oc ^ *D*D*F*FP&&D.E.E.G.GQ**T  ]c l $>$>$@$@J  d(?(?(A(AK$$t  kc x )C)C)E)EO%%4QUVWyc | $>$>$@$@J  d(?(?(A(AK$$t  {c H T8O8O8Q8Q[44W[\]Ic J 6P6P6R6R\22X\^bcdKc L >Mc N -G-G-I-IS))tUYZ[Oc P EQc T  #CZCZC\C\%f%?%?bf g Sc Z d6M6M6O6OY22UYZ[[c \ @W@W@Y@Y)c)<)<_cde]c ` "!;!;!=!=G 4%<%<%>%>H!!D  _c n !;!;!=!=G 4%<%<%>%>H!!D  mc | $>$>$@$@J  d(?(?(A(AK$$t  {c H ,Ic J ,Kc L 5Mc N 9P9P9R9R"\"5"5X\]^Oc R #=#=#?#?IT'>'>'@'@J##d  Qc ` !;!;!=!=G 4%<%<%>%>H!!D  _c l )CZCZC\C\+f+?+?bfghmc n /:Q:Q:S:S#]#6#6Y]^_oc p &@W@W@Y@Y(c(<(<_cdeqc r AXAXAZAZ*d*=*=`defsc t *uc x ,G,G,I,IS((t-D-D-F-FvOjOjOlOlv))rv  wc D 5Ec F :Gc H ?Ic J DKc L 'CZCZC\C\)f)?)?bfghMc N ?;R;R;T;T$^$7$7Z^_`Oc R #=#=#?#?IT'>'>'@'@J##d  Qc ^ (_c ` 8R8R8T8T^44Z^`defac d )C)C)E)EO%%4-D-D-F-FP))D  cc r )C)C)E)EO%%4-D-D-F-FP))D  qc @ $>$>$@$@J  d(?(?(A(AK$$t  c L 43J3J3L3LV//RVWXMc P )C)C)E)EO%%4-D-D-F-FP))D  Oc ^ %?%?%A%AK!!t)@)@)B)BL%%  ]c l $>$>$@$@J  d(?(?(A(AK$$t  kc z $>$>$@$@J  d(?(?(A(AK$$t  yc e e N%$%9;RSS@@#=#7#=#?#?@@@#%S 42H@9= &*(,(,""ll#(bk#.>)>#?lc2;s#3345lld^ l d38n % l E$)$ % lsmlll #s(^llll^\r\r\r\r\r\r\r\r~  0r