`idZddlZddlZddlZddlmZddlmZddlm Z m Z m Z ddl m ZddlmZddlmZddlmZdd lmZmZmZmZd d lmZd d lmZd d lm Z d dl!m"Z"d dl#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-d dl.m/Z/m0Z0m1Z1e1j2e3Z4dZ5dZ6dZ7dZ8dZ9e$dz Z$eeeedZ:e5e8dZ;e0e$Gdde)Zee defd?Z; dKd@e%ee dLdFZ?xZ@S)MPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classch |dd}|dd}|dd}|dd}|dd}|di}|dd|_|r||jt d |t j|} nB||stj|} n(|rt|} n|t|d } | d d } | d } | d} t| | \} }| | t|dkr| |n|j!|dur|j|i|}t|} nZ|sI|d |_|dg|_t|d} d}nt d| |_|| |jd|_|jj}||jjd(i||d|d|d|d|d|d|d|dn|j|jj}||jjd(i||d|d|d|d|d|d|d|d|d|dt3jd(i||j|j_d |jDfd!t=|d"#DtA|j!"d$Dzfd%|j#Dz tdkrg}|j$}D]}tK|tLr|j'ptQ||vntQ||v}tK|tPrtM||&}n||_'|)||r|*| tWj,|j-j./}|d|j|jkrFtatb|d'}|j|d<|d(i||j-_.dSdS#td$rYdSwxYw))Ntokenizer_object__slow_tokenizer gguf_filer% from_slowFadded_tokens_decoderadd_prefix_spacezCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r&config model_type tokenizertokenizer_configradditional_special_tokensT) from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofcFh|]}tt|Shashrepr.0tokens x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py z3PreTrainedTokenizerFast.__init__..s&$^$^$^5T$u++%6%6$^$^$^cVg|]%\}}tt|v#|&SrDrE)rIindexrJadded_tokens_decoder_hashs rK z4PreTrainedTokenizerFast.__init__..sA   uDKK  (AAA AAArMc|dSNrrD)xs rKz2PreTrainedTokenizerFast.__init__..s STUVSWrMkeyc,g|]}t|SrD)strrHs rKrQz4PreTrainedTokenizerFast.__init__..s;b;b;b5CJJ;b;b;brMc$g|] }|v|v | SrDrD)rIrJencoder tokens_to_adds rKrQz4PreTrainedTokenizerFast.__init__..s7   5PWCWCW\aiv\v\vE\v\v\vrM)specialtyperD)3popgetr0r) ValueErrorcopydeepcopy TokenizerFast from_filerrrupdatelenr&r5 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr/sorteditemslistadded_tokens_encoderkeysall_special_tokens_extendedall_special_tokens isinstancerr]rYappend add_tokensjsonloadsbackend_tokenizer pre_tokenizer __getstate__getattrpre_tokenizers_fast Exception)selfargskwargsr+slow_tokenizerr-fast_tokenizer_filer.r/fast_tokenizer gguf_param architecturetokenizer_dictr4additional_kwargs _truncation_paddingtokensspecial_tokensrJ is_special pre_tok_state pre_tok_classrPr[r\ __class__s @@@rKrrz PreTrainedTokenizerFast.__init__bsU!::&8$??$6==JJ{D11 $jj)94@@JJ{E22 %zz*@"EE & +=u E E  /D4M4U0   '!]+;< > >   3[5L M M M M O ) ) + + +?*   *DO * 6 6X 6 6 6   k8K+@ A A A   18M3J K K K   nh{.C D D D   lHX,> ? ? ?   2H=Q4R S S S ""6"""040I-$^$^DD]$^$^$^!    &';'A'A'C'C X X X   t0557788;b;bTa;b;b;bb     #?     }   ! !F!4N& % %"%446U]Bc%jjN&BU~5 eS))/&ujAAAEE$.EM e$$$$ ('''  Jt'='K'X'X'Z'Z[[M  !3T5JKKtOddd '(;]=N=Nv=V=V W W 484I 017D}7U7U}7U7U&444ed    DD  sBV## V10V1returncdS)NTrDrs rKis_fastzPreTrainedTokenizerFast.is_faststrMcd|jvr]|jddr=t|dr+|jr$tj|jSdSdS)z `bool`: Whether or not the slow tokenizer can be saved. For a sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. r&z.modelFT)vocab_files_namesendswithhasattrr&ospathisfilers rKcan_save_slow_tokenizerz/PreTrainedTokenizerFast.can_save_slow_tokenizersi 41 1 1d6L\6Z6c6cdl6m6m 1t\** 7t 7w~~do66654rMc8|jdS)zP `int`: Size of the base vocabulary (without the added tokens). Fwith_added_tokensrhget_vocab_sizers rK vocab_sizez"PreTrainedTokenizerFast.vocab_sizes ---FFFrMc8|jdS)NTr)rh get_vocabrs rKrz!PreTrainedTokenizerFast.get_vocabs((4(@@@rMc*|SN)rrs rKvocabzPreTrainedTokenizerFast.vocabs~~rMchdt|jdDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. c$i|] \}}|j|SrDcontentrIvks rK z@PreTrainedTokenizerFast.added_tokens_encoder.. mmmA 1mmmrMc|dSrSrDitems rKrUz>PreTrainedTokenizerFast.added_tokens_encoder.. dhijdkrMrVrur/rvrs rKrxz,PreTrainedTokenizerFast.added_tokens_encoders9 nm0I0O0O0Q0QWkWk)l)l)lmmmmrMc4|jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `dict[str, int]`: The added tokens. )rhget_added_tokens_decoderrs rKr/z,PreTrainedTokenizerFast.added_tokens_decoders77999rMchdt|jdDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `dict[str, int]`: The added tokens. c$i|] \}}|j|SrDrrs rKrz;PreTrainedTokenizerFast.get_added_vocab..rrMc|dSrSrDrs rKrUz9PreTrainedTokenizerFast.get_added_vocab..rrMrVrrs rKget_added_vocabz'PreTrainedTokenizerFast.get_added_vocabs9nm0I0O0O0Q0QWkWk)l)l)lmmmmrMcdS)zN Returns True, to avoid expensive `assert tokenizer` gotchas. TrDrs rK__bool__z PreTrainedTokenizerFast.__bool__s trMc8|jdS)zD Size of the full vocabulary with the added tokens. Trrrs rK__len__zPreTrainedTokenizerFast.__len__s---EEErMc|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )rhrs rKrz)PreTrainedTokenizerFast.backend_tokenizer%s rMc|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )rhdecoderrs rKrzPreTrainedTokenizerFast.decoder,s &&rMFTencodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverbosec F| d|jv}| d|jv}|r|j |g|jz} n|g} tt} | D]} | d| j|r | d| j|r | d| j|r | d| j|r | d| j |r-| dt| j| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) Ntoken_type_idsattention_mask input_idsspecial_tokens_maskoffset_mappingrA) model_input_names overflowingrrwr}idstype_idsrroffsetsrg) rrrrrrrrr encodings encoding_dictes rK_convert_encodingz)PreTrainedTokenizerFast._convert_encoding3sV( ! ($48N$N ! ($48N$N ! $ #)=)I! X%99II! I#D))  ; ;A + & - -ae 4 4 4$ C./66qzBBB$ I./66q7GHHH) S34;;A.ps'TTTE88??TTTrM)r|rYrrrs` rKconvert_tokens_to_idsz-PreTrainedTokenizerFast.convert_tokens_to_idsbsF fc " " D;;FCC CTTTTVTTTTrMrJcL|j|}||jS|Sr)rh token_to_id unk_token_id)rrJrOs rKrz;PreTrainedTokenizerFast._convert_token_to_id_with_added_vocrs*++E22 =$ $ rMrOcP|jt|Sr)rh id_to_tokenint)rrOs rK_convert_id_to_tokenz,PreTrainedTokenizerFast._convert_id_to_tokenxs**3u::666rM new_tokenscn|r|j|S|j|Sr)rhadd_special_tokensr~)rrrs rK _add_tokensz#PreTrainedTokenizerFast._add_tokens{s7  B?55jAA A))*555rMpairc6|j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )rhnum_special_tokens_to_add)rrs rKrz1PreTrainedTokenizerFast.num_special_tokens_to_adds&88>>>rMrskip_special_tokensc<t|tr|j|Sg}|rt |jn t }|D]C}t|}||vr||j|D|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `list[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `list[str]`: The decoded token(s). )r|rrhrsetall_special_idsr})rrrr ids_to_skiprOs rKconvert_ids_to_tokensz-PreTrainedTokenizerFast.convert_ids_to_tokenss c3   4?..s33 33FQc$.///CEE  > >EJJE ## MM$/55e<< = = = = rMtextrc H|jd|||d|S)N)r text_pairrrD) encode_plusr)rrrrrs rKtokenizez PreTrainedTokenizerFast.tokenizes2tkTTN`kkdjkkrrtttrMpadding_strategyr;r7r:rBr@c |jj |jj}|tjkr |jn<|||j|jd} d} n fd|D} | |kr|jjdi||tj kr||j dSdS|tj kr|nd} | ||n|j |j|j|j|d}||kr|jjdi|dSdS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r7r:r<r9c>i|]}||dSrr`)rIrrs rKrzFPreTrainedTokenizerFast.set_truncation_and_padding..s)GGG11kooa66GGGrM)rAr9pad_idr=r?rBrD)rhrkrorDO_NOT_TRUNCATErnvaluer8rlr DO_NOT_PAD no_padding MAX_LENGTHr@ pad_token_idr=r>rp) rrr;r7r:rBr@rtargetcurrentrArs @rKset_truncation_and_paddingz2PreTrainedTokenizerFast.set_truncation_and_paddingsiBo0 ?* "4"D D D&--///) /5!1 F"GGGGGGG&  11;;F;;; 9 9 9#**,,,,,$#$47Q#Q#QZZW[F -9-E\\4K\+!^#5&8 F6!!..8888888"!rMrbatch_text_or_text_pairsis_split_into_wordsreturn_tensorsrsc t|ttfs tdt |d|||||| jj|kr |j_j|||}   fd|D}i}|ddD]fd|D}||<d|D} r;g}t|D]$\}\}}||gt|d zz }%||d <|d D]} ||t||| S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got ))rr;r7r:rBr@)ris_pretokenizedc Jg|]}|  S))rrrrrrrr)r) rIrrrrrrrrrs rKrQz>PreTrainedTokenizerFast._batch_encode_plus..5sX   " "!&;&;*C+E'=+ #   rMrc0g|]\}}|D]}|SrDrD)rIr_rrWs rKrQz>PreTrainedTokenizerFast._batch_encode_plus..Ks.NNN74DINNqQNNNNrMc"g|] \}}|D]}| SrDrD)rIrrrs rKrQz>PreTrainedTokenizerFast._batch_encode_plus..Ms)SSSWQdSSqSSSSrMroverflow_to_sample_mapping) tensor_type) r|tuplerw TypeErrorr^r rhrt encode_batch enumeraterg&_eventual_warn_about_too_long_sequencer)rrrrr;r7r:rrBr@rrrrrrrrrsrtokens_and_encodingssanitized_tokensstacksanitized_encodingsritoksrrrWs` ``````` @rK_batch_encode_plusz*PreTrainedTokenizerFast._batch_encode_pluss.2UDMBB nTRjMkMknnn  ''- 3!1% (    ? 04H H H4HDO 1O00 $1/1   & ('*1- * *CNNNN&:NNNE$) S ! !SS0DSSS % X)+ & )*> ? ? K K 9D!*qcC[8I4J4J.JJ**=W 9 :)+6 X XI  7 7 :w W W W W-/BP^____rMrc @|r||fgn|g}|j|fid|d|d|d|d|d|d| d| d | d | d | d |d |d|d|d|d||}| 3|s1td|D|j}||d|||S)Nrrrr;r7r:rBr@rrrrrrrrrsci|]>\}}|t|dkr#t|dtr|dn|?S)r)rgr|rw)rIrWrs rKrz8PreTrainedTokenizerFast._encode_plus..sW"Uc%jj1nnE!Hd9S9Sn%((Y^rMr)r%rrvrr)rrrrrr;r7r:rrBr@rrrrrrrrrsr batched_inputbatched_outputs rK _encode_plusz$PreTrainedTokenizerFast._encode_plus[s.09D$ *++tf 00    3 3  21 .-  !4 3  "z  6  21 & *> #8"7 #8"7 '@&? (B'A $:#9 (-! "G# $"6!5'  0  !*C !*&4&:&:&<&<( N 33N;4OQ[]deeerMc|jj|jj|nd|S)N )rrdecodejoinrs rKconvert_tokens_to_stringz0PreTrainedTokenizerFast.convert_tokens_to_strings@%-9  " * 1 1& 9 9 9&!! rM token_idsclean_up_tokenization_spacesc |dd|_t|tr|g}|j||}||n|j}|r||}|S|S)Nuse_source_tokenizerF)r)r_rjr|rrhr-r1clean_up_tokenization)rr0rr1rr clean_texts rK_decodezPreTrainedTokenizerFast._decodes-3JJ7Mu,U,U) i % % $" I%%iEX%YY,7 ) (2 % ( 33D99J KrMsave_directory file_names. legacy_formatfilename_prefixct|}j|durtd|dus|duojduoj}|dup|du}|rtj||r|dzndtz}fdj D}|rZt|dd 5} tj |d dd d z} | | dddn #1swxYwY||} || z|fz}|rOtj||r|dzndtz} j| || fz}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-c2i|]\}}|jk||SrD)r)rItokrOrs rKrz.s.vvv*#u]bfjfu]u]u3]u]u]urMwzutf-8)r)indent sort_keys ensure_ascii )r:)rYr)rarrrr.ADDED_TOKENS_FILErxrvopenrdumpswritesave_vocabularyTOKENIZER_FILErsave) rr7r8r9r: save_slow save_fastadded_tokens_file added_vocabfout_str vocab_filesr%s ` rK_save_pretrainedz(PreTrainedTokenizerFast._save_pretraineds^,,  $ ,$1F1F`  d " ;mt&; -)5 -,  "T)C]e-C  I " /!Q3!6!6rUf f!! wvvv8Q8W8W8Y8YvvvK %+S7CCC%q"jQ$]bcccfjjGGGG$$$%%%%%%%%%%%%%%%..~.__K#k15F4HHJ  8W\\/!Q3!6!6rUc cN  " ' ' 7 7 7#~&77Js51C22C69C6c p tj|j}|d}|d} d} |dddkri|dd<g|dd<n|ddd kr^|dd O|dd } |dd| d }  | vr| } d |dd <| d gg|dd<n;|ddd vr i|dd<nt d|ddd7d|dvr-|ddvr|dd|dd<t jtj|g} |D]} | dd}| dd}|ddd kr|sC| dvr| d| d<| td)i| || ||dddkr#d|vr|dd|dd|d<|dddkr#d|vr|dd|dd|d<|ddd kr| | |d<|dt|dddksA|dddkrPd|dvrFtd|ddDr!tj|d<t |dd}|d)|| d|}|||| ,tj}d| vr| dD]}| d|d}fd |D}|| d|d<|D](}|}|t d!)fd"|D| d|d#<d$D]L}|| vrF| |\}} |vr|}|}|t d!||g| |<M| |d<t jtj||j}t*j}|d%|D]}t1||t1||} |vr|}|j|d}t7|tr-t||j|j|j|jd&'||<|||<|j }|| |tC|d kr||d%<|j"d)d(i|S)*uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `list[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelr^r!rmergesr"unk_idrg)r#r$z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenr]idrcontinuing_subword_prefixend_of_word_suffixr ByteLevelSequence pretokenizersc3.K|]}|ddkVdS)r^r_NrD)rI pretokenizers rK zBPreTrainedTokenizerFast.train_new_from_iterator..Ps@$!(K7rMinitial_alphabet)rr)rAtrainerrrc<g|]}||SrDr)rIrJspecial_tokens_maps rKrQzCPreTrainedTokenizerFast.train_new_from_iterator..bs*![![![5"4"8"8"F"F![![![rMzQAttempted to set a token in the post processor that does not exist in the mappingc:g|]}|SrD)r)rIrJr3s rKrQzCPreTrainedTokenizerFast.train_new_from_iterator..ks)CuCuCuejIDYDYZ_D`D`CuCuCurMr)clssepr5T) single_wordlstriprstrip normalizedr]r+rD)#rrrhto_strr_rardfrom_strrHr}rextendanyrr_alphabetMODEL_TO_TRAINER_MAPPINGtrain_from_iteratorrrirbrSPECIAL_TOKENS_ATTRIBUTESremover_special_tokens_mapr`r|rlrmrnror5rgr)r text_iteratorrrAnew_special_tokensrhrtokenizer_jsonrVrWr[rZr added_tokenr]r trainer_classrftrained_tokenizer_jsonrWrrJtoken_id special_tokenspecial_tokens_listspecial_token_fullr5r3s ` @rKtrain_new_from_iteratorz/PreTrainedTokenizerFast.train_new_from_iterators;DDO$:$:$<$<==%)).99 '++,<== ' "6 *e 3 3/1N7 #G ,02N7 #H - - G $V , 9 9g&x0<'0:*73G4D3Ew'0 G $V ,0J J J/1N7 #G , ,>n]dNeflNm>>>   *~g666w' 48JJJ3EnU\F]^iFj3kN7 #K 0!*4:n+E+EFF ' = =K!ooi66Gd++Ag&v.);;G;!-+i2HL^2^2^);K 11)*:; v vC+,<=cB8LF)5![![![![TZ![![![FLN#34S9(C!'#,#8#8#?#?#+", s##, DvCuCuCuntCuCuCuN#34S9%@@!/ F F  N22-m???( 2 2EtU##/ 'e 4 4 %1mGY6Y6Y$6}$EM%)%=%A%A%%N%N"0*== 2$.%$6$B1818#5#@ $ %%%F5MM%2F5M$($B!  ) % , ,-? @ @ @ ( ) )A - -2KF. /t~CCyCFCCCrM)NNFFFFT)F)NF)FN)NN)NNN)A__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESrr)rr^r__annotations__rrpropertyboolrrrrdictrYrrrxrr/rrrrdr DecoderFastr EncodingFastrrrwrrrrrrrrrrrrr rrrrrrrr%r*r/r6rPathLikerTr __classcell__)rs@rKr(r(Qs  *@D(4(;#<=DDDzzzzzxX    X GCGGGXG A4S>AAAA tCH~   X nd38nnnnXn:d3 ?&;:::X:nc3hnnnn$ FFFFF =X ''''X'1504*/+0',#-(-(-( (~-( (~ -( $( -( %) -(!%-(-(-( tCH~tL11 2-(-(-(-(^UE#x}2D,EU%PSUYZ]U^P^J_UUUU  7#7(3-777766d5j+A&B6]`6666 ??d?s????,GLd3i(?C sDI~ 8uuSu uRVumqrumvuuuuI9)I90I9 I9  I9 %SM I9smI9I9I9I9`$(,;,F2D2T$($),0&*(,0404*/+0',#%*+Y`Y`"' OT-0$7H2I4PeKf f# Y` ! Y` * Y`0Y`SMY`Y`"Y`%SMY`smY`! Y` (~Y` (~Y` $(!Y`"%)#Y`$!%%Y`&'Y`()Y`*#+Y`, -Y`Y`Y`Y`|DH#',;,F2D2T$($),0&*)-0404*/+0',#%*);;I001;E)->">?@;! ; * ; 0 ;SM;;";%SM;sm;!; (~; (~;$(; %)!;"!%#;$%;&';(#);, -;;;;z tCy S    %*7; d3i("'/tn  8)-)- //c2;.//#s(O/ ~ / "# / sCx ////j rDrDrDrDrDrDrDrDrMr()=rrbrr collectionsrcollections.abcrtypingrrrtokenizers.pre_tokenizerspre_tokenizersr tokenizersrrr rdtokenizers.decodersr rtokenizers.trainersr r r rrintegrations.ggmlrmodeling_gguf_pytorch_utilsrtokenization_utilsrtokenization_utils_baserrrrrrrrrrutilsrrr get_loggerrloggerrKSPECIAL_TOKENS_MAP_FILETOKENIZER_CONFIG_FILETIKTOKEN_VOCAB_FILErFrurr(rDrMrKrs  ######$$$$$$''''''''''777777//////111111666666^^^^^^^^^^^^::::::555555======333333                        @?????????  H % %"3/'( !! (6EXYY,--H DH DH DH DH D5H DH D.-H DH DH DrM