`iJPdZddlZddlZddlZddlmZddlmZddlZ ddl m Z m Z m Z mZmZddlmZejeZdd d Zed Zd Zd ZdZGddZGdde ZdgZdS)zTokenization classes for CLIP.N) lru_cache)Optional) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filec \tttdtddztttdtddzztttdtddzz}|dd}d }td D]8}||vr2|||d |z|dz }9d |D}t t ||S) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNrc,g|]}t|S)chr).0ns ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/clip/tokenization_clip.py z$bytes_to_unicode..:s   Q#a&&   )listrangeordappenddictzip)bscsrbs rbytes_to_unicoder(%s U3s88SXX\ * *++d5TCIIPQM3R3R.S.SSVZ[`adeiajajloptluluxyly[z[zV{V{{ AAAB A 4[[ B;; IIaLLL IIdQh    FA  "   B B  rc~t}|d}|ddD]}|||f|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charchars r get_pairsr0>sP EEEQIQRR 9d#$$$ LrcZtjdd|}|}|S)Nz\s+ )resubstrip)texts rwhitespace_cleanr7Ls' 6&#t $ $D ::<>>SWSc %%  & 6//55D"-"7t"D"D)*ABB    M MEK''%;!KKMME)66 $ 7 7 > >'; 33E::E    7 7{ K K L L L L+CHH\,B,BCC rctjd|}g}|D]2}tj|}|dkr||3d|S)z$Strips accents from a piece of text.NFDMn)rLrMcategoryr"rR)rEr6outputr/cats rrOz!BasicTokenizer._run_strip_accentssi$UD11  D&t,,Cd{{ MM$    wwvrc|jr|||vr|gSt|}d}d}g}|t|kr|||}t|r||gd}n4|r|gd}|d||dz }|t|k|d|DS)z&Splits punctuation on a piece of text.NrTFrc8g|]}d|S)r\)rR)rxs rrz5BasicTokenizer._run_split_on_punc..s"+++q +++r)rDrlenr r")rEr6rAcharsistart_new_wordr^r/s rrQz!BasicTokenizer._run_split_on_puncs$ )@T[EXEX6MT  #e**nn8Dt$$ ( tf%%%!%!&MM"%%%!&r !!$''' FA#e**nn,+F++++rc,g}|D]{}t|}||r@|d|||df|||d|S)z)Adds whitespace around any CJK character.r2r\)r!_is_chinese_charr"rRrEr6r^r/cps rrKz&BasicTokenizer._tokenize_chinese_charss $ $DTB$$R(( $ c""" d### c"""" d####wwvrc|dkr|dksT|dkr|dksH|dkr|dks<|dkr|dks0|d kr|d ks$|d kr|d ks|d kr|dks |dkr|dkrdSdS)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTFr)rErks rrizBasicTokenizer._is_chinese_chars6\\bFllf vg "--g "--g "--g "--f vg "--4urcg}|D]g}t|}|dks|dkst|r-t|r|dR||hd|S)zBPerforms invalid character removal and whitespace cleanup on text.rir2r\)r!rr r"rRrjs rrJzBasicTokenizer._clean_texts $ $DTBQww",,+d*;*;,d## $ c"""" d####wwvr)TNTNTr?) __name__ __module__ __qualname____doc__rFrXrOrQrKrirJrrrr=r=]s0# 1111 $$$$L   ,,,,,   0     rr=c eZdZdZeZddgZ dfd ZedZ d Z dd e e d e e e d e e fdZ dd e e d e e e ded e e ffd Z dd e e d e e e d e e fdZdZdZdZdZdZddede ed eefdZxZS) CLIPTokenizera Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskreplace <|endoftext|><|startoftext|>c t|trt|ddn|}t|trt|ddn|}t|trt|ddn|} ddl} | j|_nG#t $r:t dtdd|_ d|_YnwxYwt|d5} tj | |_ dddn #1swxYwYd|j D|_||_t#|_d |jD|_t|d5} | d d d } dddn #1swxYwYd | D} t/t1| t3t5| |_ddd|_t;jdt:j|_ tCj"d|||||d|dS)NF)lstriprstriprzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.)rCrDutf-8encodingci|]\}}|| Srrrkvs r z*CLIPTokenizer.__init__..4s>>>A1>>>rci|]\}}|| Srrrs rrz*CLIPTokenizer.__init__..7sHHHdaQHHHr ricPg|]#}t|$Sr)tupler9)rmerges rrz*CLIPTokenizer.__init__..:s(CCCueEKKMM**CCCrrxrw)rxrwz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)errors unk_token bos_token eos_token pad_tokenr)# isinstancestrrftfyfix_text ImportErrorloggerinfor=nlpopenjsonloadencoderitemsdecoderrr( byte_encoder byte_decoderreadr5r9r#r$r rd bpe_rankscacher3compile IGNORECASEpatsuperrF)rEr r rrrrrkwargsr vocab_handle merges_handle bpe_merges __class__s rrFzCLIPTokenizer.__init__sJTT]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir IST]_bIcIcrJyuEEEEir  ! KKK MDMM ! ! ! KKe f f f%EERRRDH DMMM ! *w / / / 3<9\22DL 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3>>););)=)=>>>  ,..HHd.?.E.E.G.GHHH + 0 0 0 [M&++--3355;;DAA!FYBYZJ [ [ [ [ [ [ [ [ [ [ [ [ [ [ [CC CCC c*eC OO.D.DEEFF):_]] : n M             s8>BACC'D  DDAG!!G%(G%c*t|jSr?)rdrrEs r vocab_sizezCLIPTokenizer.vocab_sizeLs4<   rc0t|jfi|jSr?)r#radded_tokens_encoderrs r get_vocabzCLIPTokenizer.get_vocabPsDL>>D$=>>>rN token_ids_0 token_ids_1returncX|jg}|jg}|||z|zS||z|z|z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) bos_token_id eos_token_idrErrrrs r build_inputs_with_special_tokensz.CLIPTokenizer.build_inputs_with_special_tokensSsO(&' &'  {*Y6 6;&2Y>LyXXrFalready_has_special_tokensc|r$t||dS|dgdgt|zzdgzSdgdgt|zzdgzdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrNrr)rget_special_tokens_maskrd)rErrrrs rrz%CLIPTokenizer.get_special_tokens_maskns& & 7722'[]a3   31#K 0 001QC7 7sqcC ,,,-3qc9aS3{CSCS=STXYWZZZrc|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. Nr)rrrdrs r$create_token_type_ids_from_sequencesz2CLIPTokenizer.create_token_type_ids_from_sequencessp &' &'  y;.:;;qcA A9{*Y6B[PS\\]]ab`cccrc|jvr j|St|dd|ddzfz}t|}|s|dzS t|fd}|jvrn8|\}}g}d}|t |kr |||} |||| | }n-#t$r |||dYnwxYw|||krC|t |dz kr-||dz|kr| ||z|dz }n | |||dz }|t |kt|}|}t |dkrnt|}Wd |}|j|<|S) NraTcTj|tdS)Ninf)rgetfloat)pairrEs rz#CLIPTokenizer.bpe..s 1C1CD%PU,,1W1Wrkeyrrr2) rrr0minrrdindexrP ValueErrorr"rR) rErVr,r-bigramfirstsecondnew_wordrfjs ` rbpezCLIPTokenizer.bpes DJ  :e$ $U3B3Z  E"I$6#88$ "6> ! ($W$W$W$WXXXFT^++"ME6HAc$ii-- 5!,,A OOD1I...AA "OODH---E7e##CIIM(9(9d1q5kV>S>SOOEFN333FAAOODG,,,FAc$ii-- XHD4yyA~~!$9 (:xx~~  5 sC'C/.C/c g}j.dj|}n4t |}t jj|D]{}dfd| dD}| d | dD||S)zTokenize a string.Nr2r\c32K|]}j|VdSr?)r)rr'rEs r z*CLIPTokenizer._tokenize..s<)*!!$rr|c3K|]}|VdSr?r)r bpe_tokens rrz*CLIPTokenizer._tokenize..s"TTIiTTTTTTr) rrRrrXr7rNr3findallrencoderPrr9)rEr6 bpe_tokensrVs` r _tokenizezCLIPTokenizer._tokenizes = 88DH--d3344DD#DMM$$7$788>>@@DZ$// U UEGG.3ll7.C.CE   TT%9N9Ns9S9STTT T T T Trcr|j||j|jS)z0Converts a token (str) in an id using the vocab.)rrr)rErVs r_convert_token_to_idz"CLIPTokenizer._convert_token_to_ids,|t|'7'7'G'GHHHrc6|j|S)z=Converts an index (integer) in a token (str) using the vocab.)rr)rErs r_convert_id_to_tokenz"CLIPTokenizer._convert_id_to_tokens|&&&rcd|}tfd|D}|djdd}|S)z:Converts a sequence of tokens (string) in a single string.r\c*g|]}j|Sr)r)rcrEs rrz:CLIPTokenizer.convert_tokens_to_string..s!CCC 1! 4CCCrr|)rrr2)rR bytearraydecoderrvr5)rEr:r6 byte_arrays` rconvert_tokens_to_stringz&CLIPTokenizer.convert_tokens_to_stringsowwvCCCCdCCCDD    ==EEfcRRXXZZ rsave_directoryfilename_prefixc tj|s td|ddStj||r|dzndt dz}tj||r|dzndt dz}t|dd 5}|tj |j d d d dzdddn #1swxYwYd}t|dd 5}|dt|j dD][\}} || kr td|d| }|d|dz|dz }\ dddn #1swxYwY||fS)NzVocabulary path (z) should be a directory-r\r r wr|r}rTF)indent sort_keys ensure_asciirrz#version: 0.2 c|dS)Nrr)kvs rrz/CLIPTokenizer.save_vocabulary..s Y[\]Y^rrzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r2r)ospathisdirrerrorrRVOCAB_FILES_NAMESrwriterdumpsrsortedrrwarning) rErrr merge_filefrwriterr token_indexs rsave_vocabularyzCLIPTokenizer.save_vocabularysw}}^,,  LLT^TTT U U U FW\\ oM_s222QbcoQpp  W\\ oM_s222QbcpQqq  *cG 4 4 4 c GGDJt|ATYZZZ]aa b b b c c c c c c c c c c c c c c c *cG 4 4 4  LL* + + ++1$.2F2F2H2HN^N^+_+_+_  ' KK''NNM MMM(E SXXj11D8999                 :%%s%<4C<<DDBGG G)rvrwrxrwrwr?)NF)rnrorprqrvocab_files_namesmodel_input_namesrFpropertyrrrintrrboolrrrrrrrrrr __classcell__)rs@rrsrss'2*$&67 !#!!/ / / / / / b!!X!???JNYY9Y3;DI3FY cYYYY8sx[[9[3;DI3F[ko[ c[[[[[[:JNdd9d3;DI3Fd cdddd.(((T   III'''&&c&HSM&]bcf]g&&&&&&&&rrs)rqrrrL functoolsrtypingrregexr3tokenization_utilsrrrr r utilsr get_loggerrnrrr(r0r7r;r=rs__all__rrrr s{%$ oooooooooooooo  H % %   0   ^^^^^^^^BF&F&F&F&F&'F&F&F&R  r