*`iY ddlZddlZddlZddlZddlmZddlmZddlm Z ddl m Z ddl Z ddlZddlmZmZddlmZmZmZmZddlmZejd ed ejeZd ee zd efdZ Gdde Z!Gdde Z"Gdde Z#Gdde Z$GddeZ% dde&e!de'dzd e(e)e'ffdZ*dS)N)cached_property)groupby)Path) TypedDict) AudioConfigAudioSpectrogramConfig)SpecialTokenPolicy SpecialTokens TokenizerTokenizerVersion) ImageConfigoncez%.*`get_control_token` is deprecated.*)actioncategorymessagepathreturnct|trt|}|od|jvo |jdkS)z3Check if the given path is a tekken tokenizer file.tekken.json) isinstancestrris_filenamesuffix)rs {/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/tekken.py is_tekkenr sE$Dzz <<>> Nh$)3 N w8NNc8eZdZUdZeed<eed<edzed<dS) TokenInfozToken information in the JSON file. Attributes: rank: The rank of the token. token_bytes: The token in bytes, base64 encoded. token_str: The token in string format. rank token_bytesN token_str)__name__ __module__ __qualname____doc__int__annotations__rrrr r (sC IIITzrr c2eZdZUdZeed<eed<eed<dS)SpecialTokenInfozSpecial token information in the JSON file. Attributes: rank: The rank of the token. token_str: The token in string format. is_control: Whether the token is a control token. r!r# is_controlN)r$r%r&r'r(r)rboolr*rrr,r,6s< IIINNNrr,cFeZdZUdZeed<eed<eed<eed<eed<dS) TekkenConfigaXTekken configuration in the JSON file. Attributes: pattern: The pattern of the tokenizer. num_vocab_tokens: The number of vocabulary tokens. default_vocab_size: The default vocabulary size. default_num_special_tokens: The default number of special tokens. version: The version of the tokenizer. patternnum_vocab_tokensdefault_vocab_sizedefault_num_special_tokensversionN)r$r%r&r'rr)r(r*rrr0r0DsRLLL #### LLLLLrr0cxeZdZUdZeeed<eedzed<eed<e ed<e ed<e ed<e ed <dS) ModelDataa2The data of the tekken tokenizer model. Attributes: vocab: The vocabulary of the tokenizer. config: The configuration of the tokenizer. version: The version of the tokenizer. type: The type of the tokenizer. image: The image configuration of the tokenizer. vocabNspecial_tokensconfigr5typeimageaudio) r$r%r&r'listr r)r,r0r(rr rr*rrr7r7Vs{  ?)*T1111  LLL III  rr7ceZdZdZedejdedejdedejdedej dedej ded ej ded ej ded ej ded ejded ejdedejdedejdedejdedejdedejdedejdedejdedejdedejdedejdfZdZddddddeedeededed ed!e d"ed#ee!zdzd$e"dzd%e#dzfd&Z$e%d'e!fd(Z&e'd)e(dd*ee!zd'dfd+Z)e%d'e"dzfd,Z*e*j+d-e"d'dfd.Z*e%d'e#dzfd/Z,e,j+d-e#d'dfd0Z,e%d'efd1Z-e%d'efd2Z.e%d'e fd3Z/e%d'e0fd4Z1e1j+d5e0d'dfd6Z1e2d'efd7Z3e2d'efd8Z4e2d'efd9Z5e2d'efd:Z6d'eefd;Z7de8d'eefd?Z9d@eedAe0d'eefdBZ:dCed'e8fdDZ;dzezd'e8fdGZ?d tekkenizerN)r_path image_config audio_configr8r9r1 vocab_sizenum_special_tokensr5rrVrWrXc|t||zksJ|t||f|_ttd|D} t|| ks Jd|t||ksJfdtt||D} | r8td| ddd| dd|| z}ttd |Dt|cxkr|ks nJ|||z } td | d |d t || _tt| tjksJ| jftj ||ji_ |_ | _ | _|_d|D_d|D_fdt|D_t&j_|t-|nd_dS)aInitialize the tekken tokenizer. Args: vocab: The vocabulary of the tokenizer. special_tokens: The special tokens of the tokenizer. pattern: The pattern of the tokenizer. vocab_size: The vocabulary size of the tokenizer. num_special_tokens: The number of special tokens of the tokenizer. version: The version of the tokenizer. name: The name of the tokenizer. image_config: The image configuration of the tokenizer. cg|] }|d Sr#r*.0ts r z'Tekkenizer.__init__..s-U-U-Ua n-U-U-UrzSpecial tokens must be unique: cfg|]-}t|j|d.S))idTrA)r,SPECIAL_TOKEN_TEMPLATEformatr_iselfs rraz'Tekkenizer.__init__..sN    !t/J/Q/QUV/Q/W/Wdh i i i   rzAdding special tokens rr#z, ..., cg|] }|d Sr]r*r^s rraz'Tekkenizer.__init__..s???1+???rzNon special vocabulary size is z with z special tokens.) max_vocab)rpat_strmergeable_ranksr9ch|] }|d S)r!r*r^s r z&Tekkenizer.__init__..s"E"E"E1V9"E"E"Erc,i|]}|d|dS)r#r!r*r^s r z'Tekkenizer.__init__..s"-`-`-`Aa nai-`-`-`rc:g|]}|Sr*) id_to_piecerfs rraz'Tekkenizer.__init__..s'FFFqt''**FFFrN)len _vocab_sizesetrangeloggerinfo_reload_mergeable_ranks_tekken_token2id_nospecialvaluestiktokenEncoding_model_version _image_config _audio_config_all_special_tokens_special_token_ids_special_tokens_reverse_vocab_vocabr IGNORE_special_token_policyr _file_path)rhr8r9r1rYrZr5rrVrWrXnum_defined_special_tokensspecial_fillerinner_vocab_sizes` r__init__zTekkenizer.__init__s4SZZ*<<<<<  JJ ? <<< &&)-U-Un-U-U-U)V)V%W%W">""&@@@@BtdrBtBt@@@>""&88888    3~..0BCC      KKq):;)GqqP^_aPbcnPoqq   (.83?????@@AASEXEXnnnn\nnnnnn onn&(::  r6FrrN`rrrsss*A%Sc*d*d*d'5)**++s43R3Y3Y3[3[/\/\\\\   +_ \\\' ;       ))#1 "E"En"E"E"E-`-`Q_-`-`-`*FFFFE*4E4EFFF %7%>").):$u+++rrc<|jtd|jS)zThe path to the tokenizer file.Nz)The tokenizer was not loaded from a file.)r ValueErrorrhs r file_pathzTekkenizer.file_paths# ? "HII Irclsrc 2t|trt|}|s J|t |dd5}t j|}dddn #1swxYwY|dd}|tj vr/td|d|d ttj |Jt|}|d d}|@|td krtd |d ttj }n d|D}||d <|dx}r>|tdkrtd|d|jdtdi||d<n&|dx} rtdi| |d<|dx} r2| d} t#di| } t%dd| i| |d<|} || d|| dd| dd| dd||jdd| d| d| S) zLoad the tekken tokenizer from a file. Args: path: The path to the tokenizer file. Returns: The tekken tokenizer. rutf-8)encodingNr:r5zUnknown version: z in z+. Make sure to use a valid version string: r9v7zSpecial tokens not found in zL. Please update your tokenizer file and include all special tokens you need.cg|]}|Sr*r*)r_tokens rraz(Tekkenizer.from_file..sFFFeFFFr multimodalv11z-The image config has to be called 'image' in z for tokenizers of version .r<r=audio_encoding_configencoding_configr8r1r3r4r) r8r9r1rYrZr5rrWrXrVr*)rrrexistsopenjsonloadgetr __members__rr>r@DEPRECATED_SPECIAL_TOKENSvaluer poprrrreplace) rrfuntyped _version_strr5special_tokens_dictsr9mmr<r=r model_datas r from_filezTekkenizer.from_files` dC  ::D{{}}""d""} $g . . . #!illG # # # # # # # # # # # # # # #x(,,Y77 /; ; ;aLaadaa<@AQA]<^<^aa  '''"<00>EkkJZ\`>a>a  ')$//// a4aaa "&j&J!K!KFF1EFFFN$2 !\** *2 4)%0000 uDuueleruuu +00R00GG  kk'** *U 4*33U33GG KK(( (5 U#ii(?@@O4GGGGO*TT?TeTTGG  ' sW%)x(3!(+,@A)(34PQ""7B//#00#00    sA66A:=A:c|jS)z)The image configuration of the tokenizer.)rrs rr<zTekkenizer.image+s !!rrc td)Nz!Can only set Image config at initrrhrs rr<zTekkenizer.image0<===rc|jS)zThe audio configuration of the tokenizer. Returns: The audio configuration object if it exists, otherwise None. )rrs rr=zTekkenizer.audio4s !!rc td)Nz!Can only set Audio config at initrrs rr=zTekkenizer.audio=rrc*t|jS)z.The number of special tokens of the tokenizer.)rtrrs rrZzTekkenizer.num_special_tokensAs4+,,,rc|jS)z!Vocabulary size of the tokenizer.)rurs rn_wordszTekkenizer.n_wordsFs rc|jS)zThe version of the tokenizer.)rrs rr5zTekkenizer.versionKs }rc|jS)z'The policy for handling special tokens.)rrs rspecial_token_policyzTekkenizer.special_token_policyPs ))rpolicyct|ts tdt|dt jdt ||_dS)z+Set the policy for handling special tokens.z!Expected SpecialTokenPolicy, got rzThe attributed `special_token_policy` is deprecated and will be removed in 1.10.0. Please pass a special token policy explicitly to the relevant methods.N)rr rr;warningswarn FutureWarningr)rhrs rrzTekkenizer.special_token_policyUsg&"455 RPfPPPQQ Q Y     &,"""rc,|dS)z#The beginning of sentence token id.zget_special_tokenrs rbos_idzTekkenizer.bos_ides%%e,,,rc,|dS)zThe end of sentence token id.zrrs reos_idzTekkenizer.eos_idjs%%f---rc,|dS)zThe padding token id.zrrs rpad_idzTekkenizer.pad_ido%%g...rc,|dS)zThe unknown token id.zrrs runk_idzTekkenizer.unk_idtrrc|jS)a2All tokens in the vocabulary as strings. Note: This will collapse all tokens for which we have a decoding error into the string. This is bad and results in things like len(set(vocab)) != len(vocab)). Returns: The vocabulary of the tokenizer. )rrs rr8zTekkenizer.vocabys {rsboseoscj|}fd|D}|r jg|}|r g|j}|S)aEncode a string into a list of token ids. Args: s: The string to encode. bos: Whether to add the beginning of sentence token. eos: Whether to add the end of sentence token. Returns: The list of token ids. c$g|] }|jz Sr*rZr_r`rhs rraz%Tekkenizer.encode..s!>>>!!d-->>>r)rencoderr)rhrrrtokenss` rrzTekkenizer.encodesh!K..q11>>>>v>>>  ,k+F+F  ,+v+t{+F rrrcg}t|fdD]\}}|rt|tjkr tdt |d|tjkr"|fd|Di|tjkrz{|j fd|D|S)Nc|jkSNr)r`rhs rz(Tekkenizer._decode_all..s1t?V;Vrz/Decoding `tokens` that contain special tokens (a) is not allowed. Either make sure `tokens` do not include any special tokens or, if you want to decode `tokens` that includes special tokens, change the tokenizer's special token policy to IGNORE or KEEP: ``` from mistral_common.tokens.tokenizers.mistral import MistralTokenizer from mistral_common.tokens.tokenizers.tekken import SpecialTokenPolicy tokenizer = MistralTokenizer.v3(is_tekken=True) tekken = tokenizer.instruct_tokenizer.tokenizer tekken.special_token_policy = SpecialTokenPolicy.IGNORE # or SpecialTokenPolicy.KEEP ```c3>K|]}j|dVdS)r#N)rrs r z)Tekkenizer._decode_all..s1"["[PQ4#;A#>{#K"["["["["["[rc$g|] }|jz  Sr*rrs rraz*Tekkenizer._decode_all..s"2^2^2^ST1t7N3N2^2^2^r) rr RAISErr>KEEPextendrappendrdecode)rhrrdecoded is_specialgroups` r _decode_allzTekkenizer._decode_alls!(1V1V1V1V!W!W a a J a'+=+CCC$  $u++       *-?-DDDNN"["["["[UZ"["["[[[[[)-?-FFFG t{112^2^2^2^X]2^2^2^__````rtoken_idc2d||jz cxkodkncS)z$Check if a token id is a byte token.rrrhrs ris_bytezTekkenizer.is_bytes,Ht66<<<<<<<<> >NU 8LNNOO Orc`tjdt||S)NzC`get_control_token` is deprecated. Use `get_special_token` instead.)rrrrrs rget_control_tokenzTekkenizer.get_control_tokens* []jkkk%%a(((rc|5t|ts tdt|d|*t jd|jdt|j}d| ||S)aDecode a list of token ids into a string. Args: tokens: The list of token ids to decode. special_token_policy: The policy for handling special tokens. Use the tokenizer's [attribute][mistral_common.tokens.tokenizers.tekken.Tekkenizer.special_token_policy] if `None`. Passing `None` is deprecated and will be changed to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. NzFExpected `special_token_policy` to be None or SpecialTokenPolicy, got r,Using the tokenizer's special token policy () is deprecated. It will be removed in 1.10.0. Please pass a special token policy explicitly. Future default will be SpecialTokenPolicy.IGNORE.rr) rr rr;rrrrjoinr)rhrrs rrzTekkenizer.decodes +J?SUg4h4h +vY]^rYsYsvvv  ' MH4C]HHH    $(#= wwt''EY'ZZ[[[rc`tjdt||S)z[DEPRECATED] Converts a list of token ids into a string, keeping special tokens. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead. This is a convenient method for debugging. z`to_string` is deprecated and will be removed in 1.10.0. Use `decode` with `special_token_policy=SpecialTokenPolicy.KEEP` instead.)rrr _to_stringrhrs r to_stringzTekkenizer.to_strings7  \     v&&&rcD||tjS)Nrrr rrs rrzTekkenizer._to_strings{{68J8O{PPPrcF||gtjS)z0Convert a token id to its string representation.rrrs rrszTekkenizer.id_to_pieces{{H:>>/9+FMMgVVV%);)AAA H!A!A!ABBB%);)BBBs !WAU!W!WXXX{44X@W5WXXXrr)Gr$r%r&r'r,r unkrr begin_instend_inst begin_tools end_toolsbegin_tool_resultsend_tool_results tool_callsimgpad img_breakimg_endprefixmiddler begin_system end_systembegin_tool_contentrrdr>r rr(r rr rrpropertyr classmethodr;rr<setterr=rZrr5r rrrrrrr8r.rrrrrrrrrrrrsbytesrr*rrr@r@js a=+<NNNa=+<NNNa=+<NNNa=+CPTUUUa=+AdSSSa=+DQUVVVa=+BtTTTa=+KX\]]]a=+IVZ[[[a=+CPTUUUbM,=$OOObM,=$OOObM,CPTUUUbM,AdSSSbM,@TRRRbM,@TRRRbM,@TRRRbM,FSWXXXbM,DQUVVVbM,LY]^^^)!..!#'+/+/NENENEINE-.NE NE  NE  NE"NENETzD NE"D(NE"D(NENENENE`4X E tL)E tE E E E [E N"{T)"""X" \>;>4>>>\>"{T)"""X" \>;>4>>>\>-C---X-    X )X*&8***X*  ,+= ,$ , , ,!  ,----_-...._.////_/////_/tCy"$Td3i&$s)CUZ^_bZc8======;3;3;;;;Pbj 03 6P4PPPP)3)3))))\\T#Y\>PSW>W\cf\\\\@'S 'c'''' QcQsQQQQUCUCUUUU#Y#Y#YDVY]D]#Yin#Y#Y#Y#Y#Y#Yrr@r8rkc|tt||ksJt||ft||kr5|d|}tdt|di}t|D]z\}}|hdksJ|d|ksJt j|d}|dks |t|gks J||f|d||<{t|t|ksJt| ttt|ksJ|S)zAReload our tokenizer JSON file and convert it to Tiktoken format.Nz(Cutting non special vocabulary to first z tokens.>r!r#r"r!r"r) rtrxry enumeratekeysbase64 b64decoderrvr|rw)r8rkranksrgxmerges rrzrz.sv 5zzY&&&UY(?&&& u:: ! !*9*%E KKW3u::WWW X X X!E%  !!1vvxx???????yA~~~~ =!122Cxx5E1#JJ...E ...ye  u::U # # # # u||~~  #eCJJ&7&7"8"8 8 8 8 8 Lrr)+rrloggingr functoolsr itertoolsrpathlibrtypingrnumpyrr}&mistral_common.tokens.tokenizers.audiorr%mistral_common.tokens.tokenizers.baser r r r &mistral_common.tokens.tokenizers.imager filterwarningsr getLoggerr$rxrr.rr r,r0r7r@r>r(dictrrzr*rrr-s %%%%%%VVVVVVVV ?>>>>>   4  8 $ $OC$JO4OOOO             y   9$ (AYAYAYAYAYAYAYAYL!  ?Tz %*r