*`iH ,UddlZddlmZddlmZmZmZddlmZddl m Z ddl m Z m Z mZmZmZddlmZmZddlmZdd lmZmZmZmZmZdd lmZdd lmZm Z m!Z!dd l"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)dd l*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1m2Z2m3Z3m4Z4ddl5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;ddldede:de fdZ?Gddeee eee(fZ@iddde@jAddde@jBd e@jAd!e@jBd"e@jAd#d$d%e@jCd&e@jAd'e@jBd(e@jDd)d*d+e@jBd,e@jBd-d.d/e@jCe@jCe@jCe@jAe@jAe@jBe@jBe@jBe@jBd0d1d2d3 ZEeFeGege@ffeHd4<dS)5N)Path)AnyCallableGeneric)TokenizerException) FIMRequest)UATSAssistantMessageTypeSystemMessageTypeToolMessageTypeUserMessageType)InstructRequestNormalizer normalizer_for_tokenizer_version)ChatCompletionRequest)MistralRequestValidatorMistralRequestValidatorV3MistralRequestValidatorV5MistralRequestValidatorV13ValidationMode)TranscriptionRequest) AudioConfig AudioEncoderSpecialAudioIDs)InstructRequestInstructRequestTypeInstructTokenizerSpecialTokenPolicy SpecialTokens TokenizedTypeTokenizerVersion) ImageConfig ImageEncoderSpecialImageIDs)InstructTokenizerV1InstructTokenizerV2InstructTokenizerV3InstructTokenizerV7InstructTokenizerV11InstructTokenizerV13)SentencePieceTokenizerget_image_configis_sentencepiece) Tekkenizer is_tekken)download_tokenizer_from_hf_hub image_config tokenizerreturnct|tjj|tjj|tjj}t||S)zLoad a image encoder from a config and a tokenizer. Args: image_config: The image config. tokenizer: The tokenizer. Returns: The image encoder. )img img_breakimg_end)r#get_special_tokenrr4valuer5r6r")r0r1 special_idss |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/mistral.pyload_image_encoderr;:so"  ' ' (9(? @ @--m.E.KLL++M,A,GHHK  k 2 22 audio_configcdtdtdzffd }t|tjj|tjj|tjj}t||S)zLoad a audio encoder from a config and a tokenizer. Args: audio_config: The audio config. tokenizer: The tokenizer. Returns: The audio encoder. tokenr2Nc\|sdS|SN) is_specialr7)r?r1s r:get_special_token_or_nonez5load_audio_encoder..get_special_token_or_noneWs2##E** 4**5111r<)audio begin_audio streaming_pad) strintrrrDr8rErFr)r=r1rCr9s ` r:load_audio_encoderrILs22t222222 "'' (;(ABB--m.G.MNN// 0K0QRRK  k 2 22r<cleZdZdZdeeeeefde e ee e fde e ee e effdZdeeeedfffdZedefd Zed'd Zed'd Zed(d ededdfdZed)deddfdZed)dededdfdZeddd d ej fdedeezdzdedzdedededdfdZ!eej fdeezdeddfdZ" d*de#e$de%dzdefd Z&de'defd!Z(dedefd"Z)d*d#e*e%d$e+dzdefd%Z,d#e*e%defd&Z-dS)+MistralTokenizeragMistral tokenizer. This class is a wrapper around a [InstructTokenizer][mistral_common.tokens.tokenizers.base.InstructTokenizer], a [MistralRequestValidator][mistral_common.protocol.instruct.validator.MistralRequestValidator] and a [InstructRequestNormalizer][mistral_common.protocol.instruct.normalize.InstructRequestNormalizer]. It provides a convenient interface to tokenize, validate ad normalize Mistral requests. Attributes: instruct_tokenizer: The instruct tokenizer to use. See [InstructTokenizer][mistral_common.tokens.tokenizers.instruct.InstructTokenizer]. instruct_tokenizer validatorrequest_normalizerc0||_||_||_dS)zInitializes a `MistralTokenizer`. Args: instruct_tokenizer: The instruct tokenizer to use. validator: The request validator to use. request_normalizer: The request normalizer to use. N)"_chat_completion_request_validator_instruct_request_normalizerrL)selfrLrMrNs r:__init__zMistralTokenizer.__init__us%3</,>)  r<r2.cTtj|jjj|jjffS)z Provides a recipe for pickling (serializing) this object, which is necessary for use with multiprocessing. Returns: A tuple of the factory function and the arguments to reconstruct the object from its source file. )rK from_filerLr1 file_pathrP_mode)rRs r: __reduce__zMistralTokenizer.__reduce__s. )  # - 7  3 9,   r<cFttjddz S)Ndata)r__file__parentsclss r: _data_pathzMistralTokenizer._data_pathsH~~%a(611r<c|t|dz tjS)zGet the Mistral tokenizer v1.ztokenizer.model.v1moderUrGr`rtestr^s r:v1zMistralTokenizer.v1s6}}S!1!14H!HIIP^Pc}dddr<c|t|dz tjS)zGet the Mistral tokenizer v2.z*mistral_instruct_tokenizer_240216.model.v2rbrdr^s r:v2zMistralTokenizer.v2s@}}   #OO P PWeWj   r<Fr.is_mmc|r|rd}n|r|sd}n|s|rtdd}|t||z tjS)a;Get the Mistral tokenizer v3. Args: is_tekken: Whether the tokenizer is a tekken tokenizer. See [Tekkenizer][mistral_common.tokens.tokenizers.tekken.Tekkenizer]. is_mm: Whether to load image tokenizer. Returns: The Mistral tokenizer v3. ztekken_240911.jsonztekken_240718.jsonz;Multimodal tokenizer is currently only supported for tekkenz*mistral_instruct_tokenizer_240323.model.v3rb) ValueErrorrUrGr`rre)r_r.ritokenizer_names r:v3zMistralTokenizer.v3s  J J1NN  Ju J1NN Ju JZ[[ [IN}}S!1!1N!BCC.J]}^^^r<c|rC|t|dz tjS|t|dz tjS)zGet the Mistral tokenizer v7. Args: is_mm: Whether to load the image tokenizer. Returns: The Mistral tokenizer v7. z,mistral_instruct_tokenizer_241114.model.v7m1rbz*mistral_instruct_tokenizer_241114.model.v7rd)r_ris r:v7zMistralTokenizer.v7s  ==CNN$$'UUVV]k]p! ==CNN$$'SSTT[i[n! r<modelstrictc|s[tjdttD]'\}}||vr |cS(|tvrt d|t|S)axGet the Mistral tokenizer for a given model. Args: model: The model name. strict: Whether to use strict model name matching. If `False`, the model name is matched as a substring. This is deprecated and will be removed in `mistral_common=1.10.0`. Returns: The Mistral tokenizer for the given model. aCalling `MistralTokenizer.from_model(..., strict=False)` is deprecated as it can lead to incorrect tokenizers. It is strongly recommended to use MistralTokenizer.from_model(..., strict=True)` which will become the default in `mistral_common=1.10.0`.If you are using `mistral_common` for open-sourced model weights, we recommend using `MistralTokenizer.from_file('')` instead.zUnrecognized model: )warningswarn FutureWarningMODEL_NAME_TO_TOKENIZER_CLSitemslowerr)r_rprq model_name tokenizer_clss r: from_modelzMistralTokenizer.from_models + MT     .I-N-N-P-P + +) M..(=??***/ 3 3 3$%CE%C%CDD D*51333r<Nrepo_idr?revisionforce_downloadlocal_files_onlyrccbt|||||}t||S)aODownload the Mistral tokenizer for a given Hugging Face repository ID. See [here](https://huggingface.co/mistralai/models) for a list of our OSS models. Args: repo_id: The Hugging Face repo ID. token: The Hugging Face token to use to download the tokenizer. revision: The revision of the model to use. If `None`, the latest revision will be used. mode: The validation mode to use. force_download: Whether to force the download of the tokenizer. If `True`, the tokenizer will be downloaded even if it is already cached. local_files_only: Whether to only use local files. If `True`, the tokenizer will be downloaded only if it is already cached. Returns: The Mistral tokenizer for the given model. )r|r?r}r~rrb)r/rKrU)r|r?r}r~rrctokenizer_paths r: from_hf_hubzMistralTokenizer.from_hf_hubsB48)-     )).t)DDDr<tokenizer_filenamecjt|r#tj|}|j}|j}nBt |r!t |}t|}d}ntd||t||nd}d}|/t|ts Jdt||}t|j }|j tjkrE| Jd| Jdt!t#|t%||S|j tjkrE| Jd| Jdt!t)|t%||S|j tjkr;| Jdt!t-||t/||S|j tjkr0t!t3||| t5||S|j tjkr0t!t9||| t5||S|j tjkr0t!t=||| t?||Std |) zLoads a tokenizer from a file. Args: tokenizer_filename: The path to the tokenizer file. mode: The validation mode to use. Returns: The loaded tokenizer. NzUnrecognized tokenizer file: z-Audio is only supported for tekken tokenizersz#Tokenizer version needs to be >= v3z#Tokenizer version needs to be >= v7rb)rMrN) image_encoder)r audio_encoderz!Unrecognized tokenizer filename: ) r.r-rUimagerDr,r*r+rr; isinstancerIrversionr rfrKr$rrhr%rmr&rror'rv11r(v13r)r) r_rrcr1r0r=rrrNs r:rUzMistralTokenizer.from_files1" ' ( ( [",-?@@I$?L$?LL 0 1 1 [./ABBI+,>??LLL$%YEW%Y%YZZ ZGSG_*<CCCei   #i44 e e6e e e4.|YGGM=i>OPP   0 3 3 3 ((*O((( ((*O(((##I..1t<<<#5   "2"5 5 5 ((*O((( ((*O(((##I..1t<<<#5   "2"5 5 5 ((*O(((##I]KKK3>>>#5   "2"5 5 5##I]Zghhh3>>>#5   "2"6 6 6#$Ym[hiii3>>>#5   "2"6 6 6#$Ym[hiii4$???#5  !!YEW!Y!YZZZr<requestmax_model_input_lenc|j|}||jrtd|j|}|jr||_|j|S)aDEncodes a chat completion request. Args: request: The chat completion request to encode. max_model_input_len: The maximum length of the input to the model. If `None`, the input will not be truncated. Returns: The encoded chat completion request. NzUencoding a chat completion request with truncation, but no max model len was provided) rPvalidate_requesttruncate_for_context_lengthrrQfrom_chat_completion_requesttruncate_at_max_tokensrLencode_instruct)rRrrvalidated_requestinstruct_requests r:encode_chat_completionz'MistralTokenizer.encode_chat_completionfs!CTTU\]]  &7+N &%g  <YYZkll  . J6I  3&667GHHHr<c6|j|S)zEncodes a transcription request. Args: request: The transcription request to encode. Returns: The encoded transcription request. )rLencode_transcriptionrRrs r:rz%MistralTokenizer.encode_transcriptions&;;GDDDr<c6|j|S)zEncodes a fill in the middle request. Args: request: The fill in the middle request to encode. Returns: The encoded fill in the middle request. )rL encode_fimrs r:rzMistralTokenizer.encode_fims&11':::r<tokensspecial_token_policyc:|j||S)a_Decodes a list of tokens into a string. Args: tokens: The tokens to decode. special_token_policy: The policy to use for special tokens. Passing `None` is deprecated and will be changed to `SpecialTokenPolicy.IGNORE` in `mistral_common=1.10.0`. Returns: The decoded string. )r)rLdecode)rRrrs r:rzMistralTokenizer.decodes!&--fK_-```r<c6|j|SrA)rL _to_string)rRrs r:rzMistralTokenizer._to_strings&11&999r<)r2rK)FF)FrA).__name__ __module__ __qualname____doc__rrrrr rr r r rrrStuplerrrX classmethodrr`rfrhboolrmrorGr{ staticmethodrrerrUrr rHrrrrlistrrrr<r:rKrKes   -oz=Zn.no +? E,@ E] E E E E ;* ; ; ; ; ; a aT#Y a>PSW>W acf a a a a:c:s::::::r<rKzministral-8b-2410c8tdSNT)r.rKrmrr<r:rs!1!4!4t!4!D!Dr<zmistral-tiny-2312zopen-mistral-nemo-2407c8tdSrrrr<r:rrs&6&9&9D&9&I&Ir<zmistral-tiny-2407zmistral-small-2312zopen-mixtral-8x22b-2404zmistral-small-2402zmistral-small-2409c8tdSrrrr<r:rrs"2"5"5"5"E"Er<zmistral-medium-2312zmistral-large-2402zmistral-large-2407zmistral-large-2411zpixtral-large-2411c8tdSNT)rirKrorr<r:rrs"2"5"5D"5"A"Ar<zcodestral-2405zcodestral-mamba-2407zpixtral-12b-2409c:tddSNT)r.rirrr<r:rrs 0 3 3d$ 3 O Or<zopen-mistral-7bc8tdSrrrr<r:rrs,//$/??r<c:tddSrrrr<r:rrs'**T*FFr<c8tdSrrrr<r:rrs-00t0<<r<) zopen-mixtral-8x7bz mistral-embedzmistral-small-v1zmistral-large-v1z mistral-smallz mistral-largezopen-mixtral-8x22bz codestral-22bz mistral-nemopixtralz pixtral-largerv)Irspathlibrtypingrrrmistral_common.exceptionsr#mistral_common.protocol.fim.requestr)mistral_common.protocol.instruct.messagesr r r r r *mistral_common.protocol.instruct.normalizerr(mistral_common.protocol.instruct.requestr*mistral_common.protocol.instruct.validatorrrrrr-mistral_common.protocol.transcription.requestr&mistral_common.tokens.tokenizers.audiorrr%mistral_common.tokens.tokenizers.baserrrrrrr &mistral_common.tokens.tokenizers.imager!r"r#)mistral_common.tokens.tokenizers.instructr$r%r&r'r(r).mistral_common.tokens.tokenizers.sentencepiecer*r+r,'mistral_common.tokens.tokenizers.tekkenr-r.&mistral_common.tokens.tokenizers.utilsr/r;rIrKrhrmrfrorvdictrG__annotations__rr<r:rs))))))))));:::::srrrrrrrJJJJJJONNNNN]]]]]]]]]]  JIIIIIIIQQQQQQ3[3ZJ`=`3eq3333$3[3Z3L33332D:D:D:D:D: O1?DUWd deD:D:D:N JDDJ),JIIJ), J *- J /2 J*-JEEJ+.J*-J*-J*-JAAJ&)J,/J OO!J$'*%J&*,%((+(+%(%(*-%(??FF<<;JJJT#x4D0D'E"EFr<