*`i :ddlZddlZddlmZddlmZddlmZddlZ ddl m Z ddl m Z mZddlmZmZddlmZmZejeZerddlZeGd d ZeGd d Zd eezde fdZdZdZGddeeZeGddZ de j de j fdZ!de j"de#e$e$e$fde#e$e$e$fde j"fdZ%de j de#e&e&fde j"fdZ'GddZ(dS) N) dataclass)Enum)BytesIO)Image)SerializableImagedownload_image)assert_opencv_installedis_opencv_installed) ImageChunk ImageURLChunkc>eZdZUdZeeed<ejed<dS) ImageEncodingzA tokenized image. Attributes: tokens: The token ids. image: The image as a numpy array. Examples: >>> import numpy as np >>> image_encoding = ImageEncoding(tokens=[1, 2, 3], image=np.array([[0., 0.5, 1.]])) tokensimageN) __name__ __module__ __qualname____doc__listint__annotations__npndarrayz/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/mistral_common/tokens/tokenizers/image.pyrrs<   I :rrc2eZdZUdZeed<eed<eed<dS)SpecialImageIDsaSpecial image tokens ids. Attributes: img: The image token id. img_break: The image break token id. img_end: The image end token id. Examples: >>> special_image_ids = SpecialImageIDs(img=1, img_break=2, img_end=3) img img_breakimg_endN)rrrrrrrrrrr&s7   HHHNNN LLLLLrrchunkreturnct|tr|jS|drb|dd}t j|}tj t|S|drHtj t| dddS|dr!t|Std |) zGet a serializable image from a chunk. Args: chunk: The chunk to get the image from. Returns: The image as a PIL Image object. z data:image,filezfile://rbhttpzUnsupported image url scheme ) isinstancer rget_url startswithsplitbase64 b64decoderopenrreplacer RuntimeError)r"data image_datas rimage_from_chunkr68s(%$${ }}!!,///}}$$S))!,%d++ z'*--... }}!!&))Nz$u}}66y"EEtLLMMM }}!!&))/emmoo... Hu}}HH I IIr)g3<4'?gwg M?gy{ ?)gB91?gwt.?g U?c.eZdZdZdZeddZdS)MultiModalVersionzVersion of the image tokenizer.m1r# ImageConfigcb|jdkrtddSt|j)Nr9i)namer:NotImplementedErrorselfs rconfigzMultiModalVersion.configYs1 9  r4(( (!TY.111rN)r#r:)rrrrr9propertyrArrrr8r8Ts=** B 222X222rr8c6eZdZUdZeed<eed<dZeed<dS)r:z'Configuration for the image tokenizers.image_patch_sizemax_image_sizer&spatial_merge_sizeN)rrrrrrrFrrrr:r:asA22rr:rc|jdkr|S|jdkr|d}tjd|jd}||d||dS)zUConvert a PIL image to RGB. We ensure transparent background becomes white. RGBRGBAWHITE)rr)modeconvertrnewsizepaste)rwhite_bgs r_convert_to_rgbrQjss  zU  zV f%%!Ifej'BBH NN5&%(((   E " ""rnp_imagemeanstdc6|dz }t|jdksJd|j|jdt|cxkrt|ksnJd|jd|d|||z |z }|dddS) aNormalize a tensor image with mean and standard deviation. Args: np_image: Image to be normalized. mean: Mean for each channel. std: Standard deviation for each channel. Returns: Normalized image with shape (C, H, W). go@znp_image.shape=z, mean=z, std=rr&)lenshape transpose)rRrSrTs r normalizer[xs%H x~  ! # # #%9%9%9 # # # >! D 5 5 5 5SXX 5 5 5 5 57\(.7\7\T7\7\VY7\7\ 5 5 543&H   aA & &&rnew_sizecttjtjt |tj|tj}t|ttS)zTransform an image to a numpy array with the given size. Args: image: Image to be transformed. new_size: New size of the image. Returns: Transformed image with shape (C, H, W). )dtype) interpolation) r cv2resizerarrayrQfloat32 INTER_CUBICr[ DATASET_MEAN DATASET_STD)rr\rRs rtransform_imagergsUz"(?5#9#9LLLhfifuvvvH X|[ 9 99rceZdZdZdededdfdZedefdZde j de e e ffd Z d e ezdefd Zede fd ZdS) ImageEncoderz&Image encoder for the image tokenizer. image_config special_idsr#Nc"||_||_dS)zInitialize the image encoder. Args: image_config: Configuration for the image tokenizer. special_ids: Special image tokens ids. N)rjrk)r@rjrks r__init__zImageEncoder.__init__s)&rc|jSN)rjr?s r mm_configzImageEncoder.mm_configs   rrcN|j\}}t||jjz ||jjz }|dkr$t ||z }t ||z }|dz |jj|jjzzdz}|dz |jj|jjzzdz}||fS)Nr&)rNmaxrjrEroundrDrF)r@rwhratio width_tokens height_tokenss r_image_to_num_tokensz!ImageEncoder._image_to_num_tokenssx1A)88!d>O>^:^__ 199a%i  Aa%i  AA4#4#EHYHl#lmpqq QD$5$FIZIm$mnqrr ]**rcontentct|}||\}}|dksJ|dksJ|jjg|z|jjgz|z}|jj|d<||jjz|jjz||jjz|jjzf}t||}t||S)zConverts an image chunk to an image encoding. Args: content: image chunk to be converted. Returns: Image encoding. r)rr) r6ryrkrr r!rjrDrFrgr)r@rzrrtru image_tokensnew_image_sizeprocessed_images r__call__zImageEncoder.__call__s!))((//11uuuu1uuuu)-.2d6F6P5QQUVV +3 R !2 2T5F5Y Y !2 2T5F5Y Y *%@@LHHHHrc|jjSro)rkrr?s r image_tokenzImageEncoder.image_tokens##r)rrrrr:rrmrBrprtuplerryr r rrrrrrriris11'[''SW''''!;!!!X! + +c3h + + + +I ] :I}IIII,$S$$$X$$$rri))r/logging dataclassesrenumriornumpyrPILrmistral_common.imagerrmistral_common.importsr r &mistral_common.protocol.instruct.chunkr r getLoggerrloggerr`rrr6rerfstrr8r:rQrrfloatr[rrgrirrrrs !!!!!!BBBBBBBBOOOOOOOOLLLLLLLL  8 $ $JJJ             "JMJ6J;LJJJJ.3 2  2 2 2 2 2T 2 2 2           #5; #5; # # # #'j' ue# $' ueU" #'Z ''''2 :5; :%S/ :bj : : : : ;$;$;$;$;$;$;$;$;$;$r