Pi!ddlZddlmZmZmZmZmZddlZddlZddl m Z ddl m Z m Z ddlmZddlmZddlmZejeZGdd ZdS) N)AnyListMappingOptionalTuple)Image)find_supported_resolutionsget_canvas_best_fit)resize_with_pad) tile_crop) functionalceZdZdZdddddejddddeeed eeed eee e e fd e d ee d ej de de ddfdZ ddee efde dee effdZdS)CLIPImageTransforma  This class accepts images of any size and dynamically resizes, pads, normalizes and tiles it based on the image aspect ratio and the number of image tiles we allow. The algorithm will NOT distort the image to fit a certain aspect ratio, because that leads to a significant degradation in image quality. The user can choose if they want to allow upscaling by using the flag ``resize_to_max_canvas``. For example, if an input image is of size 300x800, and we want to allow a maximum of 16 image tiles, with side 224px, then: If ``resize_to_max_canvas=False``, then: best_resolution = (448, 896) -> smallest canvas, up to 16 tiles, that doesn't require downscaling image is NOT resized image is padded (300, 800) -> 448,896 Image is tiled 2x4, for a final output shape of (8, 3, 224, 224) If ``resize_to_max_canvas=True``, then: best_resolution = (448, 1344) # canvas that allows maximum upscaling, with minimum padding, up to 16 tiles image is resized without distortion (300,800) -> (448, 1194) #448 is the limiting side for the resize image is padded (448, 1194) -> (448, 1344) Image is tiled 2x5, for a final output shape of (10, 3, 224, 224) Args: image_mean (Optional[List[float]]): Mean values of each channel, used for normalization. Should be the same used for the pre-trained model. If None, no normalization is performed. Default None. image_std (Optional[List[float]]): Standard deviation values of each channel, used for normalization. Should be the same used for the pre-trained model. If None, no normalization is performed. Default None. possible_resolutions (Optional[List[Tuple[int, int]]]): List of possible resolutions as tuples (height, width). where each tuple represents a possible canvas to fit the image into when calling ``get_canvas_best_fit``. If None, this will be calculated using max_num_tiles and tile_size. Default None. tile_size (int): Size of the tiles to divide the image into. Default 224. max_num_tiles (Optional[int]): Only used if possible_resolutions is NOT given. Maximum number of tiles to break an image into. This will be used to generate possible_resolutions, e.g. [(224, 224), (224, 448), (448, 224)] if max_num_tiles = 2 and tile_size = 224. Default 4. dtype (torch.dtype): Data type of the output image. Default torch.bfloat16. resample (str): Resampling method used when resizing images. Supports any enum of ``torchvision.transforms.InterpolationMode``, e.g. "nearest", "nearest_exact", "bilinear", "bicubic". Default 'bilinear'. resize_to_max_canvas (bool): "If True, the image will be upscaled without distortion to fit the largest possible resolution from possible_resolutions. If False, it will pick the resolution that minimizes downscaling, including no downscaling at all. In this case, the image will only be upscaled if it's size < tile_size. Default False. Examples: >>> image_transform = CLIPImageTransform( ... image_mean=None, ... image_std=None, ... tile_size=224, ... possible_resolutions=None, ... max_num_tiles=4, ... resample="bilinear", ... resize_to_max_canvas=True, ...) >>> # create random image >>> image = (np.random.rand(100,200,3) * 255).astype(np.uint8) >>> image = PIL.Image.fromarray(image) >>> output = image_transform(image) >>> output['image'].shape # [num_tiles, num_channels, tile_size, tile_size] torch.Size([2, 3, 224, 224]) >>> output['ar'] # image best fits the canvas 224x448 torch.tensor([1,2]) NbilinearF) image_mean image_stdpossible_resolutions tile_size max_num_tilesdtyperesampleresize_to_max_canvasrrrrrrrrreturnc||Jd|d||s|rt||}n|}tj|dd|_t d|jd||_|du|duksJd|d |||_||_ |rdn||_ ||_ tj j||_||_t$|_dS) NzUEither possible_resolutions or max_num_tiles must be given. Got possible_resolutions=z and max_num_tiles=)rrzFound possible_resolutions: z4. Will fit the images into the canvas with best fit.zINeed to provide both or none of image_mean and image_std. Got image_mean=z and image_std=)r torchtensorreshaperloggerdebugrmeanstdmax_sizer torchvision transformsInterpolationModeupperrrr ) selfrrrrrrrrs t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/clip/_transform.py__init__zCLIPImageTransform.__init__^s[ ! , 0I0I zNb z zjw z z1J0I I$ 8  8#=+y$$$ $8 $)L1E$F$F$N$NrST$U$U! z4+D z z z   %9!d"     jJ j j^g j j    !5C)  #.@AQAQR #"sample inferencec|d}t|tjs Jd|jdkr|d}t j|}t j||jd}t||j |j }t|||j |j }|jr!t j||j|j}|||j }t'j|d |jz}|||d |S) a Apply image decoding and transformations to the "image" field in the sample. Args: sample (Mapping[str, Any]): A sample with an "image" field containing a List[Message] to tokenize inference (bool): Whether the template is being used for inference or not. Returns: Mapping[str, Any]: The sample with an updated "image" filed and added "aspect_ratio" field. imagez Input image must be a PIL image.RGBT)rscale)r2rr)r2 target_sizerr&)r$r%)r2rr)r2 aspect_ratio) isinstancermodeconvertFto_imageto_dtyperr rrr rr&r$ normalizer%r rrr r!update)r+r/r0r2best_resolutionr6s r,__call__zCLIPImageTransform.__call__sQw%--QQ/QQQ- :  MM%((E 5!! 5 $???.!%!:!%!:    ']]     9 EKDI48DDDEUdnEE|O44<rSs66666666666666 VUUUUUIIIIII555555  8 $ $llllllllllr.