`i:TddlmZmZddlZddlZddlmZddl m Z m Z ddl m Z mZmZmZddlmZmZddlmZmZdd lmZdd lmZmZmZejeZd Z d Z!d Z"dZ#dZ$de%dde%e&fdZ'de&de(e&e&ffdZ)GddeZ*GddeZ+dgZ,dS))OptionalUnionN) functional) BatchFeature get_size_dict)IMAGENET_STANDARD_MEANIMAGENET_STANDARD_STDPILImageResamplingSizeDict)Unpack VideosKwargs) TensorTypelogging)BaseVideoProcessor) VideoMetadatagroup_videos_by_shapereorder_videoszYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.zgYou are provided the following series of {frame_count} frames from a {video_duration} [H:MM:SS] video. z z Frame from {timestamp}:ivideos torch.Tensorreturnctdx}}|D]A}|dd\}}t||}t||}B||fS)zH Get the maximum height and width across all videos in a batch. z-infN)floatsizemax)r max_height max_widthvideoheightwidths /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/smolvlm/video_processing_smolvlm.pyget_max_height_widthr#)sh#6]]*J** RSS) ,, y))  ""resolution_max_sidec|dd\}}tt|}|t||n|}||z }||kr#|}t ||z }|dzdkr|dz }n(||kr"|}t ||z}|dzdkr|dz }t|d}t|d}||fS)a Get the output size of the video after resizing given a dictionary specifying the max and min sizes. Args: video (`np.ndarray`): Video to resize. resolution_max_side (`int`): The longest edge of the video will be resized to this value. The shortest edge will be resized to keep the input aspect ratio. Returns: The output size of the video after resizing. rNr)rminMAX_IMAGE_SIZErint)rr%r r! aspect_ratios r"get_resize_output_image_sizer-5sJJLL%MFEn.ABB0C0K#fe,,,Qd6>L #U\)** A:?? aKF %$F\)** 19>> QJE ^^F qMME 5=r$cLeZdZUeeeefed<eeed<dS)SmolVLMVideoProcessorInitKwargsmax_image_sizedo_padN) __name__ __module__ __qualname__rdictstrr+__annotations__boolr$r"r/r/]s;T#s(^,,,, TNr$r/ceZdZejZddiZddiZeZ e Z dZ dZ dZdZdZdZeZddgZdeeffd Z d*d d d ededdedd f dZ d+d d deeefdededef dZ d,dedeedeeee fdeefdZ! d-de"d d ed!ed ededd"ed#e d$ed%ed&eee e"e fd'eee e"e fd(eee#e$ffd)Z%xZ&S).SmolVLMVideoProcessor longest_edgeiilTF pixel_valuespixel_attention_maskkwargsc tjdi|d|vrd|vr|d|dd<d|vrO|dd|_|dd|_t |dd|j|_dSdS)Nrvideo_sampling video_size max_framesfps)default_to_squarer9)super__init__ num_framesrDrrEr)selfr? __class__s r"rGzSmolVLMVideoProcessor.__init__qs""6""" V   0F : :5;F^F# $\ 2 v % %$%56|DDO./6DH%f-=&>|&L`d`vwwwDIII & %r$Nrrr interpolationzF.InterpolationMode antialiasrc ||ntjj}|tjjkr+tdtjj}|jrt||j}n0|j r|j r|j |j f}ntd|dtj ||||}|j d|j df}tj ||||}|S)a9 Resize an video to `(size["height"], size["width"])`. Args: video (`torch.Tensor`): Video to resize. size (`SizeDict`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output video. resample (`InterpolationMode`, *optional*, defaults to `InterpolationMode.BILINEAR`): `InterpolationMode` filter to use when resizing the video e.g. `InterpolationMode.BICUBIC`. Returns: `torch.Tensor`: The resized video. NzYou have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. BICUBIC resample will be used as an alternative. Please fall back to image processor if you want full consistency with the original model.)r%zHSize must contain 'height' and 'width' keys, or 'longest_edge' key. Got .)rKrLr<)FInterpolationModeBILINEARLANCZOSlogger warning_onceBICUBICr<r-r r! ValueErrorresizer0)rIrrrKrLr?new_sizemax_sizes r"rWzSmolVLMVideoProcessor.resize}s(*7)B H[Hd A/7 7 7   A    /7M   q4$($5HH[ qTZ q TZ0HHohlooopp p QZ[[[&~68KN8[[ QZ[[[ r$r padded_sizemax_num_framesfillreturn_pixel_maskc|dd}|d|dz }|d|dz }||jdz } |dks|dkrtd|d|d||kr!d|d|ddd| g} tj|| |}d} |rFt j|d dddddft j } d| d d|dd|df<|| fS) a"Pads the sample with empty video to the padded_size Args: video (`torch.Tensor`): Video to pad. padded_size (`tuple[int, int]`): Height and width to pad. max_num_frames (`int`): The maximum number of frames to which video will be padded. fill (`int`, *optional*): The value to use for the padding. return_pixel_mask (`bool`, *optional*, defaults to `True`): Whether to return a pixel mask. rNrr(zzPadding dimensions are negative. Please make sure that the padded size is larger than the original size. Got padded size: z, original size: rN)r\.dtype)rshaperVrOpadtorch zeros_likeint64) rIrrZr[r\r] original_sizepadding_height padding_width padding_framepadding pixel_masks r"rbzSmolVLMVideoProcessor.padsH* RSS) $Q-*::#Aq)99 &Q7 1   2 2b3>bbQ^bbb  K ' '-NAq!]SGE%t444E  H)%Q111 *=U[QQQJFGJs.mA..0B-2B0BB Cj  r$r(metadatarHrD skip_secsc j|t|ddtd||n|j}||n|j}|j}t t ||dz}t||}|dkrd}d} |dz } |dkrH|dd|zz ||zkr3t ||dz} t |||dzz } td| } t| |dz } | | krd|dz } } tj | | |t } tj | } | S) a! Video sampling function which: - Uses `num_frames` (if provided) or calculates it from `fps` and metadata. - Applies a basic center-skip if fewer frames than available, otherwise optionally skips `skip_secs` from both the start and end. - Uniformly samples the desired number of frames between the start and end indices. Args: metadata (`VideoMetadata`): Metadata of the video containing information about total duration, fps and total number of frames. num_frames (`int`, *optional*): Maximum number of frames to sample. Defaults to `self.num_frames`. fps (`int` or `float`, *optional*): Target frames to sample per second. Defaults to `self.fps`. skip_secs (`float`, *optional*, defaults to `1`): Number of seconds to skip from the start and end if the video is long enough. Returns: np.ndarray: Indices to sample video frames. NrDzAsked to sample frames per second but no video metadata was provided which is required when sampling in SmolVLM. Please pass in `VideoMetadata` object or set `do_sample_frames=False`durationr(rr'r_) getattrrVrHrDtotal_num_framesr+roundr)rnplinspaceunique) rIrlrHrDrmr?rqestimated_framesdesired_frames start_idxend_idxindicess r" sample_framesz#SmolVLMVideoProcessor.sample_framessu:  wx==EX  $.#9ZZt _cc$(#4uS8J+?%?@@AA-z:: A  N "Q& q==hz2Q]BzTWGWXXI788I*Y%-HHIIG9%% g/!344   !"$4q$8wI+i.LLL)G$$r$rdo_convert_rgb do_resize do_rescalerescale_factor do_normalizer1 image_mean image_stdreturn_tensorsc t|\}}i}|D];\}}|r||}|r||||}|||<z4SmolVLMVideoProcessor._preprocess..@s( J JU J J J J J Jr$)rZr[r)dimr=r>) tensor_type) ritemsconvert_to_rgbrWrrescale_and_normalizer#rrbrcstackr)rIrr|r}rrKr~rrr1rrrr?grouped_videosgrouped_videos_indexresized_videos_groupedrastacked_videosresized_videosprocessed_videos_groupedprocessed_videospad_sizer[processed_padded_mask_grouped padded_masksr>datas r" _preprocessz!SmolVLMVideoProcessor._preprocesssq 0EV/L/L,,!#%3%9%9%;%; ; ; !E> E!%!4!4^!D!D e!%^$Vc!d!d,: "5 ) )'(>@TUU/D^/T/T,,#% %3%9%9%;%; = = !E>!77 NL*V_N/= $U + +)*BDXYY  g+,<==H J J9I J J JJJN3HIY3Z3Z 0N0,. )') $)7)=)=)?)? D D%~/3xx"0800, 3A(/7C-e44-.FH\]] #12OQe#f#f CQg5;'7Q????Wg 01  *,8 0a8888) ' ( Dn====r$)NT)rT)NNr(r)'r2r3r4r rRresamplerr0r rr rr}r~rr|r1do_sample_framesr/ valid_kwargsmodel_input_namesr rGr rr8rWtupler+rbrrrr{listr6rr __classcell__)rJs@r"r;r;bs!)H G $D$c*N'J%IIJLN F2L')?@ x(G!H x x x x x x :> //// 56 /  / ////l"& )!)!)!38_)! )!  )!  )!)!)!)!\%)+/#$ ???SM?eCJ' ( ? C= ????\<@>>>>^$>>>> >>  >>  56 >>>>>>>>>>U5$u+#567>>E%e"456>>!sJ!78>>>>>>>>>>>>>>>>r$r;)-typingrrnumpyrsrctorchvision.transforms.v2rrOimage_processing_utilsrr image_utilsr r r r processing_utilsr rutilsrrvideo_processing_utilsr video_utilsrrr get_loggerr2rSDEFAULT_SYSTEM_MESSAGEDEFAULT_VIDEO_INTRODEFAULT_MEDIA_OUTTROFRAME_TIMESTAMP_MESSAGEr*rr+r#rr-r/r;__all__r9r$r"rs #""""""" 555555AAAAAAAAffffffffffff44444444((((((((888888OOOOOOOOOO  H % %Vn5 #n!5 #$s) # # # #%% 38_%%%%Pl v>v>v>v>v>.v>v>v>r # #r$