`iYdZddlmZddlmZmZddlZddlm Z ddl m Z ddl m Z dd lmZmZmZdd lmZejeZerddlZed Gd de ZdgZdS)z Processor class for SAM2. deepcopy)OptionalUnionN) ImageInput)ProcessorMixin) BatchEncoding) TensorTypeis_torch_availablelogging)requires)torch)backendsceZdZdZdgZdZd&deedeffd Z d'd ee d ee d ee e e e e e e jfd ee e e e ee jfd ee e e e e e jfdee e e e e jfdee eefdefdZ d(deddddfdZd)dZd*dZd+dZdZdZ d*de e jeje fdededed eede f d!Zd,d"Z d-d%ZxZS). Sam2Processora Constructs a SAM2 processor which wraps a SAM2 image processor and an 2D points & Bounding boxes processor into a single processor. [`Sam2Processor`] offers all the functionalities of [`Sam2ImageProcessorFast`] and [`Sam2VideoProcessor`]. See the docstring of [`~Sam2ImageProcessorFast.__call__`] and [`~Sam2VideoProcessor.__call__`] for more information. Args: image_processor (`Sam2ImageProcessorFast`): An instance of [`Sam2ImageProcessorFast`]. target_size (`int`, *optional*): The target size (target_size, target_size) to which the image will be resized. point_pad_value (`int`, *optional*, defaults to -10): The value used for padding input points. image_processorSam2ImageProcessorFastN target_sizepoint_pad_valuec tj|fi|||_||n|jjd|_dS)Nheight)super__init__rrsizer)selfrrrkwargs __class__s |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/sam2/processing_sam2.pyrzSam2Processor.__init__:sM33F333.*5*A;;tG[G`aiGjimagessegmentation_maps input_points input_labels input_boxesoriginal_sizesreturn_tensorsreturnc ||j|f||d|} ne|Tt|tjr&|}t d|i|} ntd| d}|Bt|dkr/t|t|krtd||| | |dd d d } | |d dd} | |d ddd } | | | dd } | | | dd }| | | dd | | | |krtd| =t| d kr*tfd| Drtd| i| | | d gz}tj |tj}|||d| d|i| M| | |}tj |tj}| d|i| Otj | tj}|||d| d|i| S)a This method uses [`Sam2ImageProcessorFast.__call__`] method to prepare image(s) for the model. It also prepares 2D points and bounding boxes for the model if they are provided. Args: images (`ImageInput`, *optional*): The image(s) to process. segmentation_maps (`ImageInput`, *optional*): The segmentation maps to process. input_points (`list[list[list[list[float]]]]`, `torch.Tensor`, *optional*): The points to add to the frame. input_labels (`list[list[list[int]]]`, `torch.Tensor`, *optional*): The labels for the points. input_boxes (`list[list[list[float]]]`, `torch.Tensor`, *optional*): The bounding boxes to add to the frame. original_sizes (`list[list[float]]`, `torch.Tensor`, *optional*): The original sizes of the images. return_tensors (`str` or `TensorType`, *optional*): The type of tensors to return. **kwargs: Additional keyword arguments to pass to the image processor. Returns: A [`BatchEncoding`] with the following fields: - `pixel_values` (`torch.Tensor`): The processed image(s). - `original_sizes` (`list[list[float]]`): The original sizes of the images. - `reshaped_input_sizes` (`torch.Tensor`): The reshaped input sizes of the images. - `labels` (`torch.Tensor`): The processed segmentation maps (if provided). - `input_points` (`torch.Tensor`): The processed points. - `input_labels` (`torch.Tensor`): The processed labels. - `input_boxes` (`torch.Tensor`): The processed bounding boxes. N)r#r(r') tensor_typez0Either images or original_sizes must be providedz{original_sizes must be of length 1 or len(images). If you are passing a single image, you must pass a single original_size.pointsz;[image level, object level, point level, point coordinates])expected_depth input_nameexpected_formatexpected_coord_sizerlabelsz([image level, object level, point level])r0r1r2boxesz)[image level, box level, box coordinates]zbInput points and labels have inconsistent dimensions. Please ensure they have the same dimensions.c3JK|]}t|dkVdSr,N)len).0 img_boxesboxes_max_dimss r z)Sam2Processor.__call__..s3[[is9~~q(99[[[[[[r!zInput boxes have inconsistent dimensions that would require padding, but boxes cannot be padded due to model limitations. Please ensure all images have the same number of boxes.)dtypeT)preserve_paddingr$r%is_bounding_boxr&)r isinstancerTensorcputolistr ValueErrorr8_validate_single_input_get_nested_dimensionsany_pad_nested_listtensorfloat32_normalize_tensor_coordinatesupdateint64)rr"r#r$r%r&r'r(rencoding_image_processorprocessed_pointsprocessed_labelsprocessed_boxespoints_max_dimslabels_max_dims padded_points final_points padded_labels final_labels final_boxesr;s @r __call__zSam2Processor.__call__?sV  ';t';("3-(( (( $ $  '.%,77 ?!/!3!3!5!5!!>'46F5Wes't't't $ $OPP P22BC  #n"5"5":":s>?R?RVYZ`VaVa?a?aN   #|'?;CZ#:: # ]$% ;    $:: # J ;    #99 " K$% :O +"&"="=>N"O"OPRQRPR"S+"&"="=>N"O"OPRQRPR"S*!%!z9Sam2Processor._convert_to_nested_list..2ooobf44T>=[\K\]]ooor!cDg|]}|dzSrhrirks r rnz9Sam2Processor._convert_to_nested_list..ror!cDg|]}|dzSrhrirks r rnz9Sam2Processor._convert_to_nested_list..ror!zUnsupported data type: )rArrBr8shapenumpyrDnpndarraylistintr^rEtype)rdatar0rms` ``r rjz%Sam2Processor._convert_to_nested_listsk <4 dEL ) ) E 222c$*oo6J6Jzz||**,,,oooooojnoooo bj ) ) E 222c$*oo6J6J{{}}$oooooojnoooo d # # E.. pooooojnoooo sEl + + EKCtDzzCCDD Dr!c8|g}t|ts|St|dkr#|t|n&t |dt||d<t|dkr|D]}t|tru||}t |D]P\}}|dzt|kr||1t ||dz|||dz<Q|S)a` Get the maximum dimensions at each level of nesting. Args: nested_list (`list`): Nested list structure. max_dims (`list`, *optional*): Current maximum dimensions (for recursion). Returns: `list`: A list of maximum dimensions for each nesting level. Nrr,)rArvr8appendmaxrG enumerate)r nested_listmax_dimsrlsub_dimsidims r rGz$Sam2Processor._get_nested_dimensionss+  H+t,, O x==A   OOC ,, - - - -hqk3{+;+;<.[s!#b#b#b1HX$6$6#b#b#br!c.g|]}tSrrrs r rnz2Sam2Processor._pad_nested_list..`s!#S#S#S1HX$6$6#S#S#Sr!)rr8rArvextend_create_empty_nested_structurerangerI) rr~ target_dims current_level pad_value current_sizer template_dimsrrs @r rIzSam2Processor._pad_nested_list,s "  ,I C ,, , , +t,, (&-K;'' !-0  C ,,q0 0 0    {kL.HI J J J Ja 3{#3#3a#777$/ 0A0C0C$DM#BB=R[\\HH!*{[9J-KKH""#b#b#b#bkT`F`@a@a#b#b#bcccc!,MA,=,?,? @ >>}iXX""#S#S#S#Sk@R@R#S#S#STTT 3{++a/ / /3{++,, v vk!nd33v%)%:%:;q>;XehiXikt%u%uKNr!ctdkr gdzSfdtdDS)a Create an empty nested structure with given dimensions filled with pad_value. Args: dims (`list`): The dimensions of the nested structure. pad_value (`int`): The value to fill the structure with. r,rcLg|] }dd!Sr7)r)r9rdimsrrs r rnz@Sam2Processor._create_empty_nested_structure..ws2eeeQRD77QRR)LLeeer!)r8r)rrrs```r rz,Sam2Processor._create_empty_nested_structurejsW t99>>;a( (eeeeeeV[\`ab\cVdVdeee er!c t|tr3t|dkrdSd||dzSt|tjt jfrt|jSdS)z Get the nesting level of a list structure. Args: input_list (`list`): The list to get the nesting level of. rr,) rArvr8_get_nesting_levelrtrurrBrr)r input_lists r rz Sam2Processor._get_nesting_levelysz j$ ' ' ):!##qt..z!}=== =  RZ$> ? ? )z'(( (qr!ryr0r1r2r3c |dSt|tjtjfry|j|kr!t d|d|d|d|jd |5|jd|kr$t d|d|d|jdd |||St|trM| |}||krt d|d |d |d|d |||SdS) a Validate a single input by ensuring proper nesting and raising an error if the input is not valid. Args: data (`torch.Tensor`, `np.ndarray`, or `list`): Input data to process. expected_depth (`int`): Expected nesting depth. input_name (`str`): Name of the input for error messages. expected_format (`str`): The expected format of the input. expected_coord_size (`int`, *optional*): Expected coordinate size (2 for points, 4 for boxes, None for labels). . NzInput z must be a tensor/array with z, dimensions. The expected nesting format is z. Got z dimensions.r]z as the last dimension, got .z must be a nested list with z( levels. The expected nesting format is z levels.) rArrBrtrundimrErrrjrvr)rryr0r1r2r3rms r rFz$Sam2Processor._validate_single_inputs0 <4 dU\2:6 7 7 FyN** qZqqnqqCRqqZ^Zcqqq%0:b>%888$MMMJ]MM{|FGI|JMMM//nEE E dD ! ! F 33D99M.. lZll^ll~MllUblll//nEE E  F Fr!c|r"||jk}|dd}tt|D]}||jdkr|t|kr||n|d}||j||||} |rA||} tj| ||| ||||<| ||<dS)a Helper method to normalize coordinates in a tensor across multiple images. Args: tensor (`torch.Tensor`): Input tensor with coordinates. original_sizes (`list`): Original image sizes. is_bounding_box (`bool`, *optional*, defaults to `False`): Whether coordinates are bounding boxes. preserve_padding (`bool`, *optional*, defaults to `False`): Whether to preserve padding values (for points). r]T)rkeepdimrr?N) rallrr8rrrerrwhere expand_as) rrJr'r@r>mask coord_maskimg_idxr`normalized_coordsimg_masks r rLz+Sam2Processor._normalize_tensor_coordinatess  8T11Db$77JS0011 8 8Ga((;BSEXEX;X;Xw 7 7^lmn^o $($?$?$fWo}Ve%@%%!$8)'2H&+k **6'?;;=NPVW^P_''F7OO'8F7O 8 8r!Tc 4|jj|||||||fi|S)a- Remove padding and upscale masks to the original image size. Args: masks (`Union[List[torch.Tensor], List[np.ndarray]]`): Batched masks from the mask_decoder in (batch_size, num_channels, height, width) format. original_sizes (`Union[torch.Tensor, List[Tuple[int,int]]]`): The original sizes of each image before it was resized to the model's expected input shape, in (height, width) format. mask_threshold (`float`, *optional*, defaults to 0.0): Threshold for binarization and post-processing operations. binarize (`bool`, *optional*, defaults to `True`): Whether to binarize the masks. max_hole_area (`float`, *optional*, defaults to 0.0): The maximum area of a hole to fill. max_sprinkle_area (`float`, *optional*, defaults to 0.0): The maximum area of a sprinkle to fill. apply_non_overlapping_constraints (`bool`, *optional*, defaults to `False`): Whether to apply non-overlapping constraints to the masks. Returns: (`torch.Tensor`): Batched masks in batch_size, num_channels, height, width) format, where (height, width) is given by original_size. )rpost_process_masks) rmasksr'mask_thresholdbinarize max_hole_areamax_sprinkle_area!apply_non_overlapping_constraintsrs r rz Sam2Processor.post_process_maskssCF7t#6       -      r!)Nr)NNNNNNN)F)r)N)rN)FF)rTrrF)__name__ __module__ __qualname____doc__ attributesimage_processor_classrrwrrrrvr^rrBstrr r rZrerjrGrIrrrtrurFrLr __classcell__)rs@r rr%s $$J4kkXc]k\_kkkkkk(,26UYMQNRKO;?B(B($B($J/B(uT$tDK/@*A%BEL%PQR B( uT$tCy/%:EL%HIJ B( eDd5k):$;U\$IJK B(!tDK'8%,'F!GHB(!sJ!78B( B(B(B(B(JX](6 <$E$E$E$EL####J<<<<| f f f..2 0F0FEL"*d230F0F 0F  0F &c] 0F 0F0F0F0Fd!8!8!8!8N*/, , , , , , , , r!r)rcopyrtypingrrrsrt image_utilsrprocessing_utilsr tokenization_utils_baser utilsr r r utils.import_utilsr get_loggerrloggerrr__all__rr!r rs6""""""""%%%%%%......444444<<<<<<<<<<******  H % %LLL :e e e e e Ne e e P  r!