.`ikSUddlZddlZddlmZmZddlmZddlmZm Z m Z m Z m Z m Z ddlZddlmZddlmZmZmZddlmZmZmZddlmZdd lmZdd lmZeeZ e d e Z!Gd de e!Z"Gddee e!Z#Gdde Z$de$de#fdZ%ddde&dej'dedzdefdZ(de&dej'defdZ)dZ*dede+fdZ,e dZ-e-eej.gej.fzZ/e e0d <d!e/e1zdeej.gej.ffd"Z2d#e&d!e/e1zde&fd$Z3ddddd%d&ej.e4ej.zd'ej5j6dzd(e4e&dzd)e&dzd*eej.gej.fdzd+e/dzdej.fd,Z7d-ej.d.ej5j8dej.fd/Z9 dAd1e4e&d2e&de:e4e&e4e&e4e&ffd3Z;d.ej5j8d4ej.d5e4e4e&d6e d7de:ej.d8ff d9Zej.d?ej.dej.fd@Z=dS)BN)ABCabstractmethod)Callable)FinalGenericLiteralProtocol TypeAliasTypeVar)PretrainedConfig)MultiModalConfig VllmConfigget_current_vllm_config)get_tensor_model_parallel_rank$get_tensor_model_parallel_world_size tensor_model_parallel_all_gather) init_logger)current_platform)AttentionBackendEnum_C)boundceZdZUeed<dS) _RootConfig vision_configN)__name__ __module__ __qualname__r__annotations__u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/vision.pyrrsr rceZdZdeeddffd ZedededefdZedefdZ edefd Z edefd Z xZ S) VisionEncoderInfo hf_configreturnNcnt||_|j|_dSN)super__init__r$r)selfr$ __class__s r!r)zVisionEncoderInfo.__init__!s1 "&4r image_width image_heightctr'NotImplementedError)r*r,r-s r!get_num_image_tokensz&VisionEncoderInfo.get_num_image_tokens's "!r ctr'r/r*s r!get_image_sizez VisionEncoderInfo.get_image_size0!!r ctr'r/r3s r!get_patch_sizez VisionEncoderInfo.get_patch_size4r5r ctr'r/r3s r!get_patch_grid_lengthz'VisionEncoderInfo.get_patch_grid_length8r5r ) rrrrrr)rintr1r4r7r9 __classcell__)r+s@r!r#r# s5+b/5d555555 "" " """^"""""^"""""^""s"""^"""""r r#c&eZdZUeeed<dS)VisionLanguageConfigrN)rrrrr rrr r!r=r==s$)******r r=r$r%c>ddlm}m}ddlm}m}ddlm}m}t|j |r ||St|j |r ||St|j |r ||Sdt|j }t|)N)CLIPEncoderInfoCLIPVisionConfig)PixtralHFEncoderInfoPixtralVisionConfig)SiglipEncoderInfoSiglipVisionConfigzUnsupported vision config: ) clipr@rApixtralrBrCsigliprDrE isinstancertyper0)r$r@rArBrCrDrEmsgs r!get_vision_encoder_inforLAs77777777BBBBBBBB========))+;<<*y)))))+>??/##I...))+=>>,  +++ GY-D(E(E G GC c " ""r attn_backend_override head_sizedtyperNc0tj|||S)zE Get the available attention backend for Vision Transformer. )backend)rget_vit_attn_backend)rOrPrNs r!_get_vit_attn_backendrTRs'  0 %   r c t}|j}||jnd}n#t$rd}YnwxYw||jnd}t |||}|S)z; Get the attention backend for Vision Transformer. NrM)r model_configmultimodal_configAssertionErrormm_encoder_attn_backendrT)rOrP vllm_configrVrWrN attn_backends r!rSrSbs!"9";"; "/ .:.FL * *D  !!! !  ( 11  ) 3L  # 22c t}|j}||jnd}n#t$rd}YnwxYw||jnd}|dkS)z> Get the tensor parallel type for Vision Transformer. Ndata)rrVrWrXmm_encoder_tp_mode)rZrVrWr_s r!is_vit_use_data_parallelr`s!"9";"; "/ .:.FL * *D  !!! !1B0M,,SW  ''r\rZc|jjS)zICallable to be passed to `@support_torch_compile`'s `enable_if` argument.)compilation_configcompile_mm_encoder)rZs r!should_torch_compile_mm_vitrds  ) <z._get_vision_feature_selector..sU111bqb!!!8_r rfc$|ddddddfSrlrrms r!roz._get_vision_feature_selector..sU111abb!!!8_r rgc|Sr'rrms r!roz._get_vision_feature_selector..sUr $Unexpected feature select strategy: )callable ValueError)ris r!_get_vision_feature_selectorrusp7,,,9,,,6""" HHHH I IIr num_vision_tokensct|r.tjd|d}||}|jdS|dkrdS|dkr|dz S|dkr|St d|)Nr?@rerfrgrr)rstorchemptyshapert)rvridummy_featuresdummy_selected_featuress r!get_num_selected_vision_tokensr~s0Q(92>>"*(>":":&,Q//7q9 1$$6  HHHH I IIr ) select_layersmax_possible_layers last_hs_procfeature_select_strategyencoder_outputspost_layer_normrrrrc$ |attjstd| ||t |  | |SS|tdt dz }||z fd|D}|d|dz dfv}||r||d|d<|t | fd|D}||r||d|d<tj|dS) a3Given the outputs a visual encoder module that may correspond to the output of the last layer, or a list of hidden states to be stacked, handle post normalization and resolve it into a single output tensor. Args: encoder_outputs: Output of encoder's last layer or all hidden states. post_layer_norm: Post norm to apply to the output of the encoder. select_layers: Optional layer indices to grab from the encoder outputs; if provided, encoder outputs must be a list. max_possible_layers: Total layers in the fully loaded visual encoder. last_hs_proc: Optional callable to be applied to the last layer if it is used, e.g., pooling head for Siglip. This is done prior to feature selection and layer normalization. If select_layers are provided, the output of last_hs_proc must be able to be concatenated with the other select_layers along the last dimension. feature_select_strategy: Defines how to select the hidden states from each layer. NzJExpected only a single encoder output when `select_layers` is not providedz@`max_possible_layers` must be provided alongside `select_layers`r?cBg|]}|dkr|n |zSrr).0 layer_idxroffsets r! z2resolve_visual_encoder_outputs..sK  >>  "" Y/ 0r c&g|] }|Srr)rhsselect_featuress r!rz2resolve_visual_encoder_outputs..s#9992??2&&999r dim)rIryTensorrtrulencat) rrrrrrnum_loaded_layershs_pooluses_last_layerrrs ` @@r!resolve_visual_encoder_outputsrs6/5<88 2   #*l?;;O " .:;RSSO-oo>>O  &"??33 3" N   O,,q0 #4 4F' G$B',?!,CR+HHOO"l72;// *67NOO9999999""%ogbk22 9W" % % %%r image_input vision_modelc|jd}t}||zdz |z}||z|z }dd|dz zzd|fz}tjj||}t}|||z|dz|zdf} || } | } t| d} | d|df} | S)aWRun a vision model with data parallelism (DP) sharding. The function will shard the input image tensor on the first dimension and run the vision model Args: image_input (torch.Tensor): Image input tensor. vision_model (torch.nn.Module): Vision model. Returns: torch.Tensor: Output image embeddings rr?r.rN) r{rrrynn functionalpadr contiguousr) rr num_chunks mp_world_sizenum_chunks_per_ranknum_padded_chunksrimage_input_paddedrankimage_input_per_rankvision_embeddingss r!run_dp_sharded_vision_modelrs "1%J8::M% 59mK+m;jH !{((1,- .!5F1G GC,00cBB ) + +D- ""dQh2E%EEsJ% %9::)446689JPQRRR)+:+s*:; r rsizesnum_gpusc t}|dkr gdg|zdg|zfSdt|D}dg|z tt|fdd}|D]T}tt| fd}||| |xx|z cc<Ut t }t t }t|D]E} ||| |t|| F|| fS)aq Generate load balancing assignment and metadata for distributing data across GPUs. The load is determined by the total image sizes, not the number of images. Args: sizes: The size of each image num_gpus: Number of GPUs to balance across Returns: shuffle_indices: Indices to reorder data for balanced loading gpu_sample_counts: Number of samples assigned to each GPU grouped_sizes_per_gpu: Total size assigned to each GPU Example: ``` sizes = [1000, 100, 200, 50] num_gpus = 2 ``` rc@g|]}ttSr)listr:)r_s r!rz/get_load_balance_assignment..`s <<.gs ar T)keyreversec|Sr'r)r gpu_loadss r!roz-get_load_balance_assignment..ls Yq\r )r)rrangesortedminappendrr:extend) rr n_samplesgpu_assignmentslarge_to_small_indicesidxmin_gpushuffle_indicesgpu_sample_countsgpu_idrs ` @r!get_load_balance_assignmentr;s{<E IA~~A3>A3>11=>>> . ::r pixel_values grid_thw_list rope_type)rope_3drope_2d.c$%t}t}dD}dgtj|$t ||\}}} dgtj|} || || |dz} t | dkr"t j$fd| D} n.t jdj dfj j } |dkr|j d|j dz%n|j |j z%t| %z} fd| D}|dkr| j ddkrJ|| t j|}t!|t"rt j|d}nt%|jd d }t jd%|fj j }nF| j ddkr || |}n(t jd|jfj j }|j d}|| kr| |z }|dkr;t j||j d|j d f|j |j }n.t j||j df|j |j }t j||gd}n|}t+|d}t#t j}t/|D]2}|| z}|| |%zz}||||3%fd |D}d gt z}d}t/|D]N}||}|dkr>||||z}||}d} |D]}!||!}"|| | |"z||!<| |"z } ||z }Ot3d|D}#t |#t |ks Jd|#S)aRun a vision model with data parallelism (DP) sharding. The function will shard the input image tensor on the first dimension and run the vision model. This function is used to run the vision model with mrope. Args: vision_model (torch.nn.Module): Vision model. pixel_values (torch.Tensor): Image/Video input tensor. grid_thw_list: List of grid dimensions for each image rope_type: Type of rope used in the vision model. Different rope types have different dimension to do ViT. "rope_3d" for 3D rope (e.g., Qwen2.5-VL) "rope_2d" for 2D rope (e.g., Kimi-VL) Returns: torch.Tensor: Output image embeddings Example: ``` vision_model.out_hidden_size = 64 vision_model.spatial_merge_size = 2 pixel_values.shape = (1350, channel) grid_thw_list = [[1, 10, 100], [1, 10, 10], [1, 10, 20], [1, 50]] tp_size = 2 ``` c6g|]}tj|Sr)mathprod)rgrid_thws r!rz5run_dp_sharded_mrope_vision_model..s"KKK8,,KKKr rr?cBg|]}||dzS)r?r)rrcum_patches_per_imagers r!rz5run_dp_sharded_mrope_vision_model..sD   2158MaRSe8TTU   r )devicerPrc g|] }| Srr)rrrs r!rz5run_dp_sharded_mrope_vision_model..sFFF=+FFFr r hidden_sizeNr)rPrcg|]}|zSrr)r patch_sizeembed_dim_reduction_factors r!rz5run_dp_sharded_mrope_vision_model..s,   7A1 1   r c3K|]}||V dSr'r)rembeds r! z4run_dp_sharded_mrope_vision_model..5s08I8I8I8I8Ir zFound unassigned embeddings)rr itertools accumulaterrryrrzr{rrPmerge_kernel_sizespatial_merge_sizemaxtensorrIrgetattrconfigout_hidden_sizerrrrtuple)&rrrrtp_size tp_rank_localpatches_per_imageimage_to_tp_rankrgrouped_pixel_values_lencum_gpu_sample_countsimage_idxs_localpixel_values_localmax_len_per_ranklocal_grid_thw_listimage_embeds_localout_dim current_len padding_sizepaddingimage_embeds_local_paddedgathered_embedsrank_embeddingsr start_idxend_idxpatches_per_output_imageoriginal_order_embeddings current_idxcount rank_images rank_embed embed_startimg_idx img_patchesout_embeddingsrrs& `` @@r!!run_dp_sharded_mrope_vision_modelrs]B344G344MLK]KKKI!56G!H!HI $$5w??D(*B I!56G!H!HI(m,/D]UVEV/WW  q  "Y     )     #[  "1% &&$    I  *1 - 0Nq0Q Q #"  +l.M M #3448RRFFFF5EFFFI  #A & * *!-"EL1D$E$E"" ,d33 J%*Y/Aq%I%I%I"l1=$GGG!&.8#*"("""    #A & * *!-.@BU!V!V  "'L01#*"(""" %*1-K%%%'+5  ! !k &,Q/&,Q/ ).)0GGk17:;(.)0G %*I/A7.KQR$S$S$S!!$6!77PVWXXXO5<(**OgCC++  $T *.H H  y/@ABBBB    EV    "&]);); ;Kg!!!$' 199+;u9L+LMK(.JK& + +6w? 5?+ ";;6)'2{* 5 K4N ~  #&?"@"@ @ @ @% A @ @ r r vision_idxrt_indexgrid_hsgrid_wsc*g}|||z}|||z}tj|dddt |d|} tj|dddt ||d} tj||jddd||z } tj | | | g} | | |ztj |d} | S)Nr?rr) ryarangeviewexpandrflattenrtorlongstackrr)rrrrrrllm_pos_ids_list llm_grid_h llm_grid_wh_indexw_indext_index_tensor _llm_pos_ids llm_pos_idss r!get_llm_pos_ids_for_visionr>sQ$(::J$(::J Z  aQ G b* - -     Z  aB G j" - -     W J    b! J+ , ,   ;ABBLL94555),!444K r )r)>rrabcrrcollections.abcrtypingrrrr r r ry transformersr vllm.configr rrvllm.distributedrrr vllm.loggerrvllm.platformsr#vllm.v1.attention.backends.registryrrloggerrrr#r=rLr:rPrTrSr`boolrdVisionFeatureSelectStrategyStrrrhrstrrur~rr LayerNormrModulerrrrrrr r!r&sw ########$$$$$$HHHHHHHHHHHHHHHH ))))))MMMMMMMMMM $#####++++++DDDDDD X   WT)***(2,"""""WR[""":+++++8+++#';#@Q####*:>     ; 0$6        ;:(((&=Z=D==== "))C!D#Xu|nel.J%KKY J)C/J u|nel*+JJJJ(JJ)C/J JJJJ2'+&*BFBFP&P&P&\D$66P&X'$.P&9t# P& t P& EL>5<784? P&94?P& \P&P&P&P&f-2X_ \FB;B; 9B;B; 49d3ic *+B;B;B;B;J{(/{,{S ?{ +, {  5<  {{{{|""""#Y " \ " \ " \""""""r