`iddlZddlmZddlmZmZmZddlZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd lmZm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)m*Z*edGdde j+Z, dEde j+dej-dej-dej-deej-de.de.fdZ/Gdd e j+Z0ee!d!"Gd#d$eZ1Gd%d&e j+Z2Gd'd(e j+Z3Gd)d*e j+Z4e j5e,d+Z6Gd,d-eZ7Gd.d/e j+Z8e!Gd0d1eZ9e!Gd2d3e9Z:e!Gd4d5eZ;Gd6d7e j+Zee!d>"Gd?d@eZ?e!dA"GdBdCe;eZ@gdDZAdS)FN) dataclass)CallableOptionalUnion)ACT2FN)Cache)GenerationMixin)use_kernel_forward_from_hub)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuple torch_int)check_model_inputs) AutoModel)InternVLConfigInternVLVisionConfigRMSNormc,eZdZdfd ZdZdZxZS)InternVLVisionRMSNormư>cttjt j||_||_dS)zD InternVLVisionRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/internvl/modeling_internvl.pyr$zInternVLVisionRMSNorm.__init__.sD l5:k#:#:;; #c|j}|tj}|ddd}|tj||jzz}|j||zS)NrT)keepdim) dtypetor'float32powmeanrsqrtr*r))r+ hidden_states input_dtypevariances r/forwardzInternVLVisionRMSNorm.forward6s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r0cHt|jjd|jS)Nz, eps=)tupler)shaper*r+s r/ extra_reprz InternVLVisionRMSNorm.extra_repr=s& )**II$2GIIIr0)r!)__name__ __module__ __qualname__r$r=rB __classcell__r.s@r/r r ,sb$$$$$$;;;JJJJJJJr0r modulequerykeyvalueattention_maskscalingdropoutc |}|} tj||dd|z} |$|ddddddd|jdf} | | z} tj| d} tj| ||j} tj| | } | dd } | | fS)Nrrr2dim)ptrainingr) r'matmul transposer@r% functionalsoftmaxrOrU contiguous) rIrJrKrLrMrNrOkwargs key_states value_states attn_weights causal_mask attn_outputs r/eager_attention_forwardraAsJL<z';';Aq'A'ABBWLL!$QQQ111.D 0@0D.D%DE #k1 =((2(>>L=((6?([[L,|\::K''1--88::K  $$r0cleZdZdZdeffd Z d dejdeejde e fdZ xZ S) InternVLVisionAttentionz+Attention Class for InternVL Vision Encoderconfigct||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ |j |_ |j }|j }d|_ tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|_|dkrtj|ntj|_ |rt/|jntj|_|rt/|jntj|_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFbiasr)r#r$rdr, embed_dimnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutprojection_dropout use_qk_norm is_causalr%Linearattention_biasq_projk_projv_projprojection_layerDropoutIdentityr q_normk_norm)r+rd proj_dropoutqk_normr.s r/r$z InternVLVisionAttention.__init___s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( !'!90 $i0NU[Ujkkk i0NU[Ujkkk i0NU[Ujkkk " $.$. I I>JQ>N>N"*\":":":TVT_TaTa?FY+DN;;;BKMM ?FY+DN;;;BKMM r0Nr:rMr[c |\}}}||}||}||} ||}||}||||j|j dd}||||j|j dd}| |||j|j dd} t} |j j dkrt|j j } | |||| |f|jsdn|j|jdd|\} } | |||j} || } || } | | fS)NrreagerrHF)rOrNrq)sizertrurvrzr{reshaperjrkrWviewrard_attn_implementationrrUrnrmrhrwro)r+r:rMr[ batch_sizeseq_len_ query_statesr\r]attention_interfacer`r^outputs r/r=zInternVLVisionAttention.forward{s "/!3!3!5!5 GQ{{=11 [[// {{=11 {{<00 [[,, #++JQUQ^__iijkmnoo '' GT^T][[eefgijkk #((Wdndm\\ffghjkll (? ; +w 6 6"9$+:Z"[ $7$7      % $}HCC$2HJ % %  % % ! \"))*gt~NN &&{33((00|##r0N) rCrDrE__doc__rr$r'Tensorrrrr=rFrGs@r/rcrc\s55Z3ZZZZZZ>26'$'$|'$!.'$+, '$'$'$'$'$'$'$'$r0rcz7 Class for outputs of [`InternVLVisionModel`]. ) custom_introceZdZdZdS)$InternVLVisionModelOutputWithPoolingaF pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`): Average of the last layer hidden states of the patch tokens (excluding the *[CLS]* token) if *config.use_mean_pooling* is set to True. If set to False, then the final hidden state of the *[CLS]* token will be returned. N)rCrDrErr0r/rrsr0rcFeZdZdZfdZdejdejfdZxZS)InternVLVisionPatchEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. ct|j|j}}|j|j}}|d|dz|d|dzz}|d|dz|d|dzf}||_||_||_||_||_tj |||||_ dS)Nrr) kernel_sizestride) r#r$ image_size patch_size num_channelsr, num_patches patch_shaper%Conv2d projection) r+rdrrrr,rrr.s r/r$z&InternVLVisionPatchEmbeddings.__init__s !'!2F4EJ $*$79Kk !!} 1 5*Q-:VW=:XY !!} 1 5z!} ST 7UV $$(&&)L+:^hiiir0 pixel_valuesreturnc |j\}}}}||jkrtd||}|jd|jd}}|ddd}|||ffS)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.rrr)r@rrlrflattenrW) r+rrrheightwidth embeddings patch_height patch_widths r/r=z%InternVLVisionPatchEmbeddings.forwards2>2D/ L&% 4, , ,w __\22 $.$4Q$79I!9Lk ''**44Q:: L+666r0) rCrDrErr$r'rr=rFrGs@r/rrsm j j j j j 7EL 7U\ 7 7 7 7 7 7 7 7r0rceZdZdZdeddffd Zdejdededejfd Z d d ejd e ej dejfd Z xZ S)InternVLVisionEmbeddingszc Construct the CLS token, position and patch embeddings. Optionally, also the mask token. rdrNcttjt jdd|j|_|jr3tjt jdd|j|_ nd|_ t||_ |j |_ t|jtjjr|jn |j|jf|_|j j}|jr6tjt jd|dz|j|_nd|_tj|j|_dS)Nr)r#r$r%r&r'zerosr, cls_tokenuse_mask_token mask_tokenrpatch_embeddingsr isinstancer collectionsabcIterabler use_absolute_position_embeddingsposition_embeddingsrxhidden_dropout_probrO)r+rdrr.s r/r$z!InternVLVisionEmbeddings.__init__s( ek!Q8J&K&KLL   # l5;q!V=O+P+PQQDOO"DO =f E E +&+[_-EFF 8F  #V%67  +7  2 ,')|EK;QR?TZTf4g4g'h'hD $ $'+D $z&"<== r0rrrc|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jdz} ||jdz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNr2r?rrbicubicF)rmode align_cornersrR)r@rr'jit is_tracingrrrpermuter%rX interpolatercat) r+rrrr num_positionsclass_pos_embedpatch_pos_embedrS new_height new_widthsqrt_num_positionss r/interpolate_pos_encodingz1InternVLVisionEmbeddings.interpolate_pos_encodings|!&q)A- 06q9A= y##%% ,+*F*F6UZ??+ +2111bqb592111abb59r"tq11 T_Q// &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr0rbool_masked_posc|j\}}}}||\}\}}|\} } }|R|j| | d} |d| } |d| z z| | zz}|j| dd} tj | |fd}|j || |||z}| |}|||ffS)Nr2rrR) r@rrrexpand unsqueezetype_asrr'rrrrO)r+rrrrrrrrrr mask_tokensw cls_tokenss r/r=z InternVLVisionEmbeddings.forwards +01fe262G2G 2U2U/ /\;!+!2!2 GQ  &/00WbIIK))"--55kBBA#q1u- a?J^**:r2>> Y J7Q???  # /#d&C&CJPVX]&^&^^J\\*-- L+666r0r)rCrDrErrr$r'rintrr BoolTensorr=rFrGs@r/rrs >3>>>>>>>,&D5<&D&DUX&D]b]i&D&D&D&DV7;77l7"%"237  77777777r0rcBeZdZfdZdejdejfdZxZS)InternVLVisionMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) r#r$rdr hidden_act activation_fnr%rrr,intermediate_sizefc1fc2r+rdr.s r/r$zInternVLVisionMLP.__init__9sf  #F$569V/1IJJ9V5v7IJJr0r:rc||}||}||}|Sr)rrr)r+r:s r/r=zInternVLVisionMLP.forward@s=// **=99 // r0)rCrDrEr$r'rr=rFrGs@r/rr8scKKKKKU\elr0r) layer_normrms_normceZdZdZdeddffd Zdejdee eje ejejfffdZ xZ S)InternVLVisionLayerz?This corresponds to the Block class in the timm implementation.rdrNct|j|_d|_t ||_t ||_t|j |j |j |_ t|j |j |j |_ |j}tj|t#j|j zd|_tj|t#j|j zd|_tj|j|_dS)Nrr-T) requires_grad)r#r$chunk_size_feed_forward seq_len_dimrc attentionrmlpNORM2FN norm_typer,layer_norm_epslayernorm_beforelayernorm_afterlayer_scale_init_valuer%r&r'r(lambda_1lambda_2rxrrO)r+rd init_valuesr.s r/r$zInternVLVisionLayer.__init__Ms '-'E$088$V,, '(8 9&:LRXRg h h h&v'789KQWQfggg3  [5:f>P3Q3Q%Qaefff  [5:f>P3Q3Q%Qaefff z&"<== r0r:c$|||\}}|j|z}||z}||}||}||}|j |j|z}||z}|Sr)rrrrrrOr)r+r:attention_outputr layer_outputs r/r=zInternVLVisionLayer.forward\s#nn  ! !- 0 0  ! =+;;)=8 ++M:: xx -- ||L11 = $=<7L$m3 r0) rCrDrErrr$r'rrr?r=rFrGs@r/rrJsII >3 > > > > > > >| uU\"E%, *D$EE Fr0rcReZdZdeddffd Zdejdeee ffdZ xZ S)InternVLVisionEncoderrdrNct|_tjfdt jD|_d|_dS)Nc.g|]}tSr)r).0irds r/ z2InternVLVisionEncoder.__init__..|s"#i#i#iA$7$?$?#i#i#ir0F) r#r$rdr% ModuleListrangenum_hidden_layerslayergradient_checkpointingrs `r/r$zInternVLVisionEncoder.__init__ys`  ]#i#i#i#ivOgIhIh#i#i#ijj &+###r0r:cL|jD] }||}t|S)N)last_hidden_state)rr )r+r: layer_modules r/r=zInternVLVisionEncoder.forwards@!J 8 8L(L77MM+    r0) rCrDrErr$r'rrr?r r=rFrGs@r/rrxs~,3,,,,,,,  |   uo% &         r0rcVeZdZUeed<dZdZdZdgZdZ dZ dZ dZ e edZfdZxZS)InternVLVisionPreTrainedModelrdinternvl_visionrTr)r: attentionsct|t|trl|jj|j|jj|j |jjdSdSt|trT|j j |j j |jj |j j dSdS)zInitialize the weightsN)r# _init_weightsrrrdatazero_rrrrfill_rdrr)r+rIr.s r/rz+InternVLVisionPreTrainedModel._init_weightss f%%% f6 7 7 K   ! ' ' ) ) ) ,!&,,...)5*/557777765  3 4 4 K O & &t{'I J J J O & &t{'I J J J J J K Kr0)rCrDrEr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrrc_can_record_outputsrrFrGs@r/rrs    )$O&*#./N"&-- K K K K K K K K Kr0rc eZdZdeddffd ZdZede d dej d e ej de e effd ZxZS) InternVLVisionModelrdrNcNt|||_t||_t ||_|jrtj ntj |j |j |_ |dS)Nr)r#r$rdrrrencoderuse_mean_poolingr%ry LayerNormr,r layernorm post_initrs r/r$zInternVLVisionModel.__init__s     26::,V44 $4 uBKMMM",vGY_e_t:u:u:u  r0c|jjSr)rrrAs r/get_input_embeddingsz(InternVLVisionModel.get_input_embeddingss //r0F)tie_last_hidden_statesrrc|||\}}||}|d}||}t||j|jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, num_patches)`, *optional*): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). )rr)rr:r)rrrrr:r)r+rrembedding_outputrencoder_outputssequence_outputs r/r=zInternVLVisionModel.forwardsr#oolOo\\!,,'788)!,..993-)7&1    r0r)rCrDrErr$rrrr'rrrrr?rr=rFrGs@r/rrs 3       000u5557;  l "%"23  u:: ;    ^65     r0rc:eZdZUeed<dZdZdZdZdZ dZ dZ dZ dS)InternVLPreTrainedModelrdTpast_key_valuesN) rCrDrErr r r _skip_keys_device_placementrr_can_compile_fullgraphrrrr0r/r#r#sM &*#"3N!"&r0r#c*eZdZdeffd ZdZxZS)InternVLMultiModalProjectorrdcttj|jjt d|jz dzz|_tj |jjt d|jz dzz|j j|_ t|j |_tj |j j|j j|_dS)Nrr)r#r$r%r vision_configr,rdownsample_ratiorrr text_configlinear_1rprojector_hidden_actactlinear_2rs r/r$z$InternVLMultiModalProjector.__init__s ,v';'G#aRXRiNiJjJjnoJo'opp  ,s1v7N3N/O/OST/T TV\VhVt   &56 &"4"@&BTB`aa r0c||}||}||}||}|Sr)rr.r0r1)r+image_featuresr:s r/r=z#InternVLMultiModalProjector.forwardsL77  m44 //  m44 r0)rCrDrErr$r=rFrGs@r/r)r)sZb~bbbbbbr0r)zM Base class for InternVL outputs, with hidden states and attentions. c8eZdZUdZdZeejed<dS)InternVLModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states) rCrDrErr6rr' FloatTensorr rr0r/r5r5s7  8<%"34;;;;;r0r5zx The InternVL model which consists of a vision backbone and a language model, without a language modeling head. c2eZdZddiZdeffd ZdZdZdZdZ dd e j d e e eeefd e efd Zde jde j de j fdZee dde e jd e e j de e jde e jde ede e j d e e eeefd e ede e jdeede eeffdZdde jdefdZxZS) InternVLModelzlanguage_model.modellanguage_modelrdct|tj|j|_t ||_tj|j|_ | dSr) r#r$r from_configr+ vision_towerr)multi_modal_projectorr-r:rrs r/r$zInternVLModel.__init__sm    %1&2FGG%@%H%H"'3F4FGG r0c4|jSr)r:rrAs r/rz"InternVLModel.get_input_embeddingss"77999r0c:|j|dSr)r:set_input_embeddingsr+rLs r/rAz"InternVLModel.set_input_embeddingss 0077777r0c||_dSrr:r+decoders r/ set_decoderzInternVLModel.set_decoder!s%r0c|jSrrDrAs r/ get_decoderzInternVLModel.get_decoder$s ""r0Nrvision_feature_layervision_feature_select_strategyc l||n |jj}||n |jj}||j}|jj}|dkr||j}n!||j |}|dkr|ddddddf}|j d}t|dz}|j d} | | ||d}| || }| | d|j d}||}|S) a% Obtains image last hidden states from the vision tower and apply multimodal projection. Args: pixel_values (`torch.FloatTensor]` of shape `(batch_size, channels, height, width)`) The tensors corresponding to the input images. vision_feature_layer (`int` or `list[int]`): Layer index or list of layer indices to extract features from. Returns: vision_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`. N)r4r2)rdefaultrrr) scale_factor)rdrJrKr5r4r,r=r vision_modelr:r@rr pixel_shuffler>) r+rrJrKr[r,vision_featureschannels feature_sizers r/get_image_featuresz InternVLModel.get_image_features'sh&%9$D $+Jj  .9 + *; ' $TZ88 ;7 2 % %"//\/JJ\OO"//\/JJXYmnO )Y 6 6-aaaQQQh7O#(+8S=)) $*1- *11*lLZ\]],,_K[,\\*11*b/BWXZB[\\44_EEr0 input_ids inputs_embedsr3c\|e||tj|jjtj|jk}|d}n||jjk}|}| d | |j}|j d|j dz}|| | krtd|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. N)r4devicer2rrz6Image features and image tokens do not match: tokens: z , features )rr'tensorrdimage_token_idlongrXallsumr expand_asr5r@numelrl)r+rUrVr3special_image_maskn_image_tokensn_image_featuress r/get_placeholder_maskz"InternVLModel.get_placeholder_mask]s/  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H +//11/99"==GG VVYYZgZnoo)/2^5I!5LL + , 2 2 4 48L8L8N8N N Nvvvdtvv "!r0rM position_idsr%cache_positionr[rc ||n |jj}||n |jj}|du|duz rtd|||}|f||||} | |j|j} | ||| } | | | }|j d||||| d| } t| j | j| j| j|| ndS)Nz:You must specify exactly one of input_ids or inputs_embedsrrJrK)rVr3)rMrdr%rVre)rr%r:rr6r)rdrJrKrlrrTr5rXr4rcmasked_scatterr:r5rr%r:r)r+rUrrMrdr%rVrJrKrer[r3r`outputss r/r=zInternVLModel.forwardusx %9$D $+Jj  .9 + *; '  -t"; < [YZZ Z  7D5577 BBM  #!44)%9/M5N ,..}/C]EXYYN!%!:!:~";"" *889K^\\M%$% )%+')      +%7#3!/)2>2JPT     r0rrQrNc (|\}}}}||zdks ||zdkrtd|||t||zt||z }|dddd}||t||zt||zt||dzz }|dddd}|S)a&Perform pixel shuffle downsampling on vision features. Args: vision_features (`torch.Tensor`): Input tensor of shape (batch_size, width, height, channels). scale_factor (`float`, *optional*, defaults to `0.5`): Factor by which to downsample. Default is 0.5, which halves the dimensions. Returns: vision_features (`torch.Tensor`): Downsampled tensor of shape (batch_size, height*scale_factor, width*scale_factor, channels/(scale_factor^2)). rzKHeight and width must be divisible by scale_factor for proper downsampling.rrr)rrlrrrrZ)r+rQrNrrrrRs r/rPzInternVLModel.pixel_shuffles)/>.B.B.D.D+ E68 L A % %)=)B)Bjkk k*.. s6L#8993x,?V;W;W  *11!Q1==HHJJ*.. F\122C 8L4M4MsS[_kmn_nSoOpOp   *11!Q1==HHJJr0NN) NNNNNNNNN)r) rCrDrE_checkpoint_conversion_mappingrr$rrArGrIr'r7rrrliststrrT LongTensorrcrrrr rrr?r5r=floatrPrFrGs@r/r9r9 sY '=>N%O"~:::888&&&### AE8< 44'4'uS$s)^'<=4)1 4444l")":?:K"]b]n""""015481537+/59@D8<597 7 E,-7 u017 !. 7 u/0 7 "% 7   127 'uS$s)^'<=7 )1 7 !!127 +,7  u11 27 7 7 ^7 r!!U\!!!!!!!!!r0r9zT Base class for InternVL causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<dZee ejed<dZeejed<dS) InternVLCausalLMOutputWithPasta4 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nlosslogitsr%r:rr6)rCrDrErrsrr'r7r rtr%r r:r?rr6rr0r/rrrrs  )-D(5$ %,,,*.FHU& '...'+OXe_+++8rAs r/r>z6InternVLForConditionalGeneration.multi_modal_projector,s z//r0rrUrMrdr%rVlabelsrelogits_to_keep image_sizesr[c  ||n |jj}||n |jj}|jd||||||||| | d | }|d}t | t rt | dn| }||dd|ddf}d}| |jd|| |jj j d| }t|||j |j |j|jS)ac Example: ```python >>> import torch >>> from transformers import AutoProcessor, AutoModelForImageTextToText >>> torch_device = "cuda" >>> processor = AutoProcessor.from_pretrained("OpenGVLab/InternVL3-1B-hf") >>> model = AutoModelForImageTextToText.from_pretrained( ... "OpenGVLab/InternVL3-1B-hf", dtype=torch.bfloat16, device_map=torch_device ... ) >>> messages = [ ... { ... "role": "user", ... "content": [ ... { ... "type": "image", ... "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg", ... }, ... { ... "type": "image", ... "url": "https://thumbs.dreamstime.com/b/golden-gate-bridge-san-francisco-purple-flowers-california-echium-candicans-36805947.jpg", ... }, ... {"type": "text", "text": "These images depict two different landmarks. Can you identify them?"}, ... ], ... }, ... ] >>> inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt").to(torch_device) >>> generate_ids = model.generate(**inputs, max_new_tokens=200) >>> print(processor.decode(generate_ids[0, inputs["input_ids"].shape[1] :], skip_special_tokens=True)) The images depict the Statue of Liberty and the Golden Gate Bridge. ```N) rUrrMrdr%rVrJrKrerr)rtrrz)rsrtr%r:rr6r)rdrJrKryrrslicerw loss_functionr-rzrrr%r:rr6)r+rUrrMrdr%rVrJrKrrerrr[rir: slice_indicesrtrss r/r=z(InternVLForConditionalGeneration.forward0sOl%9$D $+Jj  .9 + *; ' $*  %)%+'!5+I)#        8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %4%f9P9[_eD.#3!/) ' ;     r0c jtj|f|||||d|} |ddkr|| d<| S)N)r%rVrMrerrr)r#prepare_inputs_for_generation) r+rUr%rVrrMrerr[ model_inputsr.s r/rz>InternVLForConditionalGeneration.prepare_inputs_for_generationsg=uww<  +')))      !  ! !,8L (r0rk) NNNNNNNNNNrN)NNNNNN)&rCrDrErl_tied_weights_keysrr$rrAr%Moduler~rGrIr'r7rrrrmrnrTpropertyr:r=r>rrrorr rrr?rrr=rrFrGs@r/rvrvs"8-"?#, &&" ++~ 111///ry(((((( AE8<   '  'uS$s)^'<=  )1     ))X)''X'00X015481537+/59@D8<-15934.2\ \ E,-\ u01\ !. \ u/0 \ "% \   12\ 'uS$s)^'<=\ )1 \ )*\ !!12\ c5</0\ el+\ +,\  u44 5\ \ \ ^\ Br0rv)rrr#r9rv)rH)Bcollections.abcr dataclassesrtypingrrrr'torch.nnr% activationsr cache_utilsr generationr integrationsr modeling_layersr modeling_outputsr rrmodeling_utilsrrprocessing_utilsrutilsrrrrr utils.genericrautorconfiguration_internvlrrrr rrprarcrrrrrrrrrrr#r)r5r9rrrv__all__rr0r/rs.!!!!!!,,,,,,,,,, !!!!!! ))))))777777999999ddddddddddFFFFFFFF&&&&&&aaaaaaaaaaaaaa//////HHHHHHHHY''JJJJJBIJJ('J6%% I% <% % < % U\* %  %%%%%6F$F$F$F$F$biF$F$F$R  +E   !7!7!7!7!7BI!7!7!7L[7[7[7[7[7ry[7[7[7|        3H I I+++++4+++\     BI   &KKKKKOKKK<' ' ' ' ' 7' ' ' T ' ' ' ' 'o ' ' '")$  < < < < <"9 < <   < AAAAA+AA  AH  <<<<<[<<   <0 uuuuu'>uu  up   r0