`i0|dZddlmZddlmZddlmZmZmZddl Z ddl m Z m Z ddl m Z dd lmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZddlm Z m!Z!m"Z"erddl#m$Z$ej%e&Z'de j de j fdZ(de j de j fdZ)eeGddeZ*de de fdZ+de de fdZ,dZ-dZ.eedGd d!eZ/eed"Gd#d$eZ0Gd%d&e j1Z2Gd'd(e j1Z3Gd)d*e j1Z4Gd+d,e j1Z5Gd-d.eZ6eGd/d0eZ7Gd1d2e j1Z8Gd3d4e j1Z9Gd5d6e7Z:Gd7d8e j1Z;Gd9d:e7Ze j1Z>Gd?d@e j1Z?GdAdBe7Z@gdCZAdS)DzPyTorch OWLv2 model.) dataclass) lru_cache)AnyOptionalUnionN)Tensornn)ACT2FN) _create_4d_causal_attention_mask_prepare_4d_attention_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel) ModelOutputauto_docstringfilter_out_non_signature_kwargsis_vision_availablelogging torch_int) Owlv2ConfigOwlv2TextConfigOwlv2VisionConfig)center_to_corners_formatlogitsreturnctj|tjt ||jS)Ndevice)r functional cross_entropytorcharangelenr!)rs |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/owlv2/modeling_owlv2.pycontrastive_lossr(3s3 = & &vu|CKKPVP]/^/^/^ _ __ similaritycrt|}t|}||zdz S)Ng@)r(t)r* caption_loss image_losss r' owlv2_lossr/8s4#J//L!*,,..11J : % ,,r)ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eed<dZeed <d eefd ZdS) Owlv2Outputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Owlv2VisionModel`]. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`Owlv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Owlv2VisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputrc^tfdDS)Nc3tK|]2}|dvr|n!t|V3dS)r7r8Ngetattrto_tuple.0kselfs r' z'Owlv2Output.to_tuple..^c   LLLDGGRYZ^`aRbRbRkRkRmRm      r)tuplekeysrBs`r'r>zOwlv2Output.to_tuple]C    YY[[      r))__name__ __module__ __qualname____doc__r2rr$ FloatTensor__annotations__r3r4r5r6r7rr8rFrr>r)r'r1r1>s()-D(5$ %,,,48hu0188837OXe/0777/3K%+,33304L(5,-4444818886:3::: %*      r)r1r,c|r5|jtjtjfvr|n|S|jtjtjfvr|n|SN) is_floating_pointdtyper$float32float64floatint32int64int)r,s r'_upcastr[eseGG u}===qq17799LG U[999qqquuwwFr)boxesct|}|dddf|dddfz |dddf|dddfz zS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrr r)r[)r\s r'box_arear_nsT ENNE !!!Q$K%1+ %%1+aaad *C DDr)ct|}t|}tj|dddddf|ddddf}tj|dddddf|ddddf}||z d}|dddddf|dddddfz}|dddf|z|z }||z } | |fS)Nr^rminr)r_r$maxrbclamp) boxes1boxes2area1area2left_top right_bottom width_heightinterunionious r'box_iouros V  E V  Ey4! ,fQQQUm<.rDr)rErHs`r'r>z#Owlv2ObjectDetectionOutput.to_tuplerIr))rJrKrLrMr2rr$rNrOrzdictrr{r|r5r6r}r7rr8rFrr>rPr)r'ryrys">)-D(5$ %,,, $Ix~$$$*.FHU& '...59x 12999.2J*+222/3K%+,33304L(5,-44404L(5,-4444818886:3::: %*      r)ryzL Output type of [`Owlv2ForObjectDetection.image_guided_detection`]. c eZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZeed <dZeed <d eefd ZdS) %Owlv2ImageGuidedObjectDetectionOutputa logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image embeddings for each patch. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWLv2 represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`Owlv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Owlv2VisionModel`]. Nrr6query_image_embedstarget_pred_boxesquery_pred_boxesr}r7r8rc^tfdDS)Nc3tK|]2}|dvr|n!t|V3dSr;r<r?s r'rCzAOwlv2ImageGuidedObjectDetectionOutput.to_tuple..rDr)rErHs`r'r>z.Owlv2ImageGuidedObjectDetectionOutput.to_tuple rIr))rJrKrLrMrrr$rNrOr6rrrr}r7rr8rFrr>rPr)r'rrs8+/FHU& '...04L(5,-4446:!23:::59x 1299948hu0188804L(5,-4444818886:3::: %*      r)rczeZdZdeffd ZdejdededejfdZd d ej d e dejfd Z xZ S) Owlv2VisionEmbeddingsconfigcbt|j|_||_|j|_t jtj |j|_ t j |j |j|j|jd|_ |j|jzdz|_|jdz|_t j|j|j|_|dtj|jdddS)NF) in_channels out_channels kernel_sizestridebiasr^r position_idsr persistent)super__init__ patch_sizer hidden_size embed_dimr Parameterr$randnclass_embeddingConv2d num_channelspatch_embedding image_size num_patches num_positions Embeddingposition_embeddingregister_bufferr%expandrBr __class__s r'rzOwlv2VisionEmbeddings.__init__s  + +!|EK8J,K,KLL!y+)$    #-1BBqH!-1"$,t/A4>"R"R ^U\$:L-M-M-T-TU\-]-]jopppppr) embeddingsheightwidthrc|jddz }|jjd}|jddz }tjs&||kr ||kr||jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} | d| | | }| dddd}tj || | fdd }| dddddd| }t j||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrg?r r^bicubicF)sizemode align_cornersdim)shaperweight unsqueezer$jit is_tracingrrrreshapepermuter r" interpolateviewcat) rBrrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r'interpolate_pos_encodingz.Owlv2VisionEmbeddings.interpolate_pos_encoding+s!&q)A- !4;EEaHH*03a7 y##%% >+*F*F6UZ??**4+<== =,QQQU3,QQQU3r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCr)F pixel_valuesrcv|j\}}}}||}|ddd}|j|dd}t j||gd} |r| || ||z} n| | |j z} | S)Nr^rrr) rrflatten transposerrr$rrrr) rBrr batch_size_rr patch_embedsr}rs r'forwardzOwlv2VisionEmbeddings.forwardTs'3'9$ Avu++L99 #++A..88A>> +22:q"EE Y l;CCC # Q#d&C&CJPVX]&^&^^JJ#d&=&=d>O&P&PPJr)F) rJrKrLrrr$rrZrrNboolr __classcell__rs@r'rrsq0qqqqqq*'D5<'D'DUX'D]b]i'D'D'D'DR  E$5 QU bgbn        r)rc eZdZdeffd Z d deejdeejdeejdej fdZ xZ S) Owlv2TextEmbeddingsrc\ttj|j|j|_tj|j|j|_| dtj |j dddS)NrrFr) rrr r vocab_sizertoken_embeddingmax_position_embeddingsrrr$r%rrs r'rzOwlv2TextEmbeddings.__init__ds !|F,=v?QRR"$,v/MvOa"b"b  EL)GHHOOPWXXej      r)N input_idsr inputs_embedsrc| |jdn |jd}||jddd|f}|||}||}||z}|S)Nr)rrrr)rBrrr seq_lengthposition_embeddingsrs r'rzOwlv2TextEmbeddings.forwardns -6,AY_R((}GZ[]G^  ,QQQ  ^ )T^ ; ;'dn''N''' ]D( / i?? i?? i??  $.$.AA r)tensorseq_lenbszc||||j|jddS)Nrr^)rrrr contiguous)rBrrrs r'_shapezOwlv2Attention._shapes<{{3GGQQRSUVWWbbdddr)NF hidden_statesattention_maskcausal_attention_maskoutput_attentionsrc|\}}}|||jz}|||d|} |||d|} ||jzd|jf} ||||j| }| j| } | j| } | d} tj || dd} | ||jz|| fkr2td||jz|| fd| |||d|| fkr+td|d|| fd|| ||j|| |z} | ||jz|| } |||d|| fkr+td|d|| fd|| ||j|| |z} | ||jz|| } tj| d} |r=| ||j|| }|||jz|| } nd}tj| |j|j }|| j}tj || }|||jz||jfkr5td ||j||jfd||||j||j}| dd}||||}||}||fS) z#Input shape: Batch x Time x Channelrrr^z$Attention weights should be of size z , but is Nz!Attention mask should be of size r)ptrainingz `attn_output` should be of size )rrrrrrrrrr$bmmrrrr r"softmaxrrtorTrr)rBrrrrrtgt_lenr query_states key_states value_states proj_shapesrc_len attn_weightsattn_weights_reshaped attn_probs attn_outputs r'rzOwlv2Attention.forwardsF#0"4"4"6"6Wi{{=11DJ> [[]!;!;REE {{4;;}#=#=r3GG DN*B > Ct{{<#>>CZP $Z_j1 (|(*5 //!$$yz/C/CAq/I/IJJ     3#7'"J J J*dn8LgW^7_** %%''**  ! ,$))++Q/III 7a'8R77-224477(,,S$.'7SSVkkL',,S4>-A7GTTL  %""$$a'(BBB ta'8Rtt]k]p]p]r]rtt(,,S$.'7SSVddL',,S4>-A7GTTL},,\r,BB  ) %1$5$5c4>7T[$\$\ !055cDN6JGU\]]LL$( !]**<4 O2 u|Xel3XeEL>Q5RR S O2O2O2O2O2O2O2O2r)rcBeZdZfdZdejdejfdZxZS)Owlv2MLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSrR) rrrr hidden_act activation_fnr rrintermediate_sizefc1fc2rs r'rzOwlv2MLP.__init__sf  #F$569V/1IJJ9V5v7IJJr)rrc||}||}||}|SrR)r r r)rBrs r'rzOwlv2MLP.forwards=// **=99 // r))rJrKrLrr$rrrrs@r'rrscKKKKKU\elr)rc eZdZdeffd Z d dejdejdejdeede ej f d Z xZ S) Owlv2EncoderLayerrcDt|j|_t ||_t j|j|j|_ t||_ t j|j|j|_ dSNeps) rrrrr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rs r'rzOwlv2EncoderLayer.__init__s +'//<F & u ! &&&&&&&&r)rc>eZdZUeed<dZdZdgZdej fdZ dS)Owlv2PreTrainedModelrowlv2TrmodulecX|jj}t|trT|jjjd|dz|jjjd|dznt|trtj |j d|j dz|ztj |jj|jj|ztj |jj|jj|zn@t|t r|j dzd|jjzdzz|z}|j dz|z}tj |jj|tj |jj|tj |jj|tj |jj|nPt|t,r|jjdzd|jjzdzz|z}d|jjzdz|z}tj |jj|tj |jj|nt|t4rtj |jj|jdz|ztj |jj|jdz|z|jj |jj!t|tj"r=|j#j$|jj dt|tj%rH|jjd||j#"|j#j$dSdSdS) zInitialize the weightsg{Gz?)meanstdr)r&r^?N)&rinitializer_factor isinstancerrrdatanormal_rrr initrrrinitializer_rangernum_hidden_layersrrrrrrr r Owlv2Modeltext_projectiontext_embed_dimvisual_projectionvision_embed_dim logit_scalefill_logit_scale_init_valuerrzero_r)rBr"factor in_proj_std out_proj_stdfc_stds r' _init_weightsz"Owlv2PreTrainedModel._init_weights9s/ f1 2 2 N  " ) . 6 6CVd] 6 S S S  % , 1 9 9sQU 9 V V V V  5 6 6 N GOOF2&BRTXBX[aBaO b b b GOOF29v}?^ag?gO h h h GOOF5<&-BadjBjO k k k k  / / N!+T1q6=;Z7Z_c6cdgmmK",d2fz)Owlv2Encoder.__init__..ns"$h$h$h1%6v%>%>$h$h$hr)F)rrr ModuleListranger.layersgradient_checkpointingrs `r'rzOwlv2Encoder.__init__lsY m$h$h$h$hfNfHgHg$h$h$hii &+###r)Nrrroutput_hidden_states return_dictrc\||n |jj}||n |jj}||n |jj}|rdnd}|rdnd}|} |jD]/} |r|| fz}| | |||} | d} |r || dfz}0|r|| fz}|st d| ||fDSt | ||S)a Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`). attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) causal_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Causal mask for the text model. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. NrP)rrrc3K|]}||V dSrRrP)r@vs r'rCz'Owlv2Encoder.forward..s(eeqWXWdWdWdWdWdeer))last_hidden_stater attentions)rrrJuse_return_dictrHrFr) rBrrrrrJrKencoder_statesall_attentionsr encoder_layer layer_outputss r'rzOwlv2Encoder.forwardqs?>2C1N--TXT_Tq$8$D $+Jj &1%>N fee]NN$Seeeee e+>Vd    r)NNNNN)rJrKrLrMrrrr$rrrrFrrrrs@r'rBrBcs,{,,,,,,268<,0/3&*? ? !.?  ( 5 ? $D> ? 'tn ? d^?  uo% &? ? ? ? ? ? ? ? r)rBceZdZdeffd Ze d dejdeejdeejdee dee d ee d e e e ffd Z xZS) Owlv2TextTransformerrct||_|j}t ||_t ||_tj ||j |_ dSr) rrrrrrrBencoderr rrfinal_layer_norm)rBrrrs r'rzOwlv2TextTransformer.__init__sf  & -f55#F++ " YF ? 'tn ? d^?  u00 1? ? ? ^? ? ? ? ? r)rXceZdZUeed<deffd ZdejfdZdZ e d de j de e j d e ed e ed e edeeeff d ZxZS)Owlv2TextModelrct|t||_|dSrR)rrrX text_model post_initrs r'rzOwlv2TextModel.__init__s@    .v66 r)rc$|jjjSrRrfrrrHs r'get_input_embeddingsz#Owlv2TextModel.get_input_embeddings s)99r)c(||jj_dSrRri)rBvalues r'set_input_embeddingsz#Owlv2TextModel.set_input_embeddings s5:"222r)NrrrrJrKc6||||||S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> from transformers import AutoProcessor, Owlv2TextModel >>> model = Owlv2TextModel.from_pretrained("google/owlv2-base-patch16") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrrJrK)rf)rBrrrrJrKs r'rzOwlv2TextModel.forwards.<)/!5#    r))NNNN)rJrKrLrrOrr r@rjrmrr$rrrrrFrrrrs@r'rdrds   :bi::::;;;26,0/3&* # # <# !.# $D> # 'tn # d^ #  u00 1# # # ^# # # # # r)rdceZdZdeffd Ze d dejdee dee dee d ee d e e e ff d Z xZS) Owlv2VisionTransformerrc:t||_t||_t j|j|j|_ t||_ t j|j|j|_ dSr) rrrrrr rrr pre_layernormrBrZpost_layernormrs r'rzOwlv2VisionTransformer.__init__9s~  /77\&*<&BWXXX#F++  l6+=6CXYYYr)NFrrrJrrKrc||n |jj}||n |jj}||n |jj}|jjjj}||}|||}| |}| ||||}|d} | dddddf} | | } |s| | f|ddzSt| | |j |jS)N)r)rrrJrKrrr])rrrJrQrrrrTrrsrZrtrrrP) rBrrrJrrKexpected_input_dtyperrarOrbs r'rzOwlv2VisionTransformer.forwardBs?2C1N--TXT_Tq$8$D $+Jj &1%EK#';<<  Oghh **=99 ,,'/!5# '  ,A.)!!!Q'2 ++M::  L%}58KK K)/')7&1     r))NNFN)rJrKrLrrrr$rNrrrrFrrrrs@r'rqrq8sZ0ZZZZZZ-1/338&* ) ) ') $D>) 'tn ) #+4. ) d^ )  u00 1) ) ) ^) ) ) ) ) r)rqceZdZUeed<dZdeffd ZdejfdZ e d de e j de ed e ed ed e edeeeff d ZxZS)Owlv2VisionModelrrct|t||_|dSrR)rrrq vision_modelrgrs r'rzOwlv2VisionModel.__init__tsA    26:: r)rc$|jjjSrR)rzrrrHs r'rjz%Owlv2VisionModel.get_input_embeddingszs +;;r)NFrrJrrKc6||||||S)a Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Owlv2VisionModel >>> model = Owlv2VisionModel.from_pretrained("google/owlv2-base-patch16") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrJrrK)rz)rBrrrJrrKs r'rzOwlv2VisionModel.forward}s06  %/!5%=# !   r)NNNFN)rJrKrLrrOmain_input_namerr r@rjrrr$rNrrrFrrrrs@r'rxrxps $O0   'tn  #'  d^   u00 1    ^      r)rxceZdZUeed<deffd Zee ddej de ej dej fdZ ee dd ej d e dej fd Ze dde ejd e ej de ej d e e d e e de e d e de e de e deeeffdZxZS)r/rct|t|jts%t dt |jdt|jts%t dt |jd|j}|j}|j |_ |j |_ |j |_ t||_t||_t#j|j |j d|_t#j|j |j d|_t#jt-j|j|_|dS)NzLconfig.text_config is expected to be of type Owlv2TextConfig but is of type .zPconfig.vision_config is expected to be of type Owlv2VisionConfig but is of type F)r)rrr) text_configr TypeErrortype vision_configrprojection_dimrr1r3rXrfrqrzr rr2r0rr$rr6r4rg)rBrrrrs r'rzOwlv2Model.__init__sy    &,o>> 0+,,000  &.0ABB 2-..222  ( , $3)5 - 9.{;;2=AA!#4+@$BU\a!b!b!b!y)QX]^^^< V5R(S(STT r)Nrrrch|||}||j}|S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`]. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```)rr)rfr0r^)rBrr text_outputs text_featuress r'get_text_featureszOwlv2Model.get_text_featuress6>48??Ygu?3v3v ,,\-GHH r)Frrch|||}||j}|S)av Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Owlv2VisionModel`]. Examples: ```python >>> import torch >>> from transformers.image_utils import load_image >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```rr)rzr2r^)rBrrvision_outputsimage_featuress r'get_image_featureszOwlv2Model.get_image_featuressC86:5F5F%%=6G6 6 //0LMMr) return_lossrrJreturn_base_image_embedsrKc 2||n |jj}||n |jj}| | n |jj} |||||| } |||||| } | d} || } | d} || } | tj | dddz } | tj | dddz }|j | j}tj|| |z}|}d}|rt#|}|} | s||| | | | f}||f|zn|St%|||| | | | S) a4 return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. return_base_image_embeds (`bool`, *optional*): Whether or not to return the base image embeddings. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```Nr}rorr^rT)ordrkeepdim)r2r3r4r5r6r7r8)rrrJrQrzrfr0r2r$linalgnormr4exprr!matmulr,r/r1)rBrrrrrrJrrrKrrr5r6text_embeds_normr4r4r3r2outputs r'rzOwlv2Model.forwardsF2C1N--TXT_Tq$8$D $+Jj &1% Z 'tnZ #'Z #+4.Z d^Z  uk! "Z Z Z ^Z Z Z Z Z r)r/cNeZdZddedeffd ZdejdejfdZ xZ S) Owlv2BoxPredictionHeadrout_dimc,t|jj}t j|||_t j|||_t j|_ t j|||_ dSrR) rrrrr rdense0dense1GELUgeludense2)rBrrrrs r'rzOwlv2BoxPredictionHead.__init__nsn $0iu-- iu-- GII iw// r)rrc||}||}||}||}||}|SrR)rrrr)rBrrs r'rzOwlv2BoxPredictionHead.forwardws\^,,6""V$$6""V$$ r))r) rJrKrLrrZrr$rrNrrrs@r'rrmsw00{0S000000elu7Hr)rc eZdZdeffd Zdejdeejdeejde ejfdZ xZ S)Owlv2ClassPredictionHeadrclt|jj}|jj|_t j|j||_t j|jd|_ t j|jd|_ t j |_ dS)Nr) rrrrr query_dimr rr logit_shiftr4ELUelu)rBrrrs r'rz!Owlv2ClassPredictionHead.__init__s $0-9i88 9T^Q779T^Q77688r)r6 query_embeds query_maskrc ||}|L|j}|jdd\}}tj|||jf|}||fS|tj|dddzz }|tj|dddzz }tj d||}| |} | |} | | dz} || z| z}|v|j dkrtj|d }tj|d ktj|jj|}|tj}||fS) Nr^rT)rrgư>z...pd,...qd->...pqrrrr)rr!rr$zerosrrrreinsumrr4rndimrwherefinforTrbrU) rBr6rrimage_class_embedsr!rr pred_logitsrr4s r'rz Owlv2ClassPredictionHead.forwards "[[66  '.F&8&>rr&B #J +z;&OPPSSTZ[[K!34 405<3D3DEW]_im3D3n3nqu3uv#u|'8'82W['8'\'\_c'cd l#79K\ZZ &&|44 &&|44 hh{++a/ "[0K?  !"""_ZR@@@ +jAou{;CT7U7U7Y[fggK%..77K/00r)) rJrKrLrrr$rNrrrFrrrs@r'rrs {      !1'!1u01!1U\* !1 u ! !1!1!1!1!1!1!1!1r)rceZdZUeed<deffd Zedededej fdZ dej dej fdZ e d  d#deded eej dej fd Z d$dej d ej dedej fdZ d%dej deej deej deej fdZ d&dej dej dej deedeededeej fdZ d&dej deedeededeej f dZ d$dej dej dedej fdZe d'dej deej deedeeded eedefd!Ze d'dej dej deej deedeeded eedefd"ZxZS)(Owlv2ForObjectDetectionrct|t||_t ||_t ||_t |d|_tj |j j |j j |_tj|_||_|jj j|jj jz|_|jj j|jj jz|_||j|j|_|dS)Nr)rr)rrr/r!r class_headrbox_headobjectness_headr rrrr layer_normSigmoidsigmoidrrrnum_patches_heightnum_patches_widthcompute_box_biasbox_biasrgrs r'rz Owlv2ForObjectDetection.__init__s    '' 26::.v66 5faHHH,v';'GVMaMpqqqz||  "&+";"F$+JcJn"n!%!:!EIbIm!m--d.EtG]^^  r)rrrcftjd|dztj}tjd|dztj}tj||d\}}tj||fd}|dxx|zcc<|dxx|zcc<|dd }|S) Nr)rTxy)indexingrr.r.rr^)r$r%rUmeshgridstackr)rr x_coordinates y_coordinatesxxyybox_coordinatess r'!normalize_grid_corner_coordinatesz9Owlv2ForObjectDetection.normalize_grid_corner_coordinatess Q(9A(=U]SSS  Q(:Q(>emTTT  }tLLLB +r2hB777#44#55*..r155r)rch|}||}|d}|S)a#Predicts the probability that each image feature token is an object. Args: image_features (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_dim)`)): Features extracted from the image. Returns: Objectness scores. r)detachr)rBrr{s r'objectness_predictorz,Owlv2ForObjectDetection.objectness_predictors:(..00 00@@-f5  r)r^)maxsizeN feature_mapc|td|||}tj|dd}tj|dztj| dzz }tj|d}|dxx|zcc<|dxx|zcc<tj|dztj| dzz }tj||gd}|S) NzOfeature_map has been deprecated as an input. Please pass in num_patches insteadr$r'g-C6?rrrr)rrrr$cliploglog1p full_liker) rBrrrrbox_coord_biasbox_size box_size_biasrs r'rz(Owlv2ForObjectDetection.compute_box_biass  "noo o@@ASUfgg*_c3???T#9::U[/IY\`I`=a=aa?>377--.. (T/22U[(TAQ5R5RR 9nm<"EEEr)F image_featsrc||}|r#|j\}}}}|||}n|j}||j}||z }||}|S)a Args: image_feats: Features extracted from the image, returned by the `image_text_embedder` method. feature_map: A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method. interpolate_pos_encoding: Whether to interpolate the pre-trained position encodings. Returns: pred_boxes: List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary. )rrrrrr!r) rBrrrr|rrrrs r' box_predictorz%Owlv2ForObjectDetection.box_predictors&]];//  $ %:E:K 7A!#4a,,-?ARSSHH}H;;{122h \\*-- r)rrc>||||\}}||fS)a8 Args: image_feats: Features extracted from the `image_text_embedder`. query_embeds: Text query embeddings. query_mask: Must be provided with query_embeddings. A mask indicating which query embeddings are valid. )r)rBrrrrrs r'class_predictorz'Owlv2ForObjectDetection.class_predictor!s,-1OOKWa,b,b)(/00r)rrrrrJc T|||||||d}|r5|j\}}} } | |jjjz} | |jjjz} n|j} |j} |jd} |jj | }tj |ddddddf|ddddfj}|ddddddf|z}| |}|jd| | |jdf}| |}|d}|||fS)NT)rrrrrJrrKrrr)r!rrrrrrr8rzrtr$ broadcast_torr)rBrrrrrJrrrrrrrrOr6class_token_outnew_sizer5s r'image_text_embedderz+Owlv2ForObjectDetection.image_text_embedder5sz**%)/!5%=   $ 7"."4 Aq&%!'4;+D+O!O  %)B)M M  !%!8  $ 6 $7:z.==>OPP  ,\!!!RaR(-C\RSRSRSUXVXUXRXEYE_``$AAAqrr111H-? |44   q !    r "   $++H55 bk \733r)c:|j||d}|r5|j\}}}}||jjjz} ||jjjz} n|j} |j} |d} |jj| } tj | ddddddf| ddddfj} | ddddddf| z} | | } | jd| | | jdf}| |} | |fS)NT)rrrKrrr) r!rzrrrrrrrtr$rrr)rBrrrJrrrrrrrrOr6rrs r'image_embedderz&Owlv2ForObjectDetection.image_embedderisd00%@Xfj1   $ 7"."4 Aq&%!'4;+D+O!O  %)B)M M  !%!8  $ 6 +1-z.==>OPP  ,\!!!RaR(-C\RSRSRSUXVXUXRXEYE_``$AAAqrr111H-? |44   q !    r "   $++H55 n--r)query_image_featuresquery_feature_mapc||\}}||||}t|}g}g} |j} t |jdD]Q} t jgdg| } || } t| | \}}t j |ddkrt| | }t j |dz}|d|k }| r|| |d}t j|| d}t jd||}|t j|}||| || |S|r)t j|}t j| }nd \}}|||fS) Nr)rrrrr r$g?r)axiszd,id->iNN)rrrr!rGrr$rrorqrvrcnonzeronumelsqueezer%rargminappendr)rBrrrrr}r|pred_boxes_as_cornersbest_class_embedsbest_box_indicespred_boxes_deviceieach_query_boxeach_query_pred_boxesious iou_threshold selected_indsselected_embeddings mean_embedsmean_sim best_box_indr box_indicess r'embed_image_queryz)Owlv2ForObjectDetection.embed_image_querys ../CDD<''(<>OQijj 8 D D18+1!455 6 6A"\<<<.ARSSSN$9!$< !n.CDDGD!yaC(( R*>;PQQ"IdOOc1M!!W 5>>@@M""$$ 6&21om6K6KA6N6N&O##jaqAAA  < ;@STT,U\(-C-CD !((a)FGGG '' 555  3 ;'899L+&677KK(2 %L+[*44r)query_pixel_valuesrKc ||n |jj}||n |jj}||n |jj}|||d}|||||\}} |j\} } } } t j|| | | z| f}|j\} } } } t j|| | | z| f}||||\}}}| ||\}}| |||}|s6||||||| f}td|D}|St||||||d| S)a query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values of query image(s) to be detected. Pass in one query image per target image. Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, Owlv2ForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg" >>> query_image = Image.open(requests.get(query_url, stream=True).raw) >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt") >>> # forward pass >>> with torch.no_grad(): ... outputs = model.image_guided_detection(**inputs) >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_image_guided_detection( ... outputs=outputs, threshold=0.9, nms_threshold=0.3, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image >>> boxes, scores = results[i]["boxes"], results[i]["scores"] >>> for box, score in zip(boxes, scores): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}") Detected similar object with confidence 0.938 at location [327.31, 54.94, 547.39, 268.06] Detected similar object with confidence 0.959 at location [5.78, 360.65, 619.12, 366.39] Detected similar object with confidence 0.902 at location [2.85, 360.01, 627.63, 380.8] Detected similar object with confidence 0.985 at location [176.98, -29.45, 672.69, 182.83] Detected similar object with confidence 1.0 at location [6.53, 14.35, 624.87, 470.82] Detected similar object with confidence 0.998 at location [579.98, 29.14, 615.49, 489.05] Detected similar object with confidence 0.985 at location [206.15, 10.53, 247.74, 466.01] Detected similar object with confidence 0.947 at location [18.62, 429.72, 646.5, 457.72] Detected similar object with confidence 0.996 at location [523.88, 20.69, 586.84, 483.18] Detected similar object with confidence 0.998 at location [3.39, 360.59, 617.29, 499.21] Detected similar object with confidence 0.969 at location [4.47, 449.05, 614.5, 474.76] Detected similar object with confidence 0.966 at location [31.44, 463.65, 654.66, 471.07] Detected similar object with confidence 0.924 at location [30.93, 468.07, 635.35, 475.39] ```Nrr)rrrJr)rrc3K|]}||V dSrRrPr@xs r'rCzAOwlv2ForObjectDetection.image_guided_detection..(">> 1 >>r))r6rrrrr}r7r8)rrrJrKrrr$rrrrr>rFr)rBrrrrJrrKrrrrrr hidden_dimrquery_image_featsrrrrr}rrs r'image_guided_detectionz.Owlv2ForObjectDetection.image_guided_detectionsv2C1N--TXT_Tq$8$D $+Jj &1%PSd>dfp1qrr HYH_E &(9:!M  ,>AR,RT^_  <@;Q;Q 02J< < 8 &(8 '+&:&:{am&:&n&n#l!..{KIabb !! ''))F>>f>>>>>FM4$0/-%" .    r)c ||n |jj}||n |jj}||n |jj}|||||||\}} } | j} | j} | j\} }}}tj | | ||z|f}|jd| z}| | ||jd}| | ||jd}|ddk}| |||\}}| |}| || |}|sI||||| || | f}td|D}|St| |||||| | S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids). output_hidden_states (`bool`, *optional*): Whether or not to return the last hidden state. See `text_model_last_hidden_state` and `vision_model_last_hidden_state` under returned tensors for more detail. Examples: ```python >>> import requests >>> from PIL import Image >>> import torch >>> from transformers import Owlv2Processor, Owlv2ForObjectDetection >>> processor = Owlv2Processor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text_labels = [["a photo of a cat", "a photo of a dog"]] >>> inputs = processor(text=text_labels, images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.tensor([(image.height, image.width)]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_grounded_object_detection( ... outputs=outputs, target_sizes=target_sizes, threshold=0.1, text_labels=text_labels ... ) >>> # Retrieve predictions for the first image for the corresponding text queries >>> result = results[0] >>> boxes, scores, text_labels = result["boxes"], result["scores"], result["text_labels"] >>> for box, score, text_label in zip(boxes, scores, text_labels): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected {text_label} with confidence {round(score.item(), 3)} at location {box}") Detected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35] Detected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13] ```N)rrrrrJrrrrc3K|]}||V dSrRrPr s r'rCz2Owlv2ForObjectDetection.forward..r r))r6r5r|rr{r}r7r8)rrrJrKrr7r8rr$rrrrr>rFry)rBrrrrrJrrKrrrrrrrrr rmax_text_queriesrrr}r{r|rs r'rzOwlv2ForObjectDetection.forward6s h2C1N--TXT_Tq$8$D $+Jj &1%PSd>dfp1qrr %?1-;#++J8H,J\]_J`aa %%j2BIOTVDWXX v&* '+&:&:; V`&a&a#l!55kBB'' [BZ[[  !%%'''')) F>>f>>>>>FM)$$!/%* .    r)rRrrrr~)rJrKrLrrOr staticmethodrZr$rrrNrrrrrrrFrrrrrrrryrrrs@r'rrsI {$cVY^c^j\ !53D !IZ ! ! ! !Yqko"%:=LTUZUfLg 6*/ &&#'    J59-1 11&1u011U\* 1 u ! 11112-1/3).1414<14'14 14 $D> 14 'tn 14#'14 u !14141414n-1/3). (.(.'(.$D>(.'tn (. #' (. u ! (.(.(.(.^*/ *5*5#/*5!,*5#' *5   *5*5*5*5X;?,0/3).&*s s 's %U%67s $D> s 'tn s #' s d^s  /s s s ^s j 26,0/3).&*r r <r 'r !. r $D> r 'tn r #'r d^r  $r r r ^r r r r r r)r)r/r rdrxr)BrM dataclassesr functoolsrtypingrrrr$rr activationsr modeling_attn_mask_utilsr r modeling_layersrmodeling_outputsrrmodeling_utilsrutilsrrrrrrconfiguration_owlv2rrrtransformers.image_transformsr get_loggerrJloggerr(r/r1r[r_rorvryrr@rrrrrr rBrXrdrqrxr/rrr__all__rPr)r'r!s*!!!!!!'''''''''' !!!!!!dddddddd999999KKKKKKKK------QPPPPPPPPPGFFFFFF  H % %`U\`el```` -5<-EL----  ! ! ! ! ! +! !  ! JGvG&GGGG EF Ev E E E E"   "'''0  / / / / / / /   / d  * * * * * K* *  * \JJJJJBIJJJ\")@h2h2h2h2h2RYh2h2h2X     ry    /////2///d,),),),),)?,),),)`M M M M M 29M M M bI I I I I 29I I I Z3 3 3 3 3 )3 3 3 n4 4 4 4 4 RY4 4 4 p. . . . . +. . . bF F F F F %F F F TRY(-1-1-1-1-1ry-1-1-1`x x x x x 2x x x v r q qr)