`idZddlZddlZddlmZddlmZmZddl Z ddl m Z ddl m Z ddl mZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZmZdd lmZm Z ddl!m"Z"e j#e$Z%eedGddeZ&Gdde j'Z(Gdde j'Z)Gdde j'Z*Gdde j'Z+Gdde j'Z,Gdde j'Z-Gd d!e j'Z.Gd"d#e j'Z/Gd$d%eZ0Gd&d'e j'Z1eGd(d)eZ2eGd*d+e2Z3Gd,d-e j'Z4ed.Gd/d0e2Z5Gd1d2e j'Z6Gd3d4e j'Z7ed5Gd6d7e2Z8ed8Gd9d:e2Z9ed;Gd<d=e2Z:eGd>d?e2Z;gd@ZBM8Du'8!9:;BBB;?JeE$5678?????r*rc4eZdZdZfdZddZ ddZxZS) ViltEmbeddingsz Construct the text and patch embeddings. Text embeddings are equivalent to BERT embeddings. Patch embeddings are equivalent to ViT embeddings. ctt||_t jt jdd|j|_ t||_ |j j }t jt jd|dz|j|_ t j|j|j|_t j|j|_||_dSNr)super__init__TextEmbeddingstext_embeddingsr Parameterr$zeros hidden_size cls_tokenViltPatchEmbeddingspatch_embeddings num_patchesposition_embeddings Embeddingmodality_type_vocab_sizetoken_type_embeddingsDropouthidden_dropout_probdropoutconfig)selfrBr: __class__s r+r1zViltEmbeddings.__init__Ns  .f55ek!Q8J&K&KLL 3F ; ;+7 #%< A{QPVPb0c0c#d#d %'\&2QSYSe%f%f"z&"<==  r*c   !"#|jjjj\}}}}||}|dddddddf}t j||jd|jdf}|dddf ddddf} |dddf ddddf} |j\} } #|j j |j j z} |j ddddddfddd| | | !t!j!#fdt%| | Dd}|ddd}|ddd}t!jt+t!j|jdt!j|jd d d |j }|ddddddddf}||jd|jdd d d }|dd}|d}dkst5t6s| | z}|n'| | z}t;||d "d|z d  "dddf}"fd|D} fd|D}d|D}d|D}fd|D}g}tAt%|||D]\}\}}}|dkr[t!j!t!j"|}|#|||jt!j!t!j"||d}|#t!j|||||gdt!j|d}||dddf|dddff| d | }||dddf|dddff| d }||dddf|dddff| d d}||dddf|dddff| d | }|j$| d d }t!j||fd}t!j|j dddddfdddddf| d d |fd}||z}|%|}t!jt!j"|jdd||gd}|||#fffS)Nrsizerrdimc g|]R\}}tjtj||fddd|z d|z fSS)bilinearT)rImode align_cornersr)r functionalpad interpolate).0hwheight spatial_poswidths r+ z/ViltEmbeddings.visual_embed..js   Aq !!M--#V'&* .  1fqj1   r*ij)indexingdeviceF)as_tuplec<g|]}dddf|kSNrr))rSu valid_idxs r+rYz/ViltEmbeddings.visual_embed..s/NNNQ9QQQT?a#78NNNr*c<g|]}dddf|kSrbr))rSrc non_valid_idxs r+rYz/ViltEmbeddings.visual_embed..s0ZZZ]=A+>!+CDZZZr*c8g|]}|dSrrHrSvs r+rYz/ViltEmbeddings.visual_embed..s"777AaffQii777r*c8g|]}|dSrhrHris r+rYz/ViltEmbeddings.visual_embed..s"???!&&))???r*cg|]}|z Sr)r))rSrjmax_image_lengths r+rYz/ViltEmbeddings.visual_embed..s===Q$q(===r*T) replacement)&r9 projectionweightshapefloatrrPrRlongsumrB image_size patch_sizer; transposeviewr$catzipflattenstackrarangetor_expand isinstanceintmaxminnonzerounique enumerate multinomialonesappendr7rA)$rC pixel_values pixel_maskrm_phpwxx_maskx_hx_w batch_size num_channels patch_dim pos_embed patch_indexeffective_resolution unique_rows valid_row_idxnon_valid_row_idx valid_numsnon_valid_numspad_numsselectirjnvp valid_choice pad_choice cls_tokensrVrfrWrdrXs$ ` @@@@@r+ visual_embedzViltEmbeddings.visual_embed]s,7>D 1b"  ! !, / /AAAtQQQM*0022**6QWQZ8P*QQVVXXQQQTl1%%aaad+QQQTl1%%aaad+23'/ L&%K*dk.DD .qqq!""aaax8BB1aHHMMaQ]_hjstt I       SMM      %%a((221a88 IIaLL " "1a ( (k U\&,r"233U\&,rBR5S5S^b c c cik   "FM" " " "$aaaAAA"56 !((a&,q/2rSUVV !))!Q// "" a  #3#;:N^`cCdCd#; $'9 37799  #&9 "#7#;#;#=#=?OPP NNEN22 V,,e,<< 1o,,.. NNNN+NNN ZZZZkZZZ77777 ??->???====*===&s:~x'P'PQQ f fMAz2qAvv$0A1D1D1F1FHXYY  mA.|<====".uz"~~/C/C/E/EqVZ[[[  eiq)9;LQ;OPZ;[(\bcdddeeee6q))) fQQQTlF111a4L( ) . .z2| L Lqqq!t fQQQTl2388RHH!&A,qqq!t "<=BB:rSTUU fQQQTlF111a4L89>>z2|\\ ^**:r2>> Iz1o1 - - -I  %aaaAAAg .qqq$z : A A*bRT U UW` agh     M LLOOEJv|A::==fEEvNTUVVV&;888r*rc ||||} |'||||jj\}} } n|d} |d}| |t j|t j| j z} ||t j | |t j| j z}t j | |gd} t j || gd} | | fS)N) input_idstoken_type_ids inputs_embeds)rmrdtyper_rJ) r3rrBrmr{r>r$ zeros_likersr_ full_likery)rCrattention_maskrrrr image_embedsimage_token_type_idx text_embeds image_masksr embeddingsmaskss r+forwardzViltEmbeddings.forwards0**m+   595F5Fj4;;W6G66 2L+{{%,,Q//K '#$ !D$>$>  ^5:kFX Y Y Y% %  $d&@&@ OK)=UZXcXj k k k' '  Y \:BBB  >;7Q???5  r*)rE)r)r r!r"r#r1rr __classcell__rDs@r+r-r-Est     V9V9V9V9B'!'!'!'!'!'!'!'!r*r-c*eZdZdZfdZddZxZS)r2zGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_t#|dd|_|dt)j|jdd|d t)j|jt(j ddS) N) padding_idxepsposition_embedding_typeabsolute position_ids)rr[F) persistentrr)r0r1rr< vocab_sizer6 pad_token_idword_embeddingsmax_position_embeddingsr;type_vocab_sizer> LayerNormlayer_norm_epsr?r@rAgetattrrregister_bufferr$r}rr5rrIrsrCrBrDs r+r1zTextEmbeddings.__init__sK !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTTz&"<== '.v7PR\']']$  EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      r*Nc||}n|dd}|d}||jddd|f}|mt|dr2|jddd|f}||d|}|}n+t j|t j|jj}|| |}| |} || z} |j dkr| |} | | z } | | } || } | S)Nr[rrrrr)rIrhasattrrrr$r5rsr_rr>rr;rrA) rCrrrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr>rr;s r+rzTextEmbeddings.forwardsb  #..**KK',,..ss3K ^  ,QQQ  ^ J J"%::  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r*)NNNNr r!r"r#r1rrrs@r+r2r2sRQQ     &        r*r2c(eZdZdZfdZdZxZS)r8z# Image to Patch Embedding. ct|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_||_||_||_ tj |||||_ dS)Nrr) kernel_sizestride)r0r1rurvrr6r collectionsabcIterabler:rConv2dro)rCrBrurvrr6r:rDs r+r1zViltPatchEmbeddings.__init__s !'!2F4EJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&)L+:^hiiir*c|j\}}}}||jkrtd|jjj}|||}|S)NzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.r)rqr ValueErrorrorprr~)rCrrrrVrX target_dtypers r+rzViltPatchEmbeddings.forward,si2>2D/ L&% 4, , ,w -3 OOLOO,O?? @ @r*rrs@r+r8r8sV j j j j jr*r8c&eZdZfdZddZxZS)ViltSelfAttentionct|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|j |_ tj |j|j|j |_ tj |j|j|j |_tj|j|_dS)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)r0r1r6num_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluer?attention_probs_dropout_probrArs r+r1zViltSelfAttention.__init__8s.    : :a ? ?PVXhHiHi ?76#5773777  $*#= #&v'9F>,,77  - 9O _kBB %--aAq99DDFF "/"4"4"6"6ss";t?Q>S"S* *,CD 6G]=/22mM]r*NNFr r!r"r1rrrs@r+rr7sQGGGGG$,,,,,,,,r*rcZeZdZdZdeffd ZdejdejdejfdZxZ S)ViltSelfOutputz The residual connection is defined in ViltLayer instead of here (as is the case with other models), due to the layernorm applied before each block. rBcttj|j|j|_tj|j|_dSN) r0r1rrr6denser?r@rArs r+r1zViltSelfOutput.__init__sJ Yv163EFF z&"<== r*r input_tensorreturncZ||}||}|SrrrArCrrs r+rzViltSelfOutput.forwards* =11  ]33 r*) r r!r"r#rr1r$Tensorrrrs@r+rrzs >z>>>>>> U\RWR^r*rc,eZdZfdZdZddZxZS) ViltAttentionctt||_t ||_t |_dSr)r0r1r attentionroutputset pruned_headsrs r+r1zViltAttention.__init__sI *622$V,, EEr*ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)NrrrJ)lenrrrrrrrrrrrrunion)rCheadsindexs r+ prune_headszViltAttention.prune_headss u::?? F7 4>5t~7Y[_[l  u 2$.2FNN/0BEJJ1$.2FNN.t{/@%QOOO .2^-ORUV[R\R\-\*'+~'IDNLn'n$ -33E::r*NFc|||||}||d|}|f|ddz}|S)Nrr)rr)rCrrrr self_outputsattention_outputrs r+rzViltAttention.forwardsM~~m^YPabb ;;|A FF#% QRR(88r*r)r r!r"r1rrrrs@r+rrs[""""" ;;;$r*rcHeZdZdeffd ZdejdejfdZxZS)ViltIntermediaterBcttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r0r1rrr6intermediate_sizerr hidden_actstrr intermediate_act_fnrs r+r1zViltIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r*rrcZ||}||}|Sr)rr#rCrs r+rzViltIntermediate.forwards, =11 00?? r* r r!r"rr1r$r rrrs@r+rrsj9z999999U\elr*rcVeZdZdeffd ZdejdejdejfdZxZS) ViltOutputrBcttj|j|j|_tj|j|_ dSr) r0r1rrr r6rr?r@rArs r+r1zViltOutput.__init__sJ Yv79KLL z&"<== r*rrrcd||}||}||z}|Srr r s r+rzViltOutput.forwards4 =11  ]33 % 4 r*r&rs@r+r(r(su>z>>>>>> U\RWR^r*r(c*eZdZdZfdZddZxZS) ViltLayerz?This corresponds to the Block class in the timm implementation.czt|j|_d|_t ||_t ||_t||_ tj |j |j |_tj |j |j |_dS)Nrr)r0r1chunk_size_feed_forward seq_len_dimrrr intermediater(rrrr6rlayernorm_beforelayernorm_afterrs r+r1zViltLayer.__init__s '-'E$&v..,V44 (( " V-?VEZ [ [ [!|F,>FDYZZZr*NFcH||||||}|d}|dd}|||jz}||}||}|||}|f|z}|S)N)rrr)rr1r~r_r2r0r) rCrrrrself_attention_outputsrr layer_outputs r+rzViltLayer.forwards!%  ! !- 0 0  / "0" "  2!4(,)=+;+;.s!#_#_#_!If$5$5#_#_#_r*F) r0r1rBr ModuleListrangenum_hidden_layerslayergradient_checkpointingrs `r+r1zViltEncoder.__init__s`  ]#_#_#_#_uVE]?^?^#_#_#_`` &+###r*NFTc.|rdnd}|rdnd}t|jD]=\} } |r||fz}||| nd} | ||| |} | d}|r || dfz}>|r||fz}|std|||fDSt|||S)Nr)rrc3K|]}||V dSrr)ris r+ z&ViltEncoder.forward..s(mmq_`_l_l_l_l_lmmr*)last_hidden_staterr)rr=r(r ) rCrrrroutput_hidden_states return_dictall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r+rzViltEncoder.forwards #7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO(LYjkkM)!,M  P&9]1=M>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r*N) r r!r"rr&base_model_prefixsupports_gradient_checkpointing_no_split_modulesrVr)r*r+rKrKsI &*#)+>?*****r*rKceZdZdfd ZdZdZdZe ddee j dee j d ee j d ee j d ee j d ee j d ee j dee j dee dee dee dee deeee j ffdZxZS) ViltModelTcJt|||_t||_t ||_tj|j |j |_ |rt|nd|_ |dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)r0r1rBr-rr7encoderrrr6r layernorm ViltPoolerpooler post_init)rCrBadd_pooling_layerrDs r+r1zViltModel.__init__7s     (00"6** f&8f>STTT,=Gj(((4  r*c$|jjjSrrr3rrCs r+get_input_embeddingszViltModel.get_input_embeddingsHs.>>r*c(||jj_dSrrd)rCrs r+set_input_embeddingszViltModel.set_input_embeddingsKs:?'777r*c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr]r=rr)rCheads_to_pruner=rs r+ _prune_headszViltModel._prune_headsNsU +0022 C CLE5 L u % / ; ;E B B B B C Cr*NrrrrrrrrrrrCrDrc \| | n |jj} | | n |jj} | | n |jj} ||t d|+||||} n.||dd} nt d| \}}||jn|j}|tj ||f|}||t d||t d| |j dn |j d}||krt d |-tj ||jj |jj f|}| ||jj }||||||||| \}}||| }||||| | | }|d}||}|j||nd}| s||f|d dzSt'|||j|j S)ak image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. image_token_type_idx (`int`, *optional*): - The token type ids for images. Examples: ```python >>> from transformers import ViltProcessor, ViltModel >>> from PIL import Image >>> import requests >>> # prepare image and text >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "hello world" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltModel.from_pretrained("dandelin/vilt-b32-mlm") >>> inputs = processor(image, text, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer[z5You have to specify either input_ids or inputs_embedsr^zFYou cannot specify both pixel_values and image_embeds at the same timez7You have to specify either pixel_values or image_embedsrzAThe text inputs and image inputs need to have the same batch size)r)rrrrCrDr)rB pooler_outputrr)rBrrCuse_return_dictr%warn_if_padding_and_no_attention_maskrIr_r$rrqru get_head_maskr<rget_extended_attention_maskr]r^r`r rr)rCrrrrrrrrrrrCrDrtext_batch_sizerr_image_batch_sizeembedding_outputextended_attention_maskencoder_outputssequence_output pooled_outputs r+rzViltModel.forwardVsT2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U&1#%.%:!! @T  !"Z/:)FPVWWWN  # (@eff f  !l&:VWW W4@4L<-a00R^RdefRg  . .`aa a  %5t{7Mt{Oe$fouvvvJ&&y$+2OPP +/??       !5,; , , (.150P0PQ_al0m0m,, 2/!5# '  *!,..998< 8O O444UY  J#]3oabb6II I)-')7&1     r*)T NNNNNNNNNNNN)r r!r"r1rfrhrlrrr$ LongTensorr%rboolrr r(rrrs@r+r[r[5s"???@@@CCC156:594815155948.2,0/3&*t t E,-t !!23t !!12 t u01 t U-. t E-.t   12t u01t 'smt $D>t 'tnt d^t  )51B+CC Dt t t ^t t t t t r*r[c$eZdZfdZdZxZS)r_cttj|j|j|_tj|_dSr)r0r1rrr6rTanh activationrs r+r1zViltPooler.__init__sC Yv163EFF '))r*cr|dddf}||}||}|Srb)rr)rCrfirst_token_tensorrys r+rzViltPooler.forwards@+111a40 #566  66 r*rrs@r+r_r_sG$$$$$ r*r_zU ViLT Model with a language modeling head on top as done during pretraining. ceZdZddgZfdZdZdZe ddee j dee j d ee j d ee j d ee j d ee j d ee j dee j dee j dee dee dee de eee j ffdZxZS)ViltForMaskedLMzmlm_score.decoder.weightzmlm_score.decoder.biasct|t||_t ||_|dSr)r0r1r[rL ViltMLMHead mlm_scorerars r+r1zViltForMaskedLM.__init__sQ    f%% $V,, r*c|jjSr)rdecoderres r+get_output_embeddingsz%ViltForMaskedLM.get_output_embeddingss ~%%r*c@||j_|j|j_dSr)rrr)rCnew_embeddingss r+set_output_embeddingsz%ViltForMaskedLM.set_output_embeddingss!/,1r*NrrrrrrrrlabelsrrCrDrc V| | n |jj} |||||||||| | |  } | dd\}}| |jdn |jd}|ddd|f|dd|df}}||}d}| et }| |j} ||d|jj | d}| s|f| ddz}||f|zn|St||| j | j S)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (*torch.LongTensor* of shape *(batch_size, sequence_length)*, *optional*): Labels for computing the masked language modeling loss. Indices should be in *[-100, 0, ..., config.vocab_size]* (see *input_ids* docstring) Tokens with indices set to *-100* are ignored (masked), the loss is only computed for the tokens with labels in *[0, ..., config.vocab_size]* Examples: ```python >>> from transformers import ViltProcessor, ViltForMaskedLM >>> import requests >>> from PIL import Image >>> import re >>> import torch >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "a bunch of [MASK] laying on a [MASK]." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-mlm") >>> model = ViltForMaskedLM.from_pretrained("dandelin/vilt-b32-mlm") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> tl = len(re.findall("\[MASK\]", text)) >>> inferred_token = [text] >>> # gradually fill in the MASK tokens, one by one >>> with torch.no_grad(): ... for i in range(tl): ... encoded = processor.tokenizer(inferred_token) ... input_ids = torch.tensor(encoded.input_ids) ... encoded = encoded["input_ids"][0][1:-1] ... outputs = model(input_ids=input_ids, pixel_values=encoding.pixel_values) ... mlm_logits = outputs.logits[0] # shape (seq_len, vocab_size) ... # only take into account text features (minus CLS and SEP token) ... mlm_logits = mlm_logits[1 : input_ids.shape[1] - 1, :] ... mlm_values, mlm_ids = mlm_logits.softmax(dim=-1).max(dim=-1) ... # only take into account text ... mlm_values[torch.tensor(encoded) != 103] = 0 ... select = mlm_values.argmax().item() ... encoded[select] = mlm_ids[select].item() ... inferred_token = [processor.decode(encoded)] >>> selected_token = "" >>> encoded = processor.tokenizer(inferred_token) >>> output = processor.decode(encoded.input_ids[0], skip_special_tokens=True) >>> print(output) a bunch of cats laying on a couch. ```N rrrrrrrrrCrDrGrr[rrrr) rBrorLrqrrr~r_rxrr rr)rCrrrrrrrrrrrCrDrrxry text_seq_len text_featuresr mlm_logitsmasked_lm_lossloss_fctrs r+rzViltForMaskedLM.forwardsR&1%o 'tno d^o  ~uU%677 8o o o ^o o o o o r*rc$eZdZfdZdZxZS)ViltPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dS)Nr) r0r1rrr6rrr!r"r transform_act_fnrrrs r+r1z$ViltPredictionHeadTransform.__init__is Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr*c||}||}||}|Sr)rrrr%s r+rz#ViltPredictionHeadTransform.forwardrs= =11 --m<< }55 r*rrs@r+rrhsLUUUUUr*rc,eZdZdfd ZdZdZxZS)rNcht||_t||_t j|j|jd|_ t j tj |j|_ | ||j _|j |j _ dS)NFr)r0r1rBr transformrrr6rrr4r$r5rrp)rCrBrprDs r+r1zViltMLMHead.__init__zs  4V<<y!3V5FUSSS LV->!?!?@@  "(DL !I r*c(|j|j_dSr)rrres r+ _tie_weightszViltMLMHead._tie_weightss I r*cZ||}||}|Sr)rr)rCrs r+rzViltMLMHead.forwards' NN1   LLOOr*r)r r!r"r1rrrrs@r+rrys[ & & & & & &&&&r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for visual question answering, e.g. for VQAv2. ceZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e e ejffdZ xZS)ViltForQuestionAnsweringc t||j|_t||_t jt j|j|jdzt j |jdzt j t j|jdz|j|_ | dS)NrG) r0r1 num_labelsr[rLr Sequentialrr6rGELU classifierrars r+r1z!ViltForQuestionAnswering.__init__s     +f%% - If(&*>> from transformers import ViltProcessor, ViltForQuestionAnswering >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = "How many cats are there?" >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa") >>> # prepare inputs >>> encoding = processor(image, text, return_tensors="pt") >>> # forward pass >>> outputs = model(**encoding) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: 2 ```NrrrGr)rBrorLrnrr~r_rrP binary_cross_entropy_with_logitsrqrrr)rCrrrrrrrrrrrCrDrrnrrrs r+rz ViltForQuestionAnswering.forwardsb&1%T 'tnT d^T  'u/@)AA BT T T ^T T T T T r*rz Vilt Model transformer with a classifier head on top (a linear layer on top of the final hidden state of the [CLS] token) for image-to-text or text-to-image retrieval, e.g. MSCOCO and F30K. ceZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e e ejffdZ xZS)ViltForImageAndTextRetrievalct|t||_t j|jd|_|dSr/) r0r1r[rLrrr6 rank_outputrars r+r1z%ViltForImageAndTextRetrieval.__init__s[    f%% 9V%7;; r*NrrrrrrrrrrrCrDrc B| | n |jj} d} | td|||||||||| | |  }| r|jn|d}||}| s|f|ddz}| | f|zn|St | ||j|jS)ad image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels are currently not supported. Examples: ```python >>> from transformers import ViltProcessor, ViltForImageAndTextRetrieval >>> import requests >>> from PIL import Image >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"] >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> model = ViltForImageAndTextRetrieval.from_pretrained("dandelin/vilt-b32-finetuned-coco") >>> # forward pass >>> scores = dict() >>> for text in texts: ... # prepare inputs ... encoding = processor(image, text, return_tensors="pt") ... outputs = model(**encoding) ... scores[text] = outputs.logits[0, :].item() ```NzTraining is not yet supported.rrrGr) rBroNotImplementedErrorrLrnrrrr)rCrrrrrrrrrrrCrDrrrnrrs r+rz$ViltForImageAndTextRetrieval.forwardsZ&1%M 'tnM d^M  'u/@)AA BM M M ^M M M M M r*rzq Vilt Model transformer with a classifier head on top for natural language visual reasoning, e.g. NLVR2. ceZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e e ejffdZ xZS)"ViltForImagesAndTextClassificationc t||j|_t||_|j}t jt j|j |z|j |zt j |j |zt j t j|j |z|j|_ | dSr)r0r1rr[rL num_imagesrrrr6rrrra)rCrBrrDs r+r1z+ViltForImagesAndTextClassification.__init__hs     +f%% & - If(:5v7IJ7V W W L+j8 9 9 GII If(:5v7H I I    r*NrrrrrrrrrrrCrDrc | | n |jj} | | n |jj} | | n |jj} | |jdkr|d}| |jdkr|d}| |jdnd} | | |jdnd} | |jjkrtdg}| rgnd}| rgnd}t| D]}| |||||dd|ddddddfnd||dd|ddddfnd||||dd|ddddfnd|dz| | |  }| r|j n|d}| || r| |j | r| |jtj|d}||}d}| `t%}| |j} ||d|j| d}| s|||f}||f|zn|St/|||| S) a3 image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Binary classification labels. Examples: ```python >>> from transformers import ViltProcessor, ViltForImagesAndTextClassification >>> import requests >>> from PIL import Image >>> image1 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_0.jpg", stream=True).raw) >>> image2 = Image.open(requests.get("https://lil.nlp.cornell.edu/nlvr/exs/ex0_1.jpg", stream=True).raw) >>> text = "The left image contains twice the number of dogs as the right image." >>> processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> model = ViltForImagesAndTextClassification.from_pretrained("dandelin/vilt-b32-finetuned-nlvr2") >>> # prepare inputs >>> encoding = processor([image1, image2], text, return_tensors="pt") >>> # forward pass >>> outputs = model(input_ids=encoding.input_ids, pixel_values=encoding.pixel_values.unsqueeze(0)) >>> logits = outputs.logits >>> idx = logits.argmax(-1).item() >>> print("Predicted answer:", model.config.id2label[idx]) Predicted answer: True ```Nrrz\Make sure to match the number of images in the model with the number of images in the input.) rrrrrrrrrrCrDr[rJr)rBrrCrondim unsqueezerqrrr;rLrnrrrr$ryrrr~r_rxrr)rCrrrrrrrrrrrCrDrpooler_outputsrrrrrnryrrrrs r+rz*ViltForImagesAndTextClassification.forwardzs^2C1N--TXT_Tq$8$D $+Jj &1%(>'11!44L  # (9Q(>(>'11!44L.:.F\'**D  2>2J+A..PTJ / / /n 2< ,6RR$ z"" 6 6Aii--p 'tnp d^p  7u?P9QQ Rp p p ^p p p p p r*rceZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e e ejffdZ xZS)ViltForTokenClassificationc:t||j|_t|d|_t j|j|_t j |j |j|_ | dS)NF)rb) r0r1rr[rLrr?r@rArr6rrars r+r1z#ViltForTokenClassification.__init__s~     +f>>> z&"<== )F$68IJJ r*NrrrrrrrrrrrCrDrc L| | n |jj} |||||||||| | |  } | d}| |jdn |jd}||}||ddd|f}d}| `t }| |j} || d|j | d}| s|f| ddz}||f|zn|St||| j | j S)a/ image_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`, *optional*): Optionally, instead of passing `pixel_values`, you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `pixel_values` into patch embeddings. labels (`torch.LongTensor` of shape `(batch_size, text_sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrr[rGr)rBrorLrqrArrr~r_rxrrrr)rCrrrrrrrrrrrCrDrrxtext_input_sizerrrrs r+rz"ViltForTokenClassification.forwardsa0&1% > E,-> !!23> !!12 > u01 > U-. > E-.>   12> u01> )*> $D>> 'tn> d^>  $eE,=&>> ?> > > ^> > > > > r*r)rrrrrr,r[rK)=r#collections.abcrr dataclassesrtypingrrr$rtorch.nnr activationsr modeling_layersr modeling_outputsr r r rrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_viltr get_loggerr loggerrModuler-r2r8rrrrr(r,r7rKr[r_rrrrrrr__all__r)r*r+rs6 !!!!!!"""""""" %%%%%%!!!!!!999999.-----[[[[[[[[[[,,,,,,,,******  H % %  @@@@@{@@   @$W!W!W!W!W!RYW!W!W!t66666RY666r")>????? ???FRY"BIF     ry           #####*###L) ) ) ) ) ")) ) ) X*****/***.U U U U U #U U U p         C C C C C )C C  C L")""), g g g g g 2g g  g T Z Z Z Z Z #6Z Z  Z z D D D D D )<D D  D NL L L L L !4L L L ^   r*