`iwddlZddlmZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZmZmZdd lmZmZdd lmZdd lmZmZmZddl m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&ddl'm(Z(m)Z)ddl*m+Z+m,Z,m-Z-ddl.m/Z/m0Z0m1Z1e%j2e3Z4Gdde j5Z6e#GddeZ7 dHde j5dej8dej8dej8deej8de9de9fd Z:Gd!d"e j5Z;Gd#d$e j5Z<Gd%d&eZ=Gd'd(e j5Z>Gd)d*e7Z?Gd+d,e j5Z@Gd-d.e j5ZAGd/d0e j5ZBGd1d2e j5ZCGd3d4e j5ZDGd5d6eZEGd7d8e j5ZFGd9d:e j5ZGGd;dGd?d@e!ZIe#dA>GdBdCe7ZJe#dD>GdEdFe7e ZKgdGZLdS)IN) dataclass)AnyCallableOptionalUnion)nn)ACT2FN)GenerationMixin)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsBaseModelOutputWithPooling,BaseModelOutputWithPoolingAndCrossAttentions)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging torch_int)OutputRecordercheck_model_inputs) AutoModelAutoModelForCausalLMAutoModelForSeq2SeqLM)InstructBlipVideoConfigInstructBlipVideoQFormerConfigInstructBlipVideoVisionConfigczeZdZdeffd ZdejdededejfdZd d ej d e dejfd Z xZ S) !InstructBlipVideoVisionEmbeddingsconfigct||_|j|_|j|_|j|_tjtj dd|j|_ tj d|j|j|j|_ |j|jzdz|_|jdz|_tjtj d|j|j|_dS)Nr$r ) in_channels out_channels kernel_sizestrider )super__init__r* hidden_size embed_dim image_size patch_sizer Parametertorchrandnclass_embeddingConv2dpatch_embedding num_patches num_positionsposition_embeddingselfr* __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/instructblipvideo/modeling_instructblipvideo.pyr1z*InstructBlipVideoVisionEmbeddings.__init__8s  + + +!|EK1dn,M,MNN!yDO\`\k   !Ot>1D!-1"$,u{1d>PRVR`/a/a"b"b embeddingsheightwidthreturnc|jddz }|jjddz }tjs||kr ||kr|jS|jddddf}|jddddf}|jd}||jz} ||jz} t |dz} |d| | |}|dddd}tj || | fdd }|dddd dd|}tj ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 r$Ng?rr r bicubicF)sizemode align_cornersdim)shaper>r7jit is_tracingr5rreshapepermuter functional interpolateviewcat) r@rDrErFr<r=class_pos_embedpatch_pos_embedrO new_height new_widthsqrt_num_positionss rBinterpolate_pos_encodingz:InstructBlipVideoVisionEmbeddings.interpolate_pos_encodingJsr!&q)A- /5a81< y##%% ++*F*F6UZ??* *1!!!RaR%81!!!QRR%8r"t. T_, &}c'9::)11!5GI[]`aa)11!Q1==-33 i( 4  *11!Q1==BB1b#NNy/?;CCCCrCF pixel_valuesr^c(|j\}}}}|jjj}|||}|ddd}|j|dd|} tj | |gd} |r| | ||} n|j } | | ddd| dddf|z} | S)Ndtyper r$rIrN)rPr;weightrbtoflatten transposer9expandr7rXr^r>rK) r@r_r^ batch_size_rErF target_dtype patch_embeds class_embedsrDr>s rBforwardz)InstructBlipVideoVisionEmbeddings.forwardrs'3'9$ Avu+28 ++LOO,O,O,OPP #++A..88A>> +22:q"EEHHVV Y l;CCC # 9!%!>!>z6SX!Y!Y  !%!8 "4QQQ8L*//!:L:L8Laaa5O"P"S"ST`"a"aa rCF) __name__ __module__ __qualname__r'r1r7Tensorintr^ FloatTensorboolrm __classcell__rAs@rBr)r)7sc<cccccc$&D5<&D&DUX&D]b]i&D&D&D&DP  E$5 QU bgbn        rCr)cDeZdZUeed<dZdZdZdZdZ dZ dZ gdZ dZ dS) InstructBlipVideoPreTrainedModelr*blipT)"InstructBlipVideoQFormerEmbeddingsInstructBlipVideoAttention*InstructBlipVideoQFormerMultiHeadAttention"InstructBlipVideoQFormerSelfOutputcl|jj}t|tjtjfrJ|jjd||j |j j dSdSt|tj r#|jjd|dSt|tj r?|j j |jj ddSt|trPtj|jd|tj|jd|dSt|t&t(fr |jj dSdS)zInitialize the weights)meanstdN?)r*initializer_range isinstancerLinearr:rcdatanormal_biaszero_ Embedding LayerNormfill_r)init trunc_normal_r>r9)InstructBlipVideoForConditionalGenerationInstructBlipVideoModel query_tokens)r@modulefactors rB _init_weightsz.InstructBlipVideoPreTrainedModel._init_weightss. fry")4 5 5 - M  & &CV & < < <{&  &&((((('&  - - - M  & &CV & < < < < <  - - - K  " " $ $ $ M  $ $S ) ) ) ) )  A B B - G ! !&";#6 ! R R R G ! !&"8s ! O O O O O !JLb c d d -   $ * * , , , , , - -rCN)rorprqr%__annotations__base_model_prefixsupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_no_split_modulesrrCrBryrysl ####&*#"&N!-----rCryrrquerykeyvalueattention_maskscalingdropoutc ztj||dd|z}|||z}tj|d}tj|||j}tj||} | dd} | |fS)NrIrN)ptrainingr$r ) r7matmulrfrrUsoftmaxrr contiguous) rrrrrrrkwargs attn_weights attn_outputs rBeager_attention_forwardrs<s}}R'<'<==GL!#n4 =((2(>>L=((6?([[L,|U33K''1--88::K  $$rCc eZdZdZfdZdejdedefdZ d dejd e ejd e eje eje e ejffd Z xZ S) r|z=Multi-headed attention from 'Attention Is All You Need' papercVt||_|j|_|j|_|j|jz|_|j|jz|jkr td|jd|jd|jdz|_ d|_ |j |_ tj |jd|jzd|_|jrWtjt#j|j}tjt#j|j}nd}d}|It#j|t#j|d|f}tj||j_tj |j|j|_dS) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gFr )r) requires_grad)r0r1r*r2r3num_attention_heads num_headshead_dim ValueErrorscale is_causalattention_dropoutrrqkvqkv_biasr6r7zerosrX zeros_liker projection)r@r*q_biasv_biasrrAs rBr1z#InstructBlipVideoAttention.__init__s  +3$.8 =4> )T^ ; ;'dn''N''' ]D( !'!99T^Q-?eLLL ? \%+dn"="=>>F\%+dn"="=>>FFFF  y&%*:6QV*W*W*WY_!`aaHL22DHM)DNDNCCrCtensorseq_lenbszc||||j|jddS)Nr$r )rWrrrfr)r@rrrs rB_shapez!InstructBlipVideoAttention._shapes<{{3GGQQRSUVWWbbdddrCN hidden_states head_maskrGc 2|\}}}||}|||d|j||jzddddd}|d|d|d} } }t } |jjdkrt|jj} | ||| | fd|j sdn|j |j d |\} } | ||d  } | | } | | fS) z#Input shape: Batch x Time x Channelr r rr$eagerNr)rrrrI)rKrrSrrTrr*_attn_implementationrrrrrr)r@rrrrtgt_lenr3 mixed_qkv query_states key_states value_statesattention_interfacerrs rBrmz"InstructBlipVideoAttention.forwardsC#0"4"4"6"6WiHH]++ %%c7At~yTXTbGbcckk q!Q  2;1y|YWX\,j (? ; +w 6 6"9$+:Z"[ $7$7     %  #}HCC$2HJ % %  % % ! \"))#w;;FFHH ook22 L((rCN) rorprq__doc__r1r7rrrsrrtuplermrvrws@rBr|r|sGGDDDDD>eU\eCeceeee -1$)$)|$)EL)$) u|Xel3XeEL>Q5RR S $)$)$)$)$)$)$)$)rCr|cBeZdZfdZdejdejfdZxZS)InstructBlipVideoMLPct||_t|j|_t j|j|j |_ t j|j |j|_ dSr) r0r1r*r hidden_act activation_fnrrr2intermediate_sizefc1fc2r?s rBr1zInstructBlipVideoMLP.__init__ sf  #F$569V/1IJJ9V5v7IJJrCrrGc||}||}||}|Sr)rrrr@rs rBrmzInstructBlipVideoMLP.forwards=// **=99 // rCrorprqr1r7rrrmrvrws@rBrr scKKKKKU\elrCrc veZdZdeffd Zedejdejdee dej fdZ xZ S)InstructBlipVideoEncoderLayerr*cDt|j|_t ||_t j|j|j|_ t||_ t j|j|j|_ dSNeps) r0r1r2r3r| self_attnrrlayer_norm_eps layer_norm1rmlp layer_norm2r?s rBr1z&InstructBlipVideoEncoderLayer.__init__s +3F;;<F '$     q &0  ((77 // %0 rC) rorprqr%r1rr7rrrrrtrmrvrws@rBrrsS6SSSSSS| +,    ^rCrc eZdZdZdeffd Ze d deej de e de e effdZxZS) InstructBlipVideoEncodera" Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`InstructBlipVideoEncoderLayer`]. Args: config (`InstructBlipVideoConfig`): The corresponding vision configuration for the `InstructBlipVideoEncoder`. r*ct|_tjfdt jD|_d|_dS)Nc.g|]}tSr)r).0rir*s rB z5InstructBlipVideoEncoder.__init__..Js"$t$t$tq%B6%J%J$t$t$trCF) r0r1r*r ModuleListrangenum_hidden_layerslayersgradient_checkpointingr?s `rBr1z!InstructBlipVideoEncoder.__init__Gsb  m$t$t$t$tTYZ`ZrTsTs$t$t$tuu &+###rCNrrrGc P|}|jD] }||fd|i|}t|S)Nrlast_hidden_state)rr)r@ inputs_embedsrrr encoder_layers rBrmz InstructBlipVideoEncoder.forwardMsZ& ![  M)M-MM ????rCr)rorprqrr%r1rrr7rrrrrrrrmrvrws@rBrr=s,6,,,,,, 26@@!.@+, @ uo% & @@@^@@@@@rCrc eZdZUdZeed<eedZdeffd Z e de d de e jded eed eeeffd Zd ZxZS)InstructBlipVideoVisionModelr_r*)r attentionsct|||_|j}t ||_t ||_tj ||j |_ | dSr) r0r1r*r2r)rDrencoderrrrpost_layernorm post_init)r@r*r3rAs rBr1z%InstructBlipVideoVisionModel.__init__gsx     & ;FCC/77  l9&:OPPP rCF)tie_last_hidden_statesNr^rrGc |td|||}|jdd|i|}|j}||}|dddddf}||}t ||S)Nz You have to specify pixel_values)r^rrr pooler_outputr)rrDrrrr)r@r_r^rrencoder_outputsr pooled_outputs rBrmz$InstructBlipVideoVisionModel.forwardrs  ?@@ @ Oghh +74<, , ', , ,  ,= //0ABB)!!!Q'2 ++M:: )/'    rCc|jSr)rDr@s rBget_input_embeddingsz1InstructBlipVideoVisionModel.get_input_embeddingss rC)NF)rorprqmain_input_namer'rrr|_can_record_outputsr1rrrr7rtrurrrrrrmr rvrws@rBrr_s$O ))))60 <      u55559).  u01 #' +,  u00 1    ^65 6rCrc`eZdZd fd ZdZdZdZdZdZ d d e e fd Z xZ S) r}Fct||_|j|jzdkr.t |dst d|j|jfz|j|_t|j|jz |_|j|jz|_ tj |j|j |_ |rItj |j |j |_tj |j |j |_nHtj |j|j |_tj |j|j |_tj|j|_t'|dd|_|jdks |jdkr6|j|_tjd|jzd z |j|_d |_dS) Nrembedding_sizezLThe hidden size (%d) is not a multiple of the number of attention heads (%d)position_embedding_typeabsolute relative_keyrelative_key_queryr r$F)r0r1r*r2rhasattrrrsattention_head_size all_head_sizerrrencoder_hidden_sizerrDropoutattention_probs_dropout_probrgetattrrmax_position_embeddingsrdistance_embeddingsave_attentionr@r*is_cross_attentionrAs rBr1z3InstructBlipVideoQFormerMultiHeadAttention.__init__s    : :a ? ?PVXhHiHi ?^%v'ABC  $*#= #&v'9F 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD ##rCc||_dSrattn_gradients)r@r%s rBsave_attn_gradientsz>InstructBlipVideoQFormerMultiHeadAttention.save_attn_gradientss,rCc|jSrr$r s rBget_attn_gradientsz=InstructBlipVideoQFormerMultiHeadAttention.get_attn_gradientss ""rCc||_dSr attention_map)r@r+s rBsave_attention_mapz=InstructBlipVideoQFormerMultiHeadAttention.save_attention_maps*rCc|jSrr*r s rBget_attention_mapzbhlrzbhrd,lrd->bhlrrNrr r )r2rrrr7rrfrrKarangelongr5rWrrrdrbeinsummathsqrtrrSoftmaxr r, register_hookr&rrTrr)r@rrrencoder_hidden_statesencoder_attention_maskrr" key_layer value_layermixed_query_layer query_layerattention_scores seq_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_scores_dtypeattention_probsattention_probs_dropped context_layernew_context_layer_shapes rBrmz2InstructBlipVideoQFormerMultiHeadAttention.forwardsx3$>  O11$((;P2Q2QRRI33DJJ?T4U4UVVK3NN11$((=2I2IJJI33DJJ}4M4MNNK JJ}55//0ABB !< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q&++--a0J"\*EJ}OcdddiijlnoppN"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|>AABXYY  D$"5 D  # #O 4 4 4  ) )$*B C C C#',,"?"?  &= &I # %S"S* *,CD o--rCrnNNNN) rorprqr1r&r(r,r.r2rrrmrvrws@rBr}r}s$$$$$$8---###+++"""%%%"# I.I.+,I.I.I.I.I.I.I.I.rCr}cPeZdZfdZdejdejdejfdZxZS)r~cttj|j|j|_tj|j|j|_tj|j |_ dSr) r0r1rrr2denserrrhidden_dropout_probrr?s rBr1z+InstructBlipVideoQFormerSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== rCr input_tensorrGc||}||}|||z}|SrrTrrr@rrVs rBrmz*InstructBlipVideoQFormerSelfOutput.forward@ =11  ]33 }|'CDD rCrrws@rBr~r~ i>>>>> U\RWR^rCr~ceZdZd fd ZdZ ddejdeejdeejdeejd eejd e e d ejfd Z xZ S)!InstructBlipVideoQFormerAttentionFctt|||_t ||_t |_dSr)r0r1r} attentionr~outputset pruned_headsr!s rBr1z*InstructBlipVideoQFormerAttention.__init__sL CFL^__8@@ EErCct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrr$rN)lenrr_rrrbrrrrr`rTrunion)r@headsindexs rB prune_headsz-InstructBlipVideoQFormerAttention.prune_heads"s u::?? F7 4>5t~7Y[_[l  u 2$.2FNN/0BEJJ1$.2FNN.t{/@%QOOO .2^-ORUV[R\R\-\*'+~'IDNLn'n$ -33E::rCNrrrr=r>rrGc ^|jd|||||d|\}}|||} | S)N)rrrr=r>r)r_r`) r@rrrr=r>rrriattention_outputs rBrmz)InstructBlipVideoQFormerAttention.forward4sZ( ')"7#9       Q ;;{MBBrCrnrQ) rorprqr1rhr7rrrrtrrrmrvrws@rBr]r]s"""""" ;;;*7;15=A>B   | !!23 E-.  ((9:  !)): ;  +,          rCr]cBeZdZfdZdejdejfdZxZS)$InstructBlipVideoQFormerIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r0r1rrr2rrTrrstrr intermediate_act_fnr?s rBr1z-InstructBlipVideoQFormerIntermediate.__init__Jsn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rCrrGcZ||}||}|Sr)rTrors rBrmz,InstructBlipVideoQFormerIntermediate.forwardRs, =11 00?? rCrrws@rBrlrlIs^99999U\elrCrlcPeZdZfdZdejdejdejfdZxZS)InstructBlipVideoQFormerOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r0r1rrrr2rTrrrrUrr?s rBr1z'InstructBlipVideoQFormerOutput.__init__Ysf Yv79KLL f&8f>STTTz&"<== rCrrVrGc||}||}|||z}|SrrXrYs rBrmz&InstructBlipVideoQFormerOutput.forward_rZrCrrws@rBrrrrXr[rCrrcNeZdZfdZ ddeefdZdZdZxZ S) InstructBlipVideoQFormerLayerct|j|_d|_t ||_||_||jzdkrt |d|_d|_ nd|_ t||_ t||_ t||_t||_dS)Nr$rT)r"F)r0r1chunk_size_feed_forward seq_len_dimr]r_ layer_idxcross_attention_frequencycrossattentionhas_cross_attentionrl intermediaterrr`intermediate_query output_query)r@r*rzrAs rBr1z&InstructBlipVideoQFormerLayer.__init__gs '-'E$:6BB" v7 71 < <"CF_c"d"d"dD '+D $ $',D $@HH4V<< "Fv"N"N:6BBrCNrrc |j|f||d|}|dkr|ddd|ddf} |jr$|td|j| f||||d|} t |j|j|j| } |jd|kr`t |j |j|j|dd|dddf | j } tj | | gd} n!t |j |j|j|} | S)N)rrrz>encoder_hidden_states must be given for cross-attention layers)rrr=r>r$rN)r_r}rr|rfeed_forward_chunk_queryrxryrPfeed_forward_chunkrdr5r7rX) r@rrrr=r> query_lengthrrjquery_attention_output layer_outputlayer_output_texts rBrmz%InstructBlipVideoQFormerLayer.forward{s*4>  )      !  %5aaa,6I%J "' (0$%efff)<)<**#1'*?+A **  **&5-, & L %a(<77$=+0$$QQQ qqq%89 %% "\()) " %y,8I)JPQRRR 4',  L rCc\||}|||}|Sr)r~r`r@rjintermediate_outputrs rBrz0InstructBlipVideoQFormerLayer.feed_forward_chunks2"//0@AA{{#68HII rCc\||}|||}|Sr)rrrs rBrz6InstructBlipVideoQFormerLayer.feed_forward_chunk_querys4"556FGG(()<>NOO rCNNNNr) rorprqr1rrrmrrrvrws@rBrvrvfsCCCCC."#66+,6666p rCrvcReZdZfdZe ddeefdZxZS)InstructBlipVideoQFormerEncoderct|_tjfdt jD|_d|_dS)Nc0g|]}t|Sr)rv)rrzr*s rBrz.s$ o o o) *69 = = o o orCF) r0r1r*rrrrlayerrr?s `rBr1z(InstructBlipVideoQFormerEncoder.__init__sg  ] o o o ouU[UmOnOn o o o   ',###rCNrrc t|jjD]*}|j|} |||nd} | ||| |f||d|}+t |S)N)r>rr)rr*rrr) r@rrrr=r>rri layer_modulelayer_head_masks rBrmz'InstructBlipVideoQFormerEncoder.forwardst{455  A:a=L.7.CillO(L%  (>) MM9+    rCr) rorprqr1rrrrmrvrws@rBrrs,,,,,"#  +,        rCrc2eZdZdZfdZ ddZxZS)r{z;Construct the embeddings from word and position embeddings.c*ttj|j|j|j|_tj|j|j|_ tj |j|j |_ tj |j|_|dt#j|jddt)|dd|_||_dS) N) padding_idxr position_ids)r$rIF) persistentrr)r0r1rr vocab_sizer2 pad_token_idword_embeddingsrposition_embeddingsrr layernormrrUrregister_bufferr7r6rgrrr*r?s rBr1z+InstructBlipVideoQFormerEmbeddings.__init__s !|F,=v?Q_e_rsss#%<0NPVPb#c#c f&8f>STTTz&"<==   EL)GHHOOPWXXej    (/v7PR\']']$ rCNrc||d}nd}|(|jdd|||zf}|m||}|jdkr2|||j}||z}|tj ||fd}n|}||j j j }| |}| |}|S)Nr$rrrN)rKrclonerrrrdr5r7rXrrcrbr)r@ input_idsr query_embedspast_key_values_lengthrDrDrs rBrmz*InstructBlipVideoQFormerEmbeddings.forwards  "))!,JJJ  ,QQQ0FVlIl0l-lmssuuL  --i88J+z99&*&>&>|zO`?a?a&b&b#'*== '"Y j'AqIII %J]]4>#8#>?? ^^J// \\*-- rC)NNNr)rorprqrr1rmrvrws@rBr{r{s`EE$ rCr{ceZdZdZdZdZdZdZee e ddge e ddgdZ de ffd Z d Zd Zd Z dd ejdeedejdedejf dZee ddejd eejdeejdeejdeejdeejdeejdeedeeeje ffdZ!xZ"S)InstructBlipVideoQFormerModelz Querying Transformer (Q-Former), used in InstructBlipVideo. Slightly modified from BLIP-2 as it also takes the instruction as input. Fr$z .attention)rg layer_namez.crossattention)rrcross_attentionsr*ct|||_t||_t ||_|dSr)r0r1r*r{rDrrrr?s rBr1z&InstructBlipVideoQFormerModel.__init__-sV     >  rCc|jjSrrDrr s rBr z2InstructBlipVideoQFormerModel.get_input_embeddings7s ..rCc||j_dSrrr@rs rBset_input_embeddingsz2InstructBlipVideoQFormerModel.set_input_embeddings:s*/'''rCc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr_rh)r@heads_to_prunerrfs rB _prune_headsz*InstructBlipVideoQFormerModel._prune_heads=sU +0022 C CLE5 L u % / ; ;E B B B B C CrCr input_shaper5 has_queryrGc*|dkr|dddddddf}nD|dkr|ddddddf}ntd|d|jd||j}d|z d z}|S) a> Makes broadcastable attention and causal masks so that future and masked tokens are ignored. Arguments: attention_mask (`torch.Tensor`): Mask with ones indicating tokens to attend to, zeros for tokens to ignore. input_shape (`tuple[int]`): The shape of the input to the model. device: (`torch.device`): The device of the input to the model. Returns: `torch.Tensor` The extended attention mask, with a the same dtype as `attention_mask.dtype`. r Nr z!Wrong shape for input_ids (shape z) or attention_mask (shape )rarg)rOrrPrdrb)r@rrr5rextended_attention_masks rBget_extended_attention_maskz9InstructBlipVideoQFormerModel.get_extended_attention_maskEs.     1 $ $&4QQQaaa]&C # #    ! !Q & &'5QQQdAAA5E&F # #sKss\j\psss #:"<"<4:"<"N"N#&)@#@H"L&&rCNrrrrr=r>rc \||td| |jdnd} |||} | dd} | \} } | j}|t j| | f|}|| |}|t|tr|d\}}}n|\}}}||f}t|trfd|D}nF|,t j||} |}n |}nd} |j j }j| f||||| d |}|j}|dddddf}t!|| S) a$ query_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Hidden states to be used in the attention computation. If cross-attention, will be used for the query (i.e., key and value will use the encoder_hidden_states). Nz7You have to specify query_embeds when input_ids is Noner$r)rrrrI)r5c:g|]}|Sr)invert_attention_mask)rmaskr@s rBrz9InstructBlipVideoQFormerModel.forward..s(2w2w2wX\43M3Md3S3S2w2w2wrC)rrr=r>rr)rrPrDrKr5r7onesrrlistr get_head_maskr*rrrr)r@rrrrrr=r>rrembedding_outputrrhrDr5rencoder_batch_sizeencoder_sequence_lengthriencoder_hidden_shapeencoder_extended_attention_maskr sequence_outputr s` rBrmz%InstructBlipVideoQFormerModel.forwardps>$  !5VWW W0<0H|)!,,a ??%%+   '++--crc2 !, J!(  !"Z*j)A6RRRN#'"B"B>S^`f"g"g ! ,/66 ^AVWXAYA^A^A`A`>"$;QQAVA[A[A]A]>"$;Q$68O#P 0$77 e2w2w2w2w`v2w2w2w//'/).4HQW)X)X)X&262L2LMc2d2d//262L2LMc2d2d//.2 +&&y$+2OPP +74< , 2"7#B% , , , , *;'1aaa0 ;-'    rCrn)NNNNNN)#rorprqrrrrrrvrr}rr&r1r rrr7rrrrsr5rurrr LongTensorrrtrrrrrmrvrws@rBrrs4 #( N7 NEQ[g h h h  NEQ[l m m m =///000CCC )')' )'3Z)' )'  )'  )')')')'V7;37/315=A>BO O #O !!23O u/0 O u|, O E-. O  ((9:O !)): ;O +,O  uU&')UU VO O O ^O O O O O rCrzV Class defining the outputs of [`InstructBlipVideoForConditionalGeneration`]. ) custom_introceZdZUdZdZeeeje d<dZ eeeje d<dZ eeje d<dZ eeeje d<dZ eeeje d<deefd ZdS) 4InstructBlipVideoForConditionalGenerationModelOutputa loss (`torch.FloatTensor`, *optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Language modeling loss from the language model. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head of the language model. vision_outputs (`BaseModelOutputWithPooling`): Outputs of the vision encoder. qformer_outputs (`BaseModelOutputWithPoolingAndCrossAttentions`): Outputs of the Q-Former (Querying Transformer). language_model_outputs (`CausalLMOutputWithPast` or `Seq2SeqLMOutput`): Outputs of the language model. Nlosslogitsvision_outputsqformer_outputslanguage_model_outputsrGc^tfdDS)Nc3tK|]2}|dvr|n!t|V3dS)rrrN)rto_tuple)rkr@s rB zPInstructBlipVideoForConditionalGenerationModelOutput.to_tuple..sf  WWW GGq!!**,,      rC)rkeysr s`rBrz=InstructBlipVideoForConditionalGenerationModelOutput.to_tuplesE    YY[[       rC)rorprqrrrrr7rtrrrrrrrrrCrBrrs  04D(5*+ ,33315FHU5,- .55526NHU./666:>OXeE$567>>>AEHU5+<%=>EEE %*      rCrz` InstructBlipVideo base Model consisting of language model, qformer and vision encoder. c#eZdZdZdgZdeffd ZdZdZdZ dZ d e j d e j fd Zee dde j de j dee j d ee j dee j dee j dee j d ee jdeedeedeededeedeedeeeffdZxZS)rr_rr*ct|t|j|_t jtjd|j |j j |_ t|j |_t j|j j |jj |_t%j|j|_|jj$|j|jj|jj$|j|jj|dSNr$)r0r1r vision_config vision_modelrr6r7rnum_query_tokensqformer_configr2rrqformerr text_configlanguage_projectionr! from_configlanguage_modelrextend_keep_in_fp32_modulesrr?s rBr1zInstructBlipVideoModel.__init__s     89MNNLQ8OQWQfQr)s)stt4V5JKK #%9V-B-NPVPbPn#o#o '3F4FGG   0 <  " ) )$*=*O P P P   4 @  & - -d.A.W X X X rCc4|jSrrr r s rBr z+InstructBlipVideoModel.get_input_embeddings"77999rCc:|j|dSrrrrs rBrz+InstructBlipVideoModel.set_input_embeddings 0077777rCc|jjs8|jj|jj_|jj|jj_dSdSrr*use_decoder_only_language_modelrsharedr embed_tokensdecoderr s rB _tie_weightsz#InstructBlipVideoModel._tie_weights J{: R7;7J7QD  ' 47;7J7QD  ' 4 4 4 R RrCc |j}t|dkr@d|vrD  ( 7 7 7 ? ?rCrrct|e||tj|jjtj|jk}|d}n||jjk}|d | |j}|SzZ Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`. Nr4rI) r r7rr*image_token_idr7r5all unsqueeze expand_asrdr@rrspecial_image_masks rBget_placeholder_maskz+InstructBlipVideoModel.get_placeholder_mask"  !.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!;  !*dk.H!H /99"==GG VVYYZgZnoo!!rCNFqformer_input_idsqformer_attention_maskrdecoder_input_idsdecoder_attention_maskoutput_attentionsoutput_hidden_states return_dictr^ use_cacherrGc | | n |jj} |j\}}}}}|||z|||}||| | | | }|d}t j|ddt j|j }|j |jddd}t j|ddt j|j }|t j |}| |d}| |d}t j||gd}||||||| | | }|dddd|dddf}||}|||jj|zd}|I|j|}||jjk}|t j |}nd||t j|jjt j|j k}|d}|d||j }||j |j}|||}|jjr|jd ||| | | | d |}n|jd ||||| | | | d |}t;||| S) a qformer_input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary of the Q-Former. Input tokens can optionally be provided to serve as text prompt, which the Q-Former model will encode. Indices can be obtained using [`InstructBlipVideoProcessor`]. See [`InstructBlipVideoProcessor.__call__`] for details. [What are input IDs?](../glossary#input-ids) qformer_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Only relevant in case an encoder-decoder language model (like T5) is used. N)r_rrrr^rrIr4rNr$)rrrr=r>rrrrrrrrr)rrr r rrrrrr)r*use_return_dictrPrSrr7rrKr7r5rrg ones_likerepeat_interleaverXrrrrr video_token_idrrrrrdrbmasked_scatterrr)r@r_r r rrr r rrrrr^rrrhframeschannelrErFr image_embedsimage_attention_maskrquery_attention_mask query_outputs query_outputlanguage_model_inputsroutputss rBrmzInstructBlipVideoModel.forward1sR&1%->q-A-A+A111(DE !% 8 8 F F!6 = =j$+JfioJoqs t t  FD/DDFFyQQM!*dk.H!H %!&!;!;!.2M$2K2K2M2M T[7uzR_Rfggg33" "4!7!7!;!; /99"==GG VVYYZgZnoo 5 8 89M}Ob c c%445GI^__ ; 6 )d)+-"3%9'# GG*d) +-"3'="3%9'#    GD))#*    rC) NNNNNNNNNFN)rorprqrrr%r1r rrrr7rrtrrrrrrrurr rrrrmrvrws@rBrrs %O+,6&:::888RRR ???( "e.> "uO` " " " " >B15598<=A04,0/3&*).$(  ' !, !))9 :  E-.  !!12  $E$45 !))9 :   - $D> 'tn d^ #' D> -. uJJ K!   ^     rCra InstructBlipVideo Model for generating text given an image and an optional text prompt. The model consists of a vision encoder, Querying Transformer (Q-Former) and a language model. One can optionally pass `input_ids` to the model, which serve as a text prompt, to make the language model continue the prompt. Otherwise, the language model starts generating text from the [BOS] (beginning-of-sequence) token. c%eZdZUeed<dZdZdgZdeffd ZdZ dZ dZ d e j fd Zd Zd Zd ZdZ d$dejdejdeejdeedeef dZdejdejfdZee d%dejdejdeejdeejdeejdeejdeejdeejdeedeedeejdeededeed eed eee ff d!Z!ej" d&dejdeejdeejdeejdeejdeejded ejfd"Z# d$dejdejdeejdeedeef d#Z$xZ%S)'rr*r_Trct|t|j|_t jtj d|j |j j |_ t|j |_t j|j j |jj |_|jrt)j|j}nt-j|j}|j|j|j|j|j|j||_|dSr)r0r1r _from_configrrrr6r7rrrr2rrrrrrrr"rr#rrrrr)r@r*rrAs rBr1z2InstructBlipVideoForConditionalGeneration.__init__s1    8EEfFZ[[LQ8OQWQfQr)s)stt4AA&BWXX #%9V-B-NPVPbPn#o#o  1 S1=f>PQQNN2>v?QRRN  + 7  " ) ).*J K K K  / ;  & - -n.R S S S, rCc4|jSrrr s rBr z>InstructBlipVideoForConditionalGeneration.get_input_embeddingsrrCc:|j|dSrrrs rBrz>InstructBlipVideoForConditionalGeneration.set_input_embeddingsrrCc:|j|dSr)rset_output_embeddings)r@new_embeddingss rBr(z?InstructBlipVideoForConditionalGeneration.set_output_embeddingss 11.AAAAArCrGc4|jSr)rget_output_embeddingsr s rBr+z?InstructBlipVideoForConditionalGeneration.get_output_embeddingss"88:::rCc4|jSr)r get_encoderr s rBr-z5InstructBlipVideoForConditionalGeneration.get_encoder"..000rCc4|jSr)r get_decoderr s rBr0z5InstructBlipVideoForConditionalGeneration.get_decoderr.rCc|jjs8|jj|jj_|jj|jj_dSdSrrr s rBrz6InstructBlipVideoForConditionalGeneration._tie_weightsrrCc |j}t|dkr@d|vrInstructBlipVideoForConditionalGeneration.get_placeholder_maskr rCrr r rrlabelsrrc | | n |jj} ||||| d\}}}| s|n|}| s|n|}|||}|t j|}||j|j }| ||}| ||}|jj rJ|j d ||| | | |d|}| r|jn|d}d}| |jd || |jjjd|}n9|j d ||||| | | | |d |}| r|jn|d}| r|jn|d }t'||||| S) a qformer_input_ids (`torch.LongTensor` of shape (batch_size, sequence_length)): The sequence used as a prompt to be fed to the Q-Former module. qformer_attention_mask (`torch.LongTensor` of shape (batch_size, sequence_length), *optional*): Mask to avoid performing attention on padding token indices. Examples: ```python >>> from transformers import InstructBlipVideoProcessor, InstructBlipVideoForConditionalGeneration >>> import torch >>> from huggingface_hub import hf_hub_download >>> import av >>> import numpy as np >>> def read_video_pyav(container, indices): ... ''' ... Decode the video with PyAV decoder. ... Args: ... container (`av.container.input.InputContainer`): PyAV container. ... indices (`list[int]`): List of frame indices to decode. ... Returns: ... result (np.ndarray): np array of decoded frames of shape (num_frames, height, width, 3). ... ''' ... frames = [] ... container.seek(0) ... start_index = indices[0] ... end_index = indices[-1] ... for i, frame in enumerate(container.decode(video=0)): ... if i > end_index: ... break ... if i >= start_index and i in indices: ... frames.append(frame) ... return np.stack([x.to_ndarray(format="rgb24") for x in frames]) >>> model = InstructBlipVideoForConditionalGeneration.from_pretrained("Salesforce/instructblip-vicuna-7b", device_map="auto") >>> processor = InstructBlipVideoProcessor.from_pretrained("Salesforce/instructblip-vicuna-7b") >>> file_path = hf_hub_download( ... repo_id="nielsr/video-demo", filename="eating_spaghetti.mp4", repo_type="dataset" ... ) >>> container = av.open(file_path) >>> # sample uniformly 4 frames from the videWhy is this video funny?o >>> total_frames = container.streams.video[0].frames >>> indices = np.arange(0, total_frames, total_frames / 4).astype(int) >>> clip = read_video_pyav(container, indices) >>> prompt = "What is happening in the video?" >>> inputs = processor(text=prompt, images=clip, return_tensors="pt").to(model.device) >>> outputs = model.generate( ... **inputs, ... do_sample=False, ... num_beams=5, ... max_length=256, ... repetition_penalty=1.5, ... length_penalty=1.0, ... ) >>> generated_text = processor.batch_decode(outputs, skip_special_tokens=True)[0].strip() >>> print(generated_text) "A person is eating a bowl of pasta, and they are using a fork to eat it. The person is sitting at a table, and the plate of pasta is on the table in front" ```NTr r r^rrrr)rr7r) rrr r rrrr7rr$)rrrrrr)r*rget_video_featuresrr r7rrdr5rbrrrrr loss_functionrrrr)r@r_r r rrr r rrrr7rr^rrr rrrr!rrs rBrmz1InstructBlipVideoForConditionalGeneration.forward*s=f&1%8CV ..000  7D5577 BBM  !"_Y77N 5 8 89M}Ob c c!66yP]6^^%445GI^__ ; 6 C)d)+-"3%9'# G(3BW^^ FD!)t)!&T[=T=_ci *d) +-"3'="3%9'#    G$/>7<A]]`aa +t{/F/S.TT !L,uzR^Refff %,,Z;; 7D5577 BBM  !"_Y77N 5 8 89M}Ob c c!66yP]6^^%445GI^__ #0NSS")< ,"+F; .$%.KKK?KKrCc|j\}}}} } |||z|| | }|||d} | d} tj| ddtj| j} |j | jddd}tj|ddtj| j}|tj |}| |d}| |d}tj ||gd}| |||| | d }|dddd|dddf}||}|||jj|zd}|r|| |fS|S) r4T)r_r^rrNrIr4rNr$)rrrr=r>r)rPrSrr7rrKr7r5rrgrrrXrrr*r)r@r_r r r^rrhrrrErFrrrrrrrr s rBr;z->q-A-A+A111(DE !% 8 8 F F!6 = =j$+JfioJoqs t t  H(.-G G$$rC)NFF) NNNNNNNNNNFN)NNNNNF)&rorprqr%rrrrr1r rr(rModuler+r-r0rrr7rtrrrur5rrrrrrrrrmno_gradrBr;rvrws@rBrrs( $###$O!+,64:::888BBB;ry;;;;111111RRR ???0>B38&+   ' !+ !))9 :  #+4.  d^     " "e.> "uO` " " " " >B15598<=A59,0/3-1&*).$(N N 'N !,N !))9 : N E-. N !!12 N $E$45N !))9 :N   12N $D>N 'tnN )*N d^N #'N D>N +,!N " uJJ K#N N N ^N `U]__9==A045959).CC'C$E$45C!))9 : C E,- C !!12 C  12C#'C  CCC_CR>B38&+ 9%9%'9%!+9%!))9 : 9% #+4. 9% d^ 9%9%9%9%9%9%9%9%rCr)rryrrr)r)Mr9 dataclassesrtypingrrrrr7r activationsr generationr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrrmodeling_utilsrrprocessing_utilsr pytorch_utilsrrrutilsrrrrrr utils.genericrrautor!r"r#configuration_instructblipvideor%r&r' get_loggerrorrHr)ryrrfloatrr|rrrrr}r~r]rlrrrvrr{rrrr__all__rrCrBr[sp, !!!!!!111111111111 !!!!!!))))))BBBBBB999999 GFFFFFFF&&&&&&lllllllllljjjjjjjjjjjjjjjj????????IIIIIIIIII  H % %GGGGG GGGT#-#-#-#-#-#-#-#-\%% I% <% % < % U\* %  %%%%%.I)I)I)I)I)I)I)I)X     29   $>D@@@@@ry@@@D11111#C111hw.w.w.w.w.w.w.w.t        + + + + +  + + + \     29        RY   UUUUU$>UUUp$ $ $ $ $ bi$ $ $ N00000000fi i i i i $Di i i X       ;     : F F F F F =F F  F R}%}%}%}%}%0PRa}%}%}%@    rC