`i'dZddlmZddlZddlmZddlmZGdd ej j j Z Gd d ej j j Z Gd d ej j j ZdS)a Generic interface to various configurations of the Perceiver Resampler, that simply takes in a series of (potentially time-indexed) contextual embeddings, and "resamples" (compresses) them down to a pre-specified number of latents! Note that the Perceiver in general resamples based solely off the *long-range* context; there's a nice opportunity here to prime the Perceiver Resampler with say a single layer's worth of language embeddings (the target domain), and use that to softly "retrieve & compress" what we need --> this would be a novel contribution we should explore. References: - DeepMind's Flamingo: https://www.deepmind.com/blog/tackling-multiple-tasks-with-a-single-visual-language-model - Code borrowed w/ love from: https://github.com/lucidrains/flamingo-pytorch )OptionalN) shape_list) IdeficsConfigcjeZdZdededededededdffd Zfd Zd ejdejfd Z xZ S) TFIdeficsPerceiverResamplerconfig embed_dimdepthn_headshead_dim n_latentsreturnNc Ftjd i|||||f\|_|_|_|_|jj|_t|j ds |jdzn|j jdz|_ g|_ t|D]a}|j t|j|j|j|jd|dt!|j |d|dgbt"jjdd |_d S) ao Instantiates a Perceiver Resampler that operates over a sequence of embeddings (say from a ResNet or ViT or MAE) of a given dimension, performs `depth` blocks of cross-attention with a fixed `n_latents` inputs, then returns a Tensor of shape [bsz, n_latents, embed_dim]. :param embed_dim: Dimensionality of embeddings being fed to the Perceiver Resampler (also dimensionality of latent embeddings *returned* by the Perceiver Resampler. Could be e.g., VIT embed_dim, ResNet pool dim, and so on. Args: config (`IdeficsConfig`): config object embed_dim (`int`): The size of each embedding vector depth (`int`): Depth of the Perceiver Resampler (Transformer w/ cross attention). Should be shallow (< 3). n_heads (`int`): Number of heads in each Transformer block (for multi-headed self-attention). head_dim (`int`): Dimensionality of each head projection in the Transformer block. n_latents (`int`): Number of latent embeddings to resample ("compress") the input sequence to (usually < 128). r zblocks.z.0namez.1h㈵> layer_normepsilonrN)super__init__r r rrperceiver_configqk_layer_norms_perceiverqk_layer_normshasattr vision_configintermediate_dimblocksrangeappendTFIdeficsPerceiverAttention TFIdeficsMLPtfkeraslayersLayerNormalizationr) selfr r r r rrkwargsi __class__s |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/idefics/perceiver_tf.pyrz$TFIdeficsPerceiverResampler.__init__1sD( ""6"""FOQXZbdmFmC dmT^$5N6/== 4DNQ  %/!3   u  A K  / dmTEX_nij_n_n_n!!6_q___UUU     (/<4>2[_fo'     k"""""r0contextctj|jd}tj|tj|dddg}|jD]"\}}||||z}|||z}#||S)zWResample arbitrary length context & *compress* down to self.n_latents latent embeddingsraxisr)r' expand_dimsr3tiler4r"r)r+r:r3attnffs r/callz TFIdeficsPerceiverResampler.callcs.A666''BHW$5$5a$8!Q#?@@  , ,HD"d7G,,w6GbkkG+GGw'''r0) __name__ __module__ __qualname__rintrr8r'TensorrB __classcell__r.s@r/r r 0s)^#)^03)^ let long-form inputs be `context`, resampled embeddings be `latents`rcontext_layer_normrlatents_layer_norm q_layer_norm k_layer_normgFq_projuse_biasrk_projv_proj output_projNr)rrr r rrr'r(r)r*rLrMrNrOqk_scaleDenserPrSrTrU)r+r r rrr,r.s r/rz$TFIdeficsPerceiverAttention.__init__ps""6"""6?(3 dm,"$(/"D"DTXl"D"m"m"$(/"D"DTXl"D"m"m   f " B B4Vd B e eD  " B B4Vd B e eD  t+ ho++DL4=,HSX_g+hh ho++DL4=,HSX_g+hh ho++DL4=,HSX_g+hh 8?00UQ^0__r0r:r3c ^ |}|}t|\ }}|}t j||gd}t j||gd} fd|||fD\}}}jr* |} |}t j d|j z|}|t j |ddz } t j| d} t j d| |} t jt j| gd   djjzfS) a= Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: context (`tf.Tensor`): Tensor of shape `[bsz, seq, embed_dim]` representing long-form context to resample. latents (`tf.Tensor`): Tensor of shape `[bsz, n_latents, embed_dim]` representing fixed length latents to compress to. Returns: `tf.Tensor`: Tensor of shape `[bsz, n_latents, embed_dim]` representing attention over latents w/ cross from context. r<c g|]F}tjtj||jdjjfgdGS)rrrrperm)r' transposereshaper4r r).0x batch_sizer+s r/ z4TFIdeficsPerceiverAttention.call..s^    LA AGAJ dm'\]]dpdpdp q q q   r0z... i d, ... j d -> ... i jT)r=keepdimsz... i j, ... j d -> ... i dr[r])rLrMrrPrSr'concatrTrrNrOeinsumrV reduce_maxnnsoftmaxrUr`r_r r) r+r:r3 seq_lengthr qkvscoresstabilized_scoresr@ resampledrcs ` @r/rBz TFIdeficsPerceiverAttention.calls))'22))'22,6w,?,?) J  KK  KK 7G"42>>> ? ? KK 7G"42>>> ? ?     AY   1a   %!!!$$A!!!$$A8!dm:KQOO"R]6T%R%R%RRu}}.R}88I;T1EE  Jr|ILLLAAAJPRTXT`cgcpTpCq r r   r0) rCrDrErFboolrr'rGrBrHrIs@r/r%r%os`#``s`TX`gk``````*+ BI+  + bi+ + + + + + + + r0r%c`eZdZdeffd ZdeeejdejfdZ xZ S)r&r c tjd i||jj|_tjjdd|_tjj |dd|_ tjj d|_ tjj |jdd |_ d S) z:Simple MLP block with intermediate_size and embedding sizerlnrFfcrQactrc_projNr)rrr r r'r(r)r*rvrWrwReLUrxry)r+intermediate_sizer r,r.s r/rzTFIdeficsMLP.__init__s""6"""-7(/44T4MM(/''(9EPT'UU8?''U'33ho++DNUQY+ZZ r0 hidden_statesrc||}||}||}||}|S)N)rvrwrxry)r+r|s r/rBzTFIdeficsMLP.callsL ..  .. //  M22 r0) rCrDrErrrtupler'rGrBrHrIs@r/r&r&s{[-[[[[[[(5+;"<r0r&)__doc__typingr tensorflowr'modeling_tf_utilsrconfiguration_ideficsrr(r)Layerr r%r&rr0r/rs4  ++++++000000<(<(<(<(<("(/"7<(<(<(~A A A A A "(/"7A A A H28?(r0