`im~ dZddlZddlZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZmZdd lmZdd lmZmZdd lmZd d lmZejeZGddejZejjdZ dZ!GddejZ"d4dej#de$de%dej#fdZ&GddejZ'GddejZ(Gdd ejZ)Gd!d"ejZ*d#Z+d$Z,Gd%d&e Z-Gd'd(ejZ.d)ejddfd*Z/eGd+d,eZ0eGd-d.e0Z1ed/0Gd1d2e0eZ2gd3Z3dS)5zPyTorch ViTDet backbone.N)OptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BackboneOutputBaseModelOutput)PreTrainedModel)auto_docstringlogging) BackboneMixin) VitDetConfigcLeZdZdZfdZdZdejdejfdZxZ S)VitDetEmbeddingsz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) to be consumed by a Transformer. cXt|j|j}}|j|j}}t |tjj r|n||f}t |tjj r|n||f}|d|dz|d|dzz}||_ ||_||_||_ |j r8|dz}tjtjd||j|_nd|_tj|||||_dS)Nrr) kernel_sizestride)super__init__pretrain_image_size patch_size num_channels hidden_size isinstance collectionsabcIterable image_size num_patches use_absolute_position_embeddingsr Parametertorchzerosposition_embeddingsConv2d projection) selfconfigr rrrr! num_positions __class__s ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/vitdet/modeling_vitdet.pyrzVitDetEmbeddings.__init__*s$ !'!;V=NJ $*$79Kk #-j+/:R#S#SqZZZdfpYq #-j+/:R#S#SqZZZdfpYq !!} 1 5*Q-:VW=:XY $$(&  2 ,'!OM')|EK=RXRd4e4e'f'fD $ $'+D $)L+:^hiiic|r|ddddf}|jd}ttj|}||z|krt dt js ||ks||kritj | d||d dddd||fdd }| ddddS| d||dS) a Calculate absolute positional embeddings. If needed, resize embeddings and remove cls_token dimension for the original embeddings. Args: abs_pos_embeddings (`torch.Tensor`): Absolute positional embeddings with (1, num_position, num_channels). has_cls_token (`bool`): If true, has 1 embedding in abs_pos_embeddings for cls token. height (`int`): Height of input image tokens. width (`int`): Width of input image tokens. Returns: Absolute positional embeddings after processing with shape (1, height, width, num_channels) Nrz5Absolute position embeddings must be a square number.rrbicubicF)sizemode align_corners) shapeintmathsqrt ValueErrorr$jit is_tracingr functional interpolatereshapepermute)r)abs_pos_embeddings has_cls_tokenheightwidth num_positionr3new_abs_pos_embeddingss r-get_absolute_positionsz'VitDetEmbeddings.get_absolute_positions@s$  ;!3AAAqrrE!: )/2 49\**++ $;, & &TUU U 9   ! ! Ddfnn %']%>%>"**1dD"==EEaAqQQe_# &?&& "*11!Q1== =%--aCC Cr. pixel_valuesreturncp|jd}||jkrtd|jd|d||}|jf|dddd}|||jd|jd|jdz}|dddd}|S) NrzoMake sure that the channel dimension of the pixel values match with the one set in the configuration. Expected z but got .rr1rT)r6rr:r(r&r@rG)r)rHr embeddingss r-forwardzVitDetEmbeddings.forwardfs#)!, 4, , ,I!.II9EIII __\22  # /#++Aq!Q77J#d&A&A($ 0@0CZEUVWEX''J$++Aq!Q77Jr.) __name__ __module__ __qualname____doc__rrGr$TensorrM __classcell__r,s@r-rr$s jjjjj,$D$D$DLELU\r.rctdt||zdz }|jd|krtj|d|jddddd|d}|d|dd}n|}tj |dddft||z dz}tj |dddft||z dz}||z |dz t||z dzz}|| S) a Get relative positional embeddings according to the relative positions of query and key sizes. Args: q_size (`int`): Size of query q. k_size (`int`): Size of key k. rel_pos (`torch.Tensor`): Relative position embeddings (num_embeddings, num_channels). Returns: Extracted positional embeddings according to relative positions. r1rrr0linear)r3r4N?) r7maxr6rr=r>r?r@r$arangelong)q_sizek_sizerel_pos max_rel_distrel_pos_resizedq_coordsk_coordsrelative_coordss r- get_rel_posrc|sQ q3vv...233L}Q<''-33 OOAw}Q/ 4 4 < bhwkzbhwc,wkc->bhwkN)rcr6r?r$einsumview)attnqueries rel_pos_h rel_pos_wr[r\queries_height queries_width keys_height keys_widthrelative_heightrelative_width batch_size_dimr_qrelative_weights r-!add_decomposed_relative_positionsrvs,%+!NM$K!.+yIIO  IFFN J3 //*nmS I ICl#3S/JJOl#3S.IIO *nm[*UU !!!QQQ111d* + , !!!QQQ4* + , d:~ 5{Z7OPP  Kr.c,eZdZdZdfd ZddZxZS)VitDetAttentionz=Multi-head Attention block with relative position embeddings.Nct|j}|j}||_||z}|dz|_t j||dz|j|_ t j|||_ |j |_ |j rrt j tjd|dzdz ||_t j tjd|dzdz ||_dSdS)z Args: config (`VitDetConfig`): Model configuration. input_size (`tuple[int]`, *optional*): Input resolution, only required in case relative position embeddings are added. grbiasr1rrN)rrrnum_attention_heads num_headsscalerLinearqkv_biasqkvproj use_relative_position_embeddingsr#r$r%rirj)r)r* input_sizersr}head_dimr,s r-rzVitDetAttention.__init__s  . ")#t^ 9S#'@@@Ic3'' 060W-  0 X\%+a*Q-6G!6KX*V*VWWDN\%+a*Q-6G!6KX*V*VWWDNNN X Xr.Fc 4|j\}}}}|||||zd|jdddddd}|d||jz||zdd\}} } ||jz| ddz} |jr"t| ||j |j ||f||f} | d} | | z}| ||j||d}|ddddd}||||d}||}|r8| ||j| jd| jd} || f} n|f} | S) Nrr0r1rr)rs)r6rr?r}r@unbindr~ transposerrvrirjsoftmaxrfr)r) hidden_stateoutput_attentionsrqrCrDrrrrhkeysvaluesattention_scoresattention_probsoutputss r-rMzVitDetAttention.forwards'3'9$ FE1hh|$$,,Z%DN\^__gghiklnoqrtuvv # AzDN/JFUZN\^ _ _ f fgh i iv#dj0DNN2r4J4JJ  0 @ '4>4>FTY?]cej\k   +22r2::&/ #((T^VUTVWW #++Aq!Q:: #++JrJJ yy..  &-55DNO,A",EG\]_G`O$_5GG#oGr.N)FrNrOrPrQrrMrSrTs@r-rxrxs]GGXXXXXX4r.rxFinput drop_probtrainingrIc|dks|s|Sd|z }|jdfd|jdz zz}|tj||j|jz}||||z}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rrr)r)dtypedevice)r6ndimr$randrrfloor_div)rrr keep_probr6 random_tensoroutputs r- drop_pathr sCx I I [^  Q 7 7E 5 EL Y Y YYM YYy ! !M 1F Mr.cjeZdZdZd deeddffd ZdejdejfdZ de fdZ xZ S) VitDetDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).NrrIcVt||_dSr)rrr)r)rr,s r-rzVitDetDropPath.__init__!s$ "r. hidden_statesc8t||j|jSr)rrr)r)rs r-rMzVitDetDropPath.forward%s FFFr.cd|jS)Nzp=)rr)s r- extra_reprzVitDetDropPath.extra_repr(s$DN$$$r.r) rNrOrPrQrfloatrr$rRrMstrrrSrTs@r-rrsbb##(5/#T######GU\GelGGGG%C%%%%%%%%r.rc*eZdZdZdfd ZdZxZS)VitDetLayerNormaL A LayerNorm variant, popularized by Transformers, that performs point-wise mean and variance normalization over the channel dimension for inputs that have shape (batch_size, channels, height, width). https://github.com/facebookresearch/ConvNeXt/blob/d1fa8f6fef0a165b27399986cc2bdacc92777e40/models/convnext.py#L119 ư>cttjt j||_tjt j||_||_ |f|_ dSr) rrrr#r$onesweightr%r{epsnormalized_shape)r)rrr,s r-rzVitDetLayerNorm.__init__3si l5:.>#?#?@@ L-=!>!>?? !1 3r.c"|dd}||z ddd}||z tj||jzz }|jddddf|z|jddddfz}|S)NrT)keepdimr1)meanpowr$r9rrr{)r)xuss r-rMzVitDetLayerNorm.forward:s FF1dF # # UKKNN  4  0 0 UejTX.. . K4 & *TYqqq$}-E Er.)rrrTs@r-rr,sV 444444r.rc(eZdZdZfdZdZxZS)VitDetResBottleneckBlockz The standard bottleneck residual block without the last activation layer. It contains 3 conv layers with kernels 1x1, 3x3, 1x1. cttj||dd|_t ||_t|j|_ tj||ddd|_ t ||_ t|j|_ tj||dd|_ t ||_dS)ar Args: config (`VitDetConfig`): Model configuration. in_channels (`int`): Number of input channels. out_channels (`int`): Number of output channels. bottleneck_channels (`int`): Number of output channels for the 3x3 "bottleneck" conv layers. rFrzr)paddingr{N)rrrr'conv1rnorm1r hidden_actact1conv2norm2act2conv3norm3)r)r* in_channels out_channelsbottleneck_channelsr,s r-rz!VitDetResBottleneckBlock.__init__Hs Y{,?OOO $%899 6,- Y24GTU\abbb $%899 6,- Y2L!%PPP $\22 r.cX|}|D] }||}||z}|Sr)children)r)routlayers r-rMz VitDetResBottleneckBlock.forward`s;]]__  E%**CC#g r.rrTs@r-rrBsQ 333330r.rcPeZdZdededdffd ZdejdejfdZxZS) VitDetMlp in_featureshidden_featuresrINcttj|||_t |j|_tj|||_tj |j |_ dSr) rrrrfc1rractfc2Dropout dropout_probdrop)r)r*rrr,s r-rzVitDetMlp.__init__jsf 9[/::&+,9_k::Jv233 r.rc||}||}||}||}||}|Sr)rrrr)r)rs r-rMzVitDetMlp.forwardqsR HHQKK HHQKK IIaLL HHQKK IIaLLr.) rNrOrPr7rr$rRrMrSrTs@r-rrisx4C4#4$444444%,r.rc v|j\}}}}|||zz |z}|||zz |z}tj|ddd|d|f}||z||z} }||||z|| |z||}|ddddddd|||} | || ffS)a  Partition into non-overlapping windows with padding if needed. Args: hidden_state (`torch.Tensor`): Input tokens with [batch_size, height, width, num_channels]. window_size (`int`): Window size. Returns: `tuple(torch.FloatTensor)` comprising various elements: - windows: windows after partition with [batch_size * num_windows, window_size, window_size, num_channels]. - (padded_height, padded_width): padded height and width before partition rrrr1rr0)r6rr=padrfr@ contiguous) r window_sizerqrCrDr pad_height pad_width padded_height padded_widthwindowss r-window_partitionr{s/;.@+J| 44 CJu{22kAI=$$\Aq!Y:3VWWL"(:"5uy7H}dN^._._ +M+!% /"0" " /q1 (,  a  .}d>NP`cikpbqrrM!4>>-#@#@@ %txx =@Y@Y7Z7Z([([[ %--aAq99  " 9 MM-88M "W,r.)rrF)NF)rNrOrPrQrrr7boolrr$rRrrrrMrSrTs@r-rrsMMqv!!"!49!LO!im! !!!!!!L-1"' ((|(EL)( ( uU\5</0% 2EE F ((((((((r.rceZdZdeddffd Z d dejdeejd ed ed ede e e ff d Z xZ S) VitDetEncoderr*rINc t||_|j}dt jd|j|dD}g}t|D]E}|t|||||j vr|j nd||j vFtj||_d|_dS)Nc6g|]}|S)item).0rs r- z*VitDetEncoder.__init__..s jjjq!&&((jjjr.rcpu)r)rrrF)rrr*num_hidden_layersr$linspacerrangeappendrwindow_block_indicesrresidual_block_indicesr ModuleListrgradient_checkpointing)r)r*depthrlayersir,s r-rzVitDetEncoder.__init__ s  (kjEN1f>SUZch,i,i,ijjju  A MM#1!#4676;V6V6V 2 2\]'(F,I'I     ]6** &+###r.FTrrroutput_hidden_states return_dictc,|rdnd}|rdnd}t|jD]<\}} |r||fz}|||nd} | || |} | d}|r || dfz}=|r||fz}|std|||fDSt|||S)Nrrrc3K|]}||V dSrr)rvs r- z(VitDetEncoder.forward..>s(mmq_`_l_l_l_l_lmmr.last_hidden_stater attentions) enumeraterrr ) r)rrrrrall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss r-rMzVitDetEncoder.forward"s #7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO(LIZ[[M)!,M  P&9]1=M>> from transformers import VitDetConfig, VitDetModel >>> import torch >>> config = VitDetConfig() >>> model = VitDetModel(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 768, 14, 14] ```Nz You have to specify pixel_values)rrrrrrr) r*rruse_return_dictr: get_head_maskrrLr@r rr) r)rHrrrrembedding_outputencoder_outputssequence_outputs r-rMzVitDetModel.forwards82C1N--TXT_Tq$8$D $+Jj &1%r>s)|*&6****C4T#Y+?CDCCCC04,0,0/3&* = = u|,= EL)= $D> = 'tn = d^ =  uo% &= = = ^= = = = = r.r>zF ViTDet backbone, to be used with frameworks like Mask R-CNN. ) custom_introc eZdZfdZdefdZe d dejde e de e de e de f d Z xZ S) VitDetBackbonecVttt|_t |_fdtjdzD|_ | dS)Ncg|] }j Sr)r)rrrr*s r-rz+VitDetBackbone.__init__..s]]]AV/]]]r.r) rr_init_backbonerrLrr@r r num_featuresrArBs `r-rzVitDetBackbone.__init__s     v&&&*622$V,, ]]]]v?WZ[?[9\9\]]] r.rIc|jjSrrDrs r-rEz#VitDetBackbone.get_input_embeddingsrFr.NrHrrrc||n |jj}||n |jj}||n |jj}||}||d||}|r|jn|d}d}t|j|D]\} } | |j vr|| fz }|s!|r|f|ddz} n|f|ddz} | St||r|jnd|j S)a Examples: ```python >>> from transformers import VitDetConfig, VitDetBackbone >>> import torch >>> config = VitDetConfig() >>> model = VitDetBackbone(config) >>> pixel_values = torch.randn(1, 3, 224, 224) >>> with torch.no_grad(): ... outputs = model(pixel_values) >>> feature_maps = outputs.feature_maps >>> list(feature_maps[-1].shape) [1, 768, 14, 14] ```NT)rrrrrr1) feature_mapsrr) r*rNrrrLr@rzip stage_names out_featuresr r) r)rHrrrrPrrr]stagerrs r-rMzVitDetBackbone.forwardsH6&1% ; d^ ;  ; ; ; ^; ; ; ; ; r.rV)r>r,rV)rF)4rQcollections.abcrr8typingrrr$r activationsrmodeling_layersrmodeling_outputsr r modeling_utilsr utilsr r utils.backbone_utilsrconfiguration_vitdetr get_loggerrNloggerModulerr;script_if_tracingrcrvrxrRrrrrrrrrrrrr*r,r>rV__all__rr.r-rps """""""" !!!!!!999999????????------,,,,,,,,111111......  H % %UUUUUryUUUp!3!3!3H&&&R;;;;;bi;;;~U\eTV[Vb* % % % % %RY % % %bi,$$$$$ry$$$N $222@>NNNNN,NNNb8 8 8 8 8 BI8 8 8 v *RY *4 * * * *0+0+0+0+0+O0+0+0+fT T T T T 'T T T n K K K K K *MK K  K \ E D Dr.