.`icUddlZddlmZddlmZddlmZddlZddlm Z ddl m cm Z ddl mZddlmZddlmZddlmZdd lmZmZmZeeeefzZeed <eeeefejzZeed <d Z e d Z!e dZ"e dZ#e dZ$e Z%deeefdedefdZ&de'eeefdede'efdZ(Gdde j)Z*Gdde j)Z+Gdde j)Z,Gdde j-Z.Gdd eZ/Gd!d"eZ0Gd#d$eZ1Gd%d&e j)Z2Gd'd(e j)Z3dS))N)Iterable)repeat) TypeAlias) rearrange)PretrainedConfig)QuantizationConfig)default_weight_loader)InternParallelAttentionInternVisionEncoderInternVisionEncoderLayer input_dim_tnorm_tcfd}|S)Nct|tr$t|tst|Stt |SN) isinstancerstrtupler)xns t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/models/radio.pyparsez_ntuple..parse#sG a " " :a+=+= 88OVAq\\""")rrs` r_ntupler"s###### Lrsize patch_sizereturnc"|\}}||z||zzSrr)r r!hws r calc_seq_lenr&2s DAq OZ 00rsizesc fd|DS)Nc0g|]}t|Sr)r&).0r r!s r z!calc_seq_lens..8s# = = =tLz * * = = =rr)r'r!s `r calc_seq_lensr,7s = = = =u = = ==rc `eZdZ d dededededzdedzf fd Zd ejfd ZxZ S) ClsTokenrTNndim num_tokensenabledregister_multiple num_registerscft||_||_d|_||_|rV|r||_n|r |||zz |_|dz}t jtj ||jz||z|_ nd|_ |j|jz|_ dS)Nr) super__init__r/r1r3r0nn Parametertorchrandntoken num_patches)selfr/r0r1r2r3scale __class__s rr7zClsToken.__init__<s   $   %2""" %6!22&"$JE J);;TBBUJDJJ DJ?T-??rrc|j|S|jd|jddd}t j||gd}|SNrrdim)r< unsqueezeexpandshaper:cat)r>rr<s rforwardzClsToken.forward\si : H $$Q''..qwqz2rBB I     r)rTNN) __name__ __module__ __qualname__intboolr7r:TensorrJ __classcell__r@s@rr.r.;s(,$( @@@@ @ : @ Tz @@@@@@@         rr.ceZdZ d-dededed ed ed ed edzd ededededzdedzdeffd Z d.dej de e eefdzdej fdZ dej de e eefde ej ej dzffdZ dej de e eefdej fdZedZedZedZedZedZd ej d!ejfd"Zd#ej d$ej fd%Zdej dej fd&Z d/dej d'ej dzd(e eefdzdej fd)Z d/d*ed'ej dzd(e eefdzdej fd+Zd*ede eeffd,ZxZS)0ViTPatchGeneratorTFNrr! embed_dim input_dimsabs_posnormalize_patches cls_tokenmax_input_dims pos_dropoutreturn_pos_encnum_cls_tokensr2r3 patch_biascttt|tr||f}||}t|tr||f}t fd|D}||k|_||_| |_t||}|_ ||_ ||_ |dz|_ |dz|_ t fd|D|_|j |j z|_||_t#|_t'|fd| i||_|r8|dz}t+jt/jd|j|fi||z|_t5|| || | |_|rt+j|nt+j|_dS) Nc3fK|]+}ttj|z zV,dSr)rNmathceilr*dr!s r z-ViTPatchGenerator.__init__..sO  <=C !j.))J6 7 7      r)devicedtyperrc3"K|] }|zV dSrrrds rrfz-ViTPatchGenerator.__init__..s'DDAZDDDDDDrbiasr5)r0r1r2r3)r6r7rrNrcpe_moder\r]dictr!rXrVnum_rowsnum_colsrWr=r[ Im2Patches im_to_patchesViTPatchLinearembedderr8r9r:r; pos_embedr.rZ LayerNormIdentitypatch_normalizer)r>r!rVrWrXrYrZr[r\r]r^r2r3r_rgrhfactoryr?r@s ` rr7zViTPatchGenerator.__init__ms&  j# & & 2$j1J  !'N nc * * >,n=N    AO     '*4 &,fE222$ "&q)Z7 &q)Z7 DDDDDDDDD=4=8,' 33&    (2 6=     tOE\ At/FFgFFNDN" %/'    (9 KBL # # #bkmm rr imgs_sizesr"cz|G||}|||\}}|||}nQ||}|||jdd\}}||}||}|jr||fS|S)Nrxr input_size) rrapply_pos_enc_dynamiccls_token_dynamic embed_patches apply_pos_encrHrZrvr])r>rrxpatchespos_encs rrJzViTPatchGenerator.forwards  !mmA&&G#99J :   GW,,W,LLGG((++G#11'agabbk1RR GWnnW--G''00   $G# #rrc |js|dfSd}g}|D]}t||j}|dd|||zddf}||jd|}||z} t j|ddd|ddf| |dd||zdddfgd}||||z }|rt j|dnd} || fS)Nrr{rrD)rXr&r! get_pos_encrHr:rIappend) r>rrxcurrent_length pos_enc_listr seq_length img_patchesrimg_patches_with_pos full_pos_encs rr}z'ViTPatchGenerator.apply_pos_enc_dynamics=| !D=   ) )D%dDO<rrxoutrseq_len class_tokens rr~z#ViTPatchGenerator.cls_token_dynamics~% N$ZAA & &G..88;;BB a "bK JJ{ # # # JJwqqq.>G3K"KQQQNO P P P g %NNy!$$$$rc|jjSr)rZr1r>s rapply_cls_tokenz!ViTPatchGenerator.apply_cls_tokens ~%%rc|jjSr)rZr0rs rr^z ViTPatchGenerator.num_cls_tokenss ~((rc|jjSr)rZr=rs rnum_cls_patchesz!ViTPatchGenerator.num_cls_patchess ~))rc|jjSr)rZr3rs rr3zViTPatchGenerator.num_registerss ~++rc |j|jzSr)r^r3rs rnum_skipzViTPatchGenerator.num_skips"T%777r src_embed targ_embedc~|j|jkrttj|jd}|dz|jdks Jdt |d||}t j||j|jfddd }t |d }|j |dS) Nrrz*Unable to interpolate non-square embeddingzb (h w) c -> b c h w)r$r%bicubicTFr mode align_corners antialiaszb c h w -> b (h w) c) rHrNrbsqrtrF interpolatermrndatacopy_)r>rrsrc_sizes r _load_embedzViTPatchGenerator._load_embed s ?j. . .49Y_Q%78899HQ;)/!"4444<544"1XI mT]3" I")-CDDIi(((((rsrc_proj_weighttarg_proj_weightc|j|jkrttj|jddz}|dzdz|jdks Jdt |dd||}t j||j|jfddd }t |d }|j |dS) Nrrrz+Unable to interpolate non-square patch sizezb (c h w) -> b c h w)cr$r%rTFrzb c h w -> b (c h w)) rHrNrbrrrrr!rr)r>rrsrc_patch_sizes r_load_projectionz"ViTPatchGenerator._load_projection s  $4$: : : ?+@+Cq+H!I!IJJN"A%*o.CA.FFFF=GFF(&  O mot7" O(9OPPO##O44444rcZ||}||}|Sr)rprr)r>rrs rrzViTPatchGenerator.embed_patches;s+$$Q''--((r patch_idxsr|c*|js|S||jd||}|jrY|jdkrNt j|jddd|j|j|jk}t j ||d}n|}||z|fS)Nrrrhrg) rXrrHtrainingr\r:randrhrgwhere)r>rrr|rkeeps pos_enc_drops rrzViTPatchGenerator.apply_pos_enc@s | N""7=#3ZLL = #T-11 M!$a'-"#  !;ugq99LL"L%w..r batch_sizec^|j}ntfd|D}||}||S|ddd|jd}t j||jdddd|}|S)Nc3,K|]}|jzVdSr)r!)r*rer>s rrfz0ViTPatchGenerator.get_pos_enc..as*HHqDO3HHHHHHrrCrr)rEindex)rWr_get_pos_embeddingsrFrGrHr:gather)r>rrr|rWrsexp_patch_idxss` rrzViTPatchGenerator.get_pos_encXs  JJHHHHZHHHHHJ,,ZDD   #--b1188RQSATUUL   Z-a0"b 9 9q   rc|j|jfkr|jS|jd|j|jddddd}fd}|jr||jrtjd}tj |dd|j d|z z|z}tj d }| }tj tj |dd|j ||z z|z} || z} |d| z z} tj| | gd dd} tj |ddd|j d| z z} tjddd|j d |dd}tjddd|j ddddf|dd}tj||gd }|| z| z}|ddt)j||ddd|d dd|j}njt3}t)j|||fdd |j}||}n ||}|jddkrBt)j|dd |j}|dddd}|S)NrrCrrrcd|jdkr|dddddf}d|jdkr|dddddf}|S)Nr.rrC)rH)rsrWs r window_selectz.window_selectwso!}yr222%c?Z]?AAA&=> !}yr222%c111o 1 o&=>  rg?rgg?rD)stepsrgNNbilinearzerosT)gridr padding_moder)r rrr)rmrnrsreshapepermuterkrrbrr:rrglogexpstackclamp_linspacerGmul_sub_r grid_samplefloattorhmaxrrHflatten)r>rrWrsr min_scaler? aspect_min aspect_maxaspectscale_xscale_yscale_xypos_xylin_xlin_ylin_xygrid_xymax_dims ` rrz%ViTPatchGenerator._get_pos_embeddingsos M4= )Z 7 7> !N**1dmT]BOOWW q!Q        =8 1}5 5 IcNN Jz1a 8HIII9}&  "Xe__ ([ Jz1a 8HIII!J.0 !  &.1v:. ;'9rBBBII!QOOJ1a @PQQQLq 1 i6F$fZACCq 1 i6F4-!!' B 1 !F!FeU^<<< 8+f4 Q$$Q'''MOO%%,,ZRDD #!("&  "Y_%%  j//MOO%%!7+"&#  "Y_%% *M)44 % i00I ?233 : - - !! $Zb!! %%a((00Aq99 r) TFFNrUFrNNFNNrr)rKrLrMrNr rOrr7r:rPlistrrJr}r~propertyrr^rr3rr8r9rrrrrrrQrRs@rrTrTls"'-1 $(,$( #E E E  E  E  E  E E $d*E E E E :E TzE E E E E E E PKO+/c3h+@4+G $%|%15eCHo1F% u|U\D00 1%%%%<%|%15eCHo1F% %%%%&&&X&))X)**X*,,X,88X8)U\)r|))))*5$|5?D|55556u| +/-1 ///L4'/#s(Od* /  ////6+/-1 L4'#s(Od*   .PcPuS#XPPPPPPPPrrTcHeZdZdeffd ZdejdejfdZxZS)ror!cVt||_dSrr6r7r!)r>r!r@s rr7zIm2Patches.__init__s$ $rrr"c |jdkr.|d}|ddd}|S|jd|jz}|jd|jz}t |d||j||j}|S)NrrrrrCz*b c (py yy) (px xx) -> b (py px) (c yy xx))pyyypxxx)r!rrrHr)r>rrrrs rrJzIm2Patches.forwards ?a  iillGooaA..GN WR[DO + WR[DO + 8    r) rKrLrMrNr7r:rPrJrQrRs@rrorosj%3%%%%%%%,rroc.eZdZddededeffd ZxZS)rqFr!rVrjc \tjd|dzz|fd|i|||_dS)Nrrrjr)r>r!rVrjrwr@s rr7zViTPatchLinear.__init__s=j!m,iNNdNgNNN$r)F)rKrLrMrNrOr7rQrRs@rrqrqsW%%3%3%d%%%%%%%%%%rrqcTeZdZ ddejdejdzdejffd ZxZS)RadioParallelAttentionNr attn_maskr"c|!t|S|j\}}}||\}}|dd\}}} |jr|||\}}||||j|j }||||j|j }| |||j|j } d||| fD\}}} tj ||| ||j } | dd||d} || \} }| S)NrrCrDc3BK|]}|ddVdS)rrN) transposer*ts rrfz1RadioParallelAttention.forward..s0881;;q!$$888888r)rr?rr)r6rJrHqkvchunkqk_normalization_apply_qk_normviewnum_heads_per_partitionhead_dimrscaled_dot_product_attentionr?rrproj) r>rrBN_rqkvrr@s rrJzRadioParallelAttention.forwardsV  77??1%% %'1a!Q))A2)&&1a   -&&q!,,DAq FF1a5t} E E FF1a5t} E E FF1a5t} E E88q!Qi8881a, q!y    mmAq!!))!Q333Q rr)rKrLrMr:rPrJrQrRs@rrrsh@D*/,*= rrcNeZdZdfd Z ddejdejdzfdZxZS) RadioVisionEncoderLayerr"NcHtj|dti|dS)Nattn_cls)r6r7rr>argskwargsr@s rr7z RadioVisionEncoderLayer.__init__s+$J)?J6JJJJJr hidden_statesrc||||||jzz}|||||jzz}|SN)r)attnnorm1ls1mlpnorm2ls2)r>rrs rrJzRadioVisionEncoderLayer.forwardsi ii =11YiGG$(R S  &M1J1J(K(Kdh(VV rr"NrrKrLrMr7r:rPrJrQrRs@rrrs|KKKKKK *.  | <$&        rrcNeZdZdfd Z ddejdejdzfdZxZS) RadioVisionEncoderr"NcHtj|dti|dS)N layer_cls)r6r7rr s rr7zRadioVisionEncoder.__init__s+$L*ALVLLLLLr inputs_embedsrc8|}|jD]}|||}|Sr)layers)r>rrr encoder_layers rrJzRadioVisionEncoder.forwards8 & ![ N NM)M-9MMMMMrrrrrRs@rrr s|MMMMMM *.|<$&rrceZdZddgiZ ddddddededzdedzd ed ed df fd Zd ee eefzfdZ dZ de e eefde jd e jfdZ dde jde jdzd e jfdZxZS)RadioInternVisionModelrNrnum_hidden_layers_overridenum_dummy_headsprefixconfig quant_configr%r&r'r"c 6t||_|t |j|j\|_|_|_ tt|j |jz |jz}td|jD}t|j|j|j|d|jrt%|nd|j|_t+|||||d|_dS)Nc3&K|] }|dV dS)nameNrrs rrfz2RadioInternVisionModel.__init__..3s&AAAaiAAAAAArTr)rWr[rZr^r2z.encoderr(r)r%r&r')r6r7r(_init_img_size to_2tupler! image_sizeimg_size grid_sizer=rNround cpe_max_sizesetteachersrT hidden_sizecls_token_per_teacherlenr2patch_generatorrencoder) r>r(r)r%r&r' max_img_sizeunique_teachersr@s rr7zRadioInternVisionModel.__init__!s(  :>:M:M f' ( (&*;; ; 7 t~t'7 &%(99 : :V=N N  AAAAAAA0    }'393OV3///UV$6   *%'A+&&&     rr1c|dSt|}tdt||D}|d|dz}|||fS)N)NNNcg|] \}}||z Srr)r*sps rr+z9RadioInternVisionModel._init_img_size..Js HHHda16HHHrrr)r/rzip)r>r!r1r2r=s rr.z%RadioInternVisionModel._init_img_sizeFs`  ##X&&HHc(J.G.GHHHII lYq\1 K//rc|jSr) embeddingsrs rget_input_embeddingsz+RadioInternVisionModel.get_input_embeddingsNs rrxrgc |jj}|jj t||} fd|D}t |}t j||t j|}d}|D]} || z} d||| || f<| }|S)Ncg|]}|zSrr)r*rrs rr+zLRadioInternVisionModel.create_inter_image_attention_mask..XsCCCw(*CCCrrrT)r:r!rr,sumr:rrO) r>rxrgr!seq_lens patch_counts total_patchesmask start_idx patch_countend_idxrs @r!create_inter_image_attention_maskz8RadioInternVisionModel.create_inter_image_attention_maskQs)4 '0 Z88CCCC(CCC L)) { = 6     '  K+-G9=D7"Ig$55 6II rrc|||}d}|/t|dkr|||j}|||}|S)Nrzrr)rr)r:r9rPrgr;)r>rrxrrencoder_outputss rrJzRadioInternVisionModel.forwardjsu ,,Q:,FF   !c*oo&9&9>>18?I,,]i,XXrrr)rKrLrMpacked_modules_mappingrrrNrr7rr.rErr:rgrPrP FloatTensorrJrQrRs@rr"r"sl w $(26# 26 # # #  # )4/# %($J #  # #  # # # # # # J03sCx3H0000uS#X/9> 8+/   < L4'           rr"cTeZdZddgiZ ddddddededzdedzd ed ed df fd Z ddd de j dzde j dzde j dzd e e j e j ffdZ d eefdZ dde j dee eefdzd e e j e j ffdZxZS) RadioModelrNrr#r$r(r)r%r&r'r"cdt||_t||||||_d}|jr^t jdt|jD}| dkr| d|||_ dS)Nr-cDg|]\}}|dd|S) use_summaryT)get)r*irs rr+z'RadioModel.__init__..s0XXXtq!QUU=RV=W=WXXXXrr summary_idxs) r6r7r(r"modelr6r:tensor enumeratenumelregister_bufferr\)r>r(r)r%r&r'r\r@s rr7zRadioModel.__init__s  +%'A+      ? C <XXy99XXXL!!##a''$$^\BBB(rrz pixel_values pixel_embedsrxc^|||}|||S)Nrz)r]_extract_final)r>rbrcrxys rrJzRadioModel.forwards2 JJ| J ; ;""1"<<r\zinput_conditioner.zmodel.patch_generator..rrCrz model.blocks.rr>rr)zls1.zls2.zmodel.encoder.layers. weight_loader) r5rlnamed_parametersrritems startswithr9splitjoingetattrr add)r>weights loaded_params params_dict weights_listr,weightsubvllm_keyparts layer_idxsuffixparamris r load_weightszRadioModel.load_weightss#"%%% 4002233 gt $ $ ) 00LL==L($ ,$ ,LD&??>22 s>**,,-C&&&~~233 H~~677 LKCIIc14E4Eb4IKK 455 LG #q0A0A"0EGG00 L #u::?? %aI XXeABBi00F//63D3DEU3V3V/ KyKK6KKH ,H 33#H- '@U V V  eV,,,!!(+++rrfcT|jjj}|jjj}|jjj}||ddd|f}|dd|df}ng}g} d} t ||D]b} |dd| |z| |z| zddf} || |dd| | |zddf} | | | || zz } ctj| d}tj|d}|j |dd|j f}n|}| d|fS)NrrrD) r]r:rr!r^r,rr:rIr\r)r>rfrxrr!r^ all_summaryall_feat all_patches summaries current_posr=rsummary bb_summarys rrezRadioModel._extract_finals:-6Z/: 3B  AAA./KHIIHHKIK,ZDD 6 6 AA{X- h0F0TTVWVWVWW""7+++AAA{[>-II111LM  )))x+55 )I1555Ky!444H   ($QQQ(9%9:JJ$J!!!$$h..rrr)rKrLrMrSrrrNrr7r:rPrrTrJr5r|rrerQrRs@rrVrVzs w37) 26 ))) ))4/) %($J )  )) ))))))>-1,0= +/ ===lT)=lT)= L4' = u %"33 4 ====/s3x////dKO///+/c3h+@4+G/ u %"33 4////////rrV)4rbcollections.abcr itertoolsrtypingrr:torch.nnr8torch.nn.functional functionalreinopsr transformersr'vllm.model_executor.layers.quantizationr-vllm.model_executor.model_loader.weight_utilsr %vllm.model_executor.models.intern_vitr r r rNrr __annotations__rrPrr to_1tupler/ to_3tuple to_4tuple to_ntupler&rr,Moduler.rTroLinearrqrrrr"rVrrrrs $$$$$$ ))))))FFFFFFOOOOOO uS#X. Y...%-.= === GAJJ GAJJ GAJJ GAJJ  1uS#X1C1C1111 >eCHo.>C>DI>>>>.....ry...bSSSSS SSSl 0%%%%%RY%%% 446&     ,   [[[[[RY[[[|z/z/z/z/z/z/z/z/z/z/r