`iY'ddlZddlmZddlmZmZmZmZddlZddl m Z ddl m cm Z ddl mZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-ddl.m/Z/ddl0m1Z1m2Z2m3Z3edGdde j4Z5Gdde j4Z6Gdde j4Z7Gdde j4Z8Gdde j4Z9Gd d!e j4Z:d"Z;d#ej<d$ej<d%ej<d&ej<d'e=ej<ej<ff d(Z>d)ej<d*e?d'ej<fd+Z@ dVd-e j4d.ej<d/ej<d0ej<d1eej<d2eAd3eAd4e(e*fd5ZBGd6d7e j4ZCGd8d9eZDGd:d;e j4ZEd<ZFdWd=ZGGd>d?e j4ZHGd@dAe j4ZIGdBdCeZJee+dDEGdFdGe ZKe+GdHdIe&ZLGdJdKeLZMe+GdLdMeLZNe+GdNdOeLZOee+dPEGdQdRe ZPGdSdTeLeZQgdUZRdS)XN) dataclass)AnyCallableOptionalUnion) LayerNorm)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compiling)check_model_inputs) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigRMSNormc,eZdZdfd ZdZdZxZS) Glm4vRMSNormư>cttjt j||_||_dS)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s |/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/glm4v/modeling_glm4v.pyr(zGlm4vRMSNorm.__init__0sD l5:k#:#:;; #c|j}|tj}|ddd}|tj||jzz}|j||zS)NTkeepdim) dtypetor+float32powmeanrsqrtr.r-)r/ hidden_states input_dtypevariances r3forwardzGlm4vRMSNorm.forward8s|#) %((77  $$Q'',,R,>>% Ht?T4T(U(UU {]--k::::r4cHt|jjd|jS)Nz, eps=)tupler-shaper.r/s r3 extra_reprzGlm4vRMSNorm.extra_repr?s& )**II$2GIIIr4)r%)__name__ __module__ __qualname__r(rCrH __classcell__r2s@r3r$r$.sb$$$$$$;;;JJJJJJJr4r$c,eZdZddeffd ZdZxZS) Glm4VisionMlpFbiasct|j|_|j|_t j|j|j||_t j|j|j||_t j|j|j||_ t|j |_ dSNrP) r'r(r0out_hidden_sizeintermediate_sizer)Linear gate_projup_proj down_projr hidden_actact_fn)r/configrPr2s r3r(zGlm4VisionMlp.__init__Ds !-!'!74#3T5KRVWWWy!143IPTUUU 4#94;KRVWWWV./ r4c||||||zSN)rYr[rWrXr/ hidden_states r3rCzGlm4VisionMlp.forwardMs>~~dkk$..*F*FGG$,,WcJdJddeeer4F)rIrJrKboolr(rCrLrMs@r3rOrOCs_00T000000fffffffr4rOcLeZdZdeddffd ZdejdejfdZxZS)Glm4vVisionPatchEmbedr\returnNct|j|_|j|_|j|_|j|_|j|j|jg}tj|j|j|||_ dS)N) kernel_sizestride) r'r( patch_sizetemporal_patch_size in_channelsr0 embed_dimr)Conv3dproj)r/r\rgr2s r3r(zGlm4vVisionPatchEmbed.__init__Rsz  +#)#= !-+/$/R Id.K`klll r4r@c|jjj}|d|j|j|j|j}|||d|j}|S)Nr7r:) rnr-r:viewrkrjrir;rl)r/r@ target_dtypes r3rCzGlm4vVisionPatchEmbed.forward\swy'- %**  $":DOT_   -"2"2"2"F"FGGLLRQUQ_`` r4 rIrJrKr!r(r+TensorrCrLrMs@r3rdrdQszm0mTmmmmmmU\elr4rdc^eZdZUejed<d dededdffd Zdedejfd Z xZ S) Glm4vVisionRotaryEmbeddinginv_freq@dimthetareNctd|tjd|dtj|z zz }|d|ddS)N?rr6rprwF persistent)r'r(r+arangefloatregister_buffer)r/ryrzrwr2s r3r(z#Glm4vVisionRotaryEmbedding.__init__hsd %ELC%+$N$N$NQT$TUV ZeDDDDDr4seqlenctj||jj|jj}tj||j}|S)Ndevicer:)r+rrwrr:outer)r/rseqfreqss r3rCz"Glm4vVisionRotaryEmbedding.forwardms:l6$-*>dmFYZZZ C// r4)rx) rIrJrKr+rt__annotations__intrr(rCrLrMs@r3rvrveslEECEEDEEEEEE celr4rvc ZeZdZd dededededdf fd Zd ejdejfd Z xZ S) Glm4vVisionPatchMergerFry context_dimrZrPreNcttj||||_t ||_tj||||_tj||||_tj||||_ tj |_ t||_ dSrR)r'r(r)rVrnrpost_projection_normrWrXrYGELUact1r r[)r/ryrrZrPr2s r3r(zGlm4vVisionPatchMerger.__init__ts Ic3T222 $-cNN!3 $???yk=== ;$???GII Z( r4r`c||}|||}||||||zSr^)rnrrrYr[rWrXr_s r3rCzGlm4vVisionPatchMerger.forward~snyy.. yy!:!:1D!-"$,t/A4>"R"R ^U\$:L-M-M-T-TU\-]-]jopppppr4rec|jj}|jd}|jd}|j} || || }}|dkrt jd|| |j} nttr!t j | t j tt j s!t j | t j |jd} t| dz} || | |dddd| t j} t jfdt't)D| t j}t jfdt't)D| t j}|| t j}|| t j}|dz|z dzdz }|dz|z dzdz }t j||fd dd}t-j| |d d d }|dddd}||j|j} || z}|S)a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. rrrg?r6cVg|]%}|df|&Srrepeat.0i image_shapeslengthss r3 z1Glm4vVisionEmbeddings.forward..4!e!e!eA,q!t"4";";GAJ"G"G!e!e!er4cVg|]%}|df|&S)r6rrs r3rz1Glm4vVisionEmbeddings.forward..rr4r7rybicubicFborder)mode align_corners padding_mode)rr-rFrr;r+emptyr: isinstancelisttensorlongrtrrqpermute unsqueezer<catrangelenstackF grid_samplesqueeze)r/ embeddingsrrh_coordsw_coordspos_embed_weightr0 total_seqradapted_pos_embed orig_size_sq orig_size pos_embed_2dtarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32s `` r3rCzGlm4vVisionEmbeddings.forwards( 29&,Q/ N1% !(&[[00(++f2E2E( >> % A{6QaQg h h h  '4(( Q,wvUZPPPlEL99 [$|LuzZZZ ,1!4LL#-..I %%iKHHAq!!1677 y!e!e!e!e!eQVWZ[bWcWcQdQd!e!e!effiiU]jHy!e!e!e!e!eQVWZ[bWcWcQdQd!e!e!effiiU]jH  {{& {FFH{{& {FFH#~1Q6:F#~1Q6:F;/R888BB1EEOOPQRRD'(md%V^''' # &=%D%DQ%G%G%O%OPR%S%S%[%[\]_`%a%a " 6 9 9:J:P Q Q T TU_Uf g g  "33 r4rsrMs@r3rrst q0 q q q q q qGPUP\GGGGGGGGr4rc|dd|jddzf}|d|jddzdf}tj| |fdS)*Rotates half the hidden dims of the input..Nr7r6r)rFr+rxx1x2s r3 rotate_halfrs] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''r4qkcossinrec|j}|j}||}}|d|d}}||zt||zz}||zt||zz}||}||}||fS)N)r:rrrr;)rrrr orig_q_dtype orig_k_dtypeq_embedk_embeds r3apply_rotary_pos_emb_visionrs7L7L 7799aggiiqA}}R  &&((#--*;*;*A*A*C*CC3w;q>>C/0G3w;q>>C/0Gjj&&Gjj&&G G r4r@n_repc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rFrreshape)r@rbatchnum_key_value_headsslenhead_dims r3 repeat_kvrs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr4modulequerykeyvalueattention_maskscalingdropoutkwargsc Rt||j}t||j} tj||dd|z} |$|ddddddd|jdf} | | z} t j| dtj  |j } t j | ||j } tj| | } | dd} | | fS)Nr6r rr7ryr:)ptrainingr)rnum_key_value_groupsr+matmul transposerFr) functionalsoftmaxr<r;r:rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r3eager_attention_forwardrs3 ;<.PsAAG FGNN$4$4!<<<r4c lg|]0\}}}|||fdjjsdnjddd1S)NrF)rrrr r)rrr )rrrvattention_interfacerr/s r3rz0Glm4vVisionAttention.forward..Ts   Aq!$#  $( L'+}PCC$:P#        r4r)rFr rrrunbindrrrrr\_attn_implementationrmaxrrr zipr+rrrn)r/r@r r rr seq_length query_statesrrrr max_seqlenr_splits attn_outputsrrs` ` @@r3rCzGlm4vVisionAttention.forward$s#(+ HH] # # + +J4>2 N N V VWXZ[]^`a b b i ijk l l / j,'S#>|ZY\^a#b#b j#--a33==a@@ ))!Q//99!<< #--a33==a@@ (? ; +w 6 6"9$+:Z"[  ; +/B B B$QRR.:crc?:??AAJ00  $ #'=Ld6L((''NK"!nz#2#6GLXZdfrKsF       #F|   L )La888K!))*b99DDFF ii ,, r4NN) rIrJrKr!r(r+rtrrErCrLrMs@r3rrs 0 T      "26KO BB|BLB!. B &eEL%,,F&GH B BBBBBBBBr4rc eZdZd fd Z d dejdejdeejdeeejejfdejf dZxZ S) Glm4vVisionBlockreNctt|j|j|_t|j|j|_t||_t|d|_ dS)Nr1FrS) r'r(r$r0 rms_norm_epsnorm1norm2rattnrOmlprs r3r(zGlm4vVisionBlock.__init__jst !&"4&:MNNN !&"4&:MNNN (00  e444r4r@r r rc ||j||f|||d|z}||||z}|S)N)r r r)r.r,r/r-)r/r@r r rrs r3rCzGlm4vVisionBlock.forwardqst&  JJ} % %) !) 3 ) )  ) )  &M1J1J(K(KK r4reNr&) rIrJrKr(r+rtrrErCrLrMs@r3r(r(is55555526KO |L!.  &eEL%,,F&GH  r4r(c|eZdZUejed<ddeffd Zeje dZ xZ S)Glm4vTextRotaryEmbeddingrwNr\ctt|dr@|j9|jd|jd|_nd|_|j|_|j|_||_ t|j|_ | |j |\}|_ | d|d|j|_dS)N rope_scaling rope_typetypedefaultrwFr})r'r(hasattrr5getr6max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr\r rope_init_fnattention_scalingrrworiginal_inv_freq)r/r\rrwr2s r3r(z!Glm4vTextRotaryEmbedding.__init__s  6> * * 'v/B/N#044[&BUBYBYZ`BaBabbDNN&DN"("@$*$B! /?+/+<+DDFFMMaQ]QcdeQfhjlmnn ,QQQ4] ; A A C C'1!(-'E'Ek!(-[`J`J`ahmmfk ^ U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)UEN333C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s>BEEEr^) rIrJrKr+rtrr r(no_gradrrCrLrMs@r3r3r3sl////////"U]__ < <_ < < < < .$VVV$!QQq1uXVVVr4r7rc*g|]\}}||dzSrPrQrRs r3rz3apply_multimodal_rotary_pos_emb..rTr4.N)r+r enumeraterrrFrepeat_interleaverM) rrrr mrope_section unsqueeze_dim rotary_dimq_rotq_passk_rotk_passrrs r3apply_multimodal_rotary_pos_embr_s:"A%M )VV)CIImQSI4T4T*U*UVVV\^ _ _ _ i i  C )VV)CIImQSI4T4T*U*UVVV\^ _ _ _ i i  C c'SYr]a''' ( : :1" : E EC c'SYr]a''' ( : :1" : E EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{u55;@PW\]]] r4r@rrrpast_key_valuescache_positionrrec |\}} } ||} ||} ||} | || d|jdd} | || d|jdd} | || d|jdd} |\}}t| | |||jd\} } |&|||d}| | | |j |\} } t}|j j dkrt|j j }||| | | |f|jsdn|j|jd|\}}||| d}||}||fS) Nr7rr6rX)rrrkrr)rr)sizererfrgrqrrr_r5updaterbrr\rrrr rrrrh)r/r@rrrrjrkrbszq_lenr#r!rrrr cache_kwargsrrrs r3rCzGlm4vTextAttention.forwards&**,, UA{{=11 [[// {{=11 #((eRGGQQRSUVWW __S%T]CCMMaQRSS #((eRGGQQRSUVWW &S#B *c30A/0R$ $  j  &#&snUUL'6'='=j,X\Xfht'u'u $J (? ; +w 6 6"9$+:Z"[ $7$7      % $}HCC$2HL % %  % % ! \"))#ub99DDFF kk+.. L((r4r^NNNN)rIrJrK__doc__r rrr(r+rtrE LongTensorr rrrCrLrMs@r3raras. ^^^8C=^^^^^^02637+/59.).)|.)#5<#=>.)!. .) u/0 .) "% .)!!12.)-..) u|Xel3XeEL>Q5RR S.).).).).).).).)r4racBeZdZfdZdejdejfdZxZS) Glm4vTextMLPc"t||_tj|jd|jzd|_tj|j|jd|_t|j |_ dS)Nr6FrS) r'r(r\r)rVr0rU gate_up_projrYr rZ activation_fnrs r3r(zGlm4vTextMLP.__init__6sz  If&8!f>V:V]bccc6#;V=OV[\\\#F$56r4r@rec||}|dd\}}|||z}||S)Nr6r7r)rxchunkryrY)r/r@ up_statesgates r3rCzGlm4vTextMLP.forward>sX%%m44 #//!/44i 2 24 8 88 ~~i(((r4)rIrJrKr(r+ FloatTensorrCrLrMs@r3rvrv5s`77777)U%6)5;L))))))))r4rvc@eZdZdedeffd Z ddejdeejejfde ejd e ej d e e d e e d e e d e ej deej e eej ej fffdZxZS)Glm4vTextDecoderLayerr\rbct|j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ dS)Nr*) r'r(r0ra self_attnrvr/r$r+input_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormris r3r(zGlm4vTextDecoderLayer.__init__Hs !-+FI>>''+F,>FDWXXX(4V5GVM`(a(a(a%(4V5GVM`(a(a(a%".v/AvGZ"["["[r4NFr@rrrrjoutput_attentions use_cacherkrec &|} ||}|jd||||||||d| \}} ||}| |z}|} ||}||}||}| |z}|S)N)r@rrrrjrrrkrQ)rrrrr/r) r/r@rrrrjrrrkrresidualr#s r3rCzGlm4vTextDecoderLayer.forwardRs!,,];; *4>  ' 3)%+/)       q55mDD  =0 !55mDD // // >>  =0 r4)NNNFFN)rIrJrKr rr(r+rtrErrtr rbr~rCrLrMs@r3rrGs-\\3\\\\\\2637+/,1$)59''|'#5<#=>'!. ' u/0 ' "% '$D>'D>'!!12' u (51BEDU1U+V"WW X''''''''r4rzJ Base class for Llava outputs, with hidden states and attentions. ) custom_introceZdZUdZdZeejed<dZ ee ed<dZ ee ejed<dZ ee ejed<dZeejed<dS)Glm4vModelOutputWithPasta[ past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_staterjr@ attentions rope_deltas)rIrJrKrsrrr+r~rrjr r@rErrrtrQr4r3rr|s6:x 12999'+OXe_+++8.s"$[$[$[!%5f%=%=$[$[$[r4)ryrrZr*)rk out_channelsrgrhF)r'r(spatial_merge_sizerirrrd patch_embedr0rrvr r) ModuleListrdepthblocksrrTrUrZmergerr$r+post_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)r/r\rr2s ` r3r(zGlm4vVisionModel.__init__sS    "("; +/77088%)998QGGm$[$[$[$[uV\GZGZ$[$[$[\\ ,&F4LY_Yj    $00BH[#\#\#\ )*/1,     +6+=6CVWWW&+# r4cg}|D]x\}}}tj|dd|}|||jz|j||jz|j}|dddd}|}tj|d|d}|||jz|j||jz|j}|dddd}|}|tj ||gd |dztj |d}|ddddf }| |} | |d} | |fS)Nrr7rr6r r)r+rrrrrrrLappendrrrrr ) r/grid_thwpos_idsthwhpos_idswpos_ids max_grid_sizerotary_pos_emb_fullr s r3 rot_pos_embzGlm4vVisionModel.rot_pos_embs S SGAq!|A0033::2qAAH''T,,'T,,' H  ''1a33H''))H|A0033::1bAAH''T,,'T,,' H  ''1a33H''))H NN5;(';DDDKKAqQQ R R R R)G+++ ABB++-- "11-@@,W5==a@@w&&r4r@rc H||}||}||\}}tj||fd}||f}tj|dddf|dddfz|dddfdtj r|j n tj }tj|dd }|dd|ddz }|||||dddf|dddf}|jD]} | ||| }||}|d|j|j|jd}|dd dd}||d|jj}||}|S) az Args: hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. r7rNrr6rr)rr)r)r rr )rrrr+rrrrWcumsumjit is_tracingr:int32rpadrrrrrqrrFrrr\rTr) r/r@rr image_type_idsrIrr seqlensblks r3rCzGlm4vVisionModel.forwards7((77 00?? )-)9)9()C)C&i8bAAA"wwyy#''))4,Xaaad^hqqq!tn-LhWXWXWXZ[W[n]]dd %*I$8$8$:$:K(.. e  U:vQ777 abb>JssO3;;== w.YZYZYZ\]Y]J^`nopopoprsos`tuu ;  CC%$7MM ++M:: %** ')@-BUVXBY  &--aAq99  66;;B @[\\  M22 r4r1) rIrJrKr!rrr(rr+rtrCrLrMs@r3rrs +,8''':.U\.U\.el........r4rc.eZdZUeed<deffd Zee ddee j dee j dee j dee dee j d eed ee j d eed eeeffd ZxZS)Glm4vTextModelr\ctj|_j|_t jjj|j|_t j fdtj D|_ tjj|_t!|_d|_|dS)Nc0g|]}t|SrQ)r)rrbr\s r3rz+Glm4vTextModel.__init__..s$ g g g) "69 5 5 g g gr4r*r\F)r'r( pad_token_id padding_idx vocab_sizer)rr0 embed_tokensrrnum_hidden_layerslayersr$r+normr3 rotary_embrrrs `r3r(zGlm4vTextModel.__init__s    !. +L):F>>O   --i88M  !CRC^==???de "\ "2]5H5K"KTaThN  )..q!R88??=CVWXCY[]^^LL     1 $ $' 299!\=OPQ=RTVWWL(;&))+%    & #oom\JJ![ * *M)M$7*) /- M*MM -00 &++    r4)NNNNNNN)rIrJrKr rr(rrrr+rtrtr r~rbrrrrErrCrLrMs@r3rrsC  151537+/59$(59@ @ E,-@ !.@ u/0 @ "% @   12 @ D>@ !!12@ -.@  u-- .@ @ @ ^@ @ @ @ @ r4rc4eZdZUdZiZdZeed<ddgZfdZ dZ dZ d Z d Z d d eejd eejdeejdeejdeejejff dZ d!dejdeejfdZd!dejd eejfdZ d"d ejdejdeejdeejfdZee d#d eejdeejdeejdeedeejdeejdeejd eejdeejdeejdeejdeedeeeffdZxZ S)$ Glm4vModelFr\rr(ct|t|j|_t |j|_d|_ | dSr^) r'r(r _from_config vision_configvisualr text_configlanguage_modelrrrs r3r(zGlm4vModel.__init__vsl    &33F4HII ,99&:LMM r4c4|jSr^)rget_input_embeddingsrGs r3rzGlm4vModel.get_input_embeddingss"77999r4c:|j|dSr^)rset_input_embeddingsr/rs r3rzGlm4vModel.set_input_embeddingss 0077777r4c||_dSr^rr/decoders r3 set_decoderzGlm4vModel.set_decoders%r4c|jSr^rrGs r3 get_decoderzGlm4vModel.get_decoders ""r4Nrimage_grid_thwvideo_grid_thwrrec |jjj}|jj}|jj}|jj}g} |6||1|} |t j| }t jd|j d|j d|j |j } d\} } d}| | j }t| D]|\}}|||dk}|}g}d}|D]d}||krd}n||krd}||kr|s|d 1||kr|r|d O|d eg}t!jt|d D]K\}}t%|}|dd}|d ddz}||||fLg}d}|D]\}}}t'|dkr|d dznd}|d kr|| d|| d|| d}!} }|| |z|!|z}$}#}"t j|"d dd |#|$z}%t j|#dd d|"d |$}&t j|$ddd |"|#d }'|t j|%|&|'g|z| dz } d}|d kr||| d|| d}!} }|| |z|!|z}$}#}"t7|"D]!}(t j|(d dd |#|$z}%t j|#dd ddd |$}&t j|$ddd d|#d }'|t j|%|&|'g|z#|dz }||| dkr| dz } d}|dz }||z })|t j|)dd dd |zd}t j|ddd }*|* | j | d|||dkf<| |*dzt'| |z ~t j| |j d} | | fS|| !d dz } | "|dkd| ddd d  |j } | dddd dd}+|+dz|j d z } nt j|j d|j ddd d|j dd } t j#|j ddg|j |j } | | fS)aU Calculate the 3D rope index based on image and video's temporal, height and width in LLM. Explanation: Each embedding sequence contains vision embedding and text embedding or just contains text embedding. For pure text embedding sequence, the rotary position embedding has no difference with modern LLMs. Examples: input_ids: [T T T T T], here T is for text. temporal position_ids: [0, 1, 2, 3, 4] height position_ids: [0, 1, 2, 3, 4] width position_ids: [0, 1, 2, 3, 4] For vision and text embedding sequence, we calculate 3D rotary position embedding for vision part and 1D rotary position embedding for text part. Examples: Temporal (Time): 3 patches, representing different segments of the video in time. Height: 2 patches, dividing each frame vertically. Width: 2 patches, dividing each frame horizontally. We also have some important parameters: fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second. tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity. temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames. interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs. input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision. vision temporal position_ids: [0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100] vision height position_ids: [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1] vision width position_ids: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1] text temporal position_ids: [101, 102, 103, 104, 105] text height position_ids: [101, 102, 103, 104, 105] text width position_ids: [101, 102, 103, 104, 105] Here we calculate the text start position_ids as the max vision position_ids plus 1. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) Nr rrr:r)rrFTimagevideotextc|dS)NrrQ)rs r3z+Glm4vModel.get_rope_index..s [\]^[_r4r7r6r.rr8r)$r\rrimage_token_idvideo_start_token_idvideo_end_token_idr+ ones_liker,rFr:rr;rVrr itertoolsgroupbyrrritemrrqrrLrrrrrrrr masked_fill_zeros),r/rrrrrrrrmrope_position_deltastotal_input_idsr image_index video_indexvideo_group_indexr input_tokensinput_token_typevideo_check_flgtokeninput_type_grouprgroup start_index end_indexllm_pos_ids_listvideo_frame_num modality_type start_idxend_idxst_idxrrr llm_grid_t llm_grid_h llm_grid_wt_indexh_indexw_indext_idxtext_len llm_positionsmax_position_idss, r3get_rope_indexzGlm4vModel.get_rope_indexsv"[6I3#{?![; "  n&@ND^'O%!&!A!A :""o ' L(, $K ! +../EFFN )/ : :W `W ` 9%nQ&71&<= (//11 #% "') 8 8E 444*."444*/...(//8888.00_0(//8888(//7777#% "+"3I>N4O4OQ_Q_"`"`KKJC KKE"'(1+K %b ! q 0I$++S+y,IJJJJ#% "#9I7,7,5M9g?BCS?T?TWX?X?X-b15577!;;^_F$//*;7:*;7:*;7: 1 FFHHFFHH(::FFHH(::1;J #(,z":":"?"?A"F"F"M"MbR\_iRi"j"j"r"r"t"t"',z":":"?"?2q"I"I"P"PQ[]_ak"l"l"t"t"v"v"',z":":"?"?1b"I"I"P"PQ[]gik"l"l"t"t"v"v(// Wgw&>r1&E&E&L&LRQ[^hQh&i&i&q&q&s&sG&+l:&>&>&C&CAr1&M&M&T&TUVXZ\f&g&g&o&o&q&qG&+l:&>&>&C&CAq"&M&M&T&TUVXbdf&g&g&o&o&q&qG,33EK'SZ@[4\4\_e4effff)Q.),{0KA0NNN'1,K01-'1,$+Y#6(// X0F0F0K0KAr0R0R0Y0YZ[]_0`0`ci0ijjj*+ % *: B B B J J1b Q Q ?L?O?OP\Pc?d?d S!^A%6!%;;<%,,]->->-@-@1-Ds?[\K]G^G^-^____$)L1FyO_$`$`$`$j$jkl$m$m !!66 6)-2244;;B??!C )).A*=qAAA+55a88??2rJJMMnNcdd #/#3#3Au#3#E#Ea#H#L#LRY]#L#^#^_`#a (81(<~?STV?W(W%%L!3I%?%?&$&B9?1+=+BO!-M-2Ea2HA2M #-E,F,F(F)t+Fq0AQ0FV#t+T/M/M/O/OST/T &' K*C KHXH`,0,?,?""#8 -@--) k $/  -:,?) J&1$A&)99==m>RSSS %|J}?STTT +00B77>>z2NN !-!33J%+a.4PVW3XXE+//66 +55a88??2rJJ %$% %)+')     (%7#3!/)(     r4rrr^r&) NNNNNNNNNNN)!rIrJrKr_checkpoint_conversion_mappingaccepts_loss_kwargsrrrr(rrrrrr+rtrtrErr~r)r-r:rrr rrrrrCrLrMs@r3rrmsH%'" 02DE:::888&&&### 15595915 |7|7E,-|7!!12|7!!12 |7 !. |7 u|U\) * |7|7|7|7~dh#(#4FNuO_F`0u/@RZ[`[kRl(7;6: '6'6#'6('6!!23 '6 !!23 '6'6'6'6R151537+/59/3;?59592659g g E,-g !.g u/0 g "% g   12 g u|,g &e&78g !!12g !!12g e./g !!12g +,g  u.. /g g g ^g g g g g r4rzQ Base class for Glm4v causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ed<dZ ee ejed<dZee ejed<dZeejed<dS) Glm4vCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlosslogitsrjr@rr)rIrJrKrsrTrr+r~rrUrjr r@rErrrtrQr4r3rSrSs  )-D(5$ %,,,*.FHU& '...'+OXe_+++8JJJr4c|jjSr^)rrrGs r3rz,Glm4vForConditionalGeneration.language_modelEs z((r4c|jjSr^)rrrGs r3rz$Glm4vForConditionalGeneration.visualIs z  r4rrrrrjrlabelsrrklogits_to_keeprrec ~|jd|||| | ||||| d |}|d}t| trt| dn| }||dd|ddf}d}|'||||jjj}t|||j |j |j |j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```) rr*r rrrrrjrrkrN)rUrbr)rTrUrjr@rrrQ)rrrslicerY loss_functionr\rrrSrjr@rr)r/rrrrjrrbr*r rrrrkrcrrOr@ slice_indicesrUrTs r3rCz%Glm4vForConditionalGeneration.forwardMsz$*  % 3))%)+')        9C>SV8W8Wk~ot444]k mAAA}aaa,?@AA  %%VFt{OfOq%rrD*#3!/)+     r4Tc  tj|f||||||| | | |d | } d| d<|ddkr d| d<d| d<| S)N) rjrrrkrr*r rrrrrr*r )r'prepare_inputs_for_generation)r/rrjrrrkrrr*r rrr model_inputsr2s r3riz;Glm4vForConditionalGeneration.prepare_inputs_for_generations"=uww<   +)')%% 3))      (, ^$ !  ! !+/L (26L. /r4cZ|||tj|jjtj|jkd}||tj|jjtj|jkd}||tj|jjtj|jkd}n0||jjk}||jjk}||jjk}tj | | z d}|dk}||z}| d} | d} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) Nr).rrrr) rr+rr\image_start_token_idrrrrrrr3) r/rris_imageis_video_start is_video_end video_level inside_videostandalone_images image_counts video_countss r3_get_image_nums_and_video_numsz#5#5#7#7,:J:J:L:L#LRSTTT "Q % 6),,,33 %))a)00 \))r4r expand_sizeis_encoder_decoderc dkrfSgdfd}fd}|d||r8dtd|dd<fS) Nr)r*rr rsecond_per_grid_tsc dd} dd}  dd\}}d}|D]}|dkrFtj|t |}d|D}|||| ||<O|dkr't |}|||| ||<||d krFtj|t |}d |D}|||| ||<|dkr't |}|||| ||<|d kr$|||t | ||<!|S) Nrrr)rctj||}|gdg|dz zztjfd|Dd}|S)Nrc$g|] }|j SrQr)rsample repeat_argss r3rzGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples..'s"#V#V#VFMFM;$?#V#V#Vr4rr)r+rryr)rr repeat_timessamplesresultr~s @r3_repeat_interleave_sampleszGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_samples$sb+a11+nsaeeggk/BB #V#V#V#Vg#V#V#V\]^^^ r4r*c^g|]*}tj|d+Srrr+r#r3rr}s r3rz{Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual../3UUU6uz&a888<<>>UUUr4)rrr c^g|]*}tj|d+Srrrs r3rz{Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual..;rr4ry)r:rur+rr) dict_to_expandrr image_nums video_numsrrrrrvr model_kwargsr/s r3"_expand_dict_for_generation_visualzgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visuals)--.>EEN)--.>EEN%)%H%H)9)9/4)P)P&I&& "J     &  .((#k.$z:J:JKKGUUWUUUG*D*D&s+W;+++N3'',,,":..G*D*D&s+W;+++N3''111#k.$z:J:JKKGUUWUUUG*D*D&s+W;+++N3'',,,":..G*D*D&s+W;+++N3''000*D*D&s+T*5E5ET_+++N3'" !r4c|D]T}|dkrL||Dt||tjr$|vr ||d||<U|S)Nrkrr)rr+rtrW)rrrv visual_keyss r3_expand_dict_for_generationz`Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationJsz% d d+++&s+7">##6 EE8;..*8*=*O*OP[ab*O*c*cN3'! !r4rrencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rWr:r)r/rvrwrrrrrs`` `` @r3_expand_inputs_for_generationz;Glm4vForConditionalGeneration._expand_inputs_for_generation s !  l* *www + "+ "+ "+ "+ "+ "+ "+ "Z " " " " " ":9,GG  !33KQ3GGI22<@@  k 122: !pqqq.I.I,WhJi.j.jL* +,&&r4r^) NNNNNNNNNNNNr) NNNNNTNNNN)rFN)'rIrJrKrP_tied_weights_keysrQr(rrrrr+r~rrtr)r-propertyrrrrrtr rrrrrErSrCrirurbrArrrrLrMs@r3rWrW#s%'"*+111///((((((dhRR#(#4RFNuO_F`RRRR KKu/@KRZ[`[kRlKKKK))X)!!X!151537+/59-1/3;?5959265934Z Z E,-Z !.Z u/0 Z "% Z   12 Z )*Z u|,Z &e&78Z !!12Z !!12Z e./Z !!12Z c5</0Z +,Z u11 2!Z Z Z ^Z ~ ''''''X156*6*E,-6*  -6* u|U\) * 6*6*6*6*t#(04 U'U'U'!U'E,- U' uc3h/ 0 U'U'U'U'U'U'U'U'r4rW)rWrrr)rr)Sr dataclassesrtypingrrrrr+torch.nnr)torch.nn.functionalrrr activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrconfiguration_glm4vrr r!Moduler$rOrdrvrrrrtrErrrrrrr(r3rMr_rarvrrrrrrrSrW__all__rQr4r3rs:*!!!!!!111111111111 !!!!!!........))))))777777//////BBBBBB999999DDDDDDDDKKKKKKKKFFFFFFFF&&&&&&cccccccccccc//////PPPPPPPPPPY''JJJJJ29JJ('J( f f f f fBI f f fBI(        fffffRYfff"TTTTTBITTTn(((  |   +0< >Cl  5< %&     UU\ U# U%, U U U U&%% I% <% % < % U\* %  %%'(%%%%4PPPPP29PPPf16#<#<#<#<#