`iI\ddlmZmZmZmZmZddlZddlmcm Z ddlmZddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$m%Z%ddl&m'Z'ddl(m)Z)m*Z*ddl+m,Z,e*rddl-m.Z.ddl/m0Z0m1Z1ndZ.e)r ddl2m3Z3m4Z4nd\Z4Z3e$r ddl5m6Z6ddl7m8Z8e%j9e:Z;dZde?dej>fdZ@ dTd ejAd!ej>d"ej>d#ej>d$eej>d%eBd&eBfd'ZCGd(d)ejAZDGd*d+ZEd,ej>d-e?fd.ZFd/ZGd0ZHeIe.e3e4fZJd1ZKGd2d3ejAZLGd4d5ejjAZMGd6d7ejAZNGd8d9ed:;ZOGd<d=ejAZPGd>d?ejAZQGd@dAejAZRGdBdCejAZSGdDdEeZTe"GdFdGeZUGdHdIejAZVe"GdJdKeUZW dUdMeej>eXej>dfdNee?d$eej>deej>e?ffdOZYGdPdQeUeZZgdRZ[dS)V)AnyCallableOptional TypedDictUnionN)nn)ACT2FN)Cache)GenerationMixin)AttentionMaskConverter)GradientCheckpointingLayer)BaseModelOutputWithPastMoeCausalLMOutputWithPastMoeModelOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringcan_return_tupleis_torch_flex_attn_availablelogging)deprecate_kwarg)is_causal_conv1d_availableis_mamba_2_ssm_available)GraniteMoeHybridConfig)selective_state_update)mamba_chunk_scan_combined mamba_split_conv1d_scan_combined)causal_conv1d_fncausal_conv1d_updateNN) BlockMask)make_flex_block_causal_maskc|dd|jddzf}|d|jddzdf}tj| |fdS)z*Rotates half the hidden dims of the input..Ndim)shapetorchcat)xx1x2s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/granitemoehybrid/modeling_granitemoehybrid.py rotate_halfr4As] 3"!'"+"" " #B 3 q "" " #B 9rc2YB ' ' ''c||}||}||zt||zz}||zt||zz}||fS)aApplies Rotary Position Embedding to the query and key tensors. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. position_ids (`torch.Tensor`, *optional*): Deprecated and unused. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. ) unsqueezer4)qkcossin position_ids unsqueeze_dimq_embedk_embeds r3apply_rotary_pos_embr@Hsc( -- & &C -- & &C3w;q>>C/0G3w;q>>C/0G G r5 hidden_statesn_repreturnc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r-expandreshape)rArBbatchnum_key_value_headsslenhead_dims r3 repeat_kvrKcs 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr5modulequerykeyvalueattention_maskscalingdropoutc Rt||j}t||j} tj||dd|z} |$|ddddddd|jdf} | | z} t j| dtj  |j } t j | ||j } tj| | } | dd} | | fS)Nr*r r))r,dtype)ptrainingr)rKnum_key_value_groupsr.matmul transposer-r functionalsoftmaxfloat32torVrSrX contiguous) rMrNrOrPrQrRrSkwargs key_states value_states attn_weights causal_mask attn_outputs r3eager_attention_forwardrgos3 ;<Q5RR S0)0)0)SR0)0)0)0)0)r5ric eZdZdZdZejdfdefdZ ddej dej de d e e e efd eej ej ff d Zd ejfd Zdde e d e fdZdS) HybridMambaAttentionDynamicCachea A dynamic cache that can handle both the attention cache (which has a seq_len dimension) and the mamba cache (which has a constant shape regardless of seq_len). This cache has two sets of lists of tensors: `key_cache` and `value_cache` for attention cache and `conv_states` and `ssm_states` for mamba cache. Each of these lists has `num_layers` tensors. The expected shape for each tensor For attention layers, `key_cache` and `value_cache` have a shape of `(batch_size, num_heads, seq_len, head_dim)`, while `conv_states` and `ssm_states` have a shape of `(batch_size, 0)` (empty tensors). For mamba layers, `key_cache` and `value_cache` have a shape of `(batch_size, 0)` (empty tensors), while `conv_states` represents the convolution state and has a shape of `(batch_size, d_inner, d_conv)`, and `ssm_states` represents the ssm state and has a shape of `(batch_size, d_inner, d_state)`. FNrjc $|j|_d|_|j}|j}g|_g|_g|_t|jD]}|j|dkrw|xjtj |j |j zd|j z|zz||gz c_|xjtj |j|j||gz c_|xjtjggzgz c_|xjtjggzgz c_|j|fdt|jD|_fdt|jD|_dS)NFmambar*devicerVrcDg|]}tjggzSrr.tensor.0r batch_sizers r3 z=HybridMambaAttentionDynamicCache.__init__..s/rrrQ%,tj'8HHHrrrr5cDg|]}tjggzSrrrs r3rz=HybridMambaAttentionDynamicCache.__init__..s/tttqEL" ):6JJJtttr5)layers_block_typehas_previous_state mamba_d_conv mamba_d_state conv_states ssm_statestransformer_layersrangenum_hidden_layersr.zeros mamba_expandrvmamba_n_groups mamba_n_heads mamba_d_headrappend key_cache value_cache)rrjrrVrconv_kernel_sizessm_state_sizeis ` ` r3rpz)HybridMambaAttentionDynamicCache.__init__s!'!9"'!.-"$v/00 2 2A%a(G33  K",v/AAAH]D]`nDnn(%# %  K",+&%#  $   U\2$2CF%S%S%S$TT  EL" 1B6$R$R$R#SS'..q1111rrrrrRWX^XpRqRqrrrtttttTYZ`ZrTsTstttr5rbrcrkrrCcD|j|jddkr||j|<||j|<nVtj|j||gd|j|<tj|j||gd|j|<|j||j|fS)Nr)rr*r+)rr-rr.r/)rrbrcrkrs r3rz'HybridMambaAttentionDynamicCache.updates >) $ *2 .! 3 3(2DN9 %*6D Y ' '(- 4>)3Lj2Y_`(a(a(aDN9 %*/)T5Ei5PR^4_ef*g*g*gD Y '~i($*:9*EEEr5beam_idxctt|jD];}|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<|j|j}|j|d|||j|<=dS)zDReorders the cache for beam search, given the selected beam indices.rN) rlenrr index_selectr_rrr)rrrkrs r3 reorder_cachez.HybridMambaAttentionDynamicCache.reorder_cache*sEs4>2233 i iI^I.5F(,y(A(N(NqRZR]R]^dReRe(f(fDN9 %%i07F*.*:9*E*R*RSTV^VaVabhViVi*j*jD Y '%i07F*.*:9*E*R*RSTV^VaVabhViVi*j*jD Y '_Y/6F)-)C)P)PQRT\T_T_`fTgTg)h)hDOI & & i ir5rc||jvr |jdn|}t|j|krdS|j|jdS)zYReturns the sequence length of the cached states. A layer index can be optionally passed.rrU)rrrr-)rrks r3get_seq_lengthz/HybridMambaAttentionDynamicCache.get_seq_length7sS3<4CZ2Z2ZD+A..`i t~  ) + +1~i(.r22r5N)r)rtrrris_compileabler.float16rrprrrdictstrrrrrrrr5r3rrs  NIN_c$u$u5$u$u$u$uV26 FFLFlF F tCH~. F u|U\) * FFFF" ie&6 i i i i33 3c333333r5r input_tensorpad_sizect|jdkr ddddd|ddfnddd|ddf}tjj||ddS)z Padding x tensor with `pad_size` on the seq_len dim (dim=1) Assumes that we only have tensors of either size 4 or 3 rconstant)moderP)rr-r.rr\pad)rr pad_shapes r3pad_tensor_by_sizerCsj 47|7I3J3Ja3O3OAq!Q!Q//VWYZ\]_gijlmUnI 8  " "<ST " U UUr5c"t||}t|jdkr.||jdd||jdS||jdd||jd|jdS)z Padding input_tensor with `pad_size` on the seq_len dim (dim=1) and simultaneously splitting it into chunk sequences. Assumes that we only have tensors of either size 4 or 3 r rr)r*)rrr-rF)rr chunk_sizes r3reshape_into_chunksrNs&lH==L < !####L$6q$92z