`i\ddlmZmZmZddlZddlmZddlmZddl m Z m Z ddl m Z ddlmZmZddlmZdd lmZdd lmZmZdd lmZmZdd lmZmZdd lmZddl m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'ddl(m)Z)Gddej*Z+Gddej*Z,dej-de.dej-fdZ/ d4dej*dej-dej-dej-d eej-d!e0d"e0d#ee!fd$Z1d%Z2d5d&Z3Gd'd(ej*Z4Gd)d*ej*Z5Gd+d,eZ6e"Gd-d.eZ7e"Gd/d0e7Z8e"Gd1d2e7e Z9gd3Z:dS)6)CallableOptionalUnionN)ACT2FN)Cache DynamicCache)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuple)deprecate_kwarg)check_model_inputs) Cohere2Configc|eZdZUejed<ddeffd Zeje dZ xZ S)Cohere2RotaryEmbeddinginv_freqNconfigctt|drSt|jt r9|jd|jd|_nd|_|j|_ |j|_ ||_ t|j|_ | |j |\}|_|d|d|j|_dS)N rope_scaling rope_typetypedefaultrF) persistent)super__init__hasattr isinstancer"dictgetr#max_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr r rope_init_fnattention_scalingregister_bufferroriginal_inv_freq)selfr devicer __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/cohere2/modeling_cohere2.pyr(zCohere2RotaryEmbedding.__init__.s  6> * * 'z&:Mt/T/T '#044[&BUBYBYZ`BaBabbDNN&DN"("@$*$B! /?+/+<+>@@GG HZ[\H]_acdee ,QQQaaaZ 8 > > @ @'1!(-'E'Ek!(-[`J`J`ahmmfk ^ U C C C 5 5&,,..1F1L1L1N1NNYYZ[]^__E)%;;;C''))d44C''))d44C  5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 vvAGv$$cff17f&;&;;;s9B EEEN) __name__ __module__ __qualname__rHTensor__annotations__rr(no_gradrrU __classcell__r6s@r7rr+sl//}//////"U]__ < <_ < < < < Fcttjt j||_||_dS)zcThe hidden size can be a tuple or an int. The tuple is used for QKNorm to normalize across head_dimN)r'r(nn ParameterrHonesweightvariance_epsilon)r4 hidden_sizeepsbiasr6s r7r(zCohere2LayerNorm.__init__PsB l5:k#:#:;; #r8c|j}|tj}|dd}||z ddd}||z tj||jzz}|jtj|z}||S)Nr:T)keepdimr?) rCrNrHfloat32meanpowrsqrtrgrf)r4 hidden_states input_dtypernvariances r7rUzCohere2LayerNorm.forwardVs#) %((77 !!"d!33!D(--a0055b$5GG&-XH]=]1^1^^  u}55 E  ,,,r8)NraFrWrXrYr(rUr]r^s@r7r`r`OsL$$$$$$ -------r8r`rqn_repreturnc|j\}}}}|dkr|S|dddddddddf|||||}||||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rFrEreshape)rqrubatchnum_key_value_headsslenhead_dims r7 repeat_kvr}`s 2?1D.E h zz!!!!QQQaaa"23::5BUW\^bdlmmM  (;e(CT8 T TTr8modulequerykeyvalueattention_maskscalingdropoutkwargsc Rt||j}t||j} tj||dd|z} |$|ddddddd|jdf} | | z} t j| dtj  |j } t j | ||j } tj| | } | dd} | | fS)Nr?rr:)rArC)ptrainingr)r}num_key_value_groupsrHmatmulrJrFrc functionalsoftmaxrmrNrCrr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r7eager_attention_forwardrls3 ;<>C/0G3w;q>>C/0G ::E: " "GJJUJ$;$; ;;r8cReZdZdZddedeeffd Zeddd  dd e j d e e j e j fd ee j dee d ee j deede e j ee j ee e j ffdZxZS)Cohere2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperNr layer_idxct||_||_t |d|j|jz|_|j|jz|_ |jdz|_ |j |_ d|_ |j |dkr|jnd|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|j|jz|j|_tj|j|jz|j|j|_dS)Nr|gTsliding_attentionrj)r'r(r rgetattrrhnum_attention_headsr|rzrrattention_dropout is_causal layer_typessliding_windowrcLinearattention_biasq_projk_projv_projo_projr4r rr6s r7r(zCohere2Attention.__init__sf  " F4F&Jd4dee $*$>&B\$\!}d* !'!97=7I)7TXk7k7kf33qui   :T] JQWQf    i   :T] JQWQf    i   :T] JQWQf    i  & 68JQWQf    r8past_key_valuepast_key_values4.58new_nameversionrqposition_embeddingsrcache_positionrrvc ^|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } |jt| | | | \} } |&| | |d}| | | |j |\} } t}|j j dkrt|j j }||| | | |f|jsdn|j|j|jd|\}}|jg|dR}||}||fS)Nr:rr?)rMrLreagerr~)rrr)rFr|rviewrJrrrrupdaterrr _attn_implementationrrrrrxrr)r4rqrrrrr input_shape hidden_shape query_statesrrrLrM cache_kwargsattention_interfacerrs r7rUzCohere2Attention.forwards$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S   *';L*VY[^'_'_ $L*  &#&snUUL'6'='=j,X\Xfht'u'u $J (? ; +w 6 6"9$+:Z"[ $7$7      % $}HCC$2HL. % %  % % ! \*k);;;;;;FFHH kk+.. L((r8rV)NN)rWrXrY__doc__rrintr(rrHrZtupler LongTensorrr rUr]r^s@r7rrs*GG  } #      0_%0A6RRR ,059 *)*)|*)#5<#=>*)!. *) "% *) !!12 *)-.*) u|Xel3XeEL>Q5RR S*)*)*)SR*)*)*)*)*)r8rc$eZdZfdZdZxZS) Cohere2MLPct||_|j|_|j|_t j|j|jd|_t j|j|jd|_t j|j|jd|_ t|j |_ dSNFr) r'r(r rhintermediate_sizercr gate_projup_proj down_projr hidden_actact_fnr4r r6s r7r(zCohere2MLP.__init__s  !-!'!94#3T5KRWXXXy!143IPUVVV 4#94;KRWXXXV./ r8c||||||z}|SrV)rrrr)r4rOrs r7rUzCohere2MLP.forwardsANN4;;t~~a/@/@#A#ADLLQROO#STT r8rtr^s@r7rrsG00000r8rcJeZdZdedeffd Zeddd dd ejd e ejejfd e ejde e d e e de ej deede eje e ejejfffdZxZS)Cohere2DecoderLayerr rct|j|_t|||_t ||_t|j|j|_ |j ||_ dS)N)r rrhri) r'r(rhr self_attnrmlpr`layer_norm_epsinput_layernormrattention_typers r7r(zCohere2DecoderLayer.__init__sy !-)9MMMf%%/V=OV\Vklll$0;r8rrrrNFrqrr use_cacherrrvc |}||}|jd||||||d|\} } ||} || z| z}|S)ar Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch_size, sequence_length)` if flash attention is used or `(batch_size, 1, query_sequence_length, key_sequence_length)` if default attention is used. past_key_values (`Cache`, *optional*): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence position_embeddings (`tuple[torch.FloatTensor, torch.FloatTensor]`, *optional*): Tuple containing the cosine and sine positional embeddings of shape `(batch_size, seq_len, head_dim)`, with `head_dim` being the embedding dimension of each attention head. )rqrrrrr)rrr) r4rqrrrrrrresidualhidden_states_attention_hidden_states_mlps r7rUzCohere2DecoderLayer.forwards<!,,];; %3T^& ' 3)+) & & & & "!HH]33 #::=NN r8)NNFN)rWrXrYrrr(rrHrZrrrboolrrr FloatTensorrUr]r^s@r7rrs0<}<<<<<<<_%0A6RRR 26+/$)59++|+#5<#=>+!. + "% + D> +!!12+-.+ u (51BEDU1U+V"WW X+++SR+++++r8rcLeZdZUeed<dZdZdgZdgZdZ dZ dZ dZ dZ eedZdS)Cohere2PreTrainedModelr modelTrr)rq attentionsN)rWrXrYrr[base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraph_supports_attention_backendrr_can_record_outputsrr8r7rr=sl &*#./#4"5N!"&,&r8rceZdZdeffd Zee ddeej deej deej dee deej d ee d eej d eed efd ZxZS) Cohere2Modelr ctj|_j|_t jjj|j|_t j fdtj D|_ tjj|_t!|_d|_|dS)Nc0g|]}t|Sr)r).0rr s r7 z)Cohere2Model.__init__..Ys$ e e e  3 3 e e er8rr F)r'r( pad_token_id padding_idx vocab_sizerc Embeddingrh embed_tokens ModuleListrangenum_hidden_layerslayersr`rnormr rotary_embgradient_checkpointing post_initrs `r7r(zCohere2Model.__init__Rs    !. +L):F>>O  !CRC^==???de "\ "2]5H5K"KTaThN  )33A66L?-FF + -"0"0#2 , K#5"C"C{"C"C%F%U%U%U%U##  & "oom\JJ![  M)M$72=3OP /#- MM -00 &++    r8)NNNNNNN)rWrXrYrr(rrrrHrrZrrrrrrrUr]r^s@r7rrPs} 151537+/59$(59< < E,-< !.< u/0 < "% <   12 < D>< !!12< +,<  !< < < ^< < < < < r8rceZdZdgZddiZddgdgfiZfdZee dd e e j d e e j d e e j d e e eee jfd e e jde e j de ede ede ede e j de ee j fdeedefdZxZS)Cohere2ForCausalLMzlm_head.weightlm_head colwise_reprqlogitsc.t|t||_|j|_t j|j|jd|_|j |_ |j |_ | dSr) r'r(rrrrcrrhr logit_scaletie_word_embeddingsr rs r7r(zCohere2ForCausalLM.__init__s    !&))  +y!3V5FUSSS !-#)#=  r8NrrrrPrrlabelsroutput_attentionsoutput_hidden_statesrlogits_to_keeprrvc ||n |jj}| | n |jj} |jd|||||||| | d | } | j}t | t rt| dn| }||dd|ddf}||j z}d}||j d|||jj d| }t||| j | j| jS)a~ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >> from transformers import AutoTokenizer, Cohere2ForCausalLM >> model = Cohere2ForCausalLM.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> tokenizer = AutoTokenizer.from_pretrained("Cohere2ForAI/c4ai-command-r-v01") >> prompt = "Hey, are you conscious? Can you talk to me?" >> inputs = tokenizer(prompt, return_tensors="pt") >> # Generate >> generate_ids = model.generate(inputs.input_ids, max_length=30) >> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```N) rrrPrrrr$r%r)rr#r)lossrrrqrr)r r$r%rrr*rslicerr! loss_functionrrrrqr)r4rrrPrrr#rr$r%rr&routputsrq slice_indicesrr(s r7rUzCohere2ForCausalLM.forwards>N2C1N--TXT_Tq$8$D $+Jj  ,64: , )%+'/!5) , ,  , ,  1 8B>SV8W8Wk~ot444]k mAAA}aaa,?@AA$**  %4%pVFt{OeppioppD%#3!/)     r8) NNNNNNNNNNr)rWrXrY_tied_weights_keys_tp_plan_pp_planr(rrrrHrrZrrlistrrrrrrrUr]r^s@r7rrs*+=)H_-z:;H     151537KO59-1$(,0/35934H H E,-H !.H u/0 H "%tE4E/F(F"GH H   12 H )*H D>H $D>H 'tnH !!12H c5</0H +,H  H H H ^H H H H H r8r)rrr)r~)Nr);typingrrrrHtorch.nnrc activationsr cache_utilsrr generationr masking_utilsr r modeling_flash_attention_utilsr modeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrutils.deprecationr utils.genericrconfiguration_cohere2rModulerr`rZrr}rDrrrrrrrrr__all__rr8r7rCs,-,,,,,,,,, !!!!!!........))))))RRRRRRRRBBBBBB999999OOOOOOOOKKKKKKKKFFFFFFFF&&&&&&IIIIIIIIII000000//////000000!<!<!<!<!