`idZddlmZmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZd d lmZd dlmZmZmZmZmZmZmZddlmZeje Z!GddeZ"GddeZ#GddeZ$GddeZ%GddeZ&GddeZ'gdZ(dS)zPyTorch BitNet model.)CallableOptionalN)Cache)FlashAttentionKwargs)CausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONS)Unpack)logging)deprecate_kwarg)GemmaMLP)LlamaAttentionLlamaDecoderLayerLlamaForCausalLM LlamaModel LlamaRMSNormapply_rotary_pos_embeager_attention_forward) BitNetConfigceZdZdS) BitNetRMSNormN__name__ __module__ __qualname__}/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/bitnet/modular_bitnet.pyrr+Drrc*eZdZdeffd ZdZxZS) BitNetMLPconfigct|t|j|j|_dSN)eps)super__init__rintermediate_size rms_norm_eps ffn_sub_norm)selfr$ __class__s r r)zBitNetMLP.__init__0s<    )&*BH[\\\rc |||||||z}|S)N) down_projr,act_fn gate_projup_proj)r-xr0s r forwardzBitNetMLP.forward4sUNN4#4#4T[[PQARAR5S5SVZVbVbcdVeVe5e#f#fgg r)rrrrr)r5 __classcell__r.s@r r#r#/sZ]|]]]]]]rr#ceZdZdedeffd Zeddd dd ejd e ejejfd e ejde e d e ej d e ede eje ejffdZxZS)BitNetAttentionr$ layer_idxct||t|j|j|_dSr&)r(r)r hidden_sizer+ attn_sub_norm)r-r$r:r.s r r)zBitNetAttention.__init__:s> +++*6+=6CVWWWrpast_key_valuepast_key_valuesz4.58)new_nameversionN hidden_statesposition_embeddingsattention_maskcache_positionkwargsreturnc n|jdd}g|d|jR}|||dd} |||dd} |||dd} |\} } t| | | | \} } |&| | |d}|| | |j |\} } t}|j j dkrt|j j }||| | | |f|jsdn|j|jd|\}}|jg|dR}||}||}||fS)Nrr )sincosrEeagerg)dropoutscaling)shapehead_dimq_projview transposek_projv_projrupdater:rr$_attn_implementationr trainingattention_dropoutrNreshape contiguousr=o_proj)r-rBrCrDr?rErF input_shape hidden_shape query_states key_states value_statesrKrJ cache_kwargsattention_interface attn_output attn_weightss r r5zBitNetAttention.forward>s$)#2#. 88b8$-88 {{=1166|DDNNqRSTT [[//44\BBLLQPQRR {{=1166|DDNNqRSTT &S#7 jRUWZ#[#[ j  &#&snUUL'6'='=j,X\Xfht'u'u $J (? ; +w 6 6"9$+:Z"[ $7$7      % $}HCC$2HL % %  % % ! \*k);;;;;;FFHH ((55 kk+.. L((r)NN)rrrrintr)r torchTensortuplerr LongTensorr rr5r6r7s@r r9r99sX|XXXXXXX_%0A6RRR ,059 +)+)|+)#5<#=>+)!. +) "% +) !!12 +)-.+) u|Xel33 4+)+)+)SR+)+)+)+)+)rr9ceZdZdS)BitNetDecoderLayerNrrrr rlrlmr!rrlceZdZdS) BitNetModelNrrrr rnrnqr!rrnc2eZdZdgZdZdZdeffd ZxZS)BitNetForCausalLMzlm_head.weightNrGc 6tjdi|S)a$ labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, transformers., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, transformers., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, BitNetForCausalLM >>> model = BitNetForCausalLM.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/bitnet-b1.58-2B-4T") >>> prompt = f'<|begin_of_text|>User: Hey, are you conscious? Can you talk to me?<|eot_id|>Assistant: ' >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=100) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "User: Hey, are you conscious? Can you talk to me?Assistant: No, I'm not conscious. I'm an artificial intelligence designed to assist with information and tasks. How can I help you today?" ```r)r(r5)r- super_kwargsr.s r r5zBitNetForCausalLM.forwardzs!4uww.....r) rrr_tied_weights_keys_tp_plan_pp_planrr5r6r7s@r rprpusZ*+HH/ //////////rrp)rprnBitNetPreTrainedModel))__doc__typingrrrg cache_utilsrmodeling_flash_attention_utilsrmodeling_outputsrmodeling_utilsr processing_utilsr utilsr utils.deprecationr gemma.modeling_gemmarllama.modeling_llamarrrrrrrconfiguration_bitnetr get_loggerrloggerrr#r9rlrnrp__all__rrr rsU%%%%%%%% BBBBBB666666555555&&&&&&000000++++++/.....  H % %     L   1)1)1)1)1)n1)1)1)h     *        *   /////(///D   r