`iddlmZmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z dd lmZmZdd lmZd d lmZmZmZmZd d lmZddlmZddlmZejeZ dZ!GddeZ"Gdde Z#GddeZ$GddeZ%GddeZ&GddeZ'gdZ(dS))OptionalUnionN)Cache)FlashAttentionKwargs)GradientCheckpointingLayer)CausalLMOutputWithPast)Unpack)TransformersKwargslogging)deprecate_kwarg) GlmAttentionGlmForCausalLMGlmForSequenceClassificationGlmForTokenClassification)Phi3MLP) Glm4Config) Glm4RMSNormzTHUDM/GLM-4-9B-0414ceZdZdS)Glm4MLPN__name__ __module__ __qualname__y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/glm4/modular_glm4.pyrr&DrrcteZdZdedeffd Zeddd dd ejd e ejd e ej de e d e e de ej de e ejejfdeede eje e ejejfffdZxZS)Glm4DecoderLayerconfig layer_idxct|j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ dS)N)r#r$)eps) super__init__ hidden_size Glm4Attention self_attnrmlpr rms_norm_epsinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernorm)selfr#r$ __class__s rr(zGlm4DecoderLayer.__init__+s !-&f JJJ6??*6+=6CVWWW(3F4FFL_(`(`(`%(3F4FFL_(`(`(`%"-f.@fFY"Z"Z"Zrpast_key_valuepast_key_valuesz4.58)new_nameversionNF hidden_statesattention_mask position_ids use_cachecache_positionposition_embeddingskwargsreturnc $|} ||}|jd|||||||d|\}} ||}| |z}|} ||}||}||}| |z}|S)N)r8r9r:r5r;r<r=r)r.r+r0r/r,r1) r2r8r9r:r5r;r<r=r>residual_s rforwardzGlm4DecoderLayer.forward6s!,,];; )4>  ')%+) 3       q55mDD  =0  55mDD // // >>  =0 r)NNNFNN)rrrrintr(r torchTensorr LongTensorrbooltupler r FloatTensorrC __classcell__r3s@rr"r"*sU [z [c [ [ [ [ [ [_%0A6RRR2637+/$)59KO!!|!!.!u/0 ! "% ! D> !!!12!&eEL%,,F&GH!-.! u (51BEDU1U+V"WW X!!!SR!!!!!rr"ceZdZdS)r*Nrrrrr*r*[r rr*cDeZdZdeedeeefffd ZxZ S)Glm4ForCausalLM super_kwargsr?c 6tjdi|S)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, Glm4ForCausalLM >>> model = Glm4ForCausalLM.from_pretrained("THUDM/GLM-4-9B-0414") >>> tokenizer = AutoTokenizer.from_pretrained("THUDM/GLM-4-9B-0414") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```r)r'rC)r2rPr3s rrCzGlm4ForCausalLM.forward`s!4uww.....r) rrrr r rrIr rCrKrLs@rrOrO_s`/12/ u,, -//////////rrOceZdZdS)Glm4ForSequenceClassificationNrrrrrSrS}r rrSceZdZdS)Glm4ForTokenClassificationNrrrrrUrUr rrU)Glm4PreTrainedModel Glm4ModelrOrSrU))typingrrrE cache_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsr processing_utilsr utilsr r utils.deprecationr glm.modeling_glmrrrrphi3.modeling_phi3rconfiguration_glm4r modeling_glm4r get_loggerrlogger_CHECKPOINT_FOR_DOCrr"r*rOrSrU__all__rrrrhs5 #""""""" BBBBBB999999666666&&&&&&00000000000000tttttttttttt((((((******&&&&&&  H % %+     g   .....1...b     L   /////n///<     $@        !:      r