*`iVddlZddlmZmZddlmZddlmZddlm Z ddl m Z ddl m Z ddlmZdd lmZdd lmZmZdd lmZdd lmZgd ZdZdZGdde ZdefdZdedefdZdedeee gee fdefdZdS)N)CallableOptional)initialize_hooked_kv_cache)forward_quantize) getattr_chain)InternalModule)Tensor)Module)RemovableHandle)PretrainedConfigPreTrainedModel)ALL_MASK_ATTENTION_FUNCTIONS)ALL_ATTENTION_FUNCTIONS)QuantizedAttentionImplinitialize_hooked_attentionregister_query_hook IMPL_ATTRimplct_hooked_attentioncDeZdZdZdZdeffd Zdedededefd Z xZ S) ra QuantizedAttentionImpl module which wraps the functionality of the original attention implementation. Unlike the original attention function, this implementation is a `torch.nn.Module` which can be hooked to trigger transforms and calibration hooks. This module works by being registered as a submodule to attention modules via `initialize_hooked_attention`, registering a new attention implementation function which calls this module, then setting the model attention implementation to the new function. After triggering hooks and quantization, this module calls the original attention implementation function. eagerconfigcVt||_dS)N)super__init__r)selfr __class__s y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/compressed_tensors/modeling/attention.pyrzQuantizedAttentionImpl.__init__:s$  modulequerykeyvaluecd}t||d}t|dd} || rt||d|}ttj||||g|Ri|S)Nz%quantization_scheme.input_activationsquantization_enabledTq)rgetattrrrr_original_impl) rr r!r"r#argskwargsquant_args_attr quant_args quant_enableds rforwardzQuantizedAttentionImpl.forward>sB"6?DAA (>EE  !m !$VUCDDE''='LM              r) __name__ __module__ __qualname____doc__r(r rr r r. __classcell__)rs@rrr*s  N/              rrr ct|tsJdtdtdt|t|g|Ri|S)NzUsing z> attention implementation, but attention module does not have z submodule.)hasattrrHOOKED_ATTENTION_NAMEr')r r)r*s r_hooked_attentionr7\s| 69 % % E& E E.7 E E E % &769 % %f >t > > >v > >>rmodelct|ts-|tt|j|jjt kr|jjt_t|jj}tj t ttj t || t |jjt ksJt||dS)a Initialize `QuantizedAttentionImpl` and `QuantizedKVCache` instances attached to attention. Assumes that only one model is hooked at a time. :param model: parent model of attention module :param module: attention module to initialize with N)r5rregister_modulerr_attn_implementationr6r(rrregisterr7set_attn_implementationr)r8r original_masks rrres 69 % %Py*@*N*NOOO |(,AAA05 0Q-4U\5VW ()>@QRRR$-.C]SSS %%&;<<<|04IIIIIuf-----rhookreturncztt}dtffd }||dS)z Register a hook which takes post-rope query states as an argument and returns the modified query states or `None` :param module: attention module to add hook to :param hook: query hook function rctj|jj|i|}|jd}| ||jd<|j|jfS)Nr!)inspect signaturer.bind argumentsr)r*)rr)r*boundr#r?r s r_hookz"register_query_hook.._hooks`4!$,//4dEfEEVU_W566  ',EOG $z5<''rT) with_kwargs)r'rrregister_forward_pre_hook)r r?rrHs`` rrrs[$+69#=#=D(*(((((((  ) )%T ) B BBr) rCtypingrr#compressed_tensors.modeling.kvcacher1compressed_tensors.quantization.lifecycle.forwardrcompressed_tensors.utilsr!compressed_tensors.utils.internalrtorchr torch.nnr torch.utils.hooksr transformersr r transformers.masking_utilsrtransformers.modeling_utilsr__all__rr6rr7rrrrrXs%%%%%%%%JJJJJJNNNNNN222222<<<<<<------::::::::CCCCCC??????     -, , , , , ^, , , d?f????......4C C"FF#3Xf5E#EFCCCCCCCr