*`iH ddlZddlmZmZmZmZmZmZddlm Z m Z ddl m Z ddl mZddlmZddlmZddlmZdd lmZdd lmZmZmZgd Zd ZGd deZdedeedeeefdeeeeeefffdZdedefdZ dedeeegeefdefdZ!dedeeegeefdefdZ"dS)N)AnyCallableDictListOptionalTuple) ReferenceTyperef)forward_quantize) getattr_chain)InternalModule)Tensor)Module)RemovableHandle)CachePretrainedConfigPreTrainedModel)QuantizedKVCacheinitialize_hooked_kv_cacheregister_key_hookregister_value_hook KV_CACHE_ATTRkv_cacheceZdZdZdedeffd ZdeeeffdZ dededeeeffd Z d e e fd Z xZS) rai QuantizedKVCache module which wraps the functionality of any existing kvcache args. Unlike transform Cache instances, this cache is a `torch.nn.Module` which can be hooked to trigger transforms and calibration hooks. This module works by being registered as a submodule to attention modules via `initialize_hooked_kv_cache`, then adding a hook which replaces `past_key_values` kwargs with this module. This module adopts the functionality of the replaced cache, preserving caching functionality such as sliding window attention, ect. :param attn_module: parent attention module config attn_modulect||_t||_d|_dSN)super__init__rr rpast_key_values)selfrr __class__s w/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/compressed_tensors/modeling/kvcache.pyr zQuantizedKVCache.__init__6s>  {++?Creturnc||i|Sr)r"argskwargss r$updatezQuantizedKVCache.update<stT$V$$$r% key_states value_statesc4|}d}t||d}t|dd}|&|r$t||d|}t||d|}|j%|j||g|Ri|} n||f} d|_| S)Nz%quantization_scheme.input_activationsquantization_enabledTkv)rr getattrr r!r+) r"r,r-r)r*modulequant_args_attr quant_args quant_enabledrets r$forwardzQuantizedKVCache.forward?s!!##A"6?DAA (>EE  !m !)&*c:NNJ+FL#zRRL   +/$&&((/L+/39CC|,C# r%r!cD|t||_dSd|_dSr)r r!)r"r!s r$add_past_key_valuesz$QuantizedKVCache.add_past_key_valuesZs,  &#&#7#7D #'D r%)__name__ __module__ __qualname____doc__rrr rrr+r8rrr: __classcell__)r#s@r$rr(s  D/DfDDDDDD %vv~)>%%%% vv~  6(8E?((((((((r%rr3r)r*r&cdtj|jjvrdnd}||d}t |t }|||||<||fS)aZ Hook which should be called before each quantized attention forward pass. This hook dynamically replaces the `past_key_values` kwarg to the attention forward function. The original kvcache object is assigned to QuantizedKVCache().past_key_values as a weakref to maintain original cache functionality and compute savings r!past_key_valueN)inspect signaturer8 parametersgetr2rr:)r3r)r* _past_kv_namer!caches r$_kv_cache_attention_hookrHds{  1&. A A L L L   (.zz-'F'FO%fm<u|V.T.TUUU(()At(TTTTTUUr%hookcztt}dtffd }||dS)z Register a hook which takes post-rope key states as an argument and returns the modified key states or `None` :param module: attention module to add hook to :param hook: key hook function rGctj|jj|i|}|jd}| ||jd<|j|jfS)Nr,rBrCr8bind argumentsr)r*rGr)r*boundvaluerPr3s r$_hookz register_key_hook.._hooks`5!%-005tFvFFVU_\:;;  ,1EOL )z5<''r%TrKr2rrrOr3rPrrYs`` r$rr[")!?!?H(%(((((((  - -e - F FFr%cztt}dtffd }||dS)z Register a hook which takes value states as an argument and returns the modified value states or `None` :param module: attention module to add hook to :param hook: value hook function rGctj|jj|i|}|jd}| ||jd<|j|jfS)Nr-rSrVs r$rYz"register_value_hook.._hooks`5!%-005tFvFFVU_^<==  .3EON +z5<''r%TrKrZr[s`` r$rrr\r%)#rBtypingrrrrrrweakrefr r 1compressed_tensors.quantization.lifecycle.forwardr compressed_tensors.utilsr !compressed_tensors.utils.internalr torchrtorch.nnrtorch.utils.hooksr transformersrrr__all__rrstrrHrrrr(r%r$rjs=================&&&&&&&&NNNNNN222222<<<<<<------AAAAAAAAAA    6(6(6(6(6(~6(6(6(x s)-1#s(^ 49d38n $%2 Uo Uv U U U UG G"FF#3Xf5E#EFGGGGG.G G"FF#3Xf5E#EFGGGGGGGr%