*`i-ddlmZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZmZddlmZmZddlmZmZddlmZmZmZdd lmZgd ZGd d eeZej ej!ej"ej#gZ$d Z%dZ&GddeZ'dS)) defaultdict)Enum) AnnotatedAnyDictListOptionalSetUnion)CompressionFormat) DynamicTypeQuantizationArgs)QuantizationSchemepreset_name_to_scheme)is_module_quantized module_type) BaseModel ConfigDictField)Module)QuantizationStatusQuantizationConfigLIFECYCLE_ORDERDEFAULT_QUANTIZATION_METHODDEFAULT_QUANTIZATION_FORMATcbeZdZdZdZdZdZdZede dfdZ dZ d Z d Z d Zd S) ra@ Enum storing the different states a quantized layer can be in Initialized: scale, zero points and observers have been attached to the layer but are set to dummy values (not yet calibrated) Calibration: scale and zero points have been calibrated through OBCQ or similar algorithm, observers are still attached Frozen: scale and zero points are finalized, observers have been deleted, weights are still in their original precision Compressed: weights have been converted to their target type or compressed to their closed approximation initialized calibrationfrozen compressedreturncdS)zG :return: list of correct quantization lifecycle order N)clss /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/compressed_tensors/quantization/quant_config.pylifecycle_orderz"QuantizationStatus.lifecycle_order9s c|dSt||jstt|t|kSNT isinstance __class__NotImplementedErrorrindexselfothers r%__ge__zQuantizationStatus.__ge__@sM =4%00 &% %$$T**o.C.CE.J.JJJr'c|dSt||jstt|t|kSr)r*r/s r%__gt__zQuantizationStatus.__gt__GsM =4%00 &% %$$T**_-B-B5-I-IIIr'c|dSt||jstt|t|kSNFr*r/s r%__lt__zQuantizationStatus.__lt__NsM =5%00 &% %$$T**_-B-B5-I-IIIr'c|dSt||jstt|t|kSr6r*r/s r%__le__zQuantizationStatus.__le__UsM =5%00 &% %$$T**o.C.CE.J.JJJr'N)__name__ __module__ __qualname____doc__ INITIALIZED CALIBRATIONFROZEN COMPRESSED classmethodrr&r2r4r7r9r#r'r%rr&s   KK FJ%9 :[ KKKJJJJJJKKKKKr'rzcompressed-tensors fakequantc eZdZUdZeeeeeeffe d<e Z ee d<dZ e ee d<eZee d<ejZee d<dZe ee d<ee Ze eee d <dZeeed fe d <dZdZe ddede eeefde dfdZ dZ!e"d Z#dS)ra= Full configuration specifying how a model is quantized. Each quantized layer is mapped to a QuantizationScheme in config_groups. :param config_groups: dict of QuantizationSchemes specifying the quantization settings for each quantized layer. A group could also be a reference to a predefined scheme name, mapped to a list of its target layers/classes :param quant_method: a constant used to differentiate compressed-tensors quantization from other quantization configs :param format: specifies how the quantized model is stored on disk :quantization_status: specifies the current status of all quantized layers. It is assumed all layers are in the same state. :param kv_cache_scheme: optional QuantizationArgs, that specify the quantization of the kv cache. If None, kv cache is not quantized. When applying kv cache quantization to transformer AutoModelForCausalLM, the kv_cache_scheme gets converted into a QuantizationScheme that: - targets the `q_proj` and `k_proj` modules of the model. The outputs of those modules are the keys and values that might be cached - quantizes the outputs of the aformentioned layers, so that keys and values are compressed before storing them in the cache There is an explicit assumption that the model contains modules with `k_proj` and `v_proj` in their names. If this is not the case and kv_cache_scheme != None, the quantization of kv cache will fail :global_compression_ratio: optional informational config to report the model compression ratio acheived by the quantization config :ignore: optional list of layers to ignore from config_groups. Layers in this list are not quantized even if they match up with a target in config_groups config_groups quant_methodNkv_cache_schemeformatquantization_statusglobal_compression_ratio)default_factoryignoreT)excluderun_compressedc|jD]4\}}t|trt |||j|<5dS)zh updates any quantization schemes defined as presets to be fully loaded schemes )nametargetsN)rEitemsr+rr)r0_QuantizationConfig__context group_nametargets_or_schemes r%model_post_initz"QuantizationConfig.model_post_initsn .2-?-E-E-G-G   )J)+-?@@ -B)...D z * *  r'c*|SN) model_dump)r0s r%to_dictzQuantizationConfig.to_dicts   r'modelr!cjddlm}ddlm}t }d}t }t t}d}|D]\} } t| } t| o||  pt| |} t| o || } | rIt| d|}| | | j |vr|| j | rt| d|}| j j}| s$| |vrg|| <|| | t!|dkr|dSg}|D]\} }| |vr||z }i}t%|D]\}}dt'|z}|||<|4|t(jkrt,jj}nSt,jj}nAt5|tr,t!|dkrt,jjn|d}t9|||d||S) a Converts a model into its associated QuantizationConfig based on the QuantizationScheme attached to each quantized module :param model: model to calculate quantization scheme of :return: filled out QuantizationScheme for the input model r) IMPL_ATTR)is_attention_moduleNrIgroup_)rErIrGrJrHrL)compressed_tensors.modelingr]4compressed_tensors.quantization.lifecycle.initializer^listsetr named_modulesrrhasattrgetattraddquantization_schemeappendinput_activationslenrR enumeratestrrrAr int_quantizedvaluedenser+mixed_precisionr)r[rHr]r^quantization_schemes model_statusquantization_type_namesrLrGrP submodule layer_typehas_config_group has_kv_cacheconsolidated_ignore ignore_namesrEidxschemerTs r%from_pretrainedz"QuantizationConfig.from_pretraineds :99999      :> -0EE(34'8'87;$2244 0 0OD)))44J 39== '' 222Sgi6S6S /y99>Q>Q??L  O&y2GVV '++J77708LLL(// 0MNNN R&y2GVV "+"?"Q# 0V++)+F:&z"))$/// $ % % * */F4!(.  4 4 $J 444#|3#  $%9:: / /KC!CHH,J(.M* % % >1<<<*8>*06  % % v;;??"177AY  "' ,+%)&     r'c|jdS|jD]?\}}|j|jjdt jfvrdS|j|jjsdS@dS)NTF)rGrErRrkdynamicr LOCALoutput_activations)r0_r}s r%requires_calibration_dataz,QuantizationConfig.requires_calibration_data s   +4+1133  IAv'3+3{?P7QQQ44(408 44ur')extrarX)$r:r;r<r=rrnr rr__annotations__rrFrGr rrrHrr>rIrJfloatrrcrLrNrrrVrZ staticmethodrr~rr model_configr#r'r%rrhs:U#5tCy#@AABBBB3L#33326OX./666-FC---.@.L+LLL04huo444"'%"="="=FHT#Y ===;?NIc55#6#6#667>>>   !!!<@f f f 'c4i(89f & 'f f f \f P   :H---LLLr'rN)( collectionsrenumrtypingrrrrr r r compressed_tensors.configr *compressed_tensors.quantization.quant_argsr r,compressed_tensors.quantization.quant_schemerr%compressed_tensors.quantization.utilsrrpydanticrrrtorch.nnr__all__rnrr>r?r@rArrrrr#r'r%rs$#####CCCCCCCCCCCCCCCCCC777777TTTTTTTTSRRRRRRR1111111111   4K4K4K4K4Kd4K4K4Kp""! 3)r.r.r.r.r.r.r.r.r.r.r'