*`ih?UddlZddlZddlmZmZmZddlZddlmZm Z m Z m Z m Z m Z mZddlmZddlmZmZmZddlmZddlmZddlmZmZmZdd lmZgd Zd gZej e!Z"ej#e$d < d7d edede deedeeeff dZ% d7dede dej&jdeefdZ'de de(defdZ)dede*fdZ+dede*fdZ,dede(fdZ-eddedeee(efddffdZ.ed d8ded"e*d#e*d$e*deee(efddff d%Z/dejde0fd&Z1dejd'd(de*fd)Z2ed*ede*fd+Z3e eej4fd,ejd-ejd.ee d/ee d0eej5f d1Z6 d9de0d2e0d3ee d4e*de0f d5Z7d0ej5de8fd6Z9dS):N) GeneratorOptionalTuple) FP4_E2M1_DATA FP8_E4M3_DATA FloatArgsQuantizationArgsQuantizationStrategyQuantizationTyperound_to_quantized_type_dtype)QuantizationScheme)generate_mxfp4_scalesmaybe_convert_from_mxfp4_expshould_generatre_mxfp4_scales) deprecated)logger) FloatTensor IntTensorTensor)Module)is_module_quantizedis_model_quantized module_typeget_torch_bit_depth can_quantizeKV_CACHE_TARGETSis_kv_cache_quant_schemeiter_named_leaf_modulesiter_named_quantizable_modulescompute_dynamic_scales_and_zpcalculate_rangecalculate_qparamsgenerate_gparam strategy_cdivzre:.*self_attn$_LOGGERmin_valsmax_valsquantization_args global_scalereturnctj|tj|}tj|tj|}|j}t ||\}}||z }|jrtjtj|tj|}t|rt|} n|t|dz z } tj | j ||j } nb|jdkr$|jt jkrt%d||z t|z } ||| z z } tj| ||} ||| z} |jt+| |j} t-|| } t/|j|jn| j } tj| d ktj| | j | | } t+| |jd } | jd kr*| d } | d } | | fS)a :param min_vals: tensor of min value(s) to calculate scale(s) and zero point(s) from :param max_vals: tensor of max value(s) to calculate scale(s) and zero point(s) from :param quantization_args: settings to quantization :param global_scale: additional global scale to scale the locally generated scale currently only applied/supported for Fp4 :return: tuple of the calculated scale(s) and zero point(s). For FP4, the calculated scale is of dtype FP8 )args)x)devicedtypez0Asymmetric Quantization is not supported for FP4Nr0r)r0r/F)r0cast_to_original_dtype)torchmin zeros_likemaxr/r! symmetricabsrrfloatzerosshaper0num_bitstyper FLOATNotImplementedErrorclamp scale_dtyper r_get_dtype_epswheretensorzp_dtypendimreshape) r&r'r(r)r/bit_minbit_max bit_range max_val_posscales zero_pointsepss /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/compressed_tensors/quantization/utils/helpers.pyr"r"Asm(y5#3H#=#=>>Hy5#3H#=#=>>H _F&'8&AAGW'!I"Ai ( 3 3UYx5H5HII (.? @ @ @ :*[999FF E)$4$4q$89Fk&,vX^TTT   &! + +!&*:*@@@%B X%y)9)99F!23 k+w@@ &$0. +7    **;V D DF   ( 4 ++ \   C [!  S V<<<F0,5eK{a""!))!,, ; valuer,modulec d}|jtjkr2ddh t fdt |jD}n|jtjkrd}n|jtjtjfvrId}d}tj |j d|j z |j f}| d|}n?tjtjtjtjf}td||stj|\}} n.tj||| }tj||| } t'|| || S) a Returns the computed scales and zero points for dynamic activation quantization. :param value: tensor to calculate quantization parameters for :param args: quantization args :param reduce_dims: optional tuple of dimensions to reduce along, returned scale and zero point will be shaped (1,) along the reduced dimensions :return: tuple of scale and zero point derived from the observed tensor Trr4c3$K|] }|v|V dSN).0idxdims rQ z0compute_dynamic_scales_and_zp..s'OOC3COOrRNFz+Dynamic quantization is only supported for )r[keepdims)r))strategyr TOKENtuplerangerHTENSOR TENSOR_GROUPGROUPmathceilr= group_size unflatten ValueErrorr5aminmaxaminamaxr") rSr,rTr) keep_dims reduce_dims reshaped_dimssupported_strategiesmin_valmax_valr[s @rQr r sx$I },222!fOOOO5+<+<OOOOO .5 5 5 )"      Iek"o7 8 8 O M22 ! & ' - &    9# %   I =//*U iHHH*U iHHH Wgt, O O OOrRr/c\|jtjkrAd|jz}t j|dz dz |}t j| dz |}n|jtjkr|jdkrAt jtj|}t jtj |}nr|jdkrAt jtj|}t jtj |}n&tdtd|j||fS)a Calculated the effective quantization range for the given Quantization Args :param quantization_args: quantization args to get range of :param device: device to store the range to :return: tuple endpoints for the given quantization range r.r4)r/r1z1Range calculation only supported for 4 and 8 bitszInvalid quantization type ) r?r INTr>r5rFr@rr8r6rrArj)r(r/rLq_maxq_mins rQr!r!s'!1!555(11  Y]Q.v>>> iZ!^F;;;  #3#9 9 9  % * *L!26BBBEL!26BBBEE  '1 , ,L!26BBBEL!26BBBEE%C N6G6LNNOOO %<rRc~t|dsdS|jjdS|jjdS|jjdSdS)z Check if a module is quantized, based on the existence of a non-empty quantization scheme :param module: pytorch module to check :return: True if module is quantized, False otherwise quantization_schemeFNT)hasattrrzweightsinput_activationsoutput_activationsrTs rQrrsT 60 1 1u !)5t !3?t !4@t 5rRmodelcXtd|DS)z Check if any modules in a model are quantized, based on the existence of a non-empty quantization scheme in at least one module :param model: pytorch model :return: True if model is quantized, False otherwise c34K|]}t|VdSrW)r)rY submodules rQr\z%is_model_quantized..s+OO)"9--OOOOOOrR)anymodules)rs rQrrs) OOu}}OOO O OOrRc*t|jS)z Gets a string representation of a module type :module: pytorch module to get type of :return: module type as a string )r?__name__rs rQrrs <<  rRzThis function will be removed in a future release. Please use `model.named_modules()` and filter by compressed_tensors.InternalModule if neceessary)messagec#K|D]\}}t|}t|dkr d|vr||fVDt|dkr+t t|\}}d}t t|D]}||}d|vrd}|s||fVdS)z Yields modules that do not have any submodules except observers. The observers themselves are not yielded :param model: model to get leaf modules of :returns: generator tuple of (name, leaf_submodule) robserverFTN) named_moduleslistchildrenlenzipnamed_childrenrb)rnamerrrhas_non_observer_childreni child_names rQrrs!..00&&i **,,-- x==A  *"4"4 / ! ! ! !8}}q  +.Y5M5M5O5O0P0P+Q((- %3x==)) 5 5+A. Z//04-, &Io%%%!&&rRTFinclude_children include_attn include_mlpc#K|D]\}}|rt|}t|dkr d|vr||fVnut|dkr+t t|\}}d}t t|D]} || } d| vrd}|s||fV|r|dr||fV|r|dr||fVdS)aU Yield name and submodule of - leaf modules, set by include_children - attention modyles, set by include_attn :param model: model to get leaf modules of :param include_children: flag to get the leaf modules :param inlcude_attn: flag to get the attention modules :returns: generator tuple of (name, submodule) rrFT self_attnmlpN)rrrrrrrbendswith) rrrrrrrrrrrs rQrr:sd(!..00&&i  *I..0011H8}}!!j&<&<Io%%%%x==1$$/2D9Q9Q9S9S4T4T/U,NH,1)s8}}--99A!/!2J!334810* /)))  &}}[)) &Io%%%  &}}U## &Io%%%/&&rRc tj|jj}n.#t$r!tj|jj}YnwxYw|S)z Determine the number of bits used to represent the dtype of a tensor :param value: tensor to check bit depth of :return: bit depth of each element in the value tensor )r5finfor0bits TypeErroriinfo)rS bit_depths rQrrhsX2K ,,1 222K ,,1 2 s!(A  A  quant_argsr ct|}|j}||jkr!td|d|d||jkS)aI Checks if value can be quantized by quant_args. :param value: tensor to check for quantization :param quant_args: QuantizationArgs to use for quantization :return: False if value is already quantized to quant_args or value is incompatible with quant_args, True if value can be quantized with quant_args z%Can't quantize tensor with bit depth z to zH.The QuantizationArgs provided are not compatible with the input tensor.)rr>r%warn)rSrrrequested_depths rQrrwsu$E**I )O:&&&  VI V V? V V V   z* **rRschemec2|jD]}|tvrdSdS)a  Check whether the QuantizationScheme targets the kv cache. It does if all the following criteria are met: - the scheme targets either exactly match the KV_CACHE_TARGETS or the match KV_CACHE_TARGETS regex pattern - the scheme quantizes output_activations (we want to quantize the outputs from the KV_CACHE_TARGETS, as their correspond to the keys and values that are to be saved in the cache) :param scheme: The QuantizationScheme to investigate :return: boolean flag TF)targetsr)rtargets rQrrs4. % % %44 & 5rRupdated_min_valupdated_max_val scale_data quant_datar0ctj|tj|}tj|tj|}tjtj|tj|}|j|jz|z }||dgS)ah Generate a global scale for an entire tensor (input_tensor). Goal of the scale is to ensure that the quantization (local) scale falls into the approproiate dtype range. E.g. for NVFP4, group (local) scales are in dtype FP8. The global_scale attempts to use the entire FP8 dtype range while mapping a per-group max to the FP4 max. r4)r5r6r7r8r:torI) rrrrr0r&r'rMr)s rQr#r#s y%*:?*K*KLLHy%*:?*K*KLLH)EIh//81D1DEEK>JN2[@L ??5 ! ! ) )1# . ..rRdivisorr_strictctj||z }||z|krF|d|d|d|}|rt|tjd||S)NzJ quantization strategy requires strict division of weight/activation size z and group/block size z[. consider reducing the group/block size or ignoring modules with weights not divisible by T)log_once)rfrgrjrbindwarning)rSrr_rdividendrs rQr$r$s y))H'U"" 2 2&+ 2 2CJ 2 2)0 2 2   8W%% % K & & & . .w 7 7 7 OrRc|tjkrdS|tjkrdStjtjg|rtj|jSdS)Ng?g?r2r4)rr0rr5is_floating_pointrFrrPr2s rQrDrDsb ###u -% % %t  b!>!>!> ? ?{5!!%%qrRrW)TFF)F):loggingrftypingrrrr5*compressed_tensors.quantization.quant_argsrrrr r r r ,compressed_tensors.quantization.quant_schemer 1compressed_tensors.quantization.utils.mxfp4_utilsrrrcompressed_tensors.utilsrlogururrrrtorch.nnr__all__r getLoggerrr%Logger__annotations__r"nnr strr!boolrrrrrintrrrfloat32r0r#r$r;rDrXrRrQrs ---------- LKKKKK 0/////0000000000   &&&+'+H55555&* SSSS(S6" S  ; !" SSSSt&* 8P8P 8P 8P HO8P6" 8P8P8P8Pv'7:4.PfPPPPP!!3!!!!  6 &6&ic6k8JDRV8V.W&&&  &4  6" &&&& &&&&&& && uS&[!4-. &&&&&&  &&R u|     + +2D+++++( %7D.'4&3#(= //\/\/#/# / EK ////6   +,    .%+%rR