*`iw/ rddlZddlmZmZmZddlZddlmZmZm Z m Z ddl m Z m Z mZmZmZmZmZddlmZddlmZddlmZmZmZmZmZmZddlmZm Z gd Z!ej"e#Z$ dd ed eed e%fdZ&d efdZ' dd ede(dedeee)dfdej*d e%f dZ+d ed ed e%fdZ,d efdZ-dS)N)OptionalTupleUnion) IMPL_ATTR KV_CACHE_ATTRQuantizedAttentionImplQuantizedKVCache)ActivationOrdering DynamicTypeQuantizationArgsQuantizationMetadataQuantizationSchemeQuantizationStatusQuantizationStrategy)wrap_module_forward_quantized) strategy_cdiv)disable_hf_hookget_execution_device get_head_dimget_num_attn_headsget_num_kv_headsregister_offload_parameter)Module Parameter)"initialize_module_for_quantizationis_attention_moduleinitialize_qparamsinitialize_attn_qparamsTmoduleschemeforce_zero_pointc|pt|dd}|dStj|t|rt |||nt |t jjs*t dt|t|dr$|j }t |t jsJn % %|%|!1       $ 0 )%|CRC0%|!1     V $ $ : : *&& 9 9 9 : : : : : : : : : : : : : : : "(F!3!?Fs'GG Gcd|jjvo/t|dpt|dpt|dS)N attentionk_projv_projqkv_proj) __class____name__lowerr4)rs r=rrsW &*399;; ; !! ' 68 $ $ ' 6: & & base_namequantization_argsr(r)c|j}|j}|j}t|} |durdS|tjkrDt tjdtj | d} t||d| |tj krdS|tj krd} n|tjkrtd |tjkr.t#|d krtd |d df} n|tjtjfvr|jJt#|dkrtd |j} t)|d| |} g|dd| R} |t*jkrLt tj|dfd| tjd}t||d|n|tjkrq|jJt#|d krtd|j}t)|d |d |}t)|d|d|}||f} nK|tjkr.t#|dkrtd|dddf} n Jd||}|tjtjtj tjfvr tj}t tj| || d}t||d||s|jsAt tj| | |j d}t||d|dSdS)a Initialize quantization parameters for a given basename according to the passed quantization args. The shape and dtype of the observed weight/activation must also be provided. Scales will always be initialized. Global scales are initialized depending on args. Zero points will be initialized if not symmetric or if `force_zero_point` is True. :param module: module to register qparams to :param base_name: base name of qparams, for example "input", "weight", "k", "v" :param quantization_args: arguments for quantization :param observed_shape: last (right-most) known dimensions of the observed weight/act :param observed_dtype: dtype of the observed weight/actt :param force_zero_point: force the zero_point parameter to be initialized TN)r8deviceF) requires_grad _global_scale)rJz(Cannot perform static token quantizationz5Channel quant requires at least 2 observed dimensionsz2Group quant requires at least 1 observed dimensionr&)rKr8_g_idxz3Block quant requires at least 2 observed dimensionsz7Attention quant requires at least 3 observed dimensionszUnknown strategy _scale _zero_point)!strategydynamicactorderrr TENSOR_GROUPrr.emptyfloat32rr LOCALTENSORTOKEN ValueErrorCHANNELlenGROUP group_sizerr fullintBLOCKblock_structure ATTN_HEADfloat16bfloat16float64 symmetriczeroszp_dtype)rrGrHr(r)r!rUrVrWrKinit_global_scaleexpected_shaperb num_groups init_g_idxrfnum_rowsnum_cols scale_dtype init_scaleinit_zero_points r=rrs.!)H'G )H !& ) )F$'444% Kv > > >    # y///1B   +###'... )/ / /CDDD )1 1 1 ~   " "TUU U(,a0 *02F2ST T T +777 ~   " "QRR R&1 ">"#5z8LL ;>#2#.; ;; )/ / /" N2.0"V59UUU#J 'v)/C/C/CZ P P P )/ / / 0<<< ~   " "RSS S+; !3_R5H(SS !3_R5H(SS"H- )3 3 3 ~   " "VWW W(,a3 54(4444u!K     m  N+fEEEJv)';';';ZHHHW0:W# Kv5F5O        #6i+D+D+DoVVVVVWWrFc4t|td}t|td}|"| tdtdtdt ||j}t |}t|}t|}|d|f} |d|f} t| j } |t|d|j | | ||6t|d|j | | |t|d|j | | |dSdS) z(Initlaize k_scale, v_scale for self_attnNz0Attention module has quantization scheme but no z or zc attributes. Please ensure that these attributes are initialized using `apply_quantization_config`.qr'kv)r+rrr^_validate_attention_schemeconfigrrrnext parametersr8rr6) rr r!implkv_cacher|num_attn_heads num_kv_headshead_dimq_observed_shapekv_observed_shaper)s r=rrs .5VY-M-MD+26=$+O+OH |( Ly L L L L L   v&&&_F'//N#F++LF##H'h7%tX6&++--..4N     $+)-         $,)-          $,)-       rFc|jtd|jtd|jtddS)NzmCannot apply weight quantization to attention. Instead, target the (q|k|v)_proj submodule layers of attentionzHCannot apply attention quantization without specifying input activationsz-Cannot apply output quantization to attention)r9r^r6r:)r s r=r{r{Nsd ~! M   ' V    ,HIII-,rF)NT)T).loggingtypingrrrr.compressed_tensors.modelingrrrr compressed_tensors.quantizationr r r r rrr1compressed_tensors.quantization.lifecycle.forwardr%compressed_tensors.quantization.utilsrcompressed_tensors.utilsrrrrrrtorch.nnrr__all__ getLoggerrDr1boolrrstrrdr8rrr{rFr=rs ))))))))))  @?????'&&&&&&&    ' H % % ,0!R@R@ R@ ' (R@R@R@R@R@j" xWxW xWxW(xW%T *+ xW K xW  xWxWxWxWv6 6 .6 BF6 6 6 6 r J'9 J J J J J JrF