fPiǍ "dZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl mZddlmZejdZ dd Zd dZd!dZd!dZd dZdZid d d iddgdfdZdZdZ d"dZid d d ddddddddgf dZdS)#zWeightOnly for onnxrt adaptor.N) numpy_helper)np_dtype_to_tensor_dtype) ONNXModel)simple_progress_barneural_compressorc 2||zdz} tj|jd| fd} |jdd|d|z} |jd| g} g} i}d}|d kr:|d d d d d f|d d dd d fd zz}|d d d | f| d d d d f<n'|dkr|} ntd |d tj| d|| f} tj|d|f}|jtjks|jtj ksJtj |jddzt|j|j|d}| |j| |||dkr|d}n|d krtj|jddzd zdd}tj|jd|z|zd}|d d d }|dd d }||d zdz||z||d z<||d zdz||d zz||d z<nt+d |d tj||ddf}tj |jddzd |j|d}| |j| ||d|d<|d|d<||d<||d<|dkr||d<tj | d | j| d}| |tj j|f| |j|jr|jdzt1|zndt1|zdd|}|| fS)aBBuild MatMulNBits node. Args: node: original matmul node weight_shape: original weight shape num_bits (int): num_bits group_size (int): how many elements share one scale/zp k_blocks (int): block number q_weight (array): quantized weight scale (array): scale zero_point (array): zero point accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). Returns: matmul_weight_only_node: MatMulNBits node new_inits: initializers of the new node ruint8dtyper_QG MatMulNBitsNz8MatMulNBits does not have kernel support for num_bits = ._scaleTname data_typedimsvalsraw_zpKNbits block_sizeaccuracy_levelz com.microsoft)inputsoutputsrdomain)npzerosshapeinputloggererrorreshaper float32float16onnxhelper make_tensorrtobytesappendrastypefullarangeravel ValueError make_nodeoutputstr)node weight_shapenum_bits group_sizek_blocksq_weightscale zero_pointr$ blob_sizepacked q_weight_name input_names new_initskwargsop_typeq_weight_pairs scale_tensor packed_zpidxeven_idxodd_idx zp_tensorq_weight_tensormatmul_weight_only_nodes /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/onnxruntime/quantization/neural_compressor/weight_only.pymake_matmul_weight_only_noderW,s8X%*I Xx~a()4G D D DFJqM$D$D$Dj$D$DDM:a=-0KI FG1}}!!!!SSqS&)HQQQ1W,=,BB%aaa)m4qqq!!!t Q [PX[[[\\\ ZXy 9 : :F Jur8n - -E ;"* $ $ rz(A(A(A A;** Z]X %*5;77 [ ]]__  +L|())) \""" q=="))'22II ]]!1!!4q!8Q >7SSSI)J,Q/8;hFGGOOPRSSC33Q3xH!$Q$iG(1(a-(@4(G:V^K_KeKeKgKg'gIh!m $'0A'>'E*U\J]JcJcJeJeijJj&kIgl # #cX`cccdd dJy<?B*?@@ K++A&!)/PYPaPaPcPcim,   9>***###q/F3Kq/F3KF6N%F<#1 k--  \ ^^    .O_%%%"k3 15 TTY H - -s8}}@T    #I --r asymint?c  tj|d|f}|dks|dkr d|zdz dn*|dkr$|dkr d|dz zdz nd |dkr d|dz z ndtj|dd |z}tj|dd |z}|dkrtjtj|tj|}tj|j} |dk} || d ztj  z z | | <|d krtj | jn#tj|jd d|dz zz} ntj|j} tj fd||z ||k D| ||k<|d kr1tj | j|z | z nitjdtj tj | j|z | z d } tj|| j } tj|| | tj| | | tj| | tj|  | | | | fS)a Quantize tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quantized weight scale: scale zero_point: zero point rrZuintrrrsymTaxiskeepdimsg@r[r r c:g|]}t|z z S)float).0imaxqminqs rV z quant_tensor..s( ^ ^ ^!U1XX % ^ ^ ^rXout)r(r.minmaxmaximumabsonesr*r6float64r)arrayflattentolistroundminimum empty_liker divideaddclip)datar@rAschemer ratiorminrmax max_rangerDmaskrErCrhris @@rV quant_tensorrs :dR, - -D 5F??({Q 5*2a--qX\"Q&&Q)1Qx!|$%%B 6$Q . . . 6D 6$Q . . . 6D Jrvd||RVD\\::  ##1} ,44RZ@@D4KPd %*e^^BHU[ ! ! !SZ9[9[9[_`empqeq_r9s   ## h ^ ^ ^ ^ ^t TT\/J/R/R/T/T/[/[/]/] ^ ^ ^  ddl ~~hu{##d*e 3 : : < < <Arz$"(5;2G2G$2NRW1W0^0^0`0`aabbiijqrr  }T555HIdEx((((F8ZX....HX8$$$$GHdDh//// UJ &&rXctj|d|ftj}d|zdz }d}tj|dzdd}tj||z }tj|tj|}tj|dd}tj |dd} tj|dd} tj||zdd} tj | j |j } || k} ||z | | || z z | | <d| z }tj tj| ||z z||}||z|z|z }tj||dzzdd}d}d }d}t|D]}tj | j |j }tj|||zz|z|z g|j d}|| k} || | || z z || <tj tj|||z z||}||z}tj|dd}tj||zdd}tj||zdd}tj| |z|dz}| |z| |zz |z }|| z||zz |z }||z|z|z }tj||dzzdd} tj| }!tj|}"tj|!|"kd}#||#d d f||#d d f<| |#||#<||#||#<||#||#<tj | |z d|d }$|tj}tj||j }%tj|||% tj|%|$|% tj|%|% tj |%|||% |%||$fS) aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 32. Returns: output: quantized weight scale: scale zero_point: zero point rrrrTr`r 皙?Nr rk)r(r.r6r/sumsqrtrzrprmrnrqr*r r{rvrangerssubtractwhererrrxry)&r|r@rArhrisum_x2av_xweightsrrsum_wsum_xiscalerrD quant_datadiffbest_madnsteprdeltarrminis_ iscale_newfactorquant_data_newmul_weights_quant_data_newsum_lsum_l2sum_xlD this_scalethis_minmadmad_1 best_mad_1idx_to_replacerErCs& rVquant_tensor_k_quant_cpursc :dR, - - 4 4RZ @ @D h;?D D VD!G!d 3 3 3F 76J& ' 'DfT26$<<((G 6$Q . . .D 6$Q . . .D F7T 2 2 2E F7T>D 9 9 9E WTZtz 2 2 2F 4??FFtzRRSTUt|!T$Z$t*%<= 4*t *D!E!EtTRR%,~%="1DIII2^C!VZ[[[2T9DQQQ Kq 1 1fnuu}49 UNUV^3q8N*X5<fWtQw&Q>>> Xh'' %*"455a8(6~qqq7H(I >111$%#&~#6 *> :n'7^D5E/0022At<<CCGLLJ LL $ $E}T555HIdEx((((F8ZX....HX8$$$$GHdDh//// UJ &&rXc ddl}ddl}|jr||}|d|f|j}d|zdz }d}||dzdd}| ||z }| || |} | |dd} | |dd} || dd} || |zdd} || j|j}| | k}||z | || |z z ||<d|z }||||| z z||}||z| z|z }|| |dzzdd}d }d }d}t'|D]}|| j|j}||||zz|z|z g|jd}| | k}|| || |z z ||<||||| z z||}| |z}||dd}|||zdd}|||zdd}|| |z|dz}| |z| |zz |z } || z||zz |z }!| |z|!z|z }|| |dzzdd}"||"}#||}$||#|$kd}%||%ddf||%ddf<|"|%||%<| |%||%<|!|%| |%<|| |z d|d }&||j}|||j}'||||' | |'|&|' ||'|' ||'|||' |'||&fSt6d t;|||S#t<$r.t6dt;|||cYSwxYw)aQuantize tensor per group based on k quant. Ref: https://github.com/ggml-org/llama.cpp/blob/64eda5deb9859e87a020e56bab5d2f9ca956f1de/ggml/src/ggml-quants.c Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. Returns: output: quantized weight scale: scale zero_point: zero point rNrrrTr`r rrr rkzqTry to use k-quant quantization on CUDA. However, CUDA is not available.Fall back to k-quant quantization on CPU.zNow we are using k-quant quantization on cpu, which is time consuming.Please consider install cupy to speed up on CUDA. See https://cupy.dev/Please also install torch to check CUDA availability.) cupytorchcuda is_availableasarrayr.r6r/rrrzrprmrnrqr*r r{rvrrsrrrrrxrygetr,warningr ImportErrorinfo)(r|r@rAcprrhrirrrrrrrrrrDrrrrrrrrrrrrrrrrrrrrrrErCs( rVquant_tensor_k_quant_cudarssID : " " $ $> H::d##D<<Z 01188DDDh;?DDVVD!G!dV;;F776J.//DffT266$<<00G66$Q666D66$Q666DFF7TF::EFF7T>DFAAEWWTZtzW::F4#E"FGGNNtzZZ[\]t|#)T$Z$t*-D#E 4 !#*t 2L)M)MtUY!Z!Z-4~-E*9DQQ :^ KRS^bcc :T ATXYYKKq99#fnuu}<A "UNUV^;q@!N2X=DffWtQw.QfFF XXh// !#%**~qqq?P0Q >111,-+.~+>((2>(Bn%'/'?^$$D5E/!8!8!:!:AtDDKKGTTJLL,,E}}T}==H IIdExI 0 0 0 FF8ZXF 6 6 6 HHX8H , , , GGHdDhG 7 7 7<<>>599;; 0@0@@ @ NN<   ,D(JGG G DDD  D   (h CCCCC DsST*T5T=<T=cv|j}t||||||\}}} tj||| z z|S)aQuant dequant tensor per group. Args: data : input weight num_bits (int, optional): num_bits. Defaults to 4. group_size (int, optional): how many elements share one scale/zp. Defaults to 4. scheme (str, optional): quantization scheme. Defaults to "asym". dtype (str, optional): data type. Defaults to "int". ratio (float, optional): percentile of clip. Defaults to 1.0. Returns: output: quant-dequant weight )r*rr(r.) r|r@rAr}r r~ org_shapeweightrDzps rV qdq_tensorrysE I$T8ZPUVVFE2 :ev{+Y 7 77rXc|dkr|S|j}||z}||dz }|dkrtj|d|fdfd}|S)aPad tensor rowi so that it can be is divisible by group_size. Args: weight (array): weight group_size (int): how many elements share one scale/zp k_blocks (int): the number of block Returns: weight: paded weight rr)rrconstant)r*r(pad)rrArB org_w_shape padded_rowspad_lens rV pad_tensorrs]R ,KZ'KKN*G{{!Wv 6 CC MrXCPUExecutionProviderk_quantc t|}|j$tj|jnd} g} g} t d|D} d} |D]}|jdvr| dz } t| | |jdvrj| |j dI| |j idkr)| |j d}tj|| }t |jd kr|j}|j |vr9||j d }||j d }||j d }|j}|d kr|n|d}|ddz |zdz}||j d}t'|||}|dkp|dk}|r|dkrt)|j||\}}}nj !|j dd|d|ztE||j|#d}|$||j |j d<|dkr|%||&| |'| |(|S)aQuant the model with round to nearst method. Args: model (ModelProto or ONNXModel): onnx model weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'RTN' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". ratios (dict, optional): percentile of clip. Defaults to {}. accuracy_level (int): accuracy level. Support 0 (unset),1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel Nc$g|] }|jdv |S)MatMul)rLrfrgs rVrjz rtn_quantize..s$III1j1H1HQ1H1H1HrXrrrfp32)base_dirrr"rAr}rrr rr^r rZ r>r?r@rArBrCrDrEr$r[rrTr))r model_pathospathdirnamelennodesrLrget_initializerr+rrrto_arraycopyr*r get_initializer_share_numrrTrrWr6add_initializersr5rr(r. transposer1r2r3rr4add_initializerremove_initializer add_nodes remove_nodestopological_sort)model weight_configr@rAr}ratiosr$ providers algorithmr new_nodesr total_numcurr_idr> weight_tensorrr rrBinit_share_numsatisfy_MatMulNBits_conditionrCrDr q_matmul_noderJrTs rV rtn_quantizersvH e  E494D4Prwu/000VXHILII IIIJJIG E8E8 <: % % qLG  7 3 3 3 LJ & &%%djm44@!!$)R00F::!11$*Q-@@M!*=8LLLQQSSF6<  A%%LEyM))(3F;*495lC &ty1(; ,K'1R'7'7[^J#A*z9A=H"<.Gs999qAF999rXrzInput number mismatch, require z but get c0g|]\}}||fSrdrd)rfrinp_datars rVrjz"prepare_inputs..Ts,aaa~tXxx'9'9 :aaarXcg|] \}}||f Srdrd)rfrinps rVrjz"prepare_inputs..Vs hhh cs hhhrXstrictc0g|]\}}||fSrdrd)rfrrrs rVrjz"prepare_inputs..Xs*ppp)$xx}} 5ppprX)importlib.utilrutilrortSessionOptionssys version_inforrregister_custom_ops_libraryis_large_modelr1 save_modelrrInferenceSessionSerializeToString get_inputs enumerate batch_sizer isinstancedictr5itemsr(ndarrayzip) r n_samples dataloaderrrsorsession inputs_namesr%rgr|rs @rVprepare_inputsr#s )(((((   B '!!ii0H&I&I!;;;;;; &&'7'7'9'9:::    K   ."&$(#     # _U[::<%> !tAw<<3|#4#4444\#l2C2C\\cRVWXRYll\\544 d1gt $ $ s MM$aaaaQUVWQXQ^Q^Q`Q`aaabb c c c c Q , , s MM$hhSPTUVPWyaf=g=g=ghhhii j j j j MM$pppps.find_params~sL  >^FNN$4$41===Fhv|A''z"&a000#66z"&a000#66 U??:bfTllD11D(Cvc{{ '!#YJS qyTQY'S S $ U??75;''4!84q8DD8TEEM**D  +7FLO,--e %A,Q#G#G#GX\cdIeIeIeff fIIIDAAAJJ"R%( "Q;r"uaaax "##qqq& RYtBCCBJ/666 *T"" gqqqjM 1agA HrXc *+,t|}|j$tj|jnd}t |||| \}}~t j|jj j }| d|Dg}| D]}}|j dvrr||jidkrS||jidddkr ||jd~t%t'|}|||jr&t-j|j|jd zd d d |js.t1j|j|| nt1j|jd z|| }t7|D]x\}}t9t;||dzg}g}|j|D]8}|j dvr+||jidkr ||jidddkr||jdtAj!||"|jjd|}t;|j#dkr||||"|j:t;|dkrd|D}d+|D]}|$|g|d**j#d,tKj&*d*j#df*+,fd|D}+,z +tKj'd+z *z**fd|D}tQ|||d D]B\}}}|j|vr9||jd}||jd}||jd}|dkr|n |j#d}|j)}tU|||||||| | |  }||jd}|+|jd} |dk}!|!r|j#}"|"d|zdz |z}#tY|||#}t[|j.|||d\}}$}%t_||"|||#|0d|$0||dkr|%nd|  \}&}'|1|'|2||3|&nt,j45|jdd|d|ztm||j#|0|7d }(|8|(|(j|jd<| dkr|9|Dz| ||jj j :||;|jr?dd!l|jd|S)"aQuant the model with GPTQ method. Args: model (ModelProto or ONNXModel): onnx model dataloader (object): dataloader for calibration. weight_config (dict): quantization config For example, weight_config = { 'fc2': { 'bits': 4, 'group_size': 32, 'scheme': 'sym', 'algorithm': 'GPTQ' } } num_bits (int, optional): num_bits. Default is 4. group_size (int, optional): how many elements share one scale/zp. Default is 32. scheme (str, optional): sym or asym. Defaults to "asym". n_samples (int, optional): calibration sample number. percdamp (float, optional): percent of the average Hessian diagonal to use for dampening. blocksize (int, optional): blocksize to quantize weight. actorder (bool, optional): whether rearrange Hessian matrix considering the diag's value. mse (bool, optional): whether get scale and zero point with mse error. perchannel (bool, optional): whether quantize weight per-channel. accuracy_level (int): accuracy level. Support 0 (unset), 1(fp32), 2(fp16), 3(bf16), or 4(int8). providers (list): providers to use Returns: model: fake quantized ONNXModel Nrcg|] }|j Srdrrs rVrjz!gptq_quantize..s&B&B&B!qv&B&B&BrXrrrGPTQrrTFrrrrcfg|].}tj|jd|jdf/Sr)r(r)r*rs rVrjz!gptq_quantize..Os3 B B BQbh AGAJ/00 B B BrXrc&g|] }|zz zSrdrd)rfrgnsamplesr's rVrjz!gptq_quantize..Us'@@@!x8c>23@@@rXcJg|]}|tjjz Srd)r(rAr)rfrgrs rVrjz!gptq_quantize..Xs,888!bis+++888rXrr"rAr})r@rAr}rDrErFr5r7rr^r rZrrrr)load_external_data_for_model)?rrrrrrrr?rgraphr<remove_tensors_from_outputsrrLrrr5r+listsetadd_tensors_to_outputsr r1r rr rrrrinput_name_to_nodesrrrget_noder*runr(r.rrr rYrrrrrWr6r remove_nodeadd_noder2r3rr4rr MergeFromronnx.external_data_helperrbsplit)-rrrr@rAr}rrErDrFr5r7r$rrr%r org_output output_namesr>rrP input_name node_listrrHsr|rCr rCrrrrrBrDrrrJrTrbrr`r's- @@@rV gptq_quantizerusG^ e  E494D4Prwu/000VXHy*iHHJFBu{0788J %%&B&Bz&B&B&BCCCL // LJ & &!!$)R00F::!!$)R0044[&IIVSS    1 . . .L))**L   ...    K   ."&$(#     # _U[::<>!%%di4488fMMQWWW))$*Q-88D%.))%..*C*C*I!*LMMx$&&v|$$))v&&&   !:!:;;; w<<1    B B' B B B 9 9D++zlD11!4C)A,C*S2sy}"566C@@@@@R@@@B OH'!h,''#-C8888R888BB GR 6 6 6 = 8= 8   yM))(3F;*495lC &ty1(;'1R'7'7V\!_JLE!%#!!%   H"11$*Q-@@M"<r}s0%$   000000!!!!!!%%%%%%  . / /p.p.p.p.f3'3'3'3'lF'F'F'F'RXDXDXDXDv8888&4   %&ttttn666x    I I I I ^     %&||||||rX