`i6{ ,dZddlmZddlmZddlmZmZmZer ddl Z ddl m Z erddl m Z dd l mZmZerdd lmZdd Zd ed eddfdZdded ededefdZGdde j jZ ddZd dZdS)!zZHIGGS through FLUTE (Flexible Lookup Table Engine for LUT-quantized LLMs) integration file)sqrt)Optional)is_flute_availableis_hadamard_availableis_torch_availableN)nn)prepare_data_transposed) TuneMetaDataqgemm_v2)hadamard_transformcdtdt|jzD}|D]-}|j|}|dz |zdz|z}||z }||d|zdz <.tj||d|S)Ncg|]}dSr).0_s s/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/integrations/higgs.py z pad_to_block..(s888a888rconstant)rangelenshaper functionalpad) tensordimshad_block_sizevaluepad_dimsdimsizenext_multiple_of_1024deltas r pad_to_blockr('s885S%6%6!677888H''|C "&(~!=!A^ S%,!&cA =  VXz5 A AArpnreturnz torch.Tensorc||fdkrtjgddgddgddgdd gd d gd d gddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gdd?gd@dAgdBdCgdDdEgdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdfdggdhdigdjdkgdldmgdndogdpdqgdrdsgdtdugdvdwgdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgdd gd d gd d gddgddgddgddgddgddgddgddgddgd d!gd"d#gd$d%gd&d'gd(d)gd*d+gd,d-gd.d/gd0d1gd2d3gd4d5gd6d7gd8d9gd:d;gd<d=gd>d?gd@dAgdBdCgdDdEgdFdGgdHdIgdJdKgdLdMgdNdOgdPdQgdRdSgdTdUgdVdWgdXdYgdZd[gd\d]gd^d_gd`dagdbdcgdddegdfdggdhdigdjdkgdldmgdndogdpdqgdrdsgdtdugdvdwgdxdygdzd{gd|d}gd~dgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgdĐdgdƐdgdȐdgdʐdgd̐dgdΐdgdАdgdҐdgdԐdgd֐dgdؐdgdڐdgdܐdgdސdgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgS||fdkrtjgddgddgddgd d gd d gd dgddgddgddgddgddgddgddgddgdd gd!d"gd#d$gd%d&gd'd(gd)d*gd+d,gd-d.gd/d0gd1d2gd3d4gd5d6gd7d8gd9d:gd;dgd?d@gdAdBgdCdDgdEdFgdGdHgdIdJgdKdLgdMdNgdOdPgdQdRgdSdTgdUdVgdWdXgdYdZgd[d\gd]d^gd_d`gdadbgdcddgdedfgdgdhgdidjgdkdlgdmdngdodpgdqdrgdsdtgdudvgdwdxgdydzgd{d|gd}d~gddgddgS||fdkrdtjddgddgddgddgddgddgddgddgddgddgddgddgddgddgddgddggS||fdkrDtjdgdgdgdgdgdgdgdgdgdgdgdgdgdgdgdggS||fdkr,tjdgdgdgdgdgdgdgdggS||fdkr tjdgdgdgdggStd|d|(N)rgg f?gBg]?g_g\?g9g@g?gg@y?gЂ @g@?g?g`Qcg2g:7gdig c?g`ֿg`ؿg?g+@g?g`(f?g@8ig=g򩣿g g+gfg gLgx?g`*(g?g/տg@/aſgEggg ?gGg{?gh?g_a?g?g?g3@g#?gygѱܿg @g׿g d,gg&gٿg`{X?gYb?g@"gيg3gTg@@gkgg`{ڿg?g>g ?gg`5g ?g?g5?g?gIؿg A?g?gѿg9?g@@gzg j g g@g@N?g}g?gZg`@gcg ҿg@=Dgg?g`gb@g{@gh@gg?g]ƿgK?ggw?g`?g 4?g b?gg`ſgp?g@?g@Q=s?g@տgg@@g1u?g͔ƿg@?g䉫?g`@g_g@g,?g`dg`T?gL?gb?g3?gȃgxg v?g`?g섥gK3g/ۿg`Eg@#?g3?g g c?g1g?g`%?g%?g@w@gI?gkg g@J߿g?gǿgFֿg9?g`g8?g` @gxg@wg x?gg@ҿg͆gvg?gn@g`9ǿgY?g g@?g[m?g F?g@ ?g`Ьg`Ggc?g@gPug@?gg@Hg4g@gwg#?gd?gMg g ?g?g@vPgſgr?gLg ?g@?g:ڿg ?g?g` ?g@_g$Rg Z?g3g`g ?g@Ogew@g LgV?gߩҿgV?g?ggŠ?g`;?g Կg`@g-g?g`gZY?gg ?g8;?g?gaοg@Rh?g`F ?gA?g@?gg`g8?g`>?g?g@pgi?g ?g`пg@@gB?gbg@@gg`\gg)W?g`?g`g`@g 0$ӿg >gi?gcg?gH?g T?g ]g.ܿgyؿg*g`F?g@g`Is?g_;?gE?g g`޿g.g@\?g`[?g #?g ?g?ggN?gjg=?gR޿g耎?g`gLg@g=g@g@?gR@grgg?g`v_?g@T?g?g@?g8?g{}gi?gSg@k?gs?g?g?g _g@3@g?g@ ?g`Xrg[gg?g`?g@?g5g?)r@ggx?g gk?g gl?gy?g6g?gпg`5?gk@g$gz?g 1J@g ?go?g.ghgӿg_?g?g$?gg6@g@T?g-?g g@ggù?g.?g?gfg?g,'?g?g^@g@Lg>ۿg4bg?g Ng޿g ?gnZgtgi~@g Jf?g^Gg`mg`6@g/?g ?gOܿg?gggg^DgFg|g1g`8g@*?gֿg%&9:?A @$%78AA B%&89CA D%&89EA F%&9:GA H$%78IA J#$67KA L#$67MA N$%78OA P&';$%89?A @%&9:AA B%&9:CA D%&89EA F%&89GA H%&89IA J$%78KA L$%89MA N$%89OA P&';>>rr1r-bits group_size hadamard_sizec Pt|jdks Jdt|d||zz|j}t j|ddz}|j}|j}|dt j }t|dg|}|jd|z} | d| |}t j|d} t|d| dddddfz }t|dg| |jd| d|}t j |jdd|t j } td|jdd D]V} t jd|| | d zz|jz|z d t j| | | d z<W~| | jdd} | t%|z } t'| t j| |||zd |||||||d \}} } }}|| | |t j|dS)Nrz%Only 2D weights are supported for now)axisT)copydtyperrdevicer=r/)r$F)num_bitsr7 vector_sizer=r?check_correctnessr=)weightscalestablestables2 tune_metadata)rrr4tor?r2linalgnormr=float32r(reshaper emptyuint8rargmaxTrr repeat_interleaveviewfloat16)rDr6r)r7r8grid grid_norm_2r?r=multrEcodesirFrGrHs rquantize_with_higgsrZs v|   ! ! !#J ! ! ! !Q1t8_ - - 0 0 ? ?D,##Dr#22a7K ]F LE YYD Y 6 6F &1#} 5 5F <?m +D ^^Bm 4 4F \  vB  / /F  * *VAAAqqq$J-? ?F&1#q ) ) 1 1&,q/4Q O OF K SbS)& L L LE 1fl1or * *pp!LVABJ-?)?$&)H;)V\^___bbchcnooa!b&j MM%+a." - -E d=)) )F5L   % 0 0-:2MSTUUU  6 6 62FFFG]<bias)super__init__r]r^r@r7r8r Parameterr2rNint16rDrErFrGrbregister_parameter workspacerH) selfr]r^r@rbr=r?r7r8 __class__s rrdzHiggsLinear.__init__s &(  $*Z'1,,,,9$$$$l K0B6 DEK`f g g g    l K{j'@AW] ^ ^ ^ns    l5;8 ~USY#Z#Z#Zjoppp | KHak15U6 R R Rbg      2 U[fTY%Z%Z%ZjopppDII  # #FD 1 1 1+/rc  t|dg|j}|jtdt ||j|j|j|j tj |j|j |jS)Nr:z,Workspace must be set before calling forwardrC)r8) r(r8rh Exceptionr rDrErFrGrSr2rLrH)rixs rforwardzHiggsLinear.forwards RD$"4 5 5 > !JKK K K K K L  EM  2 2 N  ,    r)TNNr-r5) __name__ __module__ __qualname__intrr2r=r?rdrn __classcell__)rjs@rr\r\s '+)-!'0'0'0'0 '0  $ '0&'0'0'0'0'0'0'0'0R       rr\Fc  ddlm}|D]k\}}|g}||t |t jrd| t fd|Ds|5|j }|j } t|| |j du|j |j|j|j|<d}t#||j|_|j|ddddn #1swxYwYt)t+|dkrt/||||| \} }|d m||fS) a~ Public method that recursively replaces the Linear layers of the given model with HIGGS quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successful or not. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`HiggsConfig`): The quantization config object that contains the quantization parameters. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. r)init_empty_weightsN.c3BK|]}|VdSN)endswith)rkeycurrent_key_name_strs r z,replace_with_higgs_linear..Ks2\\c+44S99\\\\\\r)rbr@r8r7TF)quantization_configcurrent_key_namehas_been_replacedmodules_to_not_convertr:) acceleraterunamed_childrenappend isinstancer Linearjoinanyr]r^r\rbr6r8r7_modulestype source_clsrequires_grad_rlistchildrenreplace_with_higgs_linearpop) modelr}r~rrrunamemoduler]r^rr{s @rrr)s0.-----,,..$!$! f  #! %%% fbi ( ( ?#&88,<#=#= \\\\E[\\\\\ ?''))??"("4K#)#6L+6#$#[4!4!9&9&G#6#A ,,,EN4()-%7;6llEN4(3N4(77>>>#???????????????$ tFOO%%&& ' '! + +#<$7!1"3'= $$$ A  R    # ##sBDD" %D" c tj5|D]A\}}|g}||t |t r|j}|j}tj |||j du|j j |j j |j|<|tj||j j |j j j|j|j_t)t+|dkrt/||}|dC|cdddS#1swxYwYdS)a Dequantizes the HiggsLinear layers in the given model by replacing them with standard torch.nn.Linear layers. Args: model (torch.nn.Module): The model containing HiggsLinear layers to be dequantized. current_key_name (list, optional): A list to keep track of the current module names during recursion. Defaults to None. Returns: torch.nn.Module: The model with HiggsLinear layers replaced by torch.nn.Linear layers. N)rbr?r=r>r)r~r:)r2no_gradrrrr\r]r^r rrbrEr?r=reyerQ contiguousrDdatarrrdequantize_higgsr)rr~rrr]r^rs rrrks !0022 % %LD&'#%  # #D ) ) )&+.. !$0 %2 ',x D0!=/ -- (7((t$4:6Ik&-2FfmNabbb44JJLLt$+04))**++a//$%5   $ $ $ $;sEE::E>E>r)r1rr-r5)NNFNrx)__doc__mathrtypingrutilsrrrr2r flute.integrations.higgsr flute.tuner r fast_hadamard_transformr r(rrr4rZModuler\rrrrrrs]\LLL2@@@@@@11111111;::::::BBBBE?cE?cE?nE?E?E?E?P 00c0#0s0ad0000f9 9 9 9 9 %(/9 9 9 | ?$?$?$?$D''''''r