Piq ddlmZmZddlmZmZddlZddlZddlm Z m Z edGddZ e Z d e d e fd Ze jfd ejd e de dejde d ejf dZe jfdejd e de dejde d ejf dZd e de d ejfdZd e d eee ee ffdZd e de d ejfdZd e d eejejffdZdS)) dataclassfield)ListTupleN) GranularityPerAxisT)frozenceZdZUdZeed<dZeed<dZeed<edZ e eed<ed Z e eed <d S) MarlinQQQConstantsTILE@ MIN_THREAD_N MAX_PARALLELcdgS)Nry/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/quantization/marlin_qqq/utils.pyzMarlinQQQConstants.s1#r)default_factorySUPPORTED_NUM_BITSc ddgS)NrrrrrzMarlinQQQConstants.s b#YrSUPPORTED_GROUP_SIZESN) __name__ __module__ __qualname__r int__annotations__rrrrrrrrrr r sD#NNNL#L#$)E++$F$F$FS FFF',u=N=N'O'O'O49OOOOOrr num_bitsreturnc8d|zdks Jd|d|zS)zCompute the packing factor for a given number of bits. Args: num_bits (int): Number of bits to pack. Returns: int: The packing factor. rzUnsupported num_bits = r)r"s rget_pack_factorr&s4 =A   CCC    >rq_wsize_ksize_npermtilec|j||fksJ||zdksJd|d|||zdksJd|d||||z|||z|f}|d}|||z||zf}|d|fdd|f|j}|S)aPermute weights to 16x64 Marlin tiles. Args: q_w (torch.Tensor): Quantized weights. size_k (int): Number of input features. size_n (int): Number of output features. perm (torch.Tensor): The computed permutation tensor to be applied. tile (int, optional): Tile size. Defaults to `TILE`. Returns: torch.Tensor: Weight tensor permuted to Marlin tiles. r size_k = , tile = rrN)shapereshapepermutenumel)r'r(r)r*r+s rmarlin_permute_weightsr7+s $ 9( ( ( ( ( D=A   B6BBDBB    D=A   B6BBDBB    ++v~tVt^TB C CC ++l # #C ++v~v}5 6 6C ++r4::<<( ) )!!!T' 2 : :39 E EC Jr q_w_unpacked reverse_permc|jd|f||z|jd|zfksJ||zdksJd|d|||zdksJd|d||d|fdd|f|j}|||z||z||f}|d}|||f}|S)aReverse permute weights from 16x64 Marlin tiles. Args: q_w_unpacked (torch.Tensor): Unpacked quantized weights. size_k (int): Number of input features. size_n (int): Number of output features. reverse_perm (torch.Tensor): The computed reverse permutation tensor to be applied. tile (int, optional): Tile size. Defaults to `TILE`. Returns: torch.Tensor: Weight tensor reverse permuted from Marlin tiles. rr1r-r.rNr/)r3r4r6r5)r8r(r)r9r+q_w_comps rreverse_marlin_permute_weightsr<KsA$  q !6 *$1%/     D=A   B6BBDBB    D=A   B6BBDBB   ##R););)=)=$>?? < gl !! 44t LMMH --H 011H Or granularityc g}tdD]}g}|dz}dD]M}d|dzzd|dzzdzd|dzzdzd|dzzdzfD]#}|d|z|zd|zz$NtdD]# | fd |D$tj|}|dkrCt |t rtjgd } n9tjgd } n"td || d t| fdd| f }tj |}|S)aPrecompute permutations for the marlin weight shuffling. Args: num_bits (int): Number of bits to pack. granularity (Granularity): The weight quantization granularity. Returns: torch.Tensor: The weight permutation tensor. r%r)rr1r1r0r2r c g|] }|dzz S)r).0pjs r z'get_qqq_weight_perm..s!999aa#'k999r)rrr1r0r2)rr0rrGr1r2rFrHnum_bits must be 4, got {}rN)rangeappendextendnumpyarray isinstancer Exceptionformatr4lenraveltorch from_numpy) r"r= perm_listiperm1colblockrowr* interleaverDs @rget_qqq_weight_permr]psI 2YY ; ;1f 9 9EQU QU aQU aQU a  9 9  R#X^a%i78888  9q ; ;A   99995999 : : : : ; ;y ! !D1}} k7 + + ?%=%=%=>>JJ%=%=%=>>JJ4;;HEEFFF <<S__- . .qqq*} = C C E ED  D ! !D KrcJ|dkr"td|g}tdD]0|fdtdD1g}tdD]#|fddD$||fS)zPrecompute permutations for the marlin scale shuffling. Args: num_bits (int): Number of bits to pack. Returns: Tuple[List[int], List[int]]: Scale permutation list and scale permutation list for a single group. rrIr?c g|] }d|zz S)r?rrBrDrWs rrEz'get_qqq_scale_perms..s!7771q1u9777rc g|] }dz|z S)r0rr`s rrEz'get_qqq_scale_perms..s!!R!R!R!a%!)!R!R!Rr)rr1r? r )rPrQrJrL)r" scale_permscale_perm_singlerWs @rget_qqq_scale_permsrhs1}}4;;HEEFFFJ 1XX997777eAhh7778888#% 1XXTT  !R!R!R!R5Q!R!R!RSSSS ( ((rcNt||}|}|S)aReverse permutation for Marlin weight shuffling from `get_qqq_weight_perm`. Args: num_bits (int): Number of bits to pack. granularity (Granularity): The weight quantization granularity. Returns: torch.Tensor: The reversed weight permutation tensor. )r]argsort)r"r=r*s rget_qqq_weight_reverse_permrks% x 5 5D <<>>D Krct|\}}tj|}tj|}||fS)a#Reverse permutation for Marlin scale shuffling from `get_qqq_scale_perms`. Args: num_bits (int): Number of bits to pack. Returns: Tuple[List[int], List[int]]: The reversed scale permutation list and the reversed scale permutation list for a single group. )rhrTtensorrj)r"rfrgs rget_qqq_scale_reverse_permsrns[%8$A$A!J!j))1133J %677??AA ( ((r) dataclassesrrtypingrrrMrT torchao.quantization.granularityrrr constr r&r Tensorr7r<r]rhrkrnrrrrtss )(((((((   $PPPPPPPP  c c    "     ,     \ J !!,! ! !, !  !  \ !!!!J$#$K$EL$$$$N)#)%S 490D*E))))&   +  \     )# )% el8R2S ) ) ) ) ) )r