Pi-ddlmZddlZddlmcmcmZddlmZddlm Z gdZ dej de de d eej ej ffd Z ejejfd e d e d e d ej fdZdej dej de de d eej ej ej ff dZdej dej dej dejde de d eej ej ffdZdej de de de d eej ej ff dZdej dej de de de d ej f dZdej de de de d ej f dZdej de de de d ej f dZdej de de de de d ej f d Zdej de de de de d ej f d!ZdS)")TupleN)const) mask_creator) inject_24marlin_24_workspacepack_to_marlin_24unpack_from_marlin_24wsize_ksize_nreturnc$|j||fksJt|}||z|fS)aInjects 2:4 sparsity into a weight tensor. The sparsity is applied in a 2:4 ratio, where for every group of 4 weights, 2 will be pruned based on their value. The mask will be created based on the ranked weight values. Args: w (torch.Tensor): The weight tensor to inject sparsity into. size_k (int): The number of input features. size_n (int): The number of output features. Returns: Tuple[torch.Tensor, torch.Tensor]: The pruned weight tensor and the mask tensor. )shapertcudabool contiguous)r r r masks t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/sparsity/marlin/__init__.pyrrsz 7vv& & & & &    " " ' ' ) ) . . 0 0D 1H " "DOO$5$5 55 out_features min_thread_n max_parallelc||zdksJd|d|||z|z}tj|tjdS)aCreates a workspace for marlin 2:4 quantization. The workspace is used to coordinate the locks during the execution of the kernel. Args: out_features (int): The number of output features. min_thread_n (int, optional): The minimum number of threads per block. Defaults to `MARLIN_24_MIN_THREAD_N`. max_parallel (int, optional): The maximum number of parallel threads. Defaults to `MARLIN_24_MAX_PARALLEL`. Returns: torch.Tensor: The workspace tensor fully initialized with zeros. rzout_features = z, min_thread_n = rdtypedevice)torchzerosint)rrrmax_workspace_sizes rrr$sa , &! + + +G,GGGG , + +',6,F ;)6 J J JJrq_w_24scalesnum_bits group_sizec|j\}}t||||\}}|dz}t||||} t|||||} | | |fS)aPacks the quantized weights and scales into the marlin 2:4 format. Args: q_w_24 (torch.Tensor): The quantized weight tensor with 2:4 sparsity applied. scales (torch.Tensor): The scale tensor. num_bits (int): The number of bits used for quantization. group_size (int): The group size that was applied during quantization. Returns: Tuple[torch.Tensor, torch.Tensor, torch.Tensor]: The packed quantized weights, the packed scales, and the meta tensor. )r_compress_quantized_24_weight_to_marlin_weights_to_marlin_scales) r"r#r$r% in_featuresr q_w_24_compmetain_features_compmarlin_24_q_w_comp marlin_24_ss rrr:s !' K6 \8K#a',%|X$ \:xK {D 00rr,r-original_shapec|\}}t|g|||R}|dz} t|| ||} t| || ||} | |fS)a7Unpacks the quantized weights and scales from the marlin 2:4 format. Args: q_w_24_comp (torch.Tensor): The packed quantized weights. scales (torch.Tensor): The packed scales. meta (torch.Tensor): The meta tensor. original_shape (torch.Size): The original shape of the weight tensor. group_size (int): The group size that was applied during quantization. num_bits (int): The number of bits used for quantization. Returns: Tuple[torch.Tensor, torch.Tensor]: The unpacked quantized weights and scales. r')_from_marlin_scale_from_marlin_weights_decompress_quantized_24_weight) r,r#r-r1r%r$r+runpacked_scalesr.unpacked_q_w_24_compunpacked_q_w_24s rr r _s&!/K)W.W*WhWWWO"a'0%|X 6d$4lHO O ++rq_24c|j||fksJd|zdz }|dzdz}||z }|}tj|\}}|}||z} ||jddz|jddz}| |fS)aCompresses the quantized weights to a 2:4 sparse format. Normalizes the weights over 0 before compressing them. Args: q_24 (torch.Tensor): The quantized weight tensor. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: Tuple[torch.Tensor, torch.Tensor]: The compressed quantized weight tensor and the meta tensor. r'r)rrrutils)sparse_semi_structured_from_dense_cutlassresize_) r9r r r$ max_q_valzp q_24_no_zpq_24_no_zp_compr- q_24_comps rr(r(s :&&) ) ) ) )h!#I a-A BJ**,,J!KJWWOT%''))4466O "$I << 1 *DJqMA,= > >D d?rrCc|j||fksJ||jddz|jddz}d|zdz }|dzdz}||z }|}t j||}|}||jddz|jddz}||z} | S)aDecompresses the quantized weights from a 2:4 sparse format and restores the original shape. Args: q_24_comp (torch.Tensor): The compressed quantized weight tensor in 2:4 sparse format. meta (torch.Tensor): The meta tensor. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The decompressed quantized weight tensor. r;r'r)rr>rrr<'sparse_semi_structured_to_dense_cutlass) rCr-r r r$r?r@rBrAr9s rr5r5s ?vv. . . . . << 1 *DJqMA,= > >Dh!#I a-A B"nO&''))4466O>PTUUJ**,,J << 1 *DJqMA,= > >D ?D Krq_wctj|\}}}tj||||}tj|}|t j}t j|jd|jd|zft j|j }t|D]}||dd|d|f||zzz}|t j }|S)aConverts a quantized and 2:4 sparse format weight tensor to the marlin 2:4 format. Args: q_w (torch.Tensor): The quantized weight tensor in 2:4 sparse format. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The weight tensor in the marlin 2:4 format. rr;rNr) r< get_perms_24marlin_permute_weightsget_pack_factortorint64rrrrangeint32) rFr r r$perm_24_ pack_factorq_packedis rr)r)s"&x00MGQ  &sFFG D DC'11K &&  C{ 1sy|{23kzH ;  ==C1>k>)*x!|<<{{{--H OrrSctj|\}}}tj|}|tj}t j|jd|jd|zftj|j}t|D]}|||zz d|zdz z|dd|d|f<|tj }tj ||||} | S)aConverts a weight tensor in the marlin 2:4 format to a regular quantized 2:4 sparse format. Args: q_packed (torch.Tensor): The weight tensor in the marlin 2:4 format. size_k (int): The number of input features. size_n (int): The number of output features. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The weight tensor in the quantized 2:4 sparse format. rr;rNrH) r<get_reverse_perms_24rKrLrrMrrrrNrOreverse_marlin_permute_weights) rSr r r$rPrQrR q_w_unpackedrTq_w_comps rr4r4s.x88MGQ'11K {{5;''H;  HN1- ;<kL ;    +31 +E (]a + QQQ;&'' ???55L3ffgH OrcZtj|\}}}||kr5|dkr/|dt|fdd|f}n.|dt|fdd|f}|d|f}|S)aConverts a scale tensor to the format necessary for marlin. Args: scales (torch.Tensor): The scale tensor. size_k (int): The number of input features. size_n (int): The number of output features. group_size (int): The group size that was applied during quantization. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The scale tensor in the marlin format. N)r<rIreshapelenrr#r r r%r$rQ scale_perm_24scale_perm_single_24s rr*r*s.3-?-I-I*A}*FzR//S%7%7 899!!!]:JKS)=%>%> ?@@ AA# # ^^RL ) ) 4 4 6 6F Mrc`tj|\}}}||krN|dkrH|dt|fdd|f}|||z|fS|dt|fdd|f}|dS)aConverts a scale tensor from the marlin format to their original format. Args: scales (torch.Tensor): The scale tensor in the marlin format. size_k (int): The number of input features. size_n (int): The number of output features. group_size (int): The group size that was applied during quantization. num_bits (int): The number of bits used for quantization. Returns: torch.Tensor: The scale tensor in their original format r[N)r;r[)r<rVr\r]r^s rr3r35s.3-G-Q-Q*A}*FzR//S%7%7 899!!!]:JK~~v3V<===S)=%>%> ?@@ AA# # ~~g&&&r)typingrrtorchao.sparsity.marlin.utilssparsitymarlinr<rtorchao.sparsity.utilsr__all__Tensorr r MIN_THREAD_N MAX_PARALLELrrSizer r(r5r)r4r*r3rrrms ------------////////////   6 |6 6*-6 5< %&6666***KKKKK \ KKKK,"1 L"1 L"1"1 "1  5<u| 34 "1"1"1"1J$,$, L$, ,$,J $,  $,  $, 5< %&$,$,$,$,N  ,  # -0