)`i ddlmZddlmZddlZddlmZddlZddlm Z m Z m Z dZ e dgd Z e dgd Zejd Ze ie e ddejdejdejdeddf dZe ie e ddejdejdejdeddf dZdS))flashinfer_api)gen_dsv3_router_gemm_moduleN)SimpleNamespace)register_custom_opsupported_compute_capabilitybackend_requirementc|dkrtd|dkrtd|dkrtd|ddkrtd|ddkrtd|ddkrtd |jd|jdkrtd |jd|jdkrtd |jd|jdkrtd d }d}d}|jd|ks|jd|krtd|d||jd|krtd||jd|krtd||jt jkrtd|jt jkrtd|j|krtd|ddS)Nrzmat_a must be a 2D tensorzmat_b must be a 2D tensorzout must be a 2D tensorzmat_a must be row-majorzout must be row-majorrzmat_b must be column-majorz.mat_a.shape[1] must be equal to mat_b.shape[0]z,out.shape[0] must be equal to mat_a.shape[0]z,out.shape[1] must be equal to mat_b.shape[1]iz,mat_a.shape[0] (num_tokens) must be between z and z-mat_a.shape[1] (hidden_dim) must be equal to z.mat_b.shape[1] (num_experts) must be equal to zmat_a must be a bfloat16 tensorzmat_b must be a bfloat16 tensorzout must be a z tensorT)dim ValueErrorstrideshapedtypetorchbfloat16) mat_amat_boutlaunch_with_pdlexpected_num_expertsexpected_out_dtypeexpected_hidden_dim min_tokens max_tokenss s/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/gemm/routergemm_dsv3.py_mm_M1_16_K7168_shape_checksr sp yy{{a4555 yy{{a4555 wwyyA~~2333 ||A!2333 zz!}}0111 ||A!5666 {1~Q''IJJJ y|u{1~%%GHHH y|u{1~%%GHHHJJ {1~ ""ek!nz&A&A X: X XJ X X    {1~,,, Q    rc ttddg d dtjdtjdtjdt ddf fd }td dg d dtjdtjdtjdt ddf fd }t || S)Nzflashinfer::ml3_router_gemm_opr) mutates_argsFrrrreturnc8||||dSN)ml3_router_gemm_oprrrrmodules rmm_M1_16_K7168_N128z8get_dsv3_router_gemm_module..mm_M1_16_K7168_N128as% !!%_EEEEErzflashinfer::dsv3_router_gemm_opc8||||dSr.)dsv3_router_gemm_opr0s rmm_M1_16_K7168_N256z8get_dsv3_router_gemm_module..mm_M1_16_K7168_N256ms% ""5%oFFFFFr)r2r5F)rbuild_and_loadrrTensorboolr)r2r5r1s @rget_dsv3_router_gemm_moduler:]sS ( * * 9 9 ; ;F(W!& FF|F|F\F F  FFFFF F)W!& GG|G|G\G G  GGGGG G //   r) common_checkFrrrrr,cNt||||dS)a0Optimized GEMM for the router operation in Mistral Large 3. This function performs a highly optimized matrix multiplication specifically tailored for the expert routing GEMM in Mistral Large 3's Mixture of Experts (MoE) architecture. It computes out = mat_a @ mat_b where mat_a contains token embeddings and mat_b contains expert routing weights. The implementation is optimized for the specific problem dimensions used in Mistral Large 3: - Hidden dimension (K): 7168 - Number of experts (N): 128 - Number of tokens (M): 1-16 Args: mat_a (torch.Tensor): Input token embeddings of shape (M, K) where M is the number of tokens (1-16) and K is the hidden dimension (7168). Must be bfloat16, row-major (contiguous). mat_b (torch.Tensor): Expert routing weights of shape (K, N) where K is the hidden dimension (7168) and N is the number of experts (128). Must be bfloat16, column-major (transposed layout). out (torch.Tensor): Pre-allocated output tensor of shape (M, N) containing the routing scores. Must be bfloat16, row-major (contiguous). This tensor is mutated in-place. launch_with_pdl (bool, optional): Whether to launch the kernel using Persistent Device-side Launch. Defaults to False. Returns: None: The result is written directly to the `out` tensor. Raises: ValueError: If tensor dimensions, strides, or data types do not match the expected Mistral Large 3 router configuration. Note: This kernel is specialized for compute capability 10.0 (Blackwell architecture). The specific problem size optimization makes this significantly faster than general-purpose GEMM implementations for the router operation. N)r:r2r%s rr2r26Z !!55 uc?rcNt||||dS)aOptimized GEMM for the router operation in DeepSeek-V3. This function performs a highly optimized matrix multiplication specifically tailored for the expert routing GEMM in DeepSeek-V3's Mixture of Experts (MoE) architecture. It computes out = mat_a @ mat_b where mat_a contains token embeddings and mat_b contains expert routing weights. The implementation is optimized for the specific problem dimensions used in DeepSeek-V3: - Hidden dimension (K): 7168 - Number of experts (N): 256 - Number of tokens (M): 1-16 Args: mat_a (torch.Tensor): Input token embeddings of shape (M, K) where M is the number of tokens (1-16) and K is the hidden dimension (7168). Must be bfloat16, row-major (contiguous). mat_b (torch.Tensor): Expert routing weights of shape (K, N) where K is the hidden dimension (7168) and N is the number of experts (256). Must be bfloat16, column-major (transposed layout). out (torch.Tensor): Pre-allocated output tensor of shape (M, N) containing the routing scores. Must be float32, row-major (contiguous). This tensor is mutated in-place. launch_with_pdl (bool, optional): Whether to launch the kernel using Persistent Device-side Launch. Defaults to False. Returns: None: The result is written directly to the `out` tensor. Raises: ValueError: If tensor dimensions, strides, or data types do not match the expected DeepSeek-V3 router configuration. Note: This kernel is specialized for compute capability 10.0 (Blackwell architecture). The specific problem size optimization makes this significantly faster than general-purpose GEMM implementations for the router operation. N)r:r5r%s rr5r5r=rr6) api_loggingrflashinfer.jitr functoolstypesrrflashinfer.utilsrrr rr&r)cacher:r8r9r2r5rrrFs((((((666666!!!!!! 333nse$$%$se$$%$ BR&GHHH " -- <- <- - -  ---IH-`R&GHHH " -- <- <- - -  ---IH---r