)`inddlmZddlZddlZddlmZmZmZddlm Z m Z m Z d dZ d d Z dd ZdS))OptionalN) gemm_kernel!gemm_kernel_descriptor_persistentgemm_kernel_persistent) check_device check_dim check_input?c t|t||gtd|td||/t|t|gtd||jd|jdks Jd|j|jks Jd|L|jd|jdks Jd|jd|jdks Jd|j\ }|j\} |j}|r|n|t jkr|n t j}||j|ks Jd |t j f|j | n|}t j d j } | nt|  fd } t| ||| ||d|d|d|d|d|d|| |S)a GEMM operation with SM constraint by Triton. C = alpha * (a @ b.T) + beta * C Args: a: The first input matrix. Shape: (M, K) b: The second input matrix. Shape: (K, N) c: The output matrix. Shape: (M, N). In-place epilogue is supported. Expected to be out_dtype (if not specified, same as a.dtype, but fp8 --> bf16). alpha: The scaling factor for the product of a and b. beta: The scaling factor for the output matrix c. out_dtype: The dtype of the output matrix. Default: fp8 --> bf16. Otherwise, same as a.dtype. num_sms: The number of SMs to use for the computation. Nrr'Incompatible dimensions between a and b#Incompatible dtypes between a and b'Incompatible dimensions between a and c'Incompatible dimensions between b and c+Incompatible dtypes between c and out_dtypedevicedtypecudac ttj|dtj|dzfSN BLOCK_SIZE_M BLOCK_SIZE_NmintritoncdivMETAMNnum_smss x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/triton/sm_constraint_gemm.pyz!gemm_persistent..GD  K4/ 0 06;q$~BV3W3W W  )alphabetaNUM_SMS)r rr shapertorch float8_e4m3fnbfloat16emptyrrget_device_propertiesmulti_processor_countrrstride) abcr)r* out_dtyper$Krr+gridr"r#s ` @@r%gemm_persistentr:sy NNN!Q aOOO aOOO}AaS!Q 71: # # #%N # # # 7ag   D   }wqzQWQZ''')R'''wqzQWQZ''')R''' 7DAq 7DAq GE    E' ' 'U ^  99,,,5-, , BC QF189====PQAj..v66LG ggc'7.C.CG      D4                    " Hr(c* t|t||gtd|td||/t|t|gtd||jd|jdks Jd|j|jks Jd|L|jd|jdks Jd|jd|jdks Jd|j\ }|j\} |j}|r|n|t jkr|n t j}||j|ks Jd |t j f|j | n|} fd }t|||| || d| d| d| d| d| d|| |S) a GEMM operation without SM constraint by Triton. C = alpha * (a @ b.T) + beta * C Args: a: The first input matrix. Shape: (M, K) b: The second input matrix. Shape: (K, N) c: The output matrix. Shape: (M, N). In-place epilogue is supported. Expected to be out_dtype (if not specified, same as a.dtype, but fp8 --> bf16). alpha: The scaling factor for the product of a and b. beta: The scaling factor for the output matrix c. out_dtype: The dtype of the output matrix. Default: fp8 --> bf16. Otherwise, same as a.dtype. num_sms: The number of SMs to use for the computation. rNrrrrrrrrcttj|dtj|dzfSr)rr)r!r"r#s r%r&zgemm..s4 AtN+,,v{1d>>R/S/SSr()r)r*) r rr r,rr-r.r/r0rrr3) r4r5r6r)r*r7r8rr9r"r#s @@r%gemmr=bs?NNN!Q aOOO aOOO}AaS!Q 71: # # #%N # # # 7ag   D   }wqzQWQZ''')R'''wqzQWQZ''')R''' 7DAq 7DAq GE    E' ' 'U ^  99,,,5-, , BC QF189====PQA     D                    Hr(Fc t|t|t||gtd|td||/t|t|gtd||jd|jdks Jd|j|jks Jd|L|jd|jdks Jd|jd|jdks Jd|j\ }|j\}|j} |r|n| t jkr| n t j}| t jkr!|d ks Jd d ks Jd n |d ks Jd d ks Jd |t j f|j | n|}t j d j } | nt| dtdtdttfd} t!j|  fd} t%| ||| |||d| t jkrdnddd dd ||S)a GEMM operation with SM constraint by Triton. Requires TMA support and descriptor creation. C = alpha * (a @ b.T) + beta * C Note: - K and N must be greater than 16B. - Support float16, float8_e4m3fn, bfloat16. - float32 is not supported due to performance issues. Args: a: The first input matrix. Shape: (M, K) b: The second input matrix. Shape: (N, K) c: The output matrix. Shape: (M, N). In-place epilogue is supported. Expected to be out_dtype (if not specified, same as a.dtype, but fp8 --> bf16). alpha: The scaling factor for the product of a and b. beta: The scaling factor for the output matrix c. out_dtype: The dtype of the output matrix. Default: fp8 --> bf16. Otherwise, same as a.dtype. num_sms: The number of SMs to use for the computation. EPILOGUE_SUBTILE: Whether to use the epilogue subtile optimization. rNrrrrrrzLeast chunk size must be 16Brrsize alignmentstreamcDtj|dtjS)Nrr)r-r0int8)rArBrCs r%alloc_fnz,gemm_descriptor_persistent..alloc_fns{4ejAAAAr(c ttj|dtj|dzfSrrr s r%r&z,gemm_descriptor_persistent..r'r(@)r+rr BLOCK_SIZE_K GROUP_SIZE_M num_stages num_warpsEPILOGUE_SUBTILE)r rr r,rr-r.r/r0rrr1r2rintrr set_allocatorrfloat32)r4r5r6r)r*r7r$rOr8rr+rFr9r"r#s ` @@r%gemm_descriptor_persistentrSs>NNNNNN!Q aOOO aOOO}AaS!Q 71: # # #%N # # # 7ag   D   }wqzQWQZ''')R'''wqzQWQZ''')R''' 7DAq 7DAq GE    E' ' 'U ^  ###Bwww6wwwBwww6wwwwAvvv5vvvAvvv5vvvAB QF189====PQAj..v66LG ggc'7.C.CGBsBsBHSMBBBB """      D&d+        !U]22SS)!$ Hr()Nr r NN)Nr r N)Nr r NNF)typingrr-rkernels.sm_constraint_gemmrrrutilsrr r r:r=rSr(r%rXs  8777777777Q Q Q Q hH H H H \  i i i i i i r(