)`i0!LddlmZddlmZddlZddlmZddlmZ ddl Z dZ n#e $rdZ d Z YnwxYwda d ej jfd ZGd d eZddddddddejdejdejdededeedeejdeejdeejdeejdeejfdZe re je jjge jeddddddddejdejdejdededeedeejdeejdeejdeejdeejfdZddddddddddejdejdejdedejdedeejdeejdeejdeedeejdeejd eejd!eejd"ejd#ejf d$Zeddd dddddd%dejdejdejdedejdedeejdeejd&edeejdeejd eejd!eejd"eejd#ejfd'ZdS)()Enum)OptionalN)flashinfer_api)get_cudnn_fmha_gen_moduleTFstreamcttjatjt|jtS)N) _cudnn_handlecudnn create_handle set_stream cuda_stream)r s k/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/cudnn/decode.py_create_cudnn_handlers2+--  ]F$6777 cFeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdS)UIDsrrrde234iiN)__name__ __module__ __qualname__RESERVED_INVALID_UIDQ_UIDK_UIDV_UIDACTUAL_SEQ_LENS_Q_UIDACTUAL_SEQ_LENS_KV_UIDBLOCK_TABLES_UIDBLOCK_TABLES_K_UIDBLOCK_TABLES_V_UID RAGGED_Q_UID RAGGED_O_UIDRAGGED_STATS_UIDO_UID STATS_UIDrrrrs[ E E E LL EIIIrr) block_sizeactual_seq_lens_qactual_seq_lens_kv block_tablesbatch_offsets_qbatch_offsets_oqk_cachev_cachescalemax_sequence_kvr0r1r2r3r4r5cVd|t|jt|jfS)Ndecode)tupleshape) r6r7r8r9r:r0r1r2r3r4r5s r_sdpa_decode_key_fnr?4s,  ag gm  r) heur_modes)key_fnc ttj} d} d} t j| 5\} } |dkr*d}|jd|jd|jd}}}nt|dkr8|jd|jd|jd|jdf\}}}}n$td||dks Jd|dks Jd|jd}| d ||||f||z|||zdftj j }| N| | }| tjj||| |}| |}| tjj| tjj| tjj|||jdd|jdd}| |}| tjj| |}| tjj|9| |}| tjj|N| |}| tjj|d |du}| d |||||nd||nd|d ||||tj j \}} | N| | }| tjj||| tjjd  ||||g!||z|||zdg"tj j dddn #1swxYwY||||g}||#|||#|| |fS)Nrrrrz#q must have 3 or 4 dimensions, got z"q must have a sequence length of 1zk_cache must have 4 dimensionsr6)namedimstride data_typeFsdpaT) rDr6kv seq_len_q seq_len_kvuse_padding_mask is_inference attn_scalepaged_attention_k_tablepaged_attention_v_tablepaged_attention_max_seq_len_kvcompute_data_type)$rtorchcudacurrent_streamr graphrEr> ValueErrortensorrGBFLOAT16 tensor_likeset_uidrr*valueset_ragged_offsetr"r#r$reshaper(r)r%r&set_is_pass_by_valuerHFLOATr+r- set_outputset_dim set_stride set_data_typeappend) r6r7r8r9r:r0r1r2r3r4r5handleg_s_qobh_qod_qkd_vocudnn_qragged_q cudnn_k_cache cudnn_v_cachend_block_tablescudnn_k_block_tablescudnn_v_block_tablescudnn_actual_seq_lens_qcudnn_actual_seq_lens_kv padding_maskOragged_otensors_to_returns r_build_decode_graphr|Ls &ej&?&?&A&ABB [ W FQuuww!|| ! AGAJ 4AGAJGAJGAJGAJ '#4tt!!Pquuww!P!PQQQ1999B999;;==A%%%'G%%%=#DhhdD)t T4$;:/2 G *==99  !2!8999))(333MM'22MMM'22M OODJ, - - -  ! !$*"2 3 3 3  ! !$*"2 3 3 3'"."6"6 &q)1l.@.CQ##()}}_'E'E$$,,T-D-JKKK'(}}_'E'E$$,,T-D-JKKK ,*+--8I*J*J''//0J0PQQQ!-+,==9K+L+L((001L1RSSS(==eDDD-T9L66/@/L++RV1C0N,,TX!-! (<("'/"7#DAq(*==99  !2!8999##H--- IIdj& ' ' 2 24 8 8 @ @D$%  j$+tTD[!<==mm(???kW W W W W W W W W W W W W W W r%m]AF  (  $ $%< = = =  )  $ $%= > > >###sQR//R36R3)r1r2r3r0r4r5batch_offsets_kbatch_offsets_vworkspace_bufferr}r~outreturnc ft||||||||| | | nd| | nd \}}ttj}t jj|t jj|t j j|t j j|i}|||t j j<|||t j j<| | |t j j<| | |t jj<|(||t jj<||t jj<|||||S)N) r6r7r8r9r:r1r2r3r0r4r5) workspacerg)r|rrTrUrVrr"r]r#r$r-r%r&r*r+r(r)execute)r6r7r8r9rr:r1r2r3r0r4r5r}r~rrWtensorshandle_var_maps r_batch_decode_with_kv_cachers;$) '+-!+:+FD+:+FD   NE7#5:#<#<#>#>??G ! ' ' # G $4E*01%5G+12"+:!'("+:!'(1='-.1='-. MM'%5gMFFF Jr)r2r3is_cuda_graph_compatibler4r5r}r~rrc |jd}|jd}|jd}| #tj||||j|j} t sG||jd}tj}||||||||||| | | | nStj |dddf|jtj }|jd}t|||||||||| | || | S) a)Performs batched decode attention with paged KV cache using cuDNN. Args: q: Query tensor of shape (batch_size, num_heads_qo, head_dim), seq_len_q is the maximum sequence length of queries in the batch k_cache: Key cache tensor of shape (total_num_pages, num_heads_kv, page_size, head_dim) v_cache: Value cache tensor of shape (total_num_pages, num_heads_kv, page_size, head_dim) scale: Scaling factor for attention scores, typically 1/sqrt(head_dim) workspace_buffer: Workspace buffer for cuDNN operations. Scales with batch size. 128 MB should be sufficient for most cases max_sequence_kv: Maximum number of tokens per key/value sequence (s_kv_max) actual_seq_lens_kv: Actual sequence lengths for key/values per batch, shape (batch_size,) on CPU block_tables: Page table mapping for KV cache, shape (batch_size, num_pages_per_seq) on GPU is_cuda_graph_compatible: Whether the decode operation is compatible with CUDA graph batch_offsets: Optional batch offsets tensor of shape (batch_size,) on GPU out: Optional pre-allocated output tensor batch_offsets_q: Optional batch offsets for query tensor of shape (batch_size,) on GPU batch_offsets_o: Optional batch offsets for output tensor of shape (batch_size,) on GPU batch_offsets_k: Optional batch offsets for key tensor of shape (batch_size,) on GPU batch_offsets_v: Optional batch offsets for value tensor of shape (batch_size,) on GPU Returns: Output tensor of shape (batch_size, num_heads_qo, head_dim) Note: Currently only supports causal attention (causal must be True) All tensors must be contiguous and on the same CUDA device Query and KV heads can have different sizes (num_heads_qo >= num_heads_kv) rrrN)devicedtypeT) non_blockingr) r6r7r8r9rr:r1r2r3r4r5r0r) r>rTemptyrrCUDNN_AVAILABLEtorr<onesint32r)r6r7r8r9rr:r2r3rr4r5r}r~rbsrlrnactual_seq_lens_kv_gpurun_funcr1r0s r cudnn_batch_decode_with_kv_cachers:\ B 71:D = D {k"dDIII ' !3!6!6qxd!6!S!S,..5       "     $    "J AqM!(%+   ]1% #-+/1%++!    Jr)enumrtypingrrT api_loggingrutilsrr r ImportErrorr rUStreamrrTensorfloatintr?jit heur_modeA graph_cacher|rboolrr/rrrs ((((((,,,,,,LLLOO EOOO  !24:!"0415+/.2.2 | \\       -!.5<(el+el+,x$UY5?,-...U1222%&4859/32626t$t$t$ <t$t$t$ t$  t$SMt$$EL1t$%U\2t$u|,t$"%,/t$"%,/t$t$t$32/.t$~1515+/ !.2.2.2.2888 |8 \8\8  8 l 88  -8!.85<(8 8el+8el+8el+8el+8 !8" \#8888v26+/%*.2.2.2.2"&]]] |] \]\]  ] l ]]!.]5<(]#]el+]el+]el+]el+] %, ]  \!]]]]]]s % 11