)`i+k9UddlmZddlmZddlZddlmZddlmZ ddl Z dZ n#e $rdZ d Z YnwxYwda iZ eejejfed <d ejfd Zd ejjfdZGddeZddddddd ddddddddddejdejdejdedeedeedeejdejdeejdeedeedeedeejdeejd eejd!eejd"eejd#eejd$eejd%eejf(d&Ze re je jjg'e je(ddddddd ddddddddd)dejdejdejdedeedeedeejdeejdeejdeedeedeejdeejd eejd!eejd"eejd#eejd$eejd%eejf&d*Z ddddddddddddd+ dejdejdejded,ejd-ededejdejdeejd.eded/eejd0eejd1eejdeejdeejd eejd!eejd"eejd#eejd$eejd%eejd2e!ejejff0d3Z"edddddddddddd ddd4dejdejdejded,ejd-ededejdejdeejd.eded/eejd0eejd1eejdeejdeejd eejd!eejd"eejd#eejd$eejd5ed6ee#d%eejd2e!ejeejff4d7Z$dS)8)Enum)OptionalN)flashinfer_api)get_cudnn_fmha_gen_moduleTF_dummy_scale_tensorsdevicect|}|Btjdg|tjdddd}|t|<|S)Ng?r dtyper)r gettorchtensorfloat32reshape)r ts l/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/cudnn/prefill.py_get_dummy_scale_tensorrsZ  ((Ay L#vU] C C C K KAqRSUV W W'(V$ Hstreamcttjatjt|jtS)N) _cudnn_handlecudnn create_handle set_stream cuda_stream)rs r_create_cudnn_handlers4+--  ]F$6777 rcneZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdS)UIDsrrrde23456iiN)__name__ __module__ __qualname__RESERVED_INVALID_UIDQ_UIDK_UIDV_UIDACTUAL_SEQ_LENS_Q_UIDACTUAL_SEQ_LENS_KV_UIDBLOCK_TABLES_UIDBLOCK_TABLES_K_UIDBLOCK_TABLES_V_UID RAGGED_Q_UID RAGGED_O_UIDRAGGED_STATS_UID RAGGED_K_UID RAGGED_V_UIDO_UID STATS_UID Q_SCALE_UID K_SCALE_UID V_SCALE_UID S_SCALE_UID S_DESCALE_UID O_SCALE_UID S_AMAX_UID O_AMAX_UIDrrr r )s E E E LLLL EIKKKKMKJJJJrr )max_token_seq_qmax_sequence_kvactual_seq_lens_q block_tables page_sizebottom_right_causal_mask return_lsebatch_offsets_qbatch_offsets_obatch_offsets_kbatch_offsets_vbatch_offsets_statsoutlse o_data_typeqk_cachev_cachescalerPrQrRactual_seq_lens_kvrSrTrUrVrWrXrYrZr[r\r]r^cH|jd}|dkr|jd|jd}}n2|dkr|jd|jd}}|dkr|jd|jd}}n2|dkr|jd|jd}}| |jd} |||j||||||||du| | | f}|S)Nrr!rr)shapedimr )r_r`rarbrPrQrRrcrSrTrUrVrWrXrYrZr[r\r]r^graph_bh_qod_qkh_kvd_vokeys r_sdpa_prefill_key_fnrnKs. %a(Guuww!||WQZd AWQZd{{}}]1%w}Q'7d !  ]1%w}Q'7dM!$         D   C Jr) heur_modes)key_fn)rPrQrRrcrSrUrVrWrXrYrZr[r\r]r^c<ttj|j}|jd}|}|}t jstdt j |j }t j |j }t j |j }||j }t j |}|t j j ks|t j jkr:t jdkr#tdt jt j|5\}}|dkr|jd|jd}}nJ|dkr|jd|jd}}nt%d |j|d ||||f||z|||zdf| }|t j j ks|t j jkr|d d d t j j } |dd d t j j }!|dd d t j j }"|dd d t j j }#|dd d t j j }$|dd d t j j }%| t,jj|!t,jj|"t,jj|#t,jj|$t,jj|%t,jj| N|| }&|&t,jj| |&|dkr'| Jd|jd|jd}(}'nJ|dkr|jd|jd}(}'nt%d|j|dkr|d||'||f|'|z|z|||'zdf| })| N|| }*|*t,j!j|) |*|d||'||(f|'|(z|z|(|(|'zdf| }+|N||},|,t,j"j|+ |,nx|dkr`|d|j|#| })|d|j|#| }+|t,j$j|)t,j%j|+t,j&j||'|jdd|jdd}-||-}.|.t,j(j||-}/|/t,j)j|N||}0|0*d|0t,j+j|N||}1|1*d|1t,j,j|duo|du}2|t j j-ks|t j j.krK|/d||)|+||0nd||1nd|2|| | ||.nd||/nd||ndt j j\}3}4ni|t j j ks|t j jkr>|0||)|+| |!|"|#|$|%d|| |2||0nd||1nd||.nd||/nd||nd\}3}4}5}6|5t,j1j2d3d 4d 5t j j|6t,j6j2d3d 4d 5t j j| N|| }7|7t,j7j|3 |7|N||}8|8t,j8j|4 |8|3t,j9j2d3||||(g4||(z|z|(|(|zdg5|| r|4t,j:j2| 5t j j3|||dg4||zd|dg||)|+|3g}9| r|9;|4||9;|0||9;|1||9fcdddS#1swxYwYdS)Nrztorch is not availablei5fzKFP8 is not supported in cuDNN backend version < 9.17.1, current version is r!rrrezInvalid query tensor shape: r_)namergstride data_typeq_scale)rrrrk_scalev_scales_scale s_descaleo_scalez+block_tables needs 4 dimensions of kv cachezInvalid kv cache tensor shape: r`rarRrcsdpa)rrr_kv seq_len_q seq_len_kvuse_padding_mask attn_scalegenerate_statsuse_causal_mask_bottom_rightpaged_attention_k_tablepaged_attention_v_tablepaged_attention_max_seq_len_kvcompute_data_typeT)r_r|r} descale_q descale_k descale_vscale_s descale_sscale_orrrrr~rrrrF)r?set_namer;r<BFLOAT16HALFr{sdpa_fp8rM set_outputset_dim set_stride set_data_typerNrArBrErFappend):r_r`rarbrPrQrRrcrSrUrVrWrXrYrZr[r\r]r^handlerh graph_s_qo graph_s_kvcudnn_q_data_typecudnn_k_data_typecudnn_v_data_typecudnn_o_data_typeg_rirjcudnn_q cudnn_q_scale cudnn_k_scale cudnn_v_scale cudnn_s_scalecudnn_s_descale cudnn_o_scaleragged_qrkrl cudnn_k_cacheragged_k cudnn_v_cacheragged_vnd_block_tablescudnn_k_block_tablescudnn_v_block_tablescudnn_actual_seq_lens_qcudnn_actual_seq_lens_kv padding_maskOStatsamax_samax_oragged_o ragged_statstensors_to_returns: r_build_prefill_graphrs 0&ej&?&?&I&IJJ#)!,$ $ 1133 9788 8!OEEagNN!OEEgmTT!OEEgmTT  'K!OEEkRR !9 9 9 EO$<<<#%%--w^c^s^u^uww [ L (FQuuww!||WQZdAWQZd !I!I!IJJJhhdJ5t T4$;:+ G"U_%===$(@@@ !"$'#o3 !)!! !""$'#o3 !)!! !""$'#o3 !)!! !""$'#o3 !)!! #$(($$'#o3 #+##!""$'#o3 !)!! %%d&6&<===%%d&6&<===%%d&6&<===%%d&6&<===''(:(@AAA%%d&6&<===*==99  !2!8999))(333{{}}!!#++A,++%]1-w}Q/?d!##M!$M!$ !!R7=!R!RSSS{{}}!! !" $ D9 4K*4dD4KK/ !)!! #. }}_==H$$T%6%<===!33H=== !" $ D9 4K*4dD4KK/ !)!! #. }}_==H$$T%6%<===!33H===!## !" ">>++/ !)!! !"" ">>++/ !)!!  OODJ, - - -  ! !$*"2 3 3 3  ! !$*"2 3 3 3'"."6"6 &q)1l.@.CQ##()}}_'E'E$$,,T-D-JKKK'(}}_'E'E$$,,T-D-JKKK ,*+--8I*J*J''001DEEE'//0J0PQQQ!-+,==9K+L+L((112FGGG(001L1RSSS"-P2DD2P  "U_%===$(<<<66##-80/!.910!%1$#-1I0<0H,,d1=0H,,d'3&> D&+o&;9"55@"U_%===$(@@@+,::##+++)-)#'$1I%1-80/!.910!1=0H,,d1=0H,,d'3&> D?,6!,!,(5&&Ft455@@GGOO *\**==9N+O+O+Ot455@@GGOO *\**==9N+O+O+O*==99  !2!8999##H---". }}-@AA $$T%:%@AAA'' 555 IIdj& ' ' 2 24 8 8 @ @$ D1  jd"T)4a@m-... > dn233>>- 566wwdJ288*j4/D!<===!(- J  0!((/// ,!(()@AAA!-!(()ABBB''YL (L (L (L (L (L (L (L (L (L (L (L (L (L (L (L (L (L (sr?rFrBrr rGrJrKrLrHrIrrrrexecute)r_r`rarbrrrQrRrcrSrrVrurvrwrWrXrYrZr[r\r]r^rtensorsvar_mapdummy_scale_tensorrs r_batch_prefill_with_kv_cachers4* !e  /.  ( ,+.-"\"(:(((( 0/!" C#$ C%& K'NE7. ! ' ' # G$4E*01%5G+12"+:!'("+:!'("+:!'("+:!'(1='-.1='-.G(+$%  *3FGD)/ 04QX>>*1 &'*< &',>"()*< &'*1 &'*1 &' !%*";";AH"E"E F FF MM'%5fMEEECxDyr)rSrurvrwrWrXrYrZr[r\r]is_cuda_graph_compatiblebackendr^rrcn|jd}|jd}|dkr|jd|jd}}n2|dkr|jd|jd}}|dkr|jd}n%|dkr |jd}| r*|(tj||||jtj}||j|||fkrt d||j}|!|||f}tj||j|}trU|d krOtd'id |d |d |d |d|d|d|d|d|d| d| d| d| d| d|d|d|d|d|d|d|d|d |S| s Jd!|d"kr| |d#kr| Jd$||}| |jd%&}| |jd%&} tj }!|!|||||||||||| | | | ||dddd|||fS)(a3 Performs batched prefill attention with paged KV cache using cuDNN. Args: q: Query tensor of shape (Total number of tokens, num_heads_qo, head_dim) k_cache: Key cache tensor of shape (total_num_pages, num_heads_kv, page_size, head_dim) if paged kv cache is enabled else (Total sequence length of kv, num_heads_kv, d_qk) v_cache: Value cache tensor of shape (total_num_pages, num_heads_kv, page_size, head_dim) if paged kv cache is enabled else (Total sequence length of kv, num_heads_kv, d_vo) scale: Scaling factor for attention scores, typically 1/sqrt(head_dim) workspace_buffer: Workspace buffer for cuDNN operations. Scales with batch size. 128 MB should be sufficient for most cases max_token_per_sequence: Maximum number of tokens per query sequence (s_qo_max) max_sequence_kv: Maximum number of tokens per key/value sequence (s_kv_max) actual_seq_lens_q: Actual number of tokens per query sequence shape (batch_size,) on cpu or device (cpu if cuda_graph is False) actual_seq_lens_kv: Actual sequence lengths for key/values per batch, shape (batch_size,) on CPU or device (cpu if cuda_graph is False) block_tables: Page table mapping for KV cache, shape (batch_size, num_pages_per_seq) on GPU causal: Whether to apply causal masking return_lse: Whether to return log-sum-exp values (must be True) out: Optional pre-allocated output tensor lse: Optional pre-allocated tensor for log-sum-exp values if return_lse is True else returns None is_cuda_graph_compatible: Whether the prefill operation is compatible with CUDA graph q_scale: Optional scale tensor for query tensor of shape (1, 1, 1, 1) on GPU k_scale: Optional scale tensor for key tensor of shape (1, 1, 1, 1) on GPU v_scale: Optional scale tensor for value tensor of shape (1, 1, 1, 1) on GPU batch_offsets_q: Optional batch offsets for query tensor of shape (batch_size,) on GPU batch_offsets_o: Optional batch offsets for output tensor of shape (batch_size,) on GPU batch_offsets_k: Optional batch offsets for key tensor of shape (batch_size,) on GPU batch_offsets_v: Optional batch offsets for value tensor of shape (batch_size,) on GPU o_data_type: Optional data type for output tensor Returns: Output tensor of shape (batch_size * seq_len_q, num_heads_qo, head_dim) If return_lse is True, also returns log-sum-exp tensor of shape (batch_size, seq_len_q, num_heads_qo) Note: Query and KV heads can have different sizes (num_heads_qo >= num_heads_kv) When using cuda graph, actual_seq_lens_q and actual_seq_lens_kv must be on the same device as q Head dimension of query and key must be 128 or 192 Head dimension of value and output must be 128 rr!rrreNr zAlse must have shape (num_sequences, max_token_per_sequence, h_qo)cubinr_r`rarbrrrQrRrcrSrrVrurvrwrWrXrYrZr[r\r]r^z)Currently only supports return_lse = TrueziCurrently only supports if d_qk = 192 and block_tables is None or d_qk = 128 and block_tables is not NoneT) non_blockingrO) rfrgremptyr rrr CUDNN_AVAILABLErtorprefill)"r_r`rarbrrrQrRrcrSrrVrurvrwrWrXrYrZr[r\r]rrr^ num_tokens num_sequencesrirjrl out_shapeactual_seq_lens_q_gpuactual_seq_lens_kv_gpurun_funcs" r!cudnn_batch_prefill_with_kv_cacher*sDJ%+A.Muuww!||WQZd AWQZd{{}}}Q !  }Q ;+&xm C 398NPT(UUU O   g  {t, k)AHKHHHB 7g--+   a G G %  .-  $:#9  ,O 0/  21 & 6 "z G G G ,O! ",O# $,O% &,O' (!4 3) *+ ,- .$ /  4FFFFFz 4 CKKL44 w54  "4O 1 4 4QXD 4 Q Q!3!6!6qxd!6!S!S,..6  "        ! "          $-   2 8Or)%enumrtypingrr api_loggingrutilsrrr Exceptionrr dictr Tensor__annotations__rrStreamrr floatintboolr rnjit heur_modeA graph_cachertuplerstrrrOrrrs  ((((((,,,,,,LLLOO EOOO  9;d5<56;;; EL    !24P&*%)04+/#/3!&.2.2.2.226"&"&)-+666 |6 \6\6  6 c] 6c]6  -6 65<(6}6'tn66el+6el+6 el+!6"el+#6$"%,/%6& %, '6( %, )6*%+&+6666r@(UY5?,-...U2333*.)-4859/337%*262626266:&*&*-1)|(|(|( <|(|(|( |( "# |("#|($EL1|(%U\2|(u|,|(#+4.|(TN|("%,/|("%,/|("%,/|( "%,/!|("&el3#|($el #%|(&el #'|((ek*)|(|(|(43/.|(T ,0'+&*&*.2.2.2.226"&"&)-1``` |` \`\`  ` l ` ``|` `5<(` ``el #`el #` el #!`"el+#`$el+%`&el+'`(el+)`*"%,/+`, %, -`. %, /`0%+&1`2 5< %&3````F,0'+&*&*.2.2.2.226"&"&%*!)-5iii |i \i\i  i l i ii|i i5<(i iiel #iel #i el #!i"el+#i$el+%i&el+'i(el+)i*"%,/+i, %, -i. %, /i0#1i2c]3i4%+&5i6 5<%,/ /07iiiiiis & 22