-`i10ddlmZmZddlmZddlmZmZddlZddl m cm Z ddlm Z ddlmZddl mZddlmZdd lmZmZdd lmZdd lmZmZmZdd lmZdd lmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-ee.Z/edZ0ej1Z2ej3Z4ej5j6j7j8Z9ej5j:j;j8Z<GddeZ=Gdde=Z>Gdde=Z?Gdde-Z@dS))ABCabstractmethod)Callable)Any ParamSpecN)fx)auto_functionalized)PatternMatcherPass) Attention) VllmConfigget_layers_from_vllm_config) init_logger)QuantKey kNvfp4DynamickStaticTensorScale)current_platform)round_up) QUANT_OPS empty_bf16 empty_fp32 empty_i32)is_func)enable_fake_mode)MatcherQuantFP8)VllmInductorPassVllmPatternMatcherPassPc |eZdZdZdededejddfdZde d e dej fd Z de d e dej fd Z e d eeejfd eejgdfdeeejffdZe dejjddfdZe dejjddfdZdeddfdZededdfdZdS)AttentionQuantPatternzQ The base class for Attn+Quant fusions. Should not be used directly. layer quant_keydtypereturnNc||_|j|_|j|_|j|_||_|j|_||_|jtvsJd|jt|j|_dS)Nz unsupported quantization scheme ) r! layer_name num_heads head_sizer"r# quant_dtyperQUANT_OP)selfr!r"r#s p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/compilation/fusion_attn.py__init__zAttentionQuantPattern.__init__.s{  *"$? ~*** ?t~ ? ?+**"$.1 argskwargsc<|jdd|}tj|i|SNcudar#device)r#torchemptyr+r/r0s r,r7zAttentionQuantPattern.emptyAs+:BB6B{D+F+++r.c<|jdd|}tj|i|Sr2)r)r6r7r8s r, empty_quantz!AttentionQuantPattern.empty_quantEs,+vHHH{D+F+++r.trace_fnprocess_fx_fnsc`dtjdtjdtjffd }|S)Nr/r0r$c8|i|}D] }|||SN)r/r0gm process_fxr<r;s r,wrappedz4AttentionQuantPattern.wrap_trace_fn..wrappedNs:4*6**B,    2Ir.)rr/r0r GraphModule)r;r<rCs`` r, wrap_trace_fnz#AttentionQuantPattern.wrap_trace_fnIsI  16 QX ".       r.rAc(ddlm}||dS)Nr)view_to_reshape)#torch._inductor.fx_passes.post_gradrG)rArGs r,fx_view_to_reshapez(AttentionQuantPattern.fx_view_to_reshapeWs*GGGGGGr.cT|jjD]}t|tjjjjs,|jd}tdt|Dr`| |jd|j |dS)Nrc3(K|] \}}||kVdSr?r@).0idims r, z=AttentionQuantPattern.remove_noop_permutes..ds*::33!8::::::r.r) graphnodesrr6opsatenpermutedefaultr/any enumeratereplace_all_uses_with erase_node)rAnodedimss r,remove_noop_permutesz*AttentionQuantPattern.remove_noop_permutes]sHN & &D4!7!?@@ 9Q"2D"89  !R^# $   \ ux3\  &!5 &$ & & &\ &$-?$D$$$$"!3""""^"""r.r c ReZdZdZ d dedejdeddffd Zd e ddfd Z xZ S) AttentionFp8StaticQuantPatterna? Fusion for Attention+Fp8StaticQuant. Only triggers when the attention implementation returns True in `fused_output_quant_supported()`. If the pattern is found, the Fp8StaticQuant op will be removed from the graph, and its scale will be passed into Attention op as the `output_scale` argument. Tr!r# symmetricr$Ncttt|}t|||t ||_dS)N)r#scalern)r FP8_DTYPErsuperr-r quant_matcher)r+r!r#rnr" __class__s r,r-z'AttentionFp8StaticQuantPattern.__init__~sU #5     5111,Y77r.r]c dtjdtjdtjdtjdtjdtjf fd }dtjdtjdtjdtjdtjdtjf fd }d jjd jjd jjd jjt d d g}t j|||t t j tj tj |dS) Nqkv output_attnrpr$c tt||||jdd}t|d|jdjjzg}||dS)Nquerykeyvalueoutputr& output_scaleoutput_block_scalerr)r ATTN_OPr& RESHAPE_OPshaper'r(rs)rvrwrxryrpat1 attn_out_viewr+s r,patternz9AttentionFp8StaticQuantPattern._register..patterns~&"?!#'   C'AT^dn%DEM%%mU;;A> >r.c .tjjj|jdjjgdj|j }tt||||j |d}t|ddjjzgS)Nrr4r{r)r6rRrSfullrUrr'r(r)r5r rr&r)rvrwrxryrprr+s r, replacementz=AttentionFp8StaticQuantPattern._register..replacements ).-55T^T^<&x 6K &"?"#'   Cc!fr4>DN+J&KLL Lr.r) r6rjr7r'r(rpmregister_replacementr rEfwd_onlyrIr\r+r]rrinputss` r,raz(AttentionFp8StaticQuantPattern._registers ?| ?| ?| ?  ? <  ? \  ? ? ? ? ? ?. M| M| M| M  M <  M \  M M M M M M6 JJq$.$. 9 9 JJq$.$. 9 9 JJq$.$. 9 9 JJq$.$. 9 9 q!         ! / / %8%:    r.)T) rfrgrhrir r6r#boolr-r ra __classcell__rts@r,rmrmts 8 8 8{ 8 8  8 8 8 8 8 8D !3D D D D D D D D D r.rmcJeZdZdZdedejddffd ZdeddfdZ xZ S) AttentionNvfp4QuantPatterna7 Fusion for Attention+Nvfp4Quant. Only triggers when the attention implementation returns True in `fused_output_quant_supported()`. If the pattern is found, the Nvfp4Quant op will be removed from the graph, and its scale will be passed into Attention op as the `output_scale` argument. r!r#r$NcXt|t|dSr?)rrr-r)r+r!r#rts r,r-z#AttentionNvfp4QuantPattern.__init__s%  u55555r.r]c2dtjdtjdtjdtjdtjdtjdtjdttjtjfffd }dtjdtjdtjdtjdtjdtjdtjdttjtjfffd }td jjtd jjtd jjtd jjd jjzd ztd tjjzdzdtddg}tj |||t tjtjtj|dS)Nrvrwrxry output_quantr input_scaler$c Xtt|||| jdd}t|d|jd j jzg}t j||||d} tj j j | dt} | d| fS)Nr{rrT)rinputrris_sf_swizzled_layout)r rr&rrr'r(r*r6rRrSviewr#rq) rvrwrxryrrrrrat2output_scale_viewr+s r,rz5AttentionNvfp4QuantPattern._register..patterns&"?!#'   C'AT^dn%DEM& ##)'&* C!&  3 9 9#a&) L L q6,, ,r.c tjjj|jd j jdzgd j|j }tjjj |t}tt|||| j||}t!|dd j jzdzg} | |dfS)Nrrrr4r{rr)r6rRrSrrUrr'r(r)r5rr#rqr rr&r) rvrwrxryrrrrrrr+s r,rz9AttentionNvfp4QuantPattern._register..replacements ).-55T^T^q-@A&x 6K!&  3 9 9, R R %"?(#4   C AT^dn-LPQ-Q(RSSF3q6> !r.rrr)r6rjtuplerr'r(r:rrrrrr rErrIr\rs` r,raz$AttentionNvfp4QuantPattern._registers -| -| -| -  -  ,  -  ,  - -5<- . - - - - - -B "| "| "| "  "  ,  "  ,  " "5<- . " " " " " "@ q$.$. 9 9 q$.$. 9 9 q$.$. 9 9 q$.$. 9 9   Q ?1 D E E Xdnt~=CQGG   q!         ! / / %8%:    r.) rfrgrhrir r6r#r-r rarrs@r,rrs6i6 6666666W !3W W W W W W W W W r.rceZdZdZededdffd Zejde j j j ddfdZ defdZxZS) AttnFusionPassa This pass fuses post-attention quantization onto attention if supported. It uses the pattern matcher and matches each layer manually, as strings cannot be wildcarded. This also lets us check support on attention layers upon registration instead of during pattern matching. Currently, only static fp8 quant is supported, but patterns could easily be added for other quant schemes and dtypes. The bigger hurdle for wider support are attention kernels, which need to support fusing output quant. configr$Nct|td|_t |t }|D]\}}t||jj }| |jtj rSttjjdr4t#||jj }| |jt%|dkrt&d|||jdS)Nattn_fusion_pass) pass_namescaled_fp4_quantrzAttention + quant fusion is enabled, but no attention layers were found in CompilationConfig.static_forward_context so no fusion patterns were registered.)rrr-r patternsr r itemsrm model_configr#rcris_cudahasattrr6rR_Crlenloggerwarning dump_patterns)r+r attn_layersr&r! pattern_fp8 pattern_nvfp4rts r,r-zAttnFusionPass.__init__Es;    *5GHHH 1&)DD !,!2!2!4!4 C C J8v*0K  - -dm < < <')) CgeilDV.W.W C :6.4!! 33DMBBB {  q NN9    64=11111r.rPc|j||_td|jdS)Nz#Fused quant onto %s attention nodes)rapply matched_countrdebug)r+rPs r,__call__zAttnFusionPass.__call__as8!]0077 :Drs$#######$$$$$$!!!!!!!! ,,,,,,,,,JJJJJJ>>>>>>******????????###### ,+++++******@@@@@@@@@@@@++++++******HHHHHHHH X   IcNN &  & ( ( K ). 6 > Y^ # + I"I"I"I"I"CI"I"I"XZ Z Z Z Z %:Z Z Z zd d d d d !6d d d N4 4 4 4 4 +4 4 4 4 4 r.