-`id+ddlZddlmZddlZddlmZddlmZddlm Z ddl m Z ddl m Z eeZGd d e ZdS) N)Iterable)auto_functionalized) init_logger)current_platform)is_func)VllmInductorPassc JeZdZdZejdejjddfdZ dejj e ejj zddfdZ ddejjdejj d e eejj ezfd eejj ezd fdzddf d Zdejj d e eejj ezfddfd Zdejj de eejj ffdZ ddejjdejj d eejj ezd fdzddfdZdS)FixFunctionalizationPassa" This pass defunctionalizes certain nodes to avoid redundant tensor copies. After this pass, DCE (dead-code elimination) should never be run, as de-functionalized nodes may appear as dead code. To add new nodes to defunctionalize, add to the if-elif chain in __call__. graphreturnNc tjrtddSg|_d}|jD]}t |ts|j}|j d}|tj j j jkr|d}|d}||}t |t jret |t jrJ|j d|j dkr-t |j dtj jjjrt)d|Dr|j dj d} |D]t} | jD]U} t | tj jjjr*| | || V|| u|||||nddd} |||| n|tj j jjkrddd} |||| nG|tj j jjkrd dd} |||| n |tj j jjkrd d dd } |||| n|tj j jjtj j j jfvrd d i} |||| nwtCtj j"d r@|tj j"j#jkr!dddddd} |||| n|tj j j$jkrd d i} |||| dn|tj j j%jkrd d i} |||| dntCtj j dr>|tj j j&jkrd dd} |||| dnA|tj j j'jkr d di} d} |||| | n|d z }|(|dtS|j}|jD]}|*|td|||j+dS)NzBXPU platform does not support fix functionalizationpass currently.rquerykeyc3|K|]7}|jD]-}t|tjjjjV.8dSN)usersrtorchopsaten slice_scatterdefault).0 getitem_nodeusers z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/compilation/fix_functionalization.py z4FixFunctionalizationPass.__call__..9sd($0$6! ein&B&JKK)rinputresidualresultscale)rrr&flashinfer_trtllm_fused_allreduce_norm allreduce_innorm_out quant_out scale_out)rrr$)r"r args)r"r r#silu_and_mul_nvfp4_quantresult_block_scale)r"r/r input_global_scaleqkv) r1 num_heads_q num_heads_k num_heads_vhead_dimepsq_weightk_weight cos_sin_cacheis_neox position_ids) mutated_argsr-before_cleanupz,De-functionalized %s nodes, removed %s nodes),ris_xpuloggerdebugnodes_to_removenodesrrkwargsr-rr_Crotary_embeddingr getitem_usersoperatorgetitemrsplit_with_sizesallvaluesrrreplace_all_uses_with_removeinsert_defunctionalizeddefunctionalizefused_add_rms_norm#fused_add_rms_norm_static_fp8_quant rms_norm_dynamic_per_token_quantrms_normrms_norm_static_fp8_quanthasattrvllmr% silu_and_mulsilu_and_mul_quantr.fused_qk_norm_rope dump_graphlen erase_nodeclear)selfr countnoderC at_targetrr getitem_nodesmm_noderuser_of_getitemr<r- count_removeds r__call__z!FixFunctionalizationPass.__call__s  " $ $  LLT    F46K@ @ D4!455 [F ! IEIL9AAAwUm $ 2 24 8 8 E8#344$DX%566$D 1 !44 1 uy~/N/VWW5,9,@,@,B,B5$jm03G - 4 4 6 6++/3z>>O& /1M1U  >!0 E Eg N N N $ _ = = = T****00===LL&&&&(/5#9#9L((lCCCCeil=EEE#*z:: $$UD,????eilNVVV#+ ;; $$UD,????eilKSSS#+JGG $$UD,???? %- 6>!"8} $$UD,???? (PQQ< 9>HPQQ&!!""    $$UD,????eil7??? !8} $$44G%eil=EEE !8} $$44P% &@AA" !F!NNN#+0DEE $$  %    eil=EEE !5z  $$UD|RV$WWWW QJEE /000D011 ( # #D   T " " " " :E=    ""$$$$$r node_or_nodesct|tjjr|j|dS|j|dS)zM Stage a node (or nodes) for removal at the end of the pass. N) isinstancerfxNoderAappendextend)r^rgs rrMz FixFunctionalizationPass._removesT mUX] 3 3 7  ' ' 6 6 6 6 6  ' ' 6 6 6 6 6rr`r<r-.c|||||||||dS)z De-functionalize a node by replacing it with a call to the original. It also replaces the getitem users with the mutated arguments. See replace_users_with_mutated_args and insert_defunctionalized. r,N)replace_users_with_mutated_argsrNrM)r^r r`r<r-s rrOz(FixFunctionalizationPass.defunctionalizesM ,,T<@@@ $$UDt$<<< Trc ||D][\}}||}t|tr |j|n|}||||\dS)a7 Replace all getitem users of the auto-functionalized node with the mutated arguments. :param node: The auto-functionalized node :param mutated_args: The mutated arguments, indexed by getitem index. If the value of an arg is a string, `node.kwargs[arg]` is used. N)rFitemsristrrCrLrM)r^r`r<idxrargs rroz8FixFunctionalizationPass.replace_users_with_mutated_argss++D117799  ICs#C&0c&:&:C$+c""C  & &s + + + LL       rcvi}|jD].}t|tjr|jd}|||</|S)z Returns the operator.getitem users of the auto-functionalized node, indexed by the index they are getting. r)rrrGrHr-)r^r`rrrss rrFz&FixFunctionalizationPass.getitem_userssI J " "DtX-.. "il!c  rcfttsJdd|5jd}|||jn2t fd|D}|||ddddS#1swxYwYdS)a> Insert a new defunctionalized node into the graph before node. If one of the kwargs is 'out', provide args directly, as node.kwargs cannot be used. See https://github.com/pytorch/pytorch/blob/a00faf440888ffb724bad413f329a49e2b6388e7/torch/_inductor/lowering.py#L351 :param graph: Graph to insert the defunctionalized node into :param node: The auto-functionalized node to defunctionalize :param args: If we cannot use kwargs, specify args directly. If an arg is a string, `node.kwargs[arg]` is used. z%node must be auto-functionalized, is z insteadrN)rCc3`K|](}t|tr j|n|V)dSr)rirrrC)rrtr`s rrzCFixFunctionalizationPass.insert_defunctionalized..sOJM 3(<(<EDK$$#rr,)rrinserting_beforer- call_functionrCtuple)r^r r`r-functions ` rrNz0FixFunctionalizationPass.insert_defunctionalizeds2"t011  BD B B B  1  # #D ) ) 9 9y|H|##HT[#AAAAQU##H4#888 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9sAB&&B*-B*r)__name__ __module__ __qualname____doc__r time_and_logrrjGraphrfrkrrMdictintrrrzrOrorFrNrrr r s"W%ehnW%W%W%W%#"W%r7UX]Xehm5L%L7QU77778< x~hm3  334  EHMC',-4    HM15c58=3;N6N1O  %(- Dehm9K4L     8< 99x~9hm9EHMC',-4 9  999999rr )rGcollections.abcrr*torch._higher_order_ops.auto_functionalizer vllm.loggerrvllm.platformsrfx_utilsrvllm_inductor_passr r|r?r rrrrs$$$$$$ JJJJJJ######++++++000000 X  w9w9w9w9w9/w9w9w9w9w9r