-`i^9ddlZddlmZmZddlmZddlZddlmcm Z ddl m Z ddlm Z ddlmZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZd d lmZd dlm Z m!Z!m"Z"d dl#m$Z$d dl%m&Z&m'Z'ee(Z)dedeej*fdedej*ffdZ+GddZ,Gdde,Z-Gdde,Z.ej/Z0Gdde,Z1Gdde,Z2Gdd e'Z3dS)!N)CallableSequence)Any)PatternMatcherPass) VllmConfig)Range) get_tp_group tensor_model_parallel_all_reduce)$get_tensor_model_parallel_world_size) init_logger)kFp8StaticTensorSym)current_platform)enable_fake_mode)MatcherFusedAddRMSNormMatcherQuantFP8MatcherRMSNorm)NoOpEliminationPass)VllmInductorPassVllmPatternMatcherPassfn.returncntjdtdtjffd }|S)Nargsrc|dS)Nr)rrs y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/compilation/sequence_parallelism.pywrapperz&get_first_out_wrapper..wrapper"sr4y|) functoolswrapsrtorchTensor)rrs` rget_first_out_wrapperr$sO_Rsu| NrceZdZdZdedejdedzddfdZdej dej fd Z dej dej fd Z dej dej fd Z dS) _SequenceParallelPatternHelperz)Helper for sequence parallelism patterns.epsilondtypedeviceNrc|||_||_||_t|_t |_dSN)r'r(r)r tp_groupr tp_size)selfr'r(r)s r__init__z'_SequenceParallelPatternHelper.__init__,s5    $ ;== rxc t|Sr+)r r.r0s r _all_reducez*_SequenceParallelPatternHelper._all_reduce8s/222rcztjjj|d|j|jjSNr)dim world_size group_name)r"opsvllmreduce_scatterdefaultr-r, unique_namer2s r_reduce_scatterz._SequenceParallelPatternHelper._reduce_scatter;s6y~,44 1$-:S5   rcztjjj|d|j|jjSr5)r"r9r: all_gatherr<r-r,r=r2s r _all_gatherz*_SequenceParallelPatternHelper._all_gather@s6y~(00 1$-:S1   r) __name__ __module__ __qualname____doc__floatr"r(strr/r#r3r>rArrrr&r&)s33 > >{ >d >  > > > >3U\3el3333  %,     U\ el      rr&creZdZdedejdedzddffd Zdeej fdZ de ddfd Z xZ S) FirstAllReduceRMSNormPatternr'r(r)Nrcvt|||t||_dSr+)superr/rrmsnorm_matcherr.r'r(r) __class__s rr/z%FirstAllReduceRMSNormPattern.__init__Gs5 %000-g66rctjgd|j|j}tjdg|j|j}||gS)Nrr)r(rRr"emptyr)r()r.inputarg3_1s r get_inputsz'FirstAllReduceRMSNormPattern.get_inputsKsF IIIdkLLLaSDJGGGvrpm_passctdtjdtjdttjtjfffd }dtjdtjdttjtjfffd }tj||tj|dS)NrVrWrcb|}||}||fSr+r3rL)rVrW all_reducermsnormr.s rpatternz6FirstAllReduceRMSNormPattern.register..patternRs8))%00J**:v>>GJ& &rc|}||}|}||fSr+)r>rLrA)rVrWr;r^r@r.s r replacementz:FirstAllReduceRMSNormPattern.register..replacement[sK"11%88N**>6BBG))'22J~- -rr"r#tuplepmregister_replacementrXfwd_onlyr.rYr_ras` rregisterz%FirstAllReduceRMSNormPattern.registerQs '< 'L '5<- . ' ' ' ' ' ' .< .L .5<- . . . . . . .  [$//"3"3R['     rrBrCrDrFr"r(rGr/listr#rXrrh __classcell__rNs@rrIrIFs77ek73:7RV777777D.  2 t        rrIcreZdZdedejdedzddffd Zdeej fdZ de ddfd Z xZ S) MiddleAllReduceRMSNormPatternr'r(r)Nrcvt|||t||_dSr+)rKr/rrLrMs rr/z&MiddleAllReduceRMSNormPattern.__init__ks5 %0005g>>rctjddg|j|j}tjddg|j|j}tjddg|j|j}|||gS)NrRrSrT)r.mm_1residualrms_norm_weightss rrXz(MiddleAllReduceRMSNormPattern.get_inputsosr{Aq6$+TZHHH;1vdkLLL ;1vdkTTT     rrYc @dtjdtjdtjdttjtjfffd }dtjdtjdtjdttjtjfffd }tj||tj|tjt|t|tj|dS)Nrrrqrsrc||}|||}|d|dfS)Nrrr\)rrrqrsr]r^r.s rr_z7MiddleAllReduceRMSNormPattern.register..pattern|sC ))$//J**:7GRRG1:wqz) )rc|}|d|ddf}|||}|d}||dfS)Nr.r)r>sizerLrA)rrrqrsr;r^r@r.s rraz;MiddleAllReduceRMSNormPattern.register..replacementsz"11$77NN$7$7$:$: :C ?@H**>;KXVVG))'!*55Jwqz) )rr"r#rcrdrerXrfr$rgs` rrhz&MiddleAllReduceRMSNormPattern.register{s) *l *, *$l *5<- .  * * * * * * *l *, *$l *5<- .  * * * * * *  [$//"3"3R['     !' * * !+ . . OO   K       rrirls@rrnrnjs??ek?3:?RV??????  D.     # 2# t# # # # # # # # rrncreZdZdedejdedzddffd Zdeej fdZ de ddfd Z xZ S) %FirstAllReduceRMSNormStaticFP8Patternr'r(r)Nrct|||t||_t t |_dSr+)rKr/rrLrr quant_matcherrMs rr/z.FirstAllReduceRMSNormStaticFP8Pattern.__init__sH %000-g66,-@AArctjgd|j|j}tjdg|j|j}tjd|jtj}|||gS)NrPrSrRg?)r"zerosr)r(rUtensorfloat32)r.rVweightscales rrXz0FirstAllReduceRMSNormStaticFP8Pattern.get_inputssc IIIdkLLLaSDJGGG SEMJJJvu%%rrYc dtjdtjdtjdttjtjfffd }dtjdtjdtjdttjtjfffd }tj||tj|dS)NrVrrrc|}||}||\}}||fSr+r3rLr|)rVrrr]rmsquant_r.s rr_z?FirstAllReduceRMSNormStaticFP8Pattern.register..patternsQ ))%00J&&z6::C))#u55HE1*$ $rc|}||}||\}}|}||fSr+)r>rLr|rA) rVrrr;rrrr@r.s rrazCFirstAllReduceRMSNormStaticFP8Pattern.register..replacementsd "11%88N&&~v>>C))#u55HE1))%00J~- -rrbrgs` rrhz.FirstAllReduceRMSNormStaticFP8Pattern.registers %< %L %< %5<- .  % % % % % % .< .L .< .5<- . . . . . . .  [$//"3"3R['     rrirls@rrzrzsBB{Bd B  BBBBBB&D.&&&&  2 t        rrzcreZdZdedejdedzddffd Zdeej fdZ de ddfd Z xZ S) &MiddleAllReduceRMSNormStaticFP8Patternr'r(r)Nrct|||t||_t t |_dSr+)rKr/rrLrr r|rMs rr/z/MiddleAllReduceRMSNormStaticFP8Pattern.__init__sF %0005g>>,-@AArc0tjddg|j|j}tjddg|j|j}tjddg|j|j}tjddg|jtj}||||gS)NrRrSr)r"rUr)r(r)r.rqrrrsrs rrXz1MiddleAllReduceRMSNormStaticFP8Pattern.get_inputss{Aq6$+TZHHH;1vdkLLL ;1vdkTTT QF4;emLLL$ 0%88rrYc pdtjdtjdtjdtjdttjtjff fd }dtjdtjdtjdtjdttjtjff fd }tj||tj|tjt|t|tj|dS)Nrrrqrsrrc |} |||\}} ||\}}||fSr+r) rrrqrsrr]r residual_outrrr.s rr_z@MiddleAllReduceRMSNormStaticFP8Pattern.register..patterns` ))$//J $ 4 4,h!! C))#u55HE1,& &rc |}|d|ddf} |||\}} ||\}} |} | |fS)Nr.)r>rwrLr|rA) rrrqrsrr;rrrrr@r.s rrazDMiddleAllReduceRMSNormStaticFP8Pattern.register..replacements"11$77NN$7$7$:$: :C ?@H $ 4 4 0(!! C))#u55HE1))%00J|+ +rrxrgs` rrhz/MiddleAllReduceRMSNormStaticFP8Pattern.registersA 'l ', '$l '< ' 5<- . ' ' ' ' ' ' ,l ,, ,$l ,<  , 5<- .  , , , , , ,*  [$//"3"3R['     !' * * !+ . . OO   K       rrirls@rrrsBBekB3:BRVBBBBBB 9D.9999- 2- t- - - - - - - - rrceZdZdZededdffd ZdedefdZ e j de j ddfd ZxZS) SequenceParallelismPassa This pass enables sequence parallelism for models. It identifies patterns where an AllReduce operation is followed by an RMSNorm (or RMSNorm and then Quantization) operation. These patterns are replaced with a ReduceScatter operation, followed by a local RMSNorm/Quantization, and then an AllGather operation. The general transformation is: Input -> AllReduce -> RMSNorm -> Output becomes Input -> ReduceScatter -> RMSNorm -> AllGather -> Output While this pass itself does not directly yield performance improvements, it lays the groundwork for subsequent fusion passes, such as GEMM + ReduceScatter and AllGather + GEMM fusions. These fusions can significantly reduce communication overhead and improve overall model performance. This pass splits up the residual tensor across TP ranks and hence divides its size. Because the pattern matcher starts at the end of the graph, the replacement contains a slice that temporarily conforms the input residual to the correct size. After all patterns have been matched, we use a NoOpEliminationPass to clean up what have now become no-op slices. Note that an older version of the pass did not need this as it operated only on custom rms_norm and fused_rms_norm_add custom ops which did not complain about mismatched shapes during replacement. So this approach has the same assumption that correctness is only maintained if all rms_norm operations are split across ranks. Correctness-wise, this is approach strictly better than before - before, the graph was incorrect semantically and shape-wise during the pass. With this approach there's only semantic incorrectness during the pass. Both approaches restore a correct graph once all patterns are matched. configrNct|t||_|jd|jj|j_t d|_dD]}t||j|j  |jt||j|j  |jt||j|j  |jt||j|j  |j|||jdS)N.sequence_parallelism_pass) pass_name)gh㈵>gư>)rKr/r noop_cleanuprrpatternsrz model_dtyper)rhrrIrn dump_patterns)r.rr'rNs rr/z SequenceParallelismPass.__init__4sG    077)-&W&W$:K:U&W&W#,>1- - -  $ & &G 1)4;  ht}%%% 2)4;  ht}%%% ))4;  ht}%%% ))4;  ht}%%%% 64=11111r compile_rangec|jjr |jjrdSt}|o |j|zdk}|S)NTr)compilation_config splitting_opsuse_inductor_graph_partitionr is_single_sizeend)r.rr-results ris_applicable_for_rangez/SequenceParallelismPass.is_applicable_for_rangeUsb'5 &C 4688%4466   '1 ,  rgraphc|j||_td|j||dS)NzReplaced %s patterns)rapply matched_countloggerdebugr)r.rs r__call__z SequenceParallelismPass.__call__nsL!]0077 +T-?@@@ %     r)rBrCrDrErrr/rboolrr time_and_logfxGraphrrkrls@rrrs""H2z2d222222@Ut2"!bh!4!!!#"!!!!!rr)4r collections.abcrrtypingrr"torch._inductor.pattern_matcher _inductorpattern_matcherrdtorch.fxrr vllm.configrvllm.config.utilsrvllm.distributedr r vllm.distributed.parallel_stater vllm.loggerr 9vllm.model_executor.layers.quantization.utils.quant_utilsr vllm.platformsr inductor_passr matcher_utilsrrrnoop_eliminationrvllm_inductor_passrrrBrr#r$r&rIrn fp8_dtype FP8_DTYPErzrrrrrrs........ ,,,,,,,,,>>>>>>""""""######KKKKKKKKPPPPPP######,+++++++++++RRRRRRRRRR111111HHHHHHHH X  hu|,,- c5<         :! ! ! ! ! #A! ! ! H4 4 4 4 4 $B4 4 4 n '  & ( ( * * * * * ,J* * * Z; ; ; ; ; -K; ; ; |d!d!d!d!d!4d!d!d!d!d!r