-`i?BdZddlZddlZddlmZGddeZdS)aA Expert parallelism load balancer (EPLB) for vLLM. This module implements the core rearrangement algorithm. The rearrangement algorithm is adapted from [DeepSeek EPLB](https://github.com/deepseek-ai/eplb). Please find at [#12](https://github.com/deepseek-ai/EPLB/issues/12) an example on how the EPLB algorithm works. N)AbstractEplbPolicyceZdZedejdedeejejffdZedejdedeejejejffdZ edejdeded ed edeejejejff d Z ed ejd ejdedejdeejejff dZ e dde j deded edede j dzdee j e j e j ffdZdS)DefaultEplbPolicyweight num_packsreturnc|j\}}||zdksJ||z}|dkrYtjtj|tj|df}tj|tj}||fStj| d}tj||fdtj}tj||fdtj}tj||ftj } tj||ftj} t|D]} | | } | | } || D]z}ttj | }||| |f<| ||| |f<| |xx|| |fz cc<| |xxdz cc<| ||krtj | |<{||fS)a Pack n weighted objects to m packs, such that each bin contains exactly n/m objects and the weights of all packs are as balanced as possible. Parameters: weight: [X, n], the weight of each item num_packs: number of packs Returns: pack_index: [X, n], the pack index of each item rank_in_pack: [X, n], the rank of the item in the pack rrdtypeaxis)shapenptilearangeint64 zeros_likeargsortfullzerosfloat64rangeintargmininf)clsrr num_layers num_groupsgroups_per_pack pack_index rank_in_packindices pack_weights pack_items layer_idx weights_row items_rowgrouppacks x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/distributed/eplb/policy/default.pybalanced_packingz"DefaultEplbPolicy.balanced_packings "( JI%****$ 1 a  :RX!F!F!FUVXXJ=28DDDL|+ +*fW2...Wj*5rJJJ w J728LLL xY 7rzJJJ Xz95RXFFF z** / /I&y1K"9-I + / /29[1122/3 9e+,1:4 Y-.D!!!VIu,<%==!!!$1$T?o55(*K% /<''num_phyc|j\}}||z }|dksJtjtj|tj|df}tj||ftj}tj||ftj}tj|tj} t||D]G} tj||z d} | |dd| f<|| | f|dd| f<|| | fxxdz cc<H|||fS)a Replicate `num_log` experts to `num_phy` replicas, such that the maximum load of all replicas is minimized. Parameters: weight: [X, num_log] num_phy: total number of experts after replication Returns: phy2log: [X, num_phy], logical expert id of each physical expert replica_idx: [X, num_phy], the index of the replica for each logical expert logcnt: [X, num_log], number of replicas for each logical expert rr rr rN) rrrrrronesrargmax) rrr/nnum_log num_redundantphy2log replica_idxlogcntarangeniredundant_indicess r,replicate_expertsz#DefaultEplbPolicy.replicate_expertsKs,"\ 7') !!!!'")G28<<.inversesb-%%Ci 1 ..qqq$w7Gi 1 RX>>>G!(C Jr.r r).Nr rN) rrndarrayreshapesumr-rrtake_along_axisr<)rrr=r r>r?rnum_logical_experts group_sizegroups_per_nodephy_experts_per_gpurGtokens_per_groupgroup_pack_indexgroup_rank_in_packlog2mlogmlog2logtokens_per_mlogphy2mlog replicas_idxmlogcnttokens_per_phyr"r#phy2pphypphy2phy pphy2mlogpphy2logpphy_replicas_idxr8s r,rebalance_experts_hierarchicalz0DefaultEplbPolicy.rebalance_experts_hierarchicaljs#4+1,' '"Z/14444(J6 I%****$ 1)#q((((#h.!33332h> "*     ">>*j*MMQQR  03/C/C i0 0 ,, "O36HH)Ti "(333  4 '*1 2 2 78$$,VXAFFFNN #y0  +.*?*? 1Y>+ + ', +Og,ExVWXXX##7 H 1$ $  L 33lB78$$&xBBB   j)R 8 8i##y0h  AAAtm   '*b ! ! %h BBB.|XANNNVV   #GOOJ$C$CXTUVVV*F22r.r6phy_replicas_idx num_ranks old_phy2logc |jd}|dks ||zdkr||fS||z}|jd}|}|} t|D]} | |z} | |z} |dd| | f} |dd| | f}|dd| | f}tj||ft }tj||ft }t|D]}|| dd|fdddfk|z}|d}tj|rjtj|d}tj|d}||}|||f||| |zf<|||f| || |zf<d|||f<d|||f<|}|}|rF|r1tj tj ||df}|dz}tj |||}tj |||}tj |d}tj |d}| d} | d}!tj| |!}"t|D]Z}#t|"|#}$|$dkr||#d|$f}%||#d|$f}&||#|%f||#| |&zf<||#|%f| |#| |&zf<[|| fS)ar Reorder the new mapping per GPU so that experts that remain on the same GPU keep their previous slot positions when possible. Incoming experts to that GPU fill any remaining available slots. This is applied only when the number of GPUs is unchanged and the slots per GPU remain the same between the old and new mappings. rrNr rT)rcopyrrrboolanyr2nonzerorrwhererrJminimumr)'rr6r`rarbnum_phy_experts slots_per_gpur post_phy2logpost_phy_replicas_idxgpu_idxstartend old_local new_localnew_ridxused_new_indicespreserved_positionsslot_idxmatcheshas_any first_idx layer_indicesmatched_new_positionsremaining_mask fill_maskidx_baselargeremaining_priority fill_priorityremaining_indices fill_indicesremaining_counts fill_counts take_countsr'ksrc_posdst_poss' r,preserve_intragpu_slotsz)DefaultEplbPolicy.preserve_intragpu_slotss"-* >>_y8A==,, ,(94 ]1% ||~~ 0 5 5 7 7Y''> > Gm+E-'C#AAAuSyL1I59 -I'59 5H!x](C4PPP "$(J +Fd"S"S"S "-00 H H% !!!X+(>qqq$w(GG%%"++1+--6'?? H " ' : : :I$&Jw$7$7$:M,5m,D)DM%'<<EL0@!@ANV%'<<N)-9I*IJNR$]4I%IJCG' x(?@/.N,,I!!##   729]#;#;j!_MM%)&(Xnh%N%N" "He D D $&J/A$J$J$J!!z-a@@@ #1#5#51#5#=#= 'mmm33  j)9;GG !&z!2!2  IK 233AAvv / 2A2 >G*9bqb=9G?H!7*@LEGO!;<IQ!7*I))UW_*DEE222r.N num_replicasold_global_expert_indicesc|j}|j\}} |} |&|nd} ||zdkr|| ||||\} } }n|| |dd|\} } }||| | || \} } || z }|dz}tj|| |fdtj }tj |dddf}tj tj |tj |df}|||| | f<tj | |}tj ||}tj ||}|||fS)a? Entry point for expert-parallelism load balancer. Parameters: weight: [layers, num_logical_experts], the load statistics for all logical experts num_replicas: number of physical experts, must be a multiple of `num_gpus` num_groups: number of expert groups num_nodes: number of server nodes, where the intra-node network (e.g, NVLink) is faster num_ranks: number of ranks, must be a multiple of `num_nodes` old_global_expert_indices: [layers, num_logical_experts], the old global expert indices. Used to avoid unnecessary weight copying for experts moving within one rank. Returns: phy2log: [layers, num_replicas], the expert index of each replica log2phy: [layers, num_logical_experts, X], the replica indices for each expert logcnt: [layers, num_logical_experts], number of physical replicas for each logical expert Nrrr r )devicerfloatcpunumpyr_rrrrrrtorch from_numpyto)rrrr r>rarrrrL weight_npold_phy2log_np phy2log_npphy_replicas_idx_np logcnt_npnum_redundant_experts maxlogcnt log2phy_nprzreplica_indicesr6log2phyr8s r,rebalance_expertsz#DefaultEplbPolicy.rebalance_experts(s B*0,' 'LLNN&&((..00 )4 & ) ) + + 1 1 3 3 3   !Q & &22|ZI 7J+YY22|Q9 7J+Y % 0.1.I.I/N// +J+!-/B B)A- W ,i 8"BH    *--aaag6 ' Il"( 3 3 3j!_  FU =*.AAB":..11&99":..11&99!),,//77''r.)N)__name__ __module__ __qualname__ classmethodrrHrtupler-r<r_rrTensorrr.r,rrs%2(Z2(,/2( rz2:% &2(2(2([2(h,Z,*-, rz2:rz1 2,,,[,<_3 _3"_3 _3  _3  _3 rz2:rz1 2_3_3_3[_3BY3Y3*Y3 Y3 Z Y3 rz2:% & Y3Y3Y3[Y3v:>O(O( O(O( O(  O(  O($)<$#6O( u|U\5<7 8O(O(O([O(O(O(r.r)__doc__rrrabstractrrrr.r,rsw   ((((((c(c(c(c(c(*c(c(c(c(c(r.