`i;(ddlZddlZddlZddlmZddlmZddlZddlZddl m Z ddl m Z m Z mZddlmZejeZGdd eZGd d eZejd efd Zde jd efdZde jd efdZde jd efdZGddeZGddeZGddeZdgdggZ dgdggdgdggdgdgggZ!gdgdgdgZ"d ee#fd Z$de jd e#fd!Z%dS)"N)IntEnum)Optional)ir)get_dtype_sizesnode_args_kwargs sympy_product)VceZdZdZdZdZdZdS) NCCL_COLLrrN)__name__ __module__ __qualname__ ALL_REDUCE ALL_GATHERREDUCE_SCATTER ALL_TO_ALLq/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torch/_inductor/comm_analysis.pyr r s"JJNJJJrr ceZdZdZdZdZdS)NVIDIA_GPU_TYPErrr N)rrrVOLTAAMPEREHOPPERrrrrrs E F FFFrrreturnctjjtjjjpd}d|vr t jSd|vr t jSd|vr t jSt jS)NV100A100H100) torchutils collect_env get_gpu_inforunrrrr)gpu_infos r get_gpu_typer* so{&33EK4K4OPPVTVH $$ 8  %% 8  %%%%rnodect|tjstd||j}|Jd|vr t jSd|vr t jSd|vr t jSd|vr t j Std|)Nz!node is not a collective kernel: all_reduce all_gatherreduce_scatterz-torch.ops._dtensor.shard_dim_alltoall.defaultzUnsupported collective kernel: ) isinstancer_CollectiveKernel ValueErrorpython_kernel_namer rrrr)r+ kernel_names rget_collective_typer5.s dB0 1 1ECTCCDDD)K  " " "{""##  $ $## [ ( ('' 8K G G##H;HHIIIrc.d}|jD]}t|jj}t |t jrt|}n&tj j |d}||t|jj zz }|S)Nr)fallback)inputsr layoutsizer0sympyIntegerintr graphsizevars size_hintrdtype)r+sz_bytesinpnumels rget_collective_input_size_bytesrE@sH{==cjo.. eU] + + BJJEEG$..uq.AAEEN3:+;<<<< Orct|tjr6t|tjsddlm}||jdStd|)Nr)_get_group_size_by_namezUnsupported collective type: )r0rr1 _WaitKernel"torch.distributed.distributed_c10drG constant_args TypeError)r+rGs rget_collective_group_sizerMMsn$,--@jr~6V6V@NNNNNN&&t'9"'=>>>>>>???rceZdZdZdZdZdS)NCCL_HWrrr N)rrrNVLINKPCINETrrrrOrO[s F C CCCrrOceZdZdZdZdS) NCCL_ALGOrrN)rrrTREERINGrrrrTrTas D DDDrrTceZdZdZdS) NCCL_PROTOrN)rrrLLrrrrXrXfs BBBrrXg333333@gffffff@g333333??g@g@)C@r[gffffff4@)gU@g6@g3@c|j}|Jt|dd}d|vsd|vsdSddlm}|jd}||}t j|}t jd|}t|}t|\} } d |vr| d d| dz} t j || 5} || i| } t j j j| dddn #1swxYwYn2#t $r%} t"| Yd} ~ dSd} ~ wwxYw| j}|dkrdS|d z }|S) Nr3r r.r/r)_resolve_process_grouprHzcuda:all_gather_into_tensor_outr)groupdevice@@)r+getattrrJr]rKr$ distributedget_rankr`evalr_time_estimatorops_c10d_functional wait_tensordefault Exceptionloginfoestimated_time)snodekernelpy_kernel_namer]pg_namepgrankr`fnargskwargstime_estimatorwe est_time_us est_time_mss r/estimate_nccl_collective_runtime_nccl_estimatorr}s ZF    V%92>>N N * *.>..P.PtIIIIII"2&G   ( (B!**2..D\.$.. ) )F n  B$U++LD&$~55ABBx$q'!    . .V/   > D#F##A I & 2 : :1 = = =  > > > > > > > > > > > > > > >   ttttt !/KQt#K s<4!D2D DDDDD E)E  Ect|}|dz dz dz }d}t|}tj||z }|}|dkrdStj}t j}t|} tj j j } tj j j } t} |dkr|dz nd} |dkr| nd}t|| }|dkr| n| }d}||z}t!|||dks| t"jkrdndz}| t"jkr d|dz z}n8| t"jkr d|dz z}n| t"jt"jfvr|dz }d|z|z }||z}|d z }t,j}| t"jkr|dkrd|z}n-d}n*| t"jt"jt"jfvr|dz }t0||}t2|||}t2t,j||}d }|dkrd}t7||}|||z |z||zzz }|d z}||z }||z}|d z } | S) a: Returns estimated NCCL collective runtime in milliseconds (ms). The following heuristics are copied from https://github.com/NVIDIA/nccl/blob/master/src/graph/tuning.cc. We aim to estimate the runtime as accurately as possible. Assumptions: - only ring algorithm (NCCL_ALGO_RING) is used - only Low-Latency protocol (NCCL_PROTO_LL) is used, i.e. Simple or LL128 is not used - 8 gpus per node # TODO: Need to find a way to get accurate "gpus per node" and "# nodes" info. - collective is one of: allreduce, reducescatter, allgather irrr g?gUUUUUU?rZgeAgrag.A)rErMmathceilrTrVrXrYr5r$ _inductorconfig intra_node_bw inter_node_bwr*llMaxBwsminr rrrrrOrPbaseLathwLatrRmax)!r+tensor_storage_size_bytestensor_storage_size_GBnum_gpus_per_node group_sizenNodesnRanks nccl_algo nccl_protocollbwIntrabwInter compCapIndexindex2index1llMaxBwbw nChannelsbusBwnstepsratio bandwidthbandwidth_GB_per_nsintraHw nInterStepslatencyintraLatinterLat netOverhead latency_ns transport_nsnsmss! r estimate_nccl_collective_runtimers!@ E E6=DtK*400J Yz$55 6 6F F {{qIJ t $ $D o$2Go$2G>>L!Q;;VaZZAF#q[[\\aFvv&GaKKWBI NE  !ty/C'C'C99) U  E  y###fqj! % % %fqj! )*I,@A A A!6\V #E I#c/nG y### A::f*KKKK )*I,@)BVW W Wqj i ,GW~i(4HW[!),Z8HK zz 8[))H $0;3IIIG3J*,??L  "B cB Ir)& functoolsloggingrenumrtypingrr;r$r rr%rrr virtualizedr getLoggerrrlr r lru_cacher*IRNoder5r=rErMrOrTrXrrrfloatr}rrrrrs  CCCCCCCCCCg!!g   &o & & & &JbiJIJJJJ$ ")     @BI@#@@@@     g                             ,   ,)huo))))Xf29fffffffr