Pi\7UddlZddlZddlZddlmZddlmZddlmZm Z ddl Z ddl Z ddl m Z ddlmZddlmZddlmZmZed Zd Ze jjje jjje jjjhZd d d d dZeed<dddddZ eed<dZ!e"ed<de"fdZ# d/de jj$de"de"de%fdZ&Gdd Z'dddde d!e d"e d#e d$dddddf d%e(d&e(d'e(d(e(d!e(d"e(d#e(d$e(d)ee%d*ee%d+ee%d,ee%dee"d-e e jj$e ffd.Z)dS)0N)partial)Path)OptionalTuple) DictConfig)_ExperimentalConfig)tensorboard_trace_handler) get_loggerget_world_size_and_rankINFOprofiler wait_steps warmup_steps active_steps num_cyclesDEFAULT_SCHEDULEFT)profile_memory with_stack record_shapes with_flopsDEFAULT_TRACE_OPTSprofiler_outputDEFAULT_PROFILE_DIRmsgclt\}}|dkrt|dSdS)Nr)r logwarning)r_ranks p/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/training/_profiler.py_warnr%2s9%''GAt qyy Cyself_cuda_time_totalprof output_dirmetric row_limitc t\}}dt|jz}tj||}tj|stj|d|dkr"t d|jtj }tj } t|d| jd| jd| jd| jd| j d} | ||dkr3t d tj |z d d |jrt*jr|dkr{ ||d |d n4#t2$r'} td| Yd} ~ nd} ~ wwxYwt*jj|d |d|jr||d |d|||j d!||} tE|d |dd5} tG| | dddn #1swxYwY|dkrt d||dkr t*j$%dSdS)a~ Handles export of artifacts from ``torch.profiler.profile``. The following artifacts are exported: - chrome / tensorboard trace - viewable through tensorboard or perfetto.dev / chrome::/tracing - trace event table - memory timeline and snapshot.pickle if ``profile_memory`` - stacks if ``with_stack`` (note that ``profile_memory`` requires ``with_stack`` to be ``True``), viewable as a flamegraph see (https://pytorch.org/docs/stable/profiler.html#torch.profiler._KinetoProfile.export_stacks). Notes: - Each profiling cycle is exported as a sub-directory in output_dir - E.g., profiling in 5-step cycle (wait=2, warmup=2, active=1, repeat=0) will result in sub-directories iteration_5, iteration_10, etc. - If profiling in a distributed setting, each artifact will be prefixed with rank. - Memory timeline is only exported for rank 0 (error if exporting from multiple ranks on single node) See profiler documentation (https://pytorch.org/docs/stable/profiler.html#torch.profiler.profile) for more details Args: prof (torch.profiler.profile): instance of torch profiler to use output_dir (str): directory to store artifacts metric (str): metric to order trace event table by, see ``torch.profiler.profile.key_averages().table`` for row_limit (int): number of rows to display in trace event table iteration_T)exist_okrzDumping traces at step zr0--) worker_nameuse_gzipzFinished dumping traces in z.2fz secondsz/rankz_memory-timeline.htmlz# Failed to export memory timeline: Nz_memory_snapshot.picklez _stacks.txt)r+r)group_by_input_shapegroup_by_stack_n)sort_byr,z_key_averages.txtw)filezSaving profiling results to r)&r strstep_numospathjoinexistsmakedirsr infotime monotonicdatetimenowr yearmonthdayhourminutertorchcuda is_availableexport_memory_timeline Exceptionwarnmemory_dump_snapshotr export_stacks key_averagesrtableopenprint distributedbarrier)r)r*r+r, world_sizer#curr_trace_dir_namecurr_trace_dirbeginrCexporterekey_avgsfs r$ trace_handlerr`8sw@/00J&T]););;W\\*.ABBN 7>>. ) )3 NT2222 qyy :4=::;;; N  E     ! !C(R#(RRSYRRRR38RRcjRRH  HTNNN qyy Ut~/?/?%/GUUUUVVV  uz6688  199 D++%GGDGGG D D DBqBBCCCCCCCC D J  , ,!EEEEE     U nDD4DDDVTTT  !/!! eFie00  ==d===s C C q hQ                qyy @@@AAAA~~ !!#####~s*F G'G  G&JJ Jc0eZdZdZdZdZdZdZdZdS) DummyProfilera{ Drop-in replacement for torch.profiler.profile that functions as a nullcontext / object with no-op methods for ``start``, ``stop``, and ``step``. This is helpful for instrumenting profiling in a recipe without requiring changes to the code independent of whether profiling is on / off. E.g., ``` profiler = DummyProfiler() #profiler = torch.profiler.profile() # Below is same regardless of profiler object type with profiler as prof: for epoch in epochs: for batch in batches: train.step() prof.step() c|SNselfs r$ __enter__zDummyProfiler.__enter__s r&cdSrdre)rgargss r$__exit__zDummyProfiler.__exit__ r&cdSrdrerfs r$startzDummyProfiler.startrlr&cdSrdrerfs r$stopzDummyProfiler.stoprlr&cdSrdrerfs r$stepzDummyProfiler.steprlr&N) __name__ __module__ __qualname____doc__rhrkrnrprrrer&r$rbrbsi*              r&rbrrrrenabledcpurJxpurrrrreturnc 4|s.tdttddifSg} |r)| tjjj|r)| tjjj|r)| tjjj t| dkrtdt} dx}x}}t|du| du| du| dug }|r[ttdd fd Dn|| | | d fd D}t|dkrj|D]}t||<td d |d fd|Dtjdddd}|rtd|p|}|p|}|rt%dnd}| tdt&t&} t)| } | ddt-| } t/t0| }tj| |||||||}t|| |||||||d }||fS)aC Sets up :class:`~torch.profiler.profile` and returns the profiler config with post-setup updates. The profiler config can be provided in configs under the ``profiler`` key with the following layout: .. code-block:: yaml profiler: _component_: torchtune.training.setup_torch_profiler enabled: bool # Output directory of trace artifacts output_dir: str # torch.profiler.ProfilerActivity types to trace cpu: bool cuda: bool # Trace options profile_memory: bool with_stack: bool record_shapes: bool with_flops: bool # torch.profiler.schedule args wait_steps: int warmup_steps: int active_steps: int num_cycles: int The profiler schedule updates with respect to an optimizer step (e.g., if ``gradient_accumulation = 2``, then the profiler will step every 2 batches). Sensible defaults will be chosen if the config is missing options: - If no activities are specified, profiler will default to CPU + CUDA - If no schedule is specified, profiler will default to ``DEFAULT_SCHEDULE`` - Certain options will be overridden (``with_stack`` and ``record_shapes``) depending on requirements of other options (e.g., ``profile_memory`` requires ``with_stack`` and ``record_shapes``). Note: - Enabling the profiler will result in training speed reduction. - Setting ``profile_memory: True`` will generate large trace files. - The profiler schedule is context dependent. Calling ``profiler.step()`` at each batch iteration but **outside** the gradient accumulation scope will ``step`` the profiler each forward / backward step. Calling ``profiler.step()`` each batch iteration but **within** the gradient accumulation scope will ``step`` the profiler each optimizer update step such that each ``step`` contains multiple forward / backward passes. Args: enabled (bool): Enable pytorch profiler. Default is False. cpu (bool): Enable cpu profiling. Default is True. cuda (bool): Enable cuda profiling. Default is True. xpu (bool): Enable xpu profiling. Default is True. profile_memory (bool): Profile memory usage. Default is False. with_stack (bool): Profile stack. Default is False. record_shapes (bool): Record shapes. Default is True. with_flops (bool): Profile flops. Default is False. wait_steps (Optional[int]): Wait time in steps. Maps to ``wait`` kwarg of ``torch.profiler.schedule``. warmup_steps (Optional[int]): Warmup time in steps. Maps to ``warmup`` kwarg of ``torch.profiler.schedule``. active_steps (Optional[int]): Active time in steps. Maps to ``active`` kwarg of ``torch.profiler.schedule``. num_cycles (Optional[int]): Number of profiling cycles. Maps to ``repeat`` kwarg of ``torch.profiler.schedule``. output_dir (Optional[str]): Tracing file output path. Returns: Tuple[torch.profiler.profile, DictConfig] z Profiling disabled.rwFrz1No activities specified, defaulting to CPU + CUDATNz. No schedule found in config, defaulting to {}z, c32K|]}|d|VdSz = Nre.0k schedule_argss r$ z'setup_torch_profiler..6s6TT!Q55=#355TTTTTTr&rc$g|] }| | Srdrer~s r$ z(setup_torch_profiler..@s#TTTa=;K;S;S;S;Sr&z= Missing keys in torch profiler schedule {}: defaulting to {}c32K|]}|d|VdSr}rer~s r$rz'setup_torch_profiler..Gs6PPa99}Q'799PPPPPPr&rrrr)waitwarmupactiverepeatzp`profile_memory` requires `with_stack` and `record_shapes`, these will be enabled since `profile_memory` is True)verbosez= No output directory found in profiler config, defaulting to )parentsr/)r*) activitiesrrrrscheduleexperimental_configon_trace_ready) rwr*rxrJryrrrr)r%rbrappendrIr ProfilerActivityCPUCUDAXPUlenDEFAULT_PROFILER_ACTIVITIESanyrformatr<keysrrrrmkdirr8rr`profile)rwrxrJryrrrrrrrrr*ruse_default_schedule missing_keysrrrcallbackr profiler_cfgrs @r$setup_torch_profilerrsn ? $%%% Iu+= > >>>J ?%.9=>>> @%.9>??? ?%.9=>>> :! ABBB0 dS # d "  $  $ d "    (  < C C TTTT}?Q?Q?S?STTTTT      %(($   UTTT=#5#5#7#7TTT |  q ! 7 7#3A#6 a  OVVIIl++IIPPPP<PPPPP    ~&& < (^,^,\* 'H  ~   -~J!3^M?IS-d;;;;t aL_ a a   ) j!!JTD111ZJ}<<rs& """"""""  222222444444????????j  N#' N#( N#' $ D-S,,,s) X$X$ . X$X$ X$ X$X$X$X$v# # # # # # # # N-.>?),7,_=),7!%"&"& $ $L$L$ L$ L$ L$  L$  L$  L$L$L$ L$3-L$3-L$ L$ L$  5> !: -.!L$L$L$L$L$L$r&