Pi[ddlZddlZddlZddlmZddlmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZddlmZmZddlmZe e jeeefZedZd ed efd Zd d ddeeefdedefdZGddeZGddeZ GddeZ!GddeZ"GddeZ#GddeZ$Gdd eZ%dS)!N)Path)AnyDictListMappingOptionalUnion)ndarray) DictConfig OmegaConf) get_loggerget_world_size_and_rank)ProtocolDEBUGconfigreturnc t|j}|dd|dz }tj|||S#t $r)}t d|dYd}~dSd}~wwxYw)a Save the OmegaConf configuration to a YAML file at `{config.output_dir}/torchtune_config.yaml`. Args: config (DictConfig): The OmegaConf config object to be saved. It must contain an `output_dir` attribute specifying where the configuration file should be saved. Returns: Path: The path to the saved configuration file. Note: If the specified `output_dir` does not exist, it will be created. Tparentsexist_okztorchtune_config.yamlzError saving config. Error: .N)r output_dirmkdirr save Exceptionlogwarning)rroutput_config_fnamees u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/training/metric_logging.py save_configr!s<&+,, 555(+BBv2333"" <<< :a:::;;;;;;;;;>> flatten_dict({"foo": {"bar": "baz"}, "qux": "quux"}, sep="--") {"foo--bar": "baz", "qux": "quux"} Returns: Dict[str, Any]: Flattened dictionary. Note: Does not unnest dictionaries within list values (i.e., {"foo": [{"bar": "baz"}]}). r#)items isinstancedictextend flatten_dictappend)r&r$r%r(kvnew_keys r r,r,3s$ E ''1-7>Z))a)))Q a   ' LLaSWEEEKKMM N N N N LL'1 & & & & ;;cfeZdZdZdedededdfdZdeddfd Z d e eefdeddfd Z d d Z dS)MetricLoggerInterfacezAbstract metric logger.namedatasteprNcdS)zLog scalar data. Args: name (str): tag name used to group scalars data (Scalar): scalar data to log step (int): step value to record Nselfr4r5r6s r rzMetricLoggerInterface.logRs  r1rcdS)z^Logs the config as file Args: config (DictConfig): config to log Nr8)r:rs r log_configz MetricLoggerInterface.log_configas r1payloadcdS)zLog multiple scalar values. Args: payload (Mapping[str, Scalar]): dictionary of tag name and scalar value step (int): step value to record Nr8r:r=r6s r log_dictzMetricLoggerInterface.log_dictis  r1cdS)zx Close log resource, flushing if necessary. Logs should not be written after `close` is called. Nr8r:s r closezMetricLoggerInterface.closers r1rN) __name__ __module__ __qualname____doc__strScalarintrr r<rr@rCr8r1r r3r3Os!!                    V 4 C D          r1r3ceZdZdZddedeefdZdefdZded e d e ddfd Z d e ddfd Z deee fd e ddfdZddZddZdS) DiskLoggeraLogger to disk. Args: log_dir (str): directory to store logs filename (Optional[str]): optional filename to write logs to. Default: None, in which case log_{unixtimestamp}.txt will be used. **kwargs: additional arguments Warning: This logger is not thread-safe. Note: This logger creates a new file based on the current time. Nlog_dirfilenamec 6t||_|jdd|s&tt j}d|d}|j|z |_t |jd|_td|jdS)NTrlog_z.txtazWriting logs to ) rrNrrKtime _file_nameopen_fileprint)r:rNrOkwargsunix_timestamps r __init__zDiskLogger.__init__sG}}  4$777 3 --N2n222H,1$/3//  22233333r1rc|jSN)rTrBs r path_to_log_filezDiskLogger.path_to_log_files r1r4r5r6c |jd|d|d|d|jdS)NStep  | : )rVwriteflushr9s r rzDiskLogger.logsN 999$99999::: r1rc$t|}dSr\r!r:r_s r r<zDiskLogger.log_config   r1r=c|jd|d|D]%\}}|j|d|d&|jd|jdS)Nr_r`ra rb)rVrcr(rdr:r=r6r4r5s r r@zDiskLogger.log_dicts ****+++!--// 0 0JD$ J  ..t... / / / /  r1c8|jdSr\rVrCrBs r __del__zDiskLogger.__del__ r1c8|jdSr\rnrBs r rCzDiskLogger.closerpr1r\rD)rErFrGrHrIrrZrr]rJrKrr r<rr@rorCr8r1r rMrMzs  444x}4444$6      V 4CDr1rMcneZdZdZdedededdfdZdeddfd Z d e eefdeddfd Z dd Z dd Z dS) StdoutLoggerzLogger to standard output.r4r5r6rNc6td|d|d|dS)Nr_r`ra)rWr9s r rzStdoutLogger.logs/ ,d,,t,,d,,-----r1rc$t|}dSr\rfrgs r r<zStdoutLogger.log_configrir1r=ctd|dd|D]\}}t|d|ddtdddS)Nr_r`r")endrarkrb)rWr(rls r r@zStdoutLogger.log_dicts| dR((((!--// - -JD$ T##D### , , , , , dr1cBtjdSr\sysstdoutrdrBs r rozStdoutLogger.__del__ r1cBtjdSr\ryrBs r rCzStdoutLogger.closer|r1rD)rErFrGrHrIrJrKrr r<rr@rorCr8r1r rsrss$$..6......      V 4CD r1rsc eZdZdZ ddedeedeedeefdZd ed dfd Zd ed e de d dfdZ de ee fde d dfdZ ddZddZdS) WandBLoggeraLogger for use w/ Weights and Biases application (https://wandb.ai/). For more information about arguments expected by WandB, see https://docs.wandb.ai/ref/python/init. Args: project (str): WandB project name. Default is `torchtune`. entity (Optional[str]): WandB entity name. If you don't specify an entity, the run will be sent to your default entity, which is usually your username. group (Optional[str]): WandB group name for grouping runs together. If you don't specify a group, the run will be logged as an individual experiment. log_dir (Optional[str]): WandB log directory. If not specified, use the `dir` argument provided in kwargs. Else, use root directory. **kwargs: additional arguments to pass to wandb.init Example: >>> from torchtune.training.metric_logging import WandBLogger >>> logger = WandBLogger(project="my_project", entity="my_entity", group="my_group") >>> logger.log("my_metric", 1.0, 1) >>> logger.log_dict({"my_metric": 1.0}, 1) >>> logger.close() Raises: ImportError: If ``wandb`` package is not installed. Note: This logger requires the wandb package to be installed. You can install it with `pip install wandb`. In order to use the logger, you need to login to your WandB account. You can do this by running `wandb login` in your terminal. torchtuneNprojectentitygrouprNc  ddl}n"#t$r}td|d}~wwxYw||_|d||_|j=t j|jst j|jt\}|_ |jj '|j dkr|jj d||||jd|} |jj r |jj dt|jddr7|jd|jd dd |d d |_dS)Nrz``wandb`` package not found. Please install wandb using `pip install wandb` to use WandBLogger.Alternatively, use the ``StdoutLogger``, which can be specified by setting metric_logger_type='stdout'.dir)rrrrr)repo define_metric global_step*T) step_metric step_syncallow_val_changeFr8)wandb ImportError_wandbpoprNospathexistsmakedirsrrankruninit_labelgetattrrgetconfig_allow_val_change) r:rrrrNrXrrrhrs r rZzWandBLogger.__init__s  LLLL   z    zz%11  < #BGNN4<,H,H # K % % %.00 49 ;? "tyA~~"$+"L   C ;? 5 KO " " " 4 4 4 4; 6 6 V K % %m 4 4 4 K % %c}PT % U U U'-zz2De'L'L$$$ &!&rrcn|jjrtj|d}|jj||jt|} |j||j dS#t$r,}t d|d|dYd}~dSd}~wwxYwdS) awSaves the config locally and also logs the config to W&B. The config is stored in the same directory as the checkpoint. You can see an example of the logged config to W&B in the following link: https://wandb.ai/capecape/torchtune/runs/6053ofw0/files/torchtune_config_j67sb73v.yaml Args: config (DictConfig): config to log Tresolve)r) base_pathzError uploading z to W&B. Error: z8.Don't worry the config will be logged the W&B workspaceN) rrr to_containerrupdaterr!rparentrrrr:rresolvedrrs r r<zWandBLogger.log_configs ;?  -fdCCCH K  % %4+G &    #.f"5"5     '3F3M!    N':NNqNNN   s!A<< B2!B--B2r4r5r6c^|jjr |j||d|idSdSNrrrrr9s r rzWandBLogger.log*s; ;? ? KOOT4= > > > > > ? ?r1r=cb|jjr"|ji|d|idSdSrrr?s r r@zWandBLogger.log_dict.sB ;? > KOO >r1cxt|dr'|jjr|jdSdSdS)Nr)hasattrrrfinishrBs r rozWandBLogger.__del__2sM 4 " " !t{ ! K    ! ! ! !r1cT|jjr|jdSdSr\)rrrrBs r rCzWandBLogger.close7s2 ;? ! K    ! !r1)rNNNrD)rErFrGrHrIrrZr r<rJrKrrr@rorCr8r1r rrs.@# $#!% ,M,M,M ,M} ,M # ,M,M,M,M\6??6??????>V 4>C>D>>>>!!!! !!!!!!r1rceZdZdZddedefdZdededed d fd Z d e d d fd Z de eefded d fdZ ddZddZd S)TensorBoardLoggeraJLogger for use w/ PyTorch's implementation of TensorBoard (https://pytorch.org/docs/stable/tensorboard.html). Args: log_dir (str): torch.TensorBoard log directory organize_logs (bool): If `True`, this class will create a subdirectory within `log_dir` for the current run. Having sub-directories allows you to compare logs across runs. When TensorBoard is passed a logdir at startup, it recursively walks the directory tree rooted at logdir looking for subdirectories that contain tfevents data. Every time it encounters such a subdirectory, it loads it as a new run, and the frontend will organize the data accordingly. Recommended value is `True`. Run `tensorboard --logdir my_log_dir` to view the logs. **kwargs: additional arguments Example: >>> from torchtune.training.metric_logging import TensorBoardLogger >>> logger = TensorBoardLogger(log_dir="my_log_dir") >>> logger.log("my_metric", 1.0, 1) >>> logger.log_dict({"my_metric": 1.0}, 1) >>> logger.close() Note: This utility requires the tensorboard package to be installed. You can install it with `pip install tensorboard`. In order to view TensorBoard logs, you need to run `tensorboard --logdir my_log_dir` in your terminal. TrN organize_logsc ddlm}d|_t\}|_|r>>DLLL ?r1r4r5r6rNcT|jr |j|||ddSdS)NT)r new_style)r add_scalarr9s r rzTensorBoardLogger.loghs> < R L # #D$DD # Q Q Q Q Q R Rr1rc$t|}dSr\rfrgs r r<zTensorBoardLogger.log_configlrir1r=ch|D]\}}||||dSr\)r(rrls r r@zTensorBoardLogger.log_dictos@!--// ' 'JD$ HHT4 & & & & ' 'r1cX|jr"|jd|_dSdSr\rrCrBs r rozTensorBoardLogger.__del__s5 < L   DLLL  r1cX|jr"|jd|_dSdSr\rrBs r rCzTensorBoardLogger.closexrr1)TrD)rErFrGrHrIboolrZrJrKrr r<rr@rorCr8r1r rr<s2???D????$RR6RRRRRR      'V 4'C'D''''          r1rc.eZdZdZ ddeedeedeedeedeed eed eed eeed ed eee ffdZ dede de ddfdZ deee fde ddfdZdeddfdZddZddZdS) CometLoggera Logger for use w/ Comet (https://www.comet.com/site/). Comet is an experiment tracking tool that helps ML teams track, debug, compare, and reproduce their model training runs. For more information about arguments expected by Comet, see https://www.comet.com/docs/v2/guides/experiment-management/configure-sdk/#for-the-experiment. Args: api_key (Optional[str]): Comet API key. It's recommended to configure the API Key with `comet login`. workspace (Optional[str]): Comet workspace name. If not provided, uses the default workspace. project (Optional[str]): Comet project name. Defaults to Uncategorized. experiment_key (Optional[str]): The key for comet experiment to be used for logging. This is used either to append data to an Existing Experiment or to control the ID of new experiments (for example to match another ID). Must be an alphanumeric string whose length is between 32 and 50 characters. mode (Optional[str]): Control how the Comet experiment is started. * ``"get_or_create"``: Starts a fresh experiment if required, or persists logging to an existing one. * ``"get"``: Continue logging to an existing experiment identified by the ``experiment_key`` value. * ``"create"``: Always creates of a new experiment, useful for HPO sweeps. online (Optional[bool]): If True, the data will be logged to Comet server, otherwise it will be stored locally in an offline experiment. Default is ``True``. experiment_name (Optional[str]): Name of the experiment. If not provided, Comet will auto-generate a name. tags (Optional[List[str]]): Tags to associate with the experiment. log_code (bool): Whether to log the source code. Defaults to True. **kwargs (Dict[str, Any]): additional arguments to pass to ``comet_ml.start``. See https://www.comet.com/docs/v2/api-and-sdk/python-sdk/reference/Experiment-Creation/#comet_ml.ExperimentConfig Example: >>> from torchtune.training.metric_logging import CometLogger >>> logger = CometLogger(project_name="my_project", workspace="my_workspace") >>> logger.log("my_metric", 1.0, 1) >>> logger.log_dict({"my_metric": 1.0}, 1) >>> logger.close() Raises: ImportError: If ``comet_ml`` package is not installed. Note: This logger requires the comet_ml package to be installed. You can install it with ``pip install comet_ml``. You need to set up your Comet.ml API key before using this logger. You can do this by calling ``comet login`` in your terminal. You can also set it as the `COMET_API_KEY` environment variable. NTapi_key workspacerexperiment_keymodeonlineexperiment_nametagslog_coderXc   ddl} n"#t$r} td| d} ~ wwxYwd| vr| d=t\} |_d|_|jdkr2| ||||||| jd| ||d| |_dSdS)Nrz``comet_ml`` package not found. Please install comet_ml using `pip install comet_ml` to use CometLogger.Alternatively, use the ``StdoutLogger``, which can be specified by setting metric_logger_type='stdout'.rN)rrr4)rrrrrrexperiment_configr8)comet_mlrrr experimentstartExperimentConfig)r:rrrrrrrrrrXrrrhs r rZzCometLogger.__init__s  OOOO   z     y!.00 49  9>>&nn#-";(";#%D##JP##-  DOOO >rr4r5r6rcR|j|j|||dSdSN)r6)r log_metricr9s r rzCometLogger.logs5 ? & O & &tT & = = = = = ' &r1r=cP|j|j||dSdSr)r log_metricsr?s r r@zCometLogger.log_dicts3 ? & O ' 'd ' ; ; ; ; ; ' &r1rcD|jtj|d}|j|t |} |j||jdS#t$r(}t d|Yd}~dSd}~wwxYwdS)NTr) file_namez0Failed to upload config to Comet assets. Error: ) rr rlog_parametersr! log_assetr4rrrrs r r<zCometLogger.log_configs ? & -fdCCCH O * *8 4 4 4#.f"5"5  T))'3F3K* T T T RqRRSSSSSSSSS T ' &s!A++ B5BBcJ|j|jdSdSr\)rrwrBs r rCzCometLogger.closes, ? & O   ! ! ! ! ! ' &r1c.|dSr\)rCrBs r rozCometLogger.__del__s r1) NNNNNNNNTrD)rErFrGrHrrIrrrrrZrJrKrrr@r r<rCror8r1r rr~s++^"&#'!%(,"!%)-$(**#*C=*# * ! * sm **"#*tCy!**sCx.****X>>6>>>>>><V 4>> logger = MLFlowLogger(experiment_name="my_experiment", run_name="run1") >>> logger.log("accuracy", 0.95, step=1) >>> logger.log_dict({"loss": 0.1, "accuracy": 0.95}, step=1) >>> logger.log_config(config) >>> logger.close() Raises: ImportError: If ``mlflow`` package is not installed. Note: This logger requires the mlflow package to be installed. You can install it with `pip install mlflow`. Nr tracking_urirun_idrun_namec ddl}n"#t$r}td|d}~wwxYwt\}|_||_|pt jd|_|pt jd|_|pt jd|_ |jdkr|j s&|j|j |j|j |j s|j rV|j|j|j|j|}|jj|_ dSdSdS)Nrz``mlflow`` package not found. Please install mlflow using `pip install mlflow` to use MLFlowLogger.Alternatively, use the ``StdoutLogger``, which can be specified by setting metric_logger_type='stdout'.MLFLOW_TRACKING_URIMLFLOW_EXPERIMENT_NAME MLFLOW_RUN_ID)r)mlflowrrr_mlflowrgetenv _tracking_uri_experiment_name_run_idis_tracking_uri_setset_tracking_uri active_run _nested_runset_experiment start_runinfor) r:rrrrrrrhrs r rZzMLFlowLogger.__init__sm  MMMM   z   /00 49 )MRY7L-M-M / V29=U3V3V;?!;!; 9>><3355 F%1L11$2DEEE|&&((0D4D0 0(4L//0EFFFl,,h,??"x  > 10rrrcx|jrtj|d}t |}|j||jt|}t|j  d}|j |||jdSdS)zSaves the config locally and also logs the config to mlflow. The config is stored in the same directory as the checkpoint. Args: config (DictConfig): config to log Tr)r/) artifact_pathrN) rrr rr, log_paramsrr!rIrlstrip log_artifact)r:rrconfig_as_paramsrrs r r<zMLFlowLogger.log_config6s < " " $ $  -fdCCCH ,H55  L # #$4T\ # J J J"-f"5"5  3 :;;BB3GGM L % %#+| &       r1r4r5r6c|jr%|j||||jdSdSN)r6r)rrrrr9s r rzMLFlowLogger.logOsL < " " $ $ P L # #D$T$, # O O O O O P Pr1r=c|jr$|j|||jdSdSr)rrrrr?s r r@zMLFlowLogger.log_dictSsJ < " " $ $ N L $ $W4 $ M M M M M N Nr1c|jdkr4|jr|jdSdSdS)zk Ends the MLflow run. After calling close, no further logging should be performed. rN)rrrend_runrBs r rCzMLFlowLogger.closeWsF 9>>dl5577> L " " " " " >>>r1ct|dr4|jr|jdSdSdS)Nr)rrrrrBs r rozMLFlowLogger.__del___sW 4 # # # (?(?(A(A # L " " " " " # # # #r1)NNNNrD)rErFrGrHrrIrZr r<rJrKrrr@rCror8r1r rrs7<*.&* $"& !/!/!#!/sm!/ !/ 3- !/!/!/!/F2PP6PPPPPPNV 4NCNDNNNN##########r1r)&rrzrSpathlibrtypingrrrrrr torchnumpyr omegaconfr r torchtune.utilsr rtyping_extensionsrTensorrKfloatrJrr!rIr,r3rMrsrrrrr8r1r r s  <<<<<<<<<<<<<<<< ++++++++????????&&&&&& u|Wc50 1j<