Pi"ddlZddlZddlZddlZddlZddlmZddlmZddl Z ddl m Z ddl m ZmZddlmZddlmZmZmZee jjjZGdd eZdS) N)Path)Optional)record)get_args_parserrun) Subcommand)Configget_all_recipesRecipeceZdZdZfdZddZedejde fdZ dejde fd Z dejfd Z d e deefd Zd e de fdZde deedeefdZdejfdZxZS)Runz2Holds all the logic for the `tune run` subcommand.c *t|dddddtjdt j|_||j |j dS)Nrztune runzLRun a recipe. For distributed recipes, this supports all torchrun arguments.zGtune run [TORCHRUN-OPTIONS] --config [RECIPE-OPTIONS]as examples: # Run a finetuning recipe on a single device w/ default values $ tune run lora_finetune_single_device --config llama2/7B_lora_single_device # Run a finetuning recipe in a distributed fashion using torchrun w/ default values $ tune run --nproc_per_node 4 full_finetune_distributed --config llama2/7B_full_finetune_distributed # Override a parameter in the config file and specify a number of GPUs for torchrun $ tune run --nproc_per_node 2 lora_finetune_single_device --config llama2/7B_lora_single_device model.lora_rank=16 Remember, you can use `tune cp` to copy a default recipe/config to your local dir and modify the values. )proghelp descriptionusageepilogformatter_class)func) super__init__ add_parsertextwrapdedentargparseRawTextHelpFormatter_parser_add_arguments set_defaults_run_cmd)self subparsers __class__s f/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/_cli/run.pyrz Run.__init__s !,, _f[?&%93-   6  !!t}!55555returnNct}|jD]\}|jdkrd|_d|_n&|jdkrd|_d|_n |jdkrB|j|]dS) aAdd arguments to the parser. This is a bit hacky since we need to add the torchrun arguments to our parser. This grabs the argparser from torchrun, iterates over it's actions, and adds them to our parser. We rename the training_script and training_script_args to recipe and recipe_args respectively. In addition, we leave out the help argument since we add it manually to ours. training_scriptrecipezjName or path to recipe to be launched followed by args. For a list of all possible recipes, run `tune ls`.training_script_args recipe_argsz Args to be passed to the recipe.rN)get_torchrun_args_parser_actionsdestrr _add_action)r!torchrun_argparseractions r$rzRun._add_arguments=s677(1 - -F{///& 6  666+ @ && L $ $V , , , , - -r%args is_builtinctd|j|_|j|_|jsd|_| |_t|dS)zRun a recipe with torchrun.zRunning with torchrun...TN) printr)r(r+r* rdzv_endpoint standalonemodulerr!r2r3s r$_run_distributedzRun._run_distributedRsV ()))#{$($4!! #"DO %n  D r%ct|jg|jzt_|r*t jt|jddSt jt|jddS)z Run a recipe on a single device.__main__)run_nameN)strr)r+sysargvrunpyrun_path run_moduler9s r$_run_single_devicezRun._run_single_devicegst $$%(88  D N3t{++j A A A A A A  S-- C C C C C Cr%ctttjdz }t|jdz}||kS)z8Check if the user is trying to run a distributed recipe.)lenr?r@r+)r!r2total script_argss r$_is_distributed_argszRun._is_distributed_argsqs4CH !$*++a/ {""r% recipe_strcFtD]}|j|kr|cSdS)zGet a recipe from the name or path. Args: recipe_str (str): The name or path of the recipe. Returns: The recipe if it's found in built-in recipes, otherwise None. N)r name)r!rLr)s r$ _get_recipezRun._get_recipews>&''  F{j(( )  r% recipe_pathcrtj|\}}|ddS)zConvert a custom recipe path to a dot path that can be run as a module. Args: recipe_path (str): The path of the recipe. Returns: The dot path of the recipe. /.)ospathsplitextreplace)r!rPfilepath_s r$_convert_to_dotpathzRun._convert_to_dotpaths3g&&{33 !S)))r% config_strspecific_recipec||jD]}|j|kr|cStD]}|jD]}|j|kr|ccSdS)a,Get a config from the name or path. Args: config_str (str): The name or path of the config. specific_recipe (Optional[Recipe]): The specific recipe to search through. Returns: The config if it's found in built-in configs, otherwise None. N)configsrNr )r!r[r\configr)s r$ _get_configzRun._get_configs  &)1 " ";*,,!MMM-&'' " "F . " ";*,,!MMMMM- " " "r%cRd}d\}} |jddz}|j|}n*#t$r|jdYnwxYw||j}|||j}d}n(ttdz |j z }|j }d}| ||} | |}n"ttdz d z | j z }||_||j|<tjt!j||r>|s#|jd |jd ||| dS||| dS) z Run a recipe.T)NNz--configrGz$The '--config' argument is required.NFrecipesr^zRecipe zL does not support distributed training.Please run without torchrun commands.)r3)r+index ValueErrorrerrorrOr)rZr>ROOT file_pathsupports_distributedr`r?rUappendrTgetcwdrKrNr:rD) r!r2rhrP config_path config_idxr[r)r3r_s r$r z Run._run_cmds $#- [ G)// ;;a?J)*5JJ G G G L  E F F F F F G!!$+.. >224;??KJJdY.1AABBK#)#> J!!*f55 >$KKdY.:V=MMNNK" '2$  $$$  $ $T * * A'  "" > > > >  # #DZ # @ @ @ @ @s*4$AA)r&N)__name__ __module__ __qualname____doc__rrrr Namespaceboolr:rDrKr>rr rOrZr r`r __classcell__)r#s@r$r r ss<<66666@----* X%7T V(Dx'9DtDDDD#);#### c hv.>     *s *s * * * *""080@" & """"0.AX/.A.A.A.A.A.A.A.Ar%r )rrTrAr?rpathlibrtypingr torchtune0torch.distributed.elastic.multiprocessing.errorsrtorch.distributed.runrr,rtorchtune._cli.subcommandrtorchtune._recipe_registryr r r __file__parentrfr r%r$r~s CCCCCCRRRRRRRR000000FFFFFFFFFF tI &-|A|A|A|A|A*|A|A|A|A|Ar%