Pi ddlZddlmZddlZddlmZddlmZ ddejj de d e d e d e d ef d Z deejj efd e fdZ dS)N)Union)LambdaLR)OptimizerInBackwardWrapper? optimizernum_warmup_stepsnum_training_steps num_cycles last_epochreturncVdtdtffd }t|||S)a Create a learning rate schedule that linearly increases the learning rate from 0.0 to lr over ``num_warmup_steps``, then decreases to 0.0 on a cosine schedule over the remaining ``num_training_steps-num_warmup_steps`` (assuming ``num_cycles`` = 0.5). This is based on the Hugging Face implementation https://github.com/huggingface/transformers/blob/v4.23.1/src/transformers/optimization.py#L104. Args: optimizer (torch.optim.Optimizer): The optimizer for which to schedule the learning rate. num_warmup_steps (int): The number of steps for the warmup phase. num_training_steps (int): The total number of training steps. num_cycles (float): The number of waves in the cosine schedule. Defaults to 0.5 (decrease from the max value to 0 following a half-cosine). last_epoch (int): The index of the last epoch when resuming training. Defaults to -1 Returns: torch.optim.lr_scheduler.LambdaLR with the appropriate schedule. current_stepr c|kr|tdz S|z tdz z }ddtjtjzdz|zzz}td|S)Nrg?g@g)maxmathcospi)rprogresscosine_lr_multipler r r s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/training/lr_schedulers.py lr_lambdaz2get_cosine_schedule_with_warmup..lr_lambda+s * * *#a)9":":: :!#33s !$448 8  ! $(47Z/#5@AA A 3*+++)intfloatr)rr r r r rs ``` rget_cosine_schedule_with_warmuprsO8 , , , , , , , , , , Iy* 5 55rct|trMg}|D]#}||dd$n|j}t |dkrtdt ||dd}|D]}|d|krtd|S)aP Full_finetune_distributed and full_finetune_single_device assume all optimizers have the same LR, here to validate whether all the LR are the same and return if True. Args: optimizer (Union[torch.optim.Optimizer, OptimizerInBackwardWrapper]): A general optimizer input that could whether be a general optimizer or an optimizer warpper based on optimizer_in_backward. Returns: lr (float): The learning rate of the input optimizers. Raises: RuntimeError: If the learning rates of the input optimizer are not the same. param_groupsrrz,Invalid optimizer param groups with len of: lrz4LR Schedulers are different across all param groups ) isinstancer state_dictvaluesappendrlen RuntimeError)rrparamr groups rget_lrr)=s$)788. ))++2244 : :E   n 5a 8 9 9 9 9 :!-  <1 N3|;L;L N N   a BWW ;"  UVV V  Ir)rr)rtypingrtorchtorch.optim.lr_schedulerrtorchtune.training.memoryroptim Optimizerrrrr)rrr1s  ------@@@@@@ +6+6{$+6+6+6 +6  +6  +6+6+6+6\"U[*,FFG" """"""r