§ `ƒisãód—ddlZddlZddlmcmcmcmZGd„dejj¦«Z dS)éNcó‚—eZdZdZdejjdejfd„Z e d„¦«Z d„Z d„Z d„Zd „Zdd efd „Zd „ZdS)ÚPostLocalSGDOptimizeraì Wraps an arbitrary :class:`torch.optim.Optimizer` and runs `post-local SGD `_, This optimizer runs local optimizer at every step. After the warm-up stage, it averages parameters periodically after the local optimizer is applied. Args: optim: The local optimizer. averager: A model averager instance to run post-localSGD algorithm. Example:: >>> # xdoctest: +SKIP("undefined variables") >>> import torch >>> import torch.distributed as dist >>> import torch.distributed.algorithms.model_averaging.averagers as averagers >>> import torch.nn as nn >>> from torch.distributed.optim import PostLocalSGDOptimizer >>> from torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook import ( >>> PostLocalSGDState, >>> post_localSGD_hook, >>> ) >>> >>> model = nn.parallel.DistributedDataParallel( >>> module, device_ids=[rank], output_device=rank >>> ) >>> >>> # Register a post-localSGD communication hook. >>> state = PostLocalSGDState(process_group=None, subgroup=None, start_localSGD_iter=100) >>> model.register_comm_hook(state, post_localSGD_hook) >>> >>> # Create a post-localSGD optimizer that wraps a local optimizer. >>> # Note that ``warmup_steps`` used in ``PostLocalSGDOptimizer`` must be the same as >>> # ``start_localSGD_iter`` used in ``PostLocalSGDState``. >>> local_optim = torch.optim.SGD(params=model.parameters(), lr=0.01) >>> opt = PostLocalSGDOptimizer( >>> optim=local_optim, >>> averager=averagers.PeriodicModelAverager(period=4, warmup_steps=100) >>> ) >>> >>> # In the first 100 steps, DDP runs global gradient averaging at every step. >>> # After 100 steps, DDP runs gradient averaging within each subgroup (intra-node by default), >>> # and post-localSGD optimizer runs global model averaging every 4 steps after applying the local optimizer. >>> for step in range(0, 200): >>> opt.zero_grad() >>> loss = loss_fn(output, labels) >>> loss.backward() >>> opt.step() ÚoptimÚaveragercóD—||_|jj|_||_dS©N)rÚ param_groupsr)Úselfrrs úƒ/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torch/distributed/optim/post_localSGD_optimizer.pyÚ__init__zPostLocalSGDOptimizer.__init__:s!€ØˆŒ Ø œJÔ3ˆÔØ ˆŒ ˆ ˆ ócó—|jjSr)rÚstate©r s r rzPostLocalSGDOptimizer.state?s €àŒzÔÐr có4—|j ¦«Sr)rÚ__repr__rs r rzPostLocalSGDOptimizer.__repr__Cs€ØŒz×"Ò"Ñ$Ô$Ð$r cóV—|j ¦«}|jj|d<|S)zí This is the same as :class:`torch.optim.Optimizer` :meth:`state_dict`, but adds an extra entry to record model averager's step to the checkpoint to ensure reload does not cause unnecessary warm up again. Ústep)rÚ state_dictrr)r Úoptim_state_dicts r rz PostLocalSGDOptimizer.state_dictFs.€ð  œ:×0Ò0Ñ2Ô2ÐØ#'¤=Ô#5Ð˜Ñ ØÐr cóª—|j |¦«d|vr|d|j_dSt jd¦«d|j_dS)aW This is the same as :class:`torch.optim.Optimizer` :meth:`load_state_dict`, but also restores model averager's step value to the one saved in the provided ``state_dict``. If there is no ``"step"`` entry in ``state_dict``, it will raise a warning and initialize the model averager's step to 0. rz]Loaded state dict does not contain a step counter for an averager. Setting step counter to 0.rN)rÚload_state_dictrrÚwarningsÚwarn)r rs r rz%PostLocalSGDOptimizer.load_state_dictPse€ð Œ ×"Ò" :Ñ.Ô.Ð.Ø ZÐ Ð Ø!+¨FÔ!3ˆDŒMÔ Ð Ð å ŒMð-ñ ô ð ð"#ˆDŒMÔ Ð Ð r cóx—|j ¦«|j |j¬¦«dS)zI Performs a single optimization step (parameter update). )ÚparamsN)rrrÚaverage_parametersr rs r rzPostLocalSGDOptimizer.stepcs8€ð Œ ŠÑÔÐØ Œ ×(Ò(°Ô0AÐ(ÑBÔBÐBÐBÐBr TÚ set_to_nonecó<—|j |¬¦«dS)N)r)rÚ zero_grad)r rs r r zPostLocalSGDOptimizer.zero_gradjs!€Ø Œ ×Ò¨ÐÑ5Ô5Ð5Ð5Ð5r có:—|j |¦«dSr)rÚadd_param_group)r Ú param_groups r r"z%PostLocalSGDOptimizer.add_param_groupms€Ø Œ ×"Ò" ;Ñ/Ô/Ð/Ð/Ð/r N)T)Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚtorchrÚ OptimizerÚ averagersÚ ModelAveragerr ÚpropertyrrrrrÚboolr r"©r r rrsÒ€€€€€ð/ð/ðb!˜eœkÔ3ð!¸yÔ?Vð!ð!ð!ð!ð ð ð ñ„Xð ð%ð%ð%ð ð ð ð#ð#ð#ð&CðCðCð6ð6 Tð6ð6ð6ð6ð0ð0ð0ð0ð0r r) rr(Ú6torch.distributed.algorithms.model_averaging.averagersÚ distributedÚ algorithmsÚmodel_averagingr*rr)rr.r r úr3s‡ðà€€€à € € € ØJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJÐJðf0ðf0ðf0ðf0ðf0˜EœKÔ1ñf0ôf0ðf0ðf0ðf0r