Pi ddlmZmZddlmZddlmZmZddl m Z dZdej de deee e fd d fd Zd S) )OptionalUnion)nn)checkpoint_wrapperCheckpointImpl) checkpointc|dkr#t|tjtddS|dkr t |}|dkst d|t jddt xj dz c_ |rt j |zdkr#t|tjtddS|Std ) NfullF)checkpoint_impl checkpoint_fn use_reentrantpreserve_rng_state selectiverzOSelective layer AC policy (every_x_layer) expects a positive integer, received _countz]Unknown AC type or AC config. Only selective op and selective layer ac implemented currently.) ptd_checkpoint_wrapperr NO_REENTRANTrint ValueErrorr__dict__ setdefaultrNotImplementedError)moduleac_modeac_style every_x_layers r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/training/activations.pyrrs&% *7$$      K   H ""qboqq  #..x;;;!!Q&!!  2 9M IQ N N) . ;(##(  M" k   modelr ac_optionreturnNcxt|jD]$\}}|dvrt|||}||j|<%dS)acUtility to setup activation checkpointing and wrap the model for checkpointing. Args: model (nn.Module): Model to setup activation checkpointing. ac_mode (str): Activation checkpointing mode. ['none', 'full', 'selective'] ac_option (Optional[Union[int, str]]): Activation checkpointing option. If ac_mode is "selective", ac_option can be an integer or a string representing the number of layers to checkpoint. If ac_mode is "selective" and ac_option is "op", then selective op ac is run. If ac_mode is "none" or "full", ac_option is ignored. )r rN) enumeratelayersr)rrr layer_idtransformer_blocks r(apply_selective_activation_checkpointingr'Bsc (1'>'>33## + + + 2!!!  "3 X33r)typingrrtorchr;torch.distributed.algorithms._checkpoint.checkpoint_wrapperrrrtorch.utils.checkpointrModulestrrr'rrr/s#""""""".----- , , , ^3 93 3c3h(3 333333r