Pi%C|UddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z ddl m Z ddlZddlmZddlmZmZddlmZdZeed <ejd d d ejd e eefdedejdedef dZdededeefdZ ejd d d ejd e eefdedejdedef dZ!ejd fdejdejdefdZ"ej#d ejde dfdZ$ej#dddd ejdedej%dejde ed e ede dfd!Z&d ejfd"Z'dS)#N) OrderedDict)partial)AnyDict GeneratorOptional)warn)FakeTensorConverterFakeTensorMode) NF4TensorF_use_low_cpu_ramTdtypeoffload_to_cpumodel state_dictargsrrkwargsc|D]Q\}}t|tr7||||<|r||||<RdS)a A state_dict hook that replaces NF4 tensors with their restored higher-precision weight and optionally offloads the restored weight to CPU. Use this hook to avoid increased peak GPU memory usage during checkpoint save when training with QLoRA. This function is meant to be used with PyTorch's ``nn.Module._register_state_dict_hook``, i.e. >>> m = MyModule() >>> m._register_state_dict_hook(reparametrize_as_dtype_state_dict_post_hook) If the hook is registered per the above process, this hook will be called _after_ the module's ``state_dict`` method is called. The hook will replace all ``NF4Tensor`` instances by unquantizing them to the original dtype, and optionally offload the restored weight to CPU. Args: model (nn.Module): the model to take ``state_dict()`` on state_dict (Dict[str, Any]): the state dict to modify *args (Any): Unused args passed when running this as a state_dict hook. dtype (torch.dtype): the dtype to restore the weight to. Default is ``torch.bfloat16``. offload_to_cpu (bool): whether to offload the restored weight to CPU. Default is ``True``. **kwargs (Any): Unused keyword args passed when running this as a state_dict hook. N)items isinstancer tocpu)rrrrrrkvs r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/modules/common_utils.py+reparametrize_as_dtype_state_dict_post_hookrst>  ""441 a # # 4DDKKJqM 4 *1 1 1 3 3 1 44 slice_strlengthreturnc~d|vsd|vs Jdd|vrVd|dD}tfd|Ds Jddgz}|D]}d||<|S|d}t|d ks Jd d \}}}t|d kr)|d dkrt|d }|d z}d }nt|dkrG|d dkrt|d nd}|d dkrt|d nd}n|t|d kri|d dkrt|d nd}|d dkrt|d nd}|ddkrt|dnd}|d |cxkrks nJd|d |cxkrks nJd||d ks Jddgz}t ||nd ||n||nd } | D]}d |cxkrkrnd||<|S)af Convert a string representing a Python slice or index into a boolean array. The resulting array will have the same length as the specified `length` parameter. Each element in the array corresponds to an index in the original sequence, with `True` indicating that the index is included in the slice and `False` otherwise. Args: slice_str (str): A string representing a Python slice or index, e.g. "1:3", ":5", "2::3", "0,4,5". length (int): The length of the original sequence. Returns: list[bool]: A boolean array representing the slice. Examples: >>> slice_str_to_array("1:3", 5) [False, True, True, False, False] >>> slice_str_to_array(":", 5) [True, True, True, True, True] >>> slice_str_to_array("::2", 5) [True, False, True, False, True] >>> slice_str_to_array("1::2", 5) [False, True, False, True, False] >>> slice_str_to_array("2:5:2", 6) [False, False, True, False, True, False] >>> slice_str_to_array("0,4,5", 7) [True, False, False, False, True, True, False] ,:zCannot mix commas and colonsc,g|]}t|S)int).0is r z&slice_str_to_array.._s888a3q66888rc38K|]}d|cxkokncVdS)rNr&)r(r)r s r z%slice_str_to_array..`s744q1????F????444444rzIndex out of rangeFTzInvalid slice formatNNNrNzStart index out of rangezEnd index out of rangezStep cannot be zero)splitalllenr'range) rr indicesresultr)partsstartendstep slice_indicess ` rslice_str_to_arrayr=>s< i  3i#7#7#79W#7#7 7 i889??3#7#78884444G44444JJ6JJJ46!  AF1II OOC E u::???2???'E3 5zzQ58r>>E!H ai Uq!&qRE!H T$Qx2~~c%(mmm4 Uq!&qRE!H T$Qx2~~c%(mmm4 %aBs58}}}D =A/////////1K// / ;!s++++V+++++-E++ + <41999&;99 $Wv F"F aM  ????F?????F1I Mrc$t}t}t}|D]\} } t | t r-||| ||| <n||| || <|r|| || <d} tj d5tj || dddn #1swxYwYtj tj5tj| dd} dddn #1swxYwY|D]s} t || t r5| | || |R| | || t|D] } | | || <dS)ax A state_dict hook that replaces NF4 tensors with their restored higher-precision weight and optionally offloads the restored weight to CPU. Use this hook to avoid increased peak GPU memory usage during checkpoint save when training with QLoRA. This hook is similar to ``reparametrize_as_dtype_state_dict_post_hook`` but uses FakeTensor and mmap(2) to avoid CPU OOM on colab. This function is meant to be used with PyTorch's ``nn.Module._register_state_dict_hook``, i.e. >>> m = MyModule() >>> m._register_state_dict_hook(reparametrize_as_dtype_state_dict_post_hook) If the hook is registered per the above process, this hook will be called _after_ the module's ``state_dict`` method is called. The hook will replace all ``NF4Tensor`` instances by unquantizing them to the original dtype, and optionally offload the restored weight to CPU. Args: model (nn.Module): the model to take ``state_dict()`` on state_dict (Dict[str, Any]): the state dict to modify *args (Any): Unused args passed when running this as a state_dict hook. dtype (torch.dtype): the dtype to restore the weight to. Default is ``torch.bfloat16``. offload_to_cpu (bool): whether to offload the restored weight to CPU. Default is ``True``. **kwargs (Any): Unused keyword args passed when running this as a state_dict hook. z/tmp/fake_state_dict.ptT)materialize_fake_tensorsN)mmap weights_only)r r rrrr from_real_tensorrrtorch serialization skip_datasaveset_default_mmap_optionsr@ MAP_SHAREDloadkeyscopy_) rrrrrrmode converterfake_state_dictrrdest_state_dict_pathdest_state_dicts r4_low_ram_reparametrize_as_dtype_state_dict_post_hookrQsF   D#%%I!mmO  ""::1 a # # E!*!;!;D!!D!D!G!G!N!NOA  !*!;!;D!!D!DOA   :!0!3!7!7!9!9OA 5   & & & E E:: ?$8999:::::::::::::::   5 5do F FYY*%9SWXXXYYYYYYYYYYYYYYY __  44 jmY / / 4 A  $ $Z]%5%5e%<%< = = = = A  $ $Z] 3 3 3 3 __  ++'* 1 ++s$!DD D7EE"Emodulectr'tjdkrtdt}nt }|t|||dS)a} Register the reparametrize state dict hooks to the module and its submodules. This function is a wrapper that is meant to toggle between the low_cpu_ram and regular versions of the ``reparametrize_as_dtype`` state dict hooks. Args: module (nn.Module): the module to register the hooks to. dtype (torch.dtype): the dtype to restore the weight to. Default is ``torch.bfloat16``. offload_to_cpu (bool): whether to offload the restored weight to CPU. Default is ``True``. Raises: RuntimeError: If the low RAM reparametrize hook is used on Windows or an incompatible torch version. win32zPLow RAM reparametrize_as_dtype_state_dict_post_hook is not supported on Windows.rN)r sysplatform RuntimeErrorrQr_register_state_dict_hookr)rRrrhooks r(_register_reparametrize_state_dict_hooksrZsq& ; <7 " "b HDD: $$E.AAArr.c#6K|std|std|D]-}t |drt |jrd|_. dV|D]-}t |drt |jrd|_.dS#|D]-}t |drt |jrd|_.wxYw)a This context manager temporarily disables KV-cacheing on a given model, which must already already have KV-caches setup. All forward passes using the model within this context manager will not use KV-caches. KV-caches will be disabled when entering the context manager, and will be enabled upon exit, without being modified. This is useful in cases where we might wish to alternate between model calls which use KV-cacheing, and model calls which do not use KV-cacheing, without the additional overhead of deleting and setting caches up every time. Example: >>> from torchtune.models.llama3_2 import llama3_2_1b >>> from torchtune.modules import disable_kv_cache >>> import torch >>> model = llama3_2_1b() >>> # setup caches >>> model.setup_caches(batch_size=1, >>> dtype=torch.float32, >>> decoder_max_seq_len=1024) >>> print(model.caches_are_setup()) True >>> print(model.caches_are_enabled()) True >>> print(model.layers[0].attn.kv_cache) KVCache() >>> # now temporarily disable caches >>> with disable_kv_cache(model): >>> print(model.caches_are_setup()) True >>> print(model.caches_are_enabled()) False >>> print(model.layers[0].attn.kv_cache) KVCache() >>> # caches are now re-enabled, and their state is untouched >>> print(model.caches_are_setup()) True >>> print(model.caches_are_enabled()) True >>> print(model.layers[0].attn.kv_cache) KVCache() Args: model (nn.Module): model to disable KV-cacheing for. Yields: None: Returns control to the caller with KV-caches disabled on the given model. Raises: ValueError: If the model does not have caches setup. Use :func:`~torchtune.modules.TransformerDecoder.setup_caches` to setup caches first. zrModel caches must be setup before calling disable_kv_cache! Please use model.setup_caches() to setup model caches.zYou are using disable_kv_cache with a model that does not have caches enabled. This is a no-op and the expected behaviour may not occur.kv_cacheFNT) caches_are_setup ValueErrorcaches_are_enabledr moduleshasattrcallabler\ cache_enabledrrRs rdisable_kv_cacheresMn  ! ! # #  E     # # % %      --//)) 6: & & )8FO+D+D )#(F , mmoo , ,Fvz** ,x/H/H ,'+$ , ,emmoo , ,Fvz** ,x/H/H ,'+$ ,s CADencoder_max_seq_lendecoder_max_seq_len batch_sizedevicergrhc#K|rtd|5|||||dddn #1swxYwY dVt|dS#t|wxYw)a This context manager temporarily enables KV-cacheing on a given model, which does not already have KV-caches setup. All forward passes using the model within this context manager will use KV-caches. KV-caches will be set-up with the given ``batch_size``, ``dtype``, and ``max_seq_len`` when entering the context manager, and will be deleted on exit. Example: >>> from torchtune.models.llama3_2 import llama3_2_1b >>> from torchtune.modules import local_kv_cache >>> import torch >>> model = llama3_2_1b() >>> print(model.caches_are_setup()) False >>> print(model.caches_are_enabled()) False >>> print(model.layers[0].attn.kv_cache) None >>> # entering cacheing mode >>> with local_kv_cache(model, >>> batch_size=1, >>> device=torch.device("cpu"), >>> dtype=torch.float32, >>> decoder_max_seq_len=1024): >>> print(model.caches_are_setup()) True >>> print(model.caches_are_enabled()) True >>> print(model.layers[0].attn.kv_cache) KVCache() >>> # exited cacheing mode >>> print(model.caches_are_setup()) False >>> print(model.caches_are_enabled()) False >>> print(model.layers[0].attn.kv_cache) None Args: model (nn.Module): model to enable KV-cacheing for. batch_size (int): batch size for the caches. device (torch.device): device to setup caches on. this should be the same device the model is on. dtype (torch.dtype): dtype for the caches. encoder_max_seq_len (Optional[int]): maximum encoder cache sequence length. decoder_max_seq_len (Optional[int]): maximum decoder cache sequence length. Yields: None: Returns control to the caller with KV-caches setup and enabled on the given model. Raises: ValueError: If the model already has caches setup. You may use :func:`~torchtune.modules.common_utils.delete_kv_caches` to delete existing caches. zModel caches must be not setup prior to entering this context manager! Please use delete_kv_caches(model) to delete model caches.rfN)r]r^ setup_cachesdelete_kv_caches)rrirjrrgrhs rlocal_kv_cachern=sB   I         3 3                     sAAAA//Bc|std|D]4}t|dr"t |jrd|_d|_5dS)a Deletes KV caches from all attention layers in a model, and also ensures ``cache_enabled`` is set to False. Example: >>> from torchtune.models.llama3_2 import llama3_2_1b >>> from torchtune.modules import delete_kv_caches >>> import torch >>> model = llama3_2_1b() >>> model.setup_caches(batch_size=1, >>> dtype=torch.float32, >>> decoder_max_seq_len=1024) >>> print(model.caches_are_setup()) True >>> print(model.caches_are_enabled()) True >>> print(model.layers[0].attn.kv_cache) KVCache() >>> delete_kv_caches(model) >>> print(model.caches_are_setup()) False >>> print(model.caches_are_enabled()) False >>> print(model.layers[0].attn.kv_cache) None Args: model (nn.Module): model to enable KV-cacheing for. Raises: ValueError: if this function is called on a model which does not have caches setup. Use :func:`~torchtune.modules.TransformerDecoder.setup_caches` to setup caches first. zuYou have tried to delete model caches, but model.caches_are_setup() is False! Please setup caches on the model first.r\FN)r]r^r`rarbr\rcrds rrmrmsF  ! ! # #  @   --//## 6: & & #8FO+D+D ##(F "FO##r)( contextlibr@rU collectionsr functoolsrtypingrrrrwarningsr rCtorch.nnnntorch._subclasses.fake_tensorr r torchao.dtypes.nf4tensorr r bool__annotations__bfloat16Modulestrrrr'listr=rQrZcontextmanagerrerjrnrmr&rrrs9 ######111111111111 MMMMMMMM......$ #4#4#4 9#4S#X#4 #4 ; #4  #4  #4#4#4#4LG#GsGtDzGGGG\ C+C+C+ 9C+S#XC+ C+ ; C+  C+  C+C+C+C+P I ;D I,BII,)4D*EI,I,I,I,X *.)-P P P 9P P  L P ; P "# P "#P  P P P P f+#BI+#+#+#+#+#+#r