Pi \ddlZddlmZddlmZmZmZmZddlZddl m Z ddl m Z m Z ddlmZejeZeejddde jdee jge jfd eee jegefd e jfd Zddd de jd eee jegefd ee d e jfdZdeee fdeed ee jegeffdZde jdedeed efdZde jdedeed efdZdS)N)partial)CallableListOptionalUnion)Float8LinearConfigFloat8LinearRecipeName) Float8Linearmodule_filter_fnmodulefrom_float_funcr returnct|tjr] |drOtt |dkrt d||S|} d dtjdttdttjffd ||S) a Generic function to swap linear layers in a module with a new type of linear layer. Note: If applied to a root-level nn.Linear, the module will not be modified in place and returned instead Args: module: Module to modify. from_float_func: Function that accepts a linear layer and returns a new type of linear layer. module_filter_fn: If specified, only the `torch.nn.Linear` subclasses that that pass the filter function will be swapped. The inputs to the filter function are the module instance, and the FQN. Returns: nn.Module: The modified module with swapped linear layers. Nrz1Does not support a root nn.Linear with children: r cur_fqn parent_modulecb|d}|D]"\}}|dkr|}n|d|} |||#t|tjrV  ||rJ| Jd||}|dd}t |||dSdSdS)Nr.z(Linear root module should return early: )named_children isinstancennLinearsplitsetattr) r rrchild_module_name child_modulenew_fqnnew_linear_modulecur_module_namerr post_order_traversals v/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/float8/float8_linear_utils.pyr"z0swap_linear_layers..post_order_traversal8s ?G/5/D/D/F/F @ @ + |"}}+$::'8:: w ? ? ? ? fbi ( ( G  $(8(8(I(I $ ,,C6CC-,,!0 7 7 %mmC004O M?4E F F F F F G G $ $)NN) rrrlenlistchildrenAssertionErrorModulerstr)r rr root_moduler"s `` @r#swap_linear_layersr,s.&")$$   $4$4VR$@$@ tFOO%%&& ' '! + + LFLL     K"&-1GG G#G  *GGGGGGGG4%%% r$)r configr-ctjdtfd}t |||S)a Swaps `torch.nn.Linear` in `module` with `Float8Linear`. Args: module: Module to modify. module_filter_fn: If specified, only the `torch.nn.Linear` subclasses that that pass the filter function will be swapped. The inputs to the filter function are the module instance and the FQN. config (Float8LinearConfig): configuration for conversion to float8 Returns: nn.Module: The modified module with swapped linear layers. z)torchao.float8.convert_to_float8_trainingNc0tj|S)N)r-)r from_float)mr-s r#z,convert_to_float8_training..ms<2 r$r )torch_C_log_api_usage_oncerr,)r r r-r0s ` r#convert_to_float8_trainingr6Vsd& H  !LMMM ~#%%J )   r$recipe filter_fqnscJt|trt|}|tjkrt t |S|tjkrt t|S|tjkrtd|td|)a?Returns function which automatically filters nn.Linear modules that meet at least one of the following criteria: 1. Dims not divisible by 16 (hardware requirement for float8). 2. Dim sizes below certain thresholds, which may result in worse performance. NOTE: the thresholds are simple heuristics based on performance testing, and may not be optimal for your model. For the best performance, we recommend defining your own module_filter_fn customized for your module, using the performance tables for the given float8 recipe here: https://github.com/pytorch/ao/tree/main/torchao/float8#performance). These benchmarks referenced for auto filtering layers were run on H100 GPUs, and may not be representative of other hardware. This is an experimental API, the design may change in the future. )r8zUnsupported recipe: zInvalid recipe: ) rr*r TENSORWISEr_auto_filter_for_tensorwiseROWWISE_auto_filter_for_rowwiseROWWISE_WITH_GW_HPNotImplementedError ValueError)r7r8s r#_auto_filter_for_reciperAys &#0'// '2222 LLLL )1 1 1/[IIII )< < <!"A"A"ABBB4F44555r$modfqnct|tjsdStfd|D}|rdS|jj\}}|dzdko|dzdk}|sdS|dkrdS|dkrdS|dkr|dkrdSdS) NFc3 K|]}|vV dSN.0 filter_fqnrCs r# z+_auto_filter_for_rowwise..(JJ *+JJJJJJr$riTrrranyweightshaperBrCr8is_filtered_fqnNKdims_multiples_of_16s ` r#r=r=s c29 % %uJJJJkJJJJJOu : DAqr6Q;61r6Q; u  Dyyu du dqDyyu 4r$ct|tjsdStfd|D}|rdS|jj\}}|dzdko|dzdk}|sdS|dkr|dkrdSdS)NFc3 K|]}|vV dSrFrGrHs r#rKz._auto_filter_for_tensorwise..rLr$rMrrOrNTrPrTs ` r#r;r;s c29 % %uJJJJkJJJJJOu : DAqr6Q;61r6Q; u  DyyQ$YYu 4r$)logging functoolsrtypingrrrrr3torch.nnrtorchao.float8.configrr torchao.float8.float8_linearr getLogger__name__log addHandler NullHandlerr)rr*boolr,r6rAr=r;rGr$r#rgse 222222222222 LLLLLLLL555555g!!"w"$$%%%DH ??? I?ryk2945?xC(8$(>?@ ? Y ????JDH+/    I xC(8$(>?@  ' (  Y    F6 #-- .6=A#Y6 ry#$%66668")#DIRV8 +/9 r$