Pi9ddlZddlmZmZmZddlZddlmZddlm Z ej j Z GddeZ GddeZ Gd d ejZd ed e d ede fdZejjGddejjZejjGddejjZdejdfdejdejdejdee deedeef dZGddejZdS)N)Dict NamedTupleOptional)DTensor)to_fp8_saturatedcLeZdZUdZdZeed<dZeed<dZeed<dZ eed<dS)ScaledMMConfiga Configuration for the scaled_mm in the forward and backward pass. Attributes: emulate (bool): Whether to emulate the matmuls in fp32. use_fast_accum (bool): Whether to use the fast-accumulation option for scaled_mm. fp8_output (bool): Whether to output the result of the scaled_mm in fp8. pad_inner_dim (bool): Whether to pad the inner dimension of a and b with 0s. This is needed for matmuls not aligned to 16. Femulateuse_fast_accum fp8_output pad_inner_dimN) __name__ __module__ __qualname____doc__r bool__annotations__r r r y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/float8/float8_training_tensor.pyr r .s^  GT ND   JM4rr ceZdZUdZeddddZeed<eddddZeed<eddddZeed<dS)LinearMMConfiga Configuration for different gemm operations in LinearMM. This configuration is not user-facing and exists for convenience, allowing Float8TrainingTensor to use the right config based on which gemm from gemms with outputs `output`, `grad_input`, `grad_weight` is being called. Attributes: output (ScaledMMConfig): Configuration for the output gemm. grad_input (ScaledMMConfig): Configuration for the grad_input gemm. grad_weight (ScaledMMConfig): Configuration for the grad_weight gemm. FToutput grad_input grad_weightN) rrrrr rrrrrrrrr@s~  ,^E4FFFNFFF!/ueU!K!KJKKK"0.ue"L"LKLLLLLrrceZdZdZdZdZdZdS) GemmInputRolea Given a Float8TrainingTensor, the enum below describes the expected role of this tensor in the three gemms present in the fw + bw pass of a Linear layer. This is used to choose the right config for a float8 gemm when the gemm is performed. inputweight grad_outputN)rrrrINPUTWEIGHT GRAD_OUTPUTrrrrrSs) E FKKKrra_rolea_linear_mm_configb_roleb_linear_mm_configc|tjur?|tjur1|j|jksJd|jd|j|jS|tjur?|tjur1|j|jksJd|jd|j|jS|tjur?|tjur1|j|jksJd|jd|j|jStd|d|)Nz"linear_mm_config.output mismatch: z vs z&linear_mm_config.grad_input mismatch: z'linear_mm_config.grad_weight mismatch: zunexpected a_role z and b_role )rr!r"rr#rrAssertionError)r$r%r&r's rchoose_scaled_mm_configr*asI $$$=3G)G)G!(,>,EEEE k1C1J k kPbPi k kFEE"(( =, , ,=;O1O1O!,0B0MMMM w5G5R w wXjXu w wNMM",, =, , ,=;N1N1N!-1C1OOOO z6H6T z zZlZx z zPOO"--N&NNfNNOOOrceZdZdZedejdfdejdejdej de e de ede e f d Z ed ZdS) _ToFloat8ConstrFuncz A differentiable conversion to fp8. * forward: convert from high precision to float8 * backward: pass the gradient without changes Ntensorscale float8_dtypelinear_mm_configgemm_input_role axiswise_dimc |tj|z}t||}t |t rt |t s Jd|j} |j} |} |} t| | |j |||} t j | | | d| | St|||j |||S)a This function will apply the scaling, and then convert to a Float8TrainingTensor Note: We will call this function with a DTensor subclass. Ideally this would be an aten OP that DTensor could overload to ensure proper semantics. There are some techincal issues with that composing with FakeTensor, so we special case here. DTensor Invariant: DTensor must always be the outer most tensor subclass z>Expected Float8 scale to be a DTensor if bits_fp8 is a DTensor)r0r1r2F) run_checkshapestride)totorchfloat32r isinstancer device_mesh placementsto_localFloat8TrainingTensordtype from_localsizer6)ctxr-r.r/r0r1r2 tensor_scaledbits_fp8 bits_meshbits_placements local_bits local_scaleinner_float8_tensors rforwardz_ToFloat8ConstrFunc.forwards,. %-0058 #M<@@ h ( ( eW--  P  -!,I&1O!**,,J..**K"6 !1 /) ### %#mmoo((  $   L-+%     rc|dddddfSNrrBgs rbackwardz_ToFloat8ConstrFunc.backwards$dD$..r)rrrr staticmethodrr!r8Tensorr?rrintrJrOrrrr,r,zs  6:3@3F&*9 9  9 |9 k 9 #>2 9 "-0 9 sm9 9 9 \9 v//\///rr,c>eZdZdZedZedZdS)_FromFloat8ConstrFuncz A differentiable conversion from fp8. * forward: convert from float8 to high precision * backward: pass the gradient without changes cP|j|j|jz SrL)_datar7 _orig_dtype_scale)rBr-s rrJz_FromFloat8ConstrFunc.forwards |v122V]BBrc |ddfSrLrrMs rrOz_FromFloat8ConstrFunc.backwards$}rN)rrrrrPrJrOrrrrTrTsW CC\C\rrT hp_tensorsr/r0r1r2c@t||||||S)a  Given a high precision tensor `hp_tensor` and a precalculated scale `s`, scales `hp_tensor` by `s` and returns a `Float8TrainingTensor` of the result. Autograd-aware, the derivative is pass-through. DTensor-aware, if the input is a DTensor the output will be DTensor(Float8TrainingTensor). Args: hp_tensor: the tensor to convert s: the scale to use to convert the tensor float8_dtype: the float8 dtype to use linear_mm_config: Defines the configuration for the scaled_mm for the 3 fwd/bwd gemms of linear gemm_input_role: Defines the role of this tensor (input, weight or grad_output) in the 3 fwd/bwd gemms of linear axiswise_dim: for rowwise scaling, contains the axis scaled across )r,apply)rZr[r/r0r1r2s rhp_tensor_and_scale_to_float8r^s*2  $ $1l$4o|  rcReZdZUdZejed<ejed<ejed<eed<e ed<e e ed<gdZ e j d fd ejd ejd ejd e ede e de e f dZdZdZedefdZdZeddZejjZd S)r>a^ Note: this is **not** a public API and is only intended to be used inside of this repository. Please file an issue if you would benefit from this being a public API. A Python-only Float8 tensor subclass. Contains: * `_data`: the underlying e4m3 or e5m2 data * `_scale`: the scale used to scale the original fp32 tensor. We multiply by scale to go from fp32 range to fp8 range, and divide by scale to go from fp8 range to fp32 range. Scale is guaranteed to have a shape compatible with `_data`. For example: - if scaling is tensorwise, `_scale` is a scalar tensor - if scaling is axiswise and _data.shape is [3, 5], `_scale` could have shape [1, 5] or [3, 1]. `axiswise_dim` defines the scaling axis. - if scaling is axiswise and _data.shape is [2, 3, 5], `_scale` could have shape [1, 1, 5] or [2, 1, 1]. `axiswise_dim` defines the scaling axis. Non-one entries which are not the first or last element are not supported. * `_orig_dtype`: the original dtype of the tensor used to create this tensor. * `_axiswise_dim`: for axiswise scaling only, contains the axis scales across. Only values of 0 or -1 are supported. Intended usage of this abstraction: 1. to bundle raw data + fp8 metadata together for easy passing through Python PyTorch systems. 2. Float8-aware user code can use the private fields on these tensors to call into float8 operations. 3. Float8-agnostic user code can use these tensors as is - they will convert to original precision in `__torch_dispatch__`. rVrXrW_linear_mm_config_gemm_input_role _axiswise_dim)rVrXrWr`rarbNdatar. orig_dtyper0r1r2c ttj||||||j|j|j}||_ ||_ ||_ ||n t|_ ||_|dvs Jd|||_|S)N)stridesstorage_offsetr?layout requires_graddevice)Nrzunsupported axiswise_dim )r8rQ_make_wrapper_subclassrAr6rgrhrirjrVrXrWrr`rarb)clsrcr.rdr0r1r2selfs r__new__zFloat8TrainingTensor.__new__"s|22  IIKKKKMM..00;,;3     % 0 <  .BRBR !0},,,.X,.X.X,,,) rc d|jjd|jd|jd|jd|jd| S)NzFloat8TrainingTensor(lp_dtype=z, scale=z, linear_mm_config=z, axiswise_dim=z gemm_input_role=z as_orig_prec=)rVr?rXr`rbrato_original_precisionrns r__repr__zFloat8TrainingTensor.__repr__As| 0@||$+||jnkA||RVRd||x|xM||^b^x^x^z^z|| |rcD|j|j|j|jd}ddg|fS)NrWr`rarbrVrXru)rnrBs r__tensor_flatten__z'Float8TrainingTensor.__tensor_flatten__Ds8+!%!7 $ 5!/    "C''r inner_tensorsc t|dksJt|d|d|d|d|d|dS)NrVrXrWr`rarb)lenr>)rwmetadata outer_size outer_strides r__tensor_unflatten__z)Float8TrainingTensor.__tensor_unflatten__Ms_=!!Q&&&&# ' " ( # ] # ( ) ' ( _ %    rc6t|SrL)rTr]rrs rrqz*Float8TrainingTensor.to_original_precisionYs$**4000rcddlm}fdtfd|DstS||vr|||||St d|d)Nr)FLOAT8_OPS_TABLEct|pGttjjj|p#ttjjj|SrL) issubclassr8 _subclasses fake_tensor FakeTensorfunctional_tensorFunctionalTensor)typerms rallowed_subclasseszCFloat8TrainingTensor.__torch_dispatch__..allowed_subclassesisP3%%e/;FMM%7H$ rc3.K|]}|VdSrLr).0trs r z:Float8TrainingTensor.__torch_dispatch__..rs/88Q%%a((888888rzattempting to run z, this is not supported)torchao.float8.float8_opsrallNotImplementedNotImplementedError)rmfunctypesargskwargsrrs` @r__torch_dispatch__z'Float8TrainingTensor.__torch_dispatch__\s ?>>>>>      8888%88888 "! ! # # #)#D)$f== =!"Tt"T"T"TUUUrrL)rrrrr8rQrr?rrrrR __slots__r!rorsrvrPrr~rq classmethodr_C_disabled_torch_function_impl__torch_function__rrrr>r>s}@ < L%%%%####C=   I4A3F&*l|K  #>2  "-0 sm>|||(((  D    \  111VVV[V:?rr>)enumtypingrrrr8torch.distributed._tensorrtorchao.float8.float8_utilsropsatenr rEnumrr*_dynamoallow_in_graphautogradFunctionr,rTr!rQr?rRr^r>rrrrs  ---------- ------ y~<     Z   $MMMMMZMMM&      DI    P P&P P' PPPP2E/E/E/E/E/%.1E/E/E/P     EN3   (26/