Pi(ddlZddlmZddlmZddlmZddlmZm Z m Z ddl m Z m Z ddlmZddlmZmZddlmZd ZGd d eZGd d e ZGdde ZdS)N)DTensor) DeviceMesh)ColwiseParallelPrepareModuleInputRowwiseParallel) ScalingType e4m3_dtype)tensor_already_casted_to_fp8)NoopFwToFloat8BwDynamichp_tensor_to_float8_dynamic) GemmInputRolecV|jtjko|jtjkS)N)scaling_type_inputrDYNAMICscaling_type_grad_output)ms y/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchao/float8/float8_tensor_parallel.py(_float8_linear_supports_float8_allgatherr"s*  33 > &+*= =cpeZdZdZedZedZdejde dejffd Z xZ S)Float8ColwiseParallelzo Like `ColwiseParallel`, but with all-gather in float8. This currently assumes tensorwise scaling. c*|d}t|tstj|||d}t|s1t ||jjj|jtj }||kr| |d}|SNrF run_checkgemm_input_roleT placementsasync_op isinstancer from_localr r configcast_config_input target_dtypelinear_mm_configr INPUT redistribute input_layoutsdesired_input_layoutsmodinputs device_mesh input_tensors r_prepare_input_fnz'Float8ColwiseParallel._prepare_input_fn0s ay ,00 "-k=EL,L99 6 ,9$ - 3 L 1 1 1'44045Lrc|j|kr||d}tj||j|jjj}|r|n|SNTr rr)r applyr'r$cast_config_grad_outputr&to_localoutput_layoutsuse_local_outputr-outputsr/s r_prepare_output_fnz(Float8ColwiseParallel._prepare_output_fnJsw   / /**)D+G */   J . ;  &6Bw!!!7Brmoduler/returncddlm}t||stdt |t||rt |st dt||SNr Float8Linearz.Expecting module to be Float8Linear but found unsupported torchao.float8.float8_linearrBr" ValueErrortyperAssertionErrorsuper_applyselfr=r/rB __class__s rrJzFloat8ColwiseParallel._apply\======&,// 0OfOO  L   0:6BB 0!// /ww~~fk222r __name__ __module__ __qualname____doc__ staticmethodr1r<nnModulerrJ __classcell__rMs@rrr*s \2CC\C" 3RY 3Z 3BI 3 3 3 3 3 3 3 3 3 3rrcpeZdZdZedZedZdejde dejffd Z xZ S)Float8RowwiseParallelzo Like `RowwiseParallel`, but with all-gather in float8. This currently assumes tensorwise scaling. c*|d}t|tstj|||d}t|s1t ||jjj|jtj }||kr| |d}|Srr!r*s rr1z'Float8RowwiseParallel._prepare_input_fnqsay ,00 "-k=EL,L99 6 ,9$ - 3 L 1 1 1'44045Lrc|j|kr||d}tj||j|jjj}|r|n|Sr3r4r8s rr<z(Float8RowwiseParallel._prepare_output_fnsp   / /**nt*TTG*/   J . ;  &6Bw!!!7Brr=r/r>cddlm}t||stdt |t||rt |st dt||Sr@rDrKs rrJzFloat8RowwiseParallel._applyrNrrOrXs@rrZrZks \.CC\C" 3RY 3Z 3BI 3 3 3 3 3 3 3 3 3 3rrZcpeZdZdZdddddejddfd ZdZdej de d ej ffd Z xZ S) PrepareFloat8ModuleInputa Like `PrepareModuleInput`, but with all-gather in float8. This currently assumes tensorwise scaling. The only difference from `PrepareModuleInput` is that after we prepare the input DTensor, we cast the input to DTensor(Float8TrainingTensor) This is to ensure the float8 cast happens before the all-gather (i.e. Shard -> Replicate) so that if there are multiple float8 users of the input activation, we perform fp8 allgather only once. FP8 Args: float8_dtype (torch.dtype, optional): control what float8 dtype to cast to when prepare the module input, we currently only support torch.float8_e4m3fn. default: torch.float8_e4m3fn fwd_config_submodule_fqn (str, optional): the fqn of the submodule that contains the forward config used for the float8 cast. If not specified, we will search for the Float8Linear in the submodules and use the forward config from that module, in this case all module's forward config must be the same. NF)r+r,input_kwarg_layoutsdesired_input_kwarg_layoutsr: float8_dtypefwd_config_submodule_fqnct|||||||_d|_||_|jt jkrtddS)N)r+r,r`rar:zFPrepareFloat8ModuleInput only support casting to float8_e4m3fn for now)rI__init__rbr'rctorch float8_e4m3fnNotImplementedError) rLr+r,r`rar:rbrcrMs rrez!PrepareFloat8ModuleInput.__init__s} '"7 3(C-    ) $(@%   3 3 3%X  4 3rcz|t|tr|}n=t|tjs Jdtj|||fd}t |t |jtj }|||kr| |f}|j r| n|S|S)Nz%expecting input to be a torch.Tensor!Frr)r) r"rrfTensorr#r r r'r r(r)r:r7)rLinputmesh input_layoutdesired_layoutdt_inps r_prepare_input_argz+PrepareFloat8ModuleInput._prepare_input_args  #%)) !%66;6!+4,E1% - 3 F )ln.L.L,,8I,JJ(,(=I6??$$$6 ILrr=r/r>cddlm}|j9||j}t ||sJ|j|_nU|D]@}t ||r.|j |j|_&|j|jks JdA|jJt|||S)NrrAz?All the Float8Linear modules should have same linear_mm_config!) rErBrc get_submoduler"r'modulesrIrJ)rLr=r/rB fwd_linearr-rMs rrJzPrepareFloat8ModuleInput._applys======  ( 4--d.KLLJj,77 7 77$.$?D ! !~~''  c<00,4030D--#48LLLL] MLL$000 v{+++ r) rPrQrRrSrfrgrerprUrVrrJrWrXs@rr_r_s*" $((!%:6RYZBIrr_)rftorch.nnrUtorch.distributed._tensorrtorch.distributed.device_meshr!torch.distributed.tensor.parallelrrrtorchao.float8.configrr torchao.float8.distributed_utilsr #torchao.float8.float8_scaling_utilsr r %torchao.float8.float8_training_tensorr rrrZr_rrr~s  ------444444 :9999999IIIIII@?????>3>3>3>3>3O>3>3>3B<3<3<3<3<3O<3<3<3~_____1_____r