)`ie$hdZ ddlmZmZmZmZmZmZmZddl m Z ddl Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZmZdd lmZdd lmZddlmZddlmZddlmZGdde Zdededededede j de!fdZ"dededededede j de!fdZ#de$edededededede j de$efdZ%e d7ded dedededede j d!ed"eed#e!de fd$Z&e d8d&e j'd'e d(ed)e!d*ee j'd+ee j'd,ee j'd-ee j'd.ee j'd/ee j'd0ee j'd1e(d2eee j'e(fd3eed4ee!d5e!de j'f"d6Z)dS)9a3 Copyright (c) 2025 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. )UnionLiteralOptionalTupleListcastAny)AllReduceFusionWorkspaceN)flashinfer_api)trtllm_allreduce_fusion)1trtllm_create_ipc_workspace_for_all_reduce_fusion)0check_trtllm_allreduce_fusion_workspace_metadata)Mapping) CommBackendSymmDeviceMemory)AllReduceFusionPattern)MNNVLAllReduceFusionWorkspace)MNNVLAllreduceFusionStrategy)trtllm_mnnvl_allreduce)(trtllm_mnnvl_fused_allreduce_add_rmsnormceZdZdZejdfdededededejdee f fd Z e d e fd Z d Z dded ededejdeed ef dZddZxZS)TRTLLMAllReduceFusionWorkspacez,TensorRT-LLM workspace for AllReduce fusion.Ntp_sizetp_rank max_token_num hidden_dimdtype comm_backendc t||t|||||d|tjkd|_t tttttj tttf|j}|d|_ |d|_|d|_|d|_dS)a Create TensorRT-LLM AllReduce fusion workspace. Args: tp_size: Tensor parallel size (world size) tp_rank: Tensor parallel rank max_token_num: Maximum number of tokens hidden_dim: Hidden dimension size dtype: Data type comm_backend: Communication backend **kwargs: Additional arguments for workspace creation T)rrrrrcreate_metadatause_fp32_lamportuse_symm_dev_memrr N)super__init__rtorchfloat32_internal_workspacerrrintTensorrdict ipc_handlesworkspace_tensor mem_handlesmetadata) selfrrrrrrworkspace_tuple __class__s m/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/comm/allreduce.pyr'z'TRTLLMAllReduceFusionWorkspace.__init__]s* '***$U'!% "em3! $ $ $   $tCy/5<6F1GM N  $  +1- / 2*1-'* returncdS)Ntrtllmr2s r5backendz&TRTLLMAllReduceFusionWorkspace.backendsxr6c|dr(tdt|jd|dt |j|S)z=Delegate attribute access to internal workspace if not found._'z' object has no attribute ') startswithAttributeErrortype__name__getattrr*)r2names r5 __getattr__z*TRTLLMAllReduceFusionWorkspace.__getattr__s\ ??3    KDJJ'KKDKKK t/666r6 num_tokens use_oneshotc t|||||jdS#t$r}td|Yd}~dSd}~wwxYw)NTz,Workspace is insufficient for problem size. F)rr1 ValueErrorprint)r2rrGrrrHes r5is_buffer_size_sufficientz8TRTLLMAllReduceFusionWorkspace.is_buffer_size_sufficientsp  <J    4    DDD E E E55555 s A>AcJt|ddrdS|`|`|`|`d|_dS)z%Destroy workspace and free resources. _destroyedFNT)rDr.r/r0r1rOr;s r5destroyz&TRTLLMAllReduceFusionWorkspace.destroys= 4u - -  F    !   Mr6)N)r7N)rC __module__ __qualname____doc__r(float16r+rrrr'propertystrr<rFr boolrMrP __classcell__)r4s@r5rrZs<66#].2,+,+,+,+ ,+  ,+ { ,+{+,+,+,+,+,+,+\X777&*   {  c]  "        r6rr< world_sizerankrrrr7c|dkS)z} Check if trtllm backend CAN be used for workspace creation. Hard requirements: - Up to 16 ranks supported. r:r<rYrZrrrs r5_trtllm_workspace_checkr^s  r6cdS)zE Check if mnnvl backend CAN be used for workspace creation. Tr:r]s r5_mnnvl_workspace_checkr`s  4r6suitable_backendscT|sgSt|dkr|Sd|vrdgS|dgS)a Select best backend for workspace creation based on performance. Called by decorator after checking which backends pass requirements. Uses benchmarking data to pick fastest option. Args: suitable_backends: List of backends that passed hard requirement checks backend: Requested backend ("auto", "trtllm", or "mnnvl") world_size: Number of ranks rank: Current rank max_token_num: Maximum number of tokens hidden_dim: Hidden dimension size dtype: Data type **kwargs: Additional arguments Note that at this point, the backend selection does not take "runtime parameters" into account, such as layout_code, and fusion pattern. Returns: List containing the selected backend (single element) r mnnvlr)lenrar<rYrZrrrs r5_workspace_creation_heuristicrfsO<   ""  ###y!!$%%r6autoF)r9rcrg gpus_per_noderforce_oneshot_supportc ||,ttj|}|dkrg} t ||||||r| dt ||||||r| d| stdt| ||||||} | d} n|} | dkrt|||||| S| dkrNt|||| } d} |r"tj ||||tj} t| ||||| Std | ) u Create workspace for AllReduce fusion operations. Backend selection uses topology-based checks and heuristics. **Important: Workspace Reusability** The workspace is allocated based on the total size (max_token_num * hidden_dim * dtype_size). You can reuse the same workspace with different shapes as long as the total size fits: - Workspace(max_token_num=2048, hidden_dim=4096) can handle: - (token_num=2048, hidden_dim=4096) ✓ - (token_num=1024, hidden_dim=4096) ✓ - (token_num=4096, hidden_dim=2048) ✓ (same total size) - (token_num=1024, hidden_dim=8192) ✓ (same total size) - (token_num=4096, hidden_dim=4096) ✗ (too large) Use `workspace.is_buffer_size_sufficient(token_num, hidden_dim, dtype)` to check before use. Args: backend: Backend to use ("trtllm", "mnnvl", or "auto") "auto" uses heuristic to select best backend world_size: Number of ranks in the process group rank: Current rank ID max_token_num: Maximum number of tokens to support hidden_dim: Hidden dimension size dtype: Data type for communication tensors gpus_per_node: Number of GPUs per node (for multi-node topology). comm_backend: Communication backend to use. force_oneshot_support: Allocate workspace for oneshot strategy vs twoshot True: Allocate workspace for oneshot strategy up to the largest problem size requested False: Allocate workspace for twoshot strategy for all problem sizes, and for oneshot strategy up to the heuristic threshold. Note that only the workspace for MNNVL backend needs to be initialized with the correct strategy. The trtllm backend will be sufficient for both strategies. Returns: Workspace object (TRTLLMAllReduceFusionWorkspace or MNNVLAllReduceFusionWorkspace) The workspace type determines which backend will be used in allreduce_fusion() Raises: BackendSupportedError: If no suitable backend available for the configuration ValueError: If problem size not supported for the specified backend Examples: >>> # Auto-select best backend >>> workspace = create_allreduce_fusion_workspace( ... backend="auto", ... world_size=8, ... rank=0, ... max_token_num=2048, ... hidden_dim=4096, ... dtype=torch.bfloat16, ... ) >>> print(workspace.backend) # "trtllm" >>> print(workspace.get_workspace_capacity()) # 8388608 elements >>> # Check if workspace can handle different problem sizes >>> workspace.is_buffer_size_sufficient(1024, 4096, 8, torch.bfloat16) # True >>> workspace.is_buffer_size_sufficient(4096, 2048, 8, torch.bfloat16) # True (same total) >>> # Explicit backend selection >>> workspace = create_allreduce_fusion_workspace( ... backend="mnnvl", ... world_size=16, ... rank=0, ... max_token_num=2048, ... hidden_dim=4096, ... dtype=torch.bfloat16, ... ) >>> print(workspace.backend) # "mnnvl" Nrgr]r9rczNo suitable backend found. rer)rrrrrr)rYrZrhr)mappingmax_num_tokensrrrbuffer_size_in_byteszUnknown backend: )minr(cuda device_countr^appendr`rJrfrrrget_required_buffer_size_bytesrONESHOT RuntimeError)r<rYrZrrrrhrriraselectedactual_backendrkrms r5!create_allreduce_fusion_workspacerwsdEJ3355zBB & "!'!     /  $ $X . . . !!'!     .  $ $W - - -  <:;; ;1/!'!   "! !!-'!%      7 " "!'     $ -L!08  !-(!%!5     ?~??@@@r6ư>input workspacepatternlaunch_with_pdloutput residual_outnorm_out quant_out scale_out residual_in rms_gammarms_eps scale_factor layout_coderHfp32_accc xt|tr|j\}}|tj|}d}||d}||d}| || dnd}| ||dnd}| ||dnd}| ||dnd}t d%id |d |jd |jd |d |d|jd|d|d|d|d|d|d|d|d|d|d|d| d| d| d| d|j ||S||S|St|tr |tj kr#|tj krtd|d| td|tj kr+|tj|}t|||||S|tj krj| td| td |tj|}|tj|}t!|| | || |||!\}}|Std"|t#d#t%|d$)&a AllReduce + RMSNorm fusion operation. Backend is automatically determined from workspace type. If you need another backend, create the workspace for the desired backend. Supports multiple fusion patterns: - AllReduce only - AllReduce + Residual + RMSNorm - AllReduce + Residual + RMSNorm + Quantization (FP8/FP4) **Note on Workspace Reusability:** You can reuse the same workspace with different (token_num, hidden_dim) combinations as long as `workspace.is_buffer_size_sufficient(token_num, hidden_dim, tp_size, dtype)` returns True. Args: input: Input tensor [token_num, hidden_dim] workspace: Workspace object (type determines backend, see create_allreduce_fusion_workspace) pattern: Fusion pattern (AllReduceFusionPattern constant, 0-5) - kAllReduce = 0 - kARResidualRMSNorm = 1 - kARResidualRMSNormFP8Quant = 2 - kARResidualRMSNormFP4Quant = 3 - kARResidualRMSNormOutFP8Quant = 4 - kARResidualRMSNormOutFP4Quant = 5 Note: MNNVL only supports patterns 0 and 1 launch_with_pdl: Use Persistent Dependency Launch # ===== OUTPUT tensors (pre-allocated, filled by function) ===== output: AllReduce output [token_num, hidden_dim] residual_out: Prenorm output (after residual add, before norm) [token_num, hidden_dim] norm_out: Normalized output [token_num, hidden_dim] quant_out: Quantized output [token_num, hidden_dim] [trtllm only] scale_out: Quantization scale factors [trtllm only] # ===== INPUT parameters ===== residual_in: Residual tensor to ADD [token_num, hidden_dim] rms_gamma: RMSNorm weight [hidden_dim] rms_eps: RMSNorm epsilon for numerical stability scale_factor: Input scale factor for quantization [trtllm only] layout_code: Scale factor layout (QuantizationSFLayout) [trtllm only] # ===== Control parameters ===== use_oneshot: Use oneshot strategy vs twoshot If None, uses internal heuristics. Note: when explicitly set to True, the MNNVL backend needs to be initialized with a sufficiently large workspace. fp32_acc: [trtllm only] Use FP32 accumulation for AllReduce Returns: Output tensor (typically norm_out for fusion cases, output otherwise) Examples: >>> # Basic AllReduce + Residual + RMSNorm >>> workspace = create_allreduce_fusion_workspace( ... backend="auto", ... world_size=8, ... rank=0, ... max_token_num=2048, ... hidden_dim=4096, ... dtype=torch.bfloat16, ... ) >>> >>> # Pre-allocate output tensors >>> prenorm = torch.empty_like(hidden_states) >>> normed = torch.empty_like(hidden_states) >>> >>> # Call fusion - backend inferred from workspace type >>> output = allreduce_fusion( ... input=hidden_states, ... workspace=workspace, ... pattern=AllReduceFusionPattern.kARResidualRMSNorm, ... launch_with_pdl=True, ... residual_out=prenorm, ... norm_out=normed, ... residual_in=residual, ... rms_gamma=norm_weight ... ) >>> # output == normed (final result) >>> # With FP8 quantization >>> quant = torch.empty_like(hidden_states, dtype=torch.float8_e4m3fn) >>> scales = torch.empty(token_num * hidden_dim // 16, dtype=torch.float16) >>> >>> output = allreduce_fusion( ... input=hidden_states, ... workspace=workspace, ... pattern=AllReduceFusionPattern.kARResidualRMSNormFP8Quant, ... norm_out=normed, ... quant_out=quant, ... scale_out=scales, ... residual_in=residual, ... rms_gamma=norm_weight, ... scale_factor=scale_tensor ... ) Ncx|st|d|dS)Nz must be contiguous) is_contiguousrJview)trEs r5_flatten_checkedz*allreduce_fusion.._flatten_checked@s;??$$ ? D!=!=!=>>>66":: r6ryr}rr~rr allreduce_inrY world_rank token_numrworkspace_ptrsr|trigger_completion_at_endr pattern_coderH allreduce_outrrrrrr1z4MNNVL AllReduce+RMS fusion does not support pattern z(. Please try the TRTLLM backend instead.zLMNNVL AllReduce does not support quantization fusion and thus no layout_code)ryrzr|r}z/MNNVL AllReduce+RMS fusion requires residual_inz-MNNVL AllReduce+RMS fusion requires rms_gamma)ryrgammarzepsilonr}r~r|z'Unsupported pattern for MNNVL backend: zUnknown workspace type: zJ. Expected TRTLLMAllReduceFusionWorkspace or MNNVLAllReduceFusionWorkspacer:) isinstancershaper( empty_liker rYrZr/r1rrkARResidualRMSNorm kAllReducerJrr TypeErrorrB)ryrzr{r|r}r~rrrrrrrrrHrrrr input_flat output_flatresidual_in_flatresidual_out_flat norm_out_flatquant_out_flat norm_resultresidual_results r5allreduce_fusionrsj);<<E !&  : >%e,,F     &%eW55 &&vx88 &  [- 8 8 8 '  \> : : :  7?6J  Xz 2 2 2PT 9B8M  Y 4 4 4SW     #  ++ !~~  i  "z  %55  ,O '6o X ! $  &+ )( +* #] %n! " i# $ i% &G' (&) *$ + ,''-  4  O  " M I< = == -@ @ @1<<<xwxxx   "^  ,7 7 7~)%00 "# /      M .A A A" !RSSS  !PQQQ +E22#$/66 ,T'#) / , , , (K PwPPQQ Q XtI X X X   r6) rgNNNNNNNF) FNNNNNNNrxNNNF)*rStypingrrrrrrr workspace_baser r(flashinfer.api_loggingr trtllm_arr rrrkrrcrrrtrtllm_mnnvl_arrrrrrrVr+rrWr^r`listrfrwr,floatrr:r6r5rs DDCCCCCCCCCCCCCCCCC444444 111111......HHHHHHGGGGGG00000000.-----::::::999999333333EEEEEE$WWWWW%=WWW~      ;  "          ;      *,&Cy,& ,&,&  ,&  ,&  ,& ;,& #Y,&,&,&,&h28*."'bAbA . /bAbA bA bA  bA ; bAbA;'bA bAbAbAbAbAT "%)+/'+(,(,*.(,9=!%"&'y y <y 'y y  y U\ " y 5<(y u|$y  %y  %y %,'y  %y y 5u!456y #!y $$%y &'y ( \)y y y y y y r6