)`i.DdZddlZddlmZddlZddlmZddlmZddl m Z m Z m Z ej dZe d1d ejd ejd ed eejdeedejf dZe ddd ejd ejd ejd edeeddf dZe dd ejd ejd ejd edeeddf dZee dd d2d ejd ejd ejded edeedejfdZe dd ejd ejd ejded edeeddfdZee dd d2d ejdejd ejd edeeddf dZe d d2d ejdejd ejd edeeddf dZee dd  d2d ejd ejdejd ejded edeeddfd!Ze d d2d ejd ejdejd ejded edeeddfd"Ze d1d ejd ejd ed eejdeedejf d#Ze d$dd ejd ejd ejd edeeddf d%Ze d$d ejd ejd ejd edeeddf d&Zee d'd d2d ejdejd ejd edeeddf d(Ze d' d2d ejdejd ejd edeeddf d)Zee d*d+ d3d ejd,ejd-ejd edejf d.Z e d* d3d ejd,ejd-ejd edejf d/Z! dd0l"m#Z#m$Z$dS#e%$rdZ#dZ$YdSwxYw)4a3 Copyright (c) 2024 by FlashInfer team. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. N)Optional)flashinfer_api)gen_norm_module)device_support_pdlregister_custom_opregister_fake_opcBtSN)rbuild_and_loadc/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/flashinfer/norm.pyget_norm_modulers    + + - --rư>inputweightepsout enable_pdlreturnc|t|j}|tj|}t ||||||S)a^Root mean square normalization. ``out[i] = (input[i] / RMS(input)) * weight[i]`` Parameters ---------- input: torch.Tensor Input tensor, 2D shape (batch_size, hidden_size) or 3D shape (batch_size, num_heads, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. out: Optional[torch.Tensor] The output tensor, if specified, the kernel will update this tensor inplace. enable_pdl: bool Whether to enable `programmatic dependent launch `_ Returns ------- output: torch.Tensor Normalized tensor, 2D shape (batch_size, hidden_size) or 3D shape (batch_size, num_heads, hidden_size). )rdevicetorch empty_like_rmsnormrrrrrs rrmsnormr sI>' 55  {u%% S%j111 Jrzflashinfer::rmsnorm)r) mutates_argsc||t|j}t|||||dSr )rrrrrrrrrs rrrGsA' 55 c5&#zBBBBBrcdSr r r!s r _rmsnorm_faker#T  Drzflashinfer::rmsnorm_quantscalec~|t|j}t||||||dS)a&Root mean square normalization. ``out[i] = (input[i] / RMS(input)) * weight[i]`` Parameters ---------- out: torch.Tensor The output tensor, will quantize the output to the dtype of this tensor. input: torch.Tensor Input tensor, 2D shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). scale: float Scale factor for quantization. eps: float Epsilon for numerical stability. enable_pdl: bool Whether to enable `programmatic dependent launch `_ Returns ------- output: torch.Tensor Normalized tensor, 2D shape (batch_size, hidden_size). N)rrr rmsnorm_quantrrrr%rrs rr'r'_sDF' 55 ##CsJOOOOOrcdSr r r(s r_rmsnorm_quant_faker*s  Drzflashinfer::fused_add_rmsnorm)rresidualr+c||t|j}t|||||dS)aFused add root mean square normalization. Step 1: ``residual[i] += input[i]`` Step 2: ``input[i] = (residual[i] / RMS(residual)) * weight[i]`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). residual: torch.Tensor Residual tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. enable_pdl: bool Whether to enable `programmatic dependent launch `_ N)rrrfused_add_rmsnormrr+rrrs rr-r-sA>' 55 ''xjQQQQQrcdSr r r.s r_fused_add_rmsnorm_faker0r$rz#flashinfer::fused_add_rmsnorm_quant)rr+c |t|j}t|||||||dS)aRFused add root mean square normalization. Step 1: ``residual[i] += input[i]`` Step 2: ``input[i] = (residual[i] / RMS(residual)) * weight[i]`` Parameters ---------- out: torch.Tensor The output tensor, will quantize the output to the dtype of this tensor. input: torch.Tensor Input tensor, shape (batch_size, hidden_size). residual: torch.Tensor Residual tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). scale: float Scale factor for quantization. eps: float Epsilon for numerical stability. enable_pdl: bool Whether to enable `programmatic dependent launch `_ N)rrrfused_add_rmsnorm_quantrrr+rr%rrs rr2r2sQN' 55 -- UHfeS*rcdSr r r3s r_fused_add_rmsnorm_quant_faker5s  Drc|t|j}|tj|}t ||||||S)aGemma-style root mean square normalization. ``out[i] = (input[i] / RMS(input)) * (weight[i] + 1)`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. out: Optional[torch.Tensor] The output tensor, if specified, the kernel will update this tensor inplace. enable_pdl: bool Whether to enable `programmatic dependent launch `_ Returns ------- output: torch.Tensor Gemma Normalized tensor, shape (batch_size, hidden_size). )rrrr_gemma_rmsnormrs r gemma_rmsnormr8sI>' 55  {u%%3vsJ777 Jrzflashinfer::gemma_rmsnormc||t|j}t|||||dSr )rrrr8r!s rr7r7$sA' 55 ##CZHHHHHrcdSr r r!s r_gemma_rmsnorm_faker;1r$rz#flashinfer::gemma_fused_add_rmsnormc||t|j}t|||||dS)aGemma-style fused add root mean square normalization. Step 1: ``residual[i] += input[i]`` Step 2: ``input[i] = (residual[i] / RMS(residual)) * (weight + 1)`` Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). residual: torch.Tensor Residual tensor, shape (batch_size, hidden_size). weight: torch.Tensor Weight tensor, shape (hidden_size,). eps: float Epsilon for numerical stability. enable_pdl: bool Whether to enable `programmatic dependent launch `_ N)rrrgemma_fused_add_rmsnormr.s rr=r=<sBB' 55 --eXvsJWWWWWrcdSr r r.s r_gemma_fused_add_rmsnorm_faker?br$rzflashinfer::layernormr gemmabetacxtj|}t||||||S)a Layer normalization. Parameters ---------- input: torch.Tensor Input tensor, shape (batch_size, hidden_size). Need to be bfloat16. gemma: torch.Tensor Gemma tensor, shape (hidden_size,). Need to be float32. beta: torch.Tensor Beta tensor, shape (hidden_size,). Need to be float32. eps: float Epsilon for numerical stability. Returns ------- output: torch.Tensor Layer Normalized tensor, shape (batch_size, hidden_size). Same dtype as input. )rrr layernorm)rr@rArrs rrCrCms;2  5 ! !CUE4=== JrcD|j\}}|||gSr )shape new_empty)rr@rArbks r_layernorm_fakerIs% ;DAq ??Aq6 " ""r)rmsnorm_fp4quantadd_rmsnorm_fp4quant)rNN)rN)r)&__doc__ functoolstypingrr api_loggingrjit.normrutilsrrr cacherTensorfloatboolrrr#r'r*r-r0r2r5r8r7r;r=r?rCrIcute_dslrJrK ImportErrorr rrrXs/   ''''''%%%%%%KKKKKKKKKK ..."&!% ## <# L# # %,  #  #  \ ####L)AAA C  C < C L C  C  C  C C CBA C'((  <  L          )( /hGGG !% #P#P #P <#P L#P  #P  #P  #P \#P#P#PHG#PL-..  <  L            /. 3BWXXX !% RR <RlR LR  R  R  RRRYXRD122 !%   < l  L          32 )8K!%'' ' <'l' L '  '  '' ''''T788!%     <  l   L             98  "&!% ## <# L# # %,  #  #  \ ####L/hGGG I  I < I L I  I  I  I I IHG I-..  <  L          /. )8M!% XX <XlX LX  X  X  XXXXD788 !%   < l  L          98 +"===   < < ,    \ >=8)**  ## <# <# ,#  #  \ ###+*# @@@@@@@@@@    sP PP