.`i': \dZddlZddlmZddlmcmZddlmZddl m Z ddl m Z m Z ddlmZdejdejd ed ejfd Zdejd ejdejd ed eejejff d Zdejdejdejd ed ejf dZ d dedejdefdZe jdGdde Ze jdGdde Ze jdGdde ZGddejZdS)!zCustom normalization layers.N)rocm_aiter_ops)CustomOp)rms_norm_batch_invariantvllm_is_batch_invariant)current_platformxweightvariance_epsilonreturncddlm}trt|||St j|}||||||SNr) _custom_ops)vllmrrrtorch empty_likerms_norm)rr r opsouts x/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/model_executor/layers/layernorm.pyrrsq('''''  E'63CDDD  1  CLL    Jresidualcddlm}trt||z||||zfS|||||||fSr )rrrrfused_add_rms_norm)rrr r rs rrr#s ('''''  ' L&"2  x<    h;rbiasclddlm}tj|}|||||||Sr )rrrr poly_norm)rr rr rrs rrr8sR('''''  1  CMM     JrFwith_fused_adddtype use_aiterc|o|tjtjfv}|r|r tjS|r tjS|rt St SN)rfloat16bfloat16rrms_norm2d_with_addrrrrrs rdispatch_rocm_rmsnorm_funcr&Hsde  (I 2^211'&&"!! OrrceZdZdZ ddedededzded ejdzd df fd Z e dd ej d ededejdej dzdej dzdedzd ej e ej ej fzfdZ dd ej dej dzd ej e ej ej fzfdZ dd ej dej dzd ej e ej ej fzfdZ dd ej dej dzd ej e ej ej fzfdZ dd ej dej dzd ej e ej ej fzfdZd efdZxZS)RMSNormzRoot mean square normalization. Computes x -> w * x / sqrt(E[x^2] + eps) where w is the learned weight. Refer to https://arxiv.org/abs/1910.07467 ư>NT hidden_sizeepsvar_hidden_size has_weightrr ct||_||_||krdn||_|pt j}||_t j|||_ |jrtj |j |_ tj rCtj}t!d|||_t!d|||_dSdS)NrFr%T)super__init__r*r variance_size_overriderget_default_dtyper-onesr nn Parameterris_rocmris_rmsnorm_enabledr&rocm_norm_funcrocm_norm_func_with_add) selfr*r+r,r-r weight_dtypeaiter_rmsnorm_enabled __class__s rr1zRMSNorm.__init__fs & ##{22DD #9 7 9 9 $jLAAA ? 4,t{33DK  # % % $2$E$G$G !"<$"/###D  ,F#s@rr(r(\s&*$( t    {T!  @ '+(,-1++ <+++K +  t# + ,% +!$d + elEL89 9+++\+`)-  < ,%  elEL89 9     ()-HH <H,%H elEL89 9 HHHH&)-SS <S,%S elEL89 9 SSSS&)-  < ,%  elEL89 9     0Crr(gemma_rms_normc eZdZdZ ddededdffd Zedej d ed ej dej fd Z edej d ed ej d ej de ej ej ff d Z dd ej d ej dzdej e ej ej fzfdZ dd ej d ej dzdej e ej ej fzfdZxZS) GemmaRMSNormzRMS normalization for Gemma. Two differences from the above RMSNorm: 1. x * (1 + w) instead of x * w. 2. (x * w).to(orig_dtype) instead of x.to(orig_dtype) * w. r)r*r+r Ncttjt j||_||_dSr!)r0r1r5r6rzerosr r )r;r*r+r>s rr1zGemmaRMSNorm.__init__sD l5;{#;#;<< #rr r rc"|j}|}|ddd}|t j||zz}|d|zz}||}|S)zGPyTorch-native implementation equivalent to forward() without residual.rCrBTrD?)rrhrKrLrrMrG)r r rr?rOs r_forward_static_no_residualz(GemmaRMSNorm._forward_static_no_residuals~W GGII5588==R=66  H'7788 8 v||~~% & DD  rrc|j}|tjkr)||zn||z}|}|}|ddd}|tj||zz}|d|zz}||}||fS)zDPyTorch-native implementation equivalent to forward() with residual.rCrBTrDrv)rrr"rhrKrLrMrG)r r rrr?rOs r_forward_static_with_residualz*GemmaRMSNorm._forward_static_with_residual'sW U]** GGII(( ( (X  GGII5588==R=66  H'7788 8 v||~~% & DD  ({rc|&||jj|j|S||jj|j||S)rA)rwr rRr ryrSs rrTzGemmaRMSNorm.forward_native@s\  33  $"7 55  $"7H rc>tjr|||St |ddsCtj|j|_tj|j|_d|_|||S)N _is_compiledFT) rcompiler is_compilingrTgetattrcompilerwryr|rSs rrXzGemmaRMSNorm.forward_cudaOs > & & ( ( 4&&q(33 3t^U33 %/4}000D ,27222D .!%D ""1h///rr)r!)rcrdrerfrgrhr1rjrrkrwrlryrTrXrnros@rrrrrs$$$$  $$$$$$     <     \   <,  u|U\) * \6)-   < ,%  elEL89 9    $)-00 <0,%0 elEL89 9 00000000rrrrms_norm_gatedceZdZdZ ddedededzded ejdzd ej dzf fd Z d Z dd ej dej dzdej fdZ dd ej dej dzdej fdZxZS) RMSNormGatedzRMS Normalization with optional gating. This is a native PyTorch implementation that supports: - Standard RMS normalization - Group RMS normalization - Optional gating with SiLU activation h㈵>NFr*r+ group_sizenorm_before_gatedevicerc"||d}t||_tjt j|fi||_|dd||_ ||_ | dS)aInitialize RMSNormGated. Args: hidden_size: Size of the hidden dimension eps: Epsilon for numerical stability group_size: If not None, do GroupNorm with each group having group_size elements. group_size=None is equivalent to group_size=hidden_size (i.e. there's only 1 group). norm_before_gate: If True and z is provided: out = norm(x) * silu(z) If False and z is provided: out = norm(x * silu(z)) device: Device to create parameters on dtype: Data type for parameters )rrrN) r0r1r+r5r6remptyr register_parameterrrreset_parameters) r;r*r+rrrrfactory_kwargsr>s rr1zRMSNormGated.__init__os.%+U;; l5;{#M#Mn#M#MNN  ---$ 0 rcXtjj|jdSr!)rr5initones_r )r;s rrzRMSNormGated.reset_parameterss"  DK(((((rrzr c(||js|tj|z}|jT|ddd}|t j||jzz}||j z}nvddl m }||d|j }|ddd}|t j||jzz}||d |j z}||jr|tj|z}|S) a Native PyTorch implementation of RMS normalization with gating. Args: x: Input tensor z: Optional gating tensor Returns: Normalized (and optionally gated) tensor If z is not None: - norm_before_gate=True: out = norm(x) * silu(z) - norm_before_gate=False: out = norm(x * silu(z)) NrCrBTrDr) rearrangez... (g d) -> ... g d)dz... g d -> ... (g d)) rFsilurrKrLrrMr+r einopsr)r;rrrOx_normedrrx_groups rrTzRMSNormGated.forward_natives$ =!6=AF1II A ? "uuQxx}}T}::H5;x$(':;;;HT[(CC ) ( ( ( ( (i#9T_MMMG{{1~~**r4*@@HX-@!A!AAH)H&<== KC =T2=q /C rc dddlm}|||j|j||j|j|jS)Nr) rmsnorm_fn)rr+rr)2vllm.model_executor.layers.fla.ops.layernorm_guardrr rr+rr)r;rrrs rrXzRMSNormGated.forward_cudasP RQQQQQz K I!2    r)rNFNNr!)rcrdrerfrgrhrirrrr1rrkrTrXrnros@rrrcs<!%!&&*$(    $J     t#  {T!      @)))9=((("',"5( ((((V9=     "',"5           rrcDeZdZdZddedeffd ZdejfdZ xZ S) LayerNormz Layer Normalization. r)rEr+c@t||_||_t jt j|t j|_ t jt j |t j|_ dS)Nr/) r0r1rEr+r5r6rr4rHr rtr)r;rEr+r>s rr1zLayerNorm.__init__so l5:c#G#G#GHH LS !F!F!FGG rrctj||jf|j|j|j|Sr!)r layer_normrhrEr rr+type_as)r;rs rforwardzLayerNorm.forwards>| GGII{DKDH  '!** rr) rcrdrerfrgrhr1rrkrrnros@rrrsHHCHeHHHHHHrr)F)rfrtorch.nnr5torch.nn.functional functionalrvllm._aiter_opsrvllm.model_executor.custom_opr*vllm.model_executor.layers.batch_invariantrrvllm.platformsrrkrhrrlrrrirr&registerr(rrrModulerrrrs#" ******222222,+++++ |"\=B \" |l L   5< %& *  | "\ 16 QV  \    "AF!&9=(:ccccchcccN#$$Z0Z0Z0Z0Z08Z0Z0%$Z0|#$$e e e e e 8e e %$e P r