`i XddlZddlmZddlmZddlmZddlmZm Z e j e Z erddl Z dZ d"deed ed d eed ed effdZ d"deed ed d eed ed effdZ d"deed ed d eed ed effdZ d#ded d d eed ed effdZ d#ded d d eed ed effdZ d#ded d d eed ed effdZeeeeeedZ d$dedededeedeef dZd#dedeefdZd#dedeefdZd#dedeefdZd#dedeefdZd#dedeefdZ d#dedeefd Z!eeeee e!dZ"d#dedeefd!Z#dS)%Nwraps)Optional)PretrainedConfig)is_torch_availableloggingcPddtfd}|S)ad Decorator function to update the RoPE parameters in the forward pass, if the model is using a dynamic RoPE (i.e. a RoPE implementation that may recompute its frequencies in the forward pass). Args: rope_forward (Callable): The forward pass of the RoPE implementation. Returns: The decorated forward pass. ctj|dz}t|jdr |jj}n |jj}||krWt|ds(||j||dz\|_}|d|jddS|j ||_ |d|j ddS) zbLongrope uses long factor if sequence is larger than original pretraining length, short otherwise.r original_max_position_embeddings long_inv_freqseq_leninv_freqF persistentN) torchmaxhasattrconfigr max_position_embeddings rope_init_fnr register_bufferoriginal_inv_freqto)self position_idsdevicerr _s t/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/modeling_rope_utils.pylongrope_frequency_updatez6dynamic_rope_update..longrope_frequency_update+s)L))A- 4; B C C S/3{/[ , ,/3{/R , 5 5 5411 (,(9(9K1QTU1U):))%"A  T-?E R R R R R&*%;%>%>v%F%FD "  T-CPU V V V V Vctj|dz}||jkrD||j||\}|_|d|d||_||jkrZ|j|jkrL|j ||_|d|jd|j|_dSdSdS)a dynamic RoPE layers should recompute `inv_freq` in the following situations: 1 - growing beyond the cached sequence length (allow scaling) 2 - the current sequence length is in the original scale (avoid losing precision with small sequences) rrrFrN) rrmax_seq_len_cachedrrattention_scalingroriginal_max_seq_lenrr)rrrrrs r dynamic_frequency_updatez5dynamic_rope_update..dynamic_frequency_update>s )L))A- T, , ,/3/@/@f^e/@/f/f ,Hd,  X% H H H&-D # T. . .43JTMf3f3f&*%;%>%>v%F%FD "  T-CPU V V V&*&?D # # # / .3f3fr"cd|jvr|||jn|jdkr|||j|||S)Ndynamic)rlongrope) rope_typer)rxrr'r! rope_forwards r wrapperz$dynamic_rope_update..wrapperQsh  & & $ $T< I I I I I ^z ) ) % %dL J J J J|D!\222r"r)r-r.r'r!s` @@r dynamic_rope_updater/shWWW&@@@& <3333333 Nr"rrz torch.devicerreturnz torch.Tensorc6|j}t|dd}t|ddp|j|jz}t ||z}d}d|t jd|dt j|t j |z zz }||fS) a  Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). partial_rotary_factor?head_dimNrdtyperr7) rope_thetagetattr hidden_sizenum_attention_headsintrarangeint64rfloat) rrrbaser2r4dimattention_factorrs r _compute_default_rope_parametersrD\s>  D#F,CSIIvz400dF4F&Jd4dH h.. / /Cdu|AsAU[IIILLTZbgbmLnnqttuvH % %%r"cV|jd}t|||\}}||z}||fS)a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). factor) rope_scalingrD)rrrrFrrCs r '_compute_linear_scaling_rope_parametersrHsD>  *F"B&&RY!Z!ZH  H % %%r"cV|j}t|dd}t|d|j|jz}t ||z}|j}|jd}d} ||}n_t|tj r5tj |tj ||j |j }nt||}|||z|z |dz z ||dz z zz}d|tjd |dtj |tj |z zz } | | fS) a Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The default sequence length used to update the dynamic RoPE at inference time * rope_scaling (`dict[str, float]`): The standard RoPE scaling parameters, from which `factor` will be accessed. The value of `factor` is used to determine the new base frequency, along with the current sequence length (seq_len), the maximum positional embeddings (max_position_embeddings), and the computed dimensionality (dim) of the rotary embeddings. If seq_len <= max_position_embeddings, this factor has no effect. If seq_len <= max_position_embeddings, this factor effectively stretches the context window using an exponent derived from `dim`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. Defaults to 1.0. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. If `None` or shorter than max_position_embeddings, this value will be overridden by max_position_embeddings. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). r2r3r4rFNr7rrr5rr6r8)r9r:r;r<r=rrG isinstancerTensormaximumtensorr7rrr>r?rr@) rrrrAr2r4rBrrFrCrs r _compute_dynamic_ntk_parametersrOsNT  D#F,CSIIvz6+=A[+[\\H h.. / /C$<   *F) GU\ * *8-  L0 gn ] ] ]   g677 FW$'>>6A:NTW[^ab[bTcd dDdu|AsAU[IIILLTZbgbmLnnqttuvH % %%r"c|j}t|dd}t|d|j|jz}t ||z}|jd}|jd}|jd} |jd} |jdp|j} dd } |6| r)| r't| || | || z }n | |}|jd pd } |jdpd }dfd}d}|tj d|d |tj|z z}d|z }d||zz }|jdd}|| |||| |\}}d ||||dz |tjz }|d |z z||zz}||fS)ak Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://huggingface.co/papers/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * rope_scaling (`dict[str, float | int]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied to the computed cos/sin. If None, the value is inferred from `factor`, `mscale`, and `mscale_all_dim` as avaialble. * `beta_fast` (`float`, *optional*, defaults to 32): Parameter to set the boundary for extrapolation (only) in the linear ramp function. * `beta_slow` (`float`, *optional*, defaults to 1): Parameter to set the boundary for interpolation (only) in the linear ramp function. * `factor` (`float`, *optional*): The scaling factor applied when interpolating the position IDs to extend the possible context length. Additionally, if `attention_factor` is None, the log of this value is used to compute a value for `attention_factor`, possibly in conjunciton with `mscale` and `mscale_all_dim`, if provided. * `mscale` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale` acts scalar augmenting `log(factor)` when computing the numerator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `mscale_all_dim` (`float`, *optional*): If `attention_factor` is None and both `mscale` and `mscale_all_dim` are provided, `mscale_all_dim` acts scalar augmenting `log(factor)` when computing the denominator for the inferred value of `attention_factor`. If not provided, `attention_factor` will be calculated based on `factor` only. * `original_max_position_embeddings` (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, the function falls back to `max_position_embeddings`. * `truncate` (`bool`, *optional*): Whether to truncate the correction range. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r2r3r4rFrCmscalemscale_all_dimr rcL|dkrdSd|ztj|zdzS)Nrr3g?)mathlog)scalerQs r get_mscalez,_compute_yarn_parameters..get_mscale:s, A::3V|dhuoo-33r"N beta_fast beta_slowc|tj||dztjzz zdtj|zz S)zPInverse dimension formula to find the dimension based on the number of rotationsr5)rTrUpi) num_rotationsrBrArs r find_correction_dimz5_compute_yarn_parameters..find_correction_dimLsAdh6-!:Kdg:UVWWW\]`d`him`n`n\noor"c||||}||||}|r(tj|}tj|}t|dt ||dz fS)z.Find dimension range bounds based on rotationsrr)rTfloorceilrmin) low_rothigh_rotrBrArtruncatelowhighr^s r find_correction_rangez7_compute_yarn_parameters..find_correction_rangePst!!'36MNN""8S$8OPP  #*S//C9T??D3{{CcAg....r"c||kr|dz }tj|tj|z ||z z }tj|dd}|S)NgMbP?r6rr)rr>float32clamp)rbrrB linear_func ramp_funcs r linear_ramp_factorz4_compute_yarn_parameters..linear_ramp_factorYsQ #:: 5LC|Cu}===Cc R K Q22 r"rr5r8reT)r) r9r:r;r<r=rGgetrr@rr>r)rrrrAr2r4rBrFrCrQrRr rWrXrZrhrn pos_freqsinv_freq_extrapolationinv_freq_interpolationrerfrginv_freq_extrapolation_factorrr^s @r _compute_yarn_parametersrtsp  D#F,CSIIvz6+=A[+[\\H h.. / /C   *F*../ABB  $ $X . .F(,,-=>>N BCCevGe%4444   2n 2$ZZ%?%?**VUcBdBd%dee  )z&11 #'' 44:I#'' 449Ippp/////aa003363UUX[[\I 9_ FY$67"&&z488H%%iCGgiqrrIC%&(:(:3cQh(O(O(R(RZ`hmhs(R(t(t$t!!&C"CD #@ @ A  % %%r"c.|j}t|dd}t|d|j|jz}t ||z}|jd}|jd}|jd} |jd} t|dd x} r |j| z } n|j} | G| dkrd} n>tj d tj | tj | z z} |r(|| kr"tj |tj | } n!tj |tj | } tjd |d tj| |z } d| || zzz }|| fS)a Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. This function assumes that the config will provide at least the following properties: * rope_theta (`float`): The base wavelength from which the inverse frequencies will be derived. * hidden_size (`int`): The numerator when deriving a head_dim, if not provided directly. * num_attention_heads (`int`): The denominator when deriving a head_dim, if not provided directly. * max_position_embeddings (`int`): The maximum length of the positional embeddings. * original_max_position_embeddings (`int`, *optional*): The original max position embeddings used during pretraining. If not provided, defaults to `max_position_embeddings`. * rope_scaling (`dict[str, float]`): The standard RoPE scaling parameters, from which the following keys will be accessed: * `attention_factor` (`float`, *optional*): The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, inferred from the value of `factor`. * `factor` (`float`, *optional*): The scaling factor to apply to the RoPE embeddings. If both `max_position_embeddings` and `original_max_position_embeddings` are provided, this value will be overridden s the ratio between those values. * `long_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is provided and greater than `original_max_position_embeddings`. * `short_factor` (`float`, *optional*): The scale factor applied when computing the inverse frequencies if `seq_len` is None or less-than-or-equal-to `original_max_position_embeddings`. Additionally, this function will make use of the following properties if they are found in the config: * head_dim (`int`, *optional*): The size of the key-value heads in the model. If None, this value will be derived as hidden_size // num_attention_heads. * partial_rotary_factor (`float`, *optional*, defaults to 1.0): If less than 1.0, inverse frequencies will be returned for the first fraction of the head_dim. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r2r3r4 long_factor short_factorrFrCr NrrJrr5)r9r:r;r<r=rGrorrTsqrtrUrrNrjr>r?r@)rrrrAr2r4rBrvrwrFrCr ext_factorsinv_freq_shapers r _compute_longrope_parametersr{ss^  D#F,CSIIvz6+=A[+[\\H h.. / /C%m4K&~6L  $ $X . .F*../ABB ,36;]_c+d+dd'J/2RR+1+I( S=="  #yTXf-=-=Ii@j@j-j)jkk U7===l;emFSSS l)*LMO&8'*::$'kH$G[+;!;X=NPXYYN$w.@EUXgEghM]*n= 1, got rGrorrrrKr@rr)rrrGr+rrrFs r (_validate_linear_scaling_rope_parametersr8s&L  l.>.>vt.L.LMMI (+M ))++,,MM=kZZZZ ( #F ~Z66~&3,,ZRXZZ[[[[[;G,r"ch|j}|d|dd}ddh}dh}t|}t ||||||d}|t |t r|dkrtd|dSdS)Nr+rrFr rr3rr)rrrGr+rrrrFs r )_validate_dynamic_scaling_rope_parametersrDs&L  l.>.>vt.L.LMMI (+M78M ))++,,MM=-]hiiii ( #F ~Z66~&3,,ZRXZZ[[[[[;G,r"c Z|j}|d|dd}ddh}hd}t|}t ||||||d}|t |t r|dkrtd||d}|8t |t r|d krtd ||d } | 2t | t std | |d } | 2t | t std| | pd| pdkr!td| d| d|jd} | 8|j | z } | |kr&t d|d| d|ddSdSt ddS)Nr+rrF>rQrerXrZrRrCr rr3rrCrL`rope_scaling`'s attention_factor field must be a float greater than 0, got rXz6`rope_scaling`'s beta_fast field must be a float, got rZz6`rope_scaling`'s beta_slow field must be a float, got rYrzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r zHThe explicitly set RoPE scaling factor (config.rope_scaling['factor'] = z) does not match the ratio implicitly set by other parameters (implicit factor = post-yarn context length / pre-yarn context length = config.max_position_embeddings / config.rope_scaling['original_max_position_embeddings'] = z). Using the explicit factor (z) in YaRN. This may cause unexpected behaviour in model usage, please correct the 'max_position_embeddings' fields in the model config.a~config.rope_scaling['original_max_position_embeddings'], the pre-yarn context length, is unset. We will **assume** config.max_position_embeddings holds the pre-yarn context length. Some use cases may expect config.max_position_embeddings to hold the post-yarn context length (pre-yarn context length * factor) -- we recommend updating both fields for optimal downstream model usage.) rGrorrrrKr@rrr warning_once) rrrGr+rrrrFrCrXrZr implicit_factors r _validate_yarn_parametersrRs&L  l.>.>vt.L.LMMI (+MM ))++,,MM=-]hiiii ( #F ~Z66~&3,,ZRXZZ[[[#''(:;;#Z8H%-P-P#TdghThTh m[k m m     --IZ 5%A%A[PY[[\\\  --IZ 5%A%A[PY[[\\\RIN++ Z^g Z Z6? Z Z Z   (.':'>'>?a'b'b$'3 8;[[ f $ $   u[auu# uuCI uuu      % $  _     r"c|j}|d|dd}hd}hd}t|}t |||||t |dd}t |d|j|jz}t||z} |d } t| ts6td | Drt d | t| | d zkr0t d | d zdt| |d} t| ts6td| Drt d| t| | d zkr0t d| d zdt| t|drtddS|d} | t dn8t| t"r| dkrt d| |d} | :t| t"r| dkr!t d| dSdSdS)Nr+r>r+rvrw>rFrCr rr2r3r4rwc3NK|] }t|ttfV!dSNrKr=r@.0r,s r z0_validate_longrope_parameters..s11d1dRS*Qe 2M2M1d1d1d1d1d1dr"zC`rope_scaling`'s short_factor field must be a list of numbers, got r5z5`rope_scaling`'s short_factor field must have length z, got rvc3NK|] }t|ttfV!dSrrrs r rz0_validate_longrope_parameters..s10b0bQRAU|1L1L0b0b0b0b0b0br"zB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length r aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.rFz1Missing required keys in `rope_scaling`: 'factor'rrCgr)rGrorrrr:r;r<r=rKlistallrrlenrrr@)rrrGr+rrrr2r4rBrwrvrFrCs r _validate_longrope_parametersrs'&L  l.>.>vt.L.LMMI@@@MVVVM ))++,,MM=-]hiiii#F,CSIIvz6+=A[+[\\H h.. / /C##N33L lD ) )mc1d1dWc1d1d1d.d.dmk]ikklll <C1H$$rsVWxrr_bco_p_prrsss""=11K k4 ( (kS0b0bVa0b0b0b-b-bki\giijjj ;3!8##pcUVhpp^abm^n^nppqqq v9:: A     !!(++ > NNN O O O OFE** `fsll NN^V\^^ _ _ _'++,>??  '.66 :JS:P:Pucsuu ( ':P:Pr"ct|j}|d|dd}hd}t|}t |||||d}|t |t r|dkrtd||d}|d }|t |t std ||t |t std |||kr td |d ||d} | t | tstd| | |j kr'td| d|j dSdS)Nr+r>rFr+r}r~r rrFr3rr}r~z<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=r zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) rGrorrrrKr@rrr=r) rrrGr+rrrFr}r~r s r _validate_llama3_parametersrs&L  l.>.>vt.L.LMMIvvvM ))++,,MM=kZZZZ ( #F ~Z66~&3,,ZRXZZ[[["#45O#$67j%&H&HgVegghhhz2BE'J'JiWgiijjj?** H H H6E H H   (44V'W$'/zBbdg7h7h/ 2/ 2 2   (6+III o/ o oNTNl o o     JIr"ct|dd}|dS|d|dd}t|}||||dStd|ddS) zO Validate the RoPE config arguments, given a `PretrainedConfig` object rGNr+rrrzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')r:roROPE_VALIDATION_FUNCTIONSrr)rrrGr+ validation_fns r rope_config_validationrs6>488L  l.>.>vy.Q.QRRI-11)<rs 111111........  H % %LLL;;;~*.'+!(&(& % &(& ^ $(&c](& >5 ! (&(&(&(&X*.'+!(&(& % &(& ^ $(&c](& >5 ! (&(&(&(&X*.'+!A&A& % &A& ^ $A&c]A& >5 ! A&A&A&A&JPTz&z& z&&4z&?G}z& >5 !z&z&z&z&|PTO&O& O&&4O&?G}O& >5 !O&O&O&O&fPT>,>, >,&4>,?G}>, >5 !>,>,>,>,J05. $,( $(!% lllllC= l # llll:[[.>[XVY][[[[ \ \5E \T\]`Ta \ \ \ \ \ \6F \U]^aUb \ \ \ \? ? &6? Xc]? ? ? ? D//*:/RU////d! ! (8! xPS}! ! ! ! L168 %-)   #3 (3-      r"