`iJdZddlZddlmZddlmZmZmZddlZddlm Z ddl m Z ddl m Z dd lmZdd lmZmZdd lmZdd lmZmZmZdd lmZmZmZddlmZeje Z! dode j"dej#dej#dej#deej#dee$de$deej#fdZ%Gdde j"Z&Gdde j"Z' dpdej#d e$d!ee(d"e)d#e*f d$Z+ dqdej#d%ee(e*fd!ee(d#e*fd&Z,Gd'd(e j"Z-Gd)d*e j"Z.Gd+d,e j"Z/eGd-d.eZ0Gd/d0e j"Z1Gd1d2e j"Z2Gd3d4e0Z3eed56Gd7d8eZ4eed96Gd:d;eZ5eed<6Gd=d>eZ6eed?6Gd@dAeZ7eedB6GdCdDeZ8eedE6GdFdGeZ9dHej:j;dIej#dJej#fdKZGdQdRe j"Z?GdSdTe j"Z@GdUdVe j"ZAeGdWdXe0ZBGdYdZe j"ZCed[6Gd\d]e0ZDGd^d_e j"ZEed`6Gdadbe0ZFedc6Gdddee j"ZGedf6Gdgdhe0ZHGdidje j"ZIedk6Gdldme0ZJgdnZKdS)szPyTorch PatchTST model.N) dataclass)CallableOptionalUnion)nn)ACT2CLS)FlashAttentionKwargs)BaseModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)NegativeBinomialOutput NormalOutputStudentTOutput) ModelOutputauto_docstringlogging)PatchTSTConfigmodulequerykeyvalueattention_maskscalingdropout head_maskc ||ddz}tj||dd|z} || |z} tj| d} || |ddddz} tj| ||j } tj| |} | dd } | | fS)Nrdimr)ptraining) sizetorchmatmul transposer functionalsoftmaxviewrr' contiguous) rrrrrrrrkwargs attn_weights attn_outputs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/patchtst/modeling_patchtst.pyeager_attention_forwardr4&s**R..D(<s}}Q':':;;gEL!#n4 =((2(>>L#innQAq&A&AA =((6?([[L,|U33K''1--88::K  $$c>eZdZdZ ddededed ed ed ed eeffd Z dde j dee j dee j dee j deede e dee j ee j eee j ffdZxZS)PatchTSTAttentionz=Multi-headed attention from 'Attention Is All You Need' paperrFTN embed_dim num_headsr is_decoderbias is_causalconfigc t||_||_||_||z|_||_|j|z|jkrtd|jd|d|jdz|_||_ ||_ tj ||||_ tj ||||_tj ||||_tj ||||_dS)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: ).r"r;)super__init__r8r9rhead_dimr= ValueErrorrr:r<rLineark_projv_projq_projout_proj) selfr8r9rr:r;r<r= __class__s r3rBzPatchTSTAttention.__init__Hs "" !Y.  MI %$. 8 83dn33%.333 }d* $"i 94@@@ i 94@@@ i 94@@@  )YTBBB r5 hidden_stateskey_value_statesrlayer_head_maskoutput_attentionsr0returnc |du}|jdd\}} |r |jdn| } || d|jf} || d|jf} ||j| dd} |r|n|}||j| dd}||j| dd}t}|jj dkrt|jj }||| |||f|j sdn|j |j ||d|\}}||| d}||}||dfS)z#Input shape: Batch x Time x ChannelNr!rr#eagerr)rrrOr)shaperCrHr.r+rFrGr4r=_attn_implementationr r'rrreshaper/rI)rJrLrMrrNrOr0is_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacer2r1s r3forwardzPatchTSTAttention.forwardgs.T9%*3B3/ W/AN"(++wgr4=9 wDM:7t{{=116 FPPQRTUVV -?R))]5T[[005~FPPQRTUVV 7t{{>227HRRSTVWXX (? ; +w 6 6"9$+:Z"[ $7$7      % $}>CC$,L/% % %  % % ! \"))#w;;FFHH mmK00 L$..r5)rFTFN)NNNF)__name__ __module__ __qualname____doc__intfloatboolrrrBr)Tensorrr tuplera __classcell__rKs@r3r7r7Es^GG  +/CCCC C  C  CC(CCCCCCD481526,1 3/3/|3/#5<03/!. 3/ "%,/ 3/ $D> 3/-.3/ u|Xel3XeEL>Q5RR S3/3/3/3/3/3/3/3/r5r7c>eZdZdZdeffd ZdejfdZxZ S)PatchTSTBatchNormzP Compute batch normalization over the sequence length (time) dimension. r=cttj|j|j|_dS)Neps)rArBr BatchNorm1dd_modelnorm_eps batchnormrJr=rKs r3rBzPatchTSTBatchNorm.__init__s7 FOLLLr5inputsc|dd}||}|ddS)a Parameters: inputs (`torch.Tensor` of shape `(batch_size, sequence_length, d_model)`): input for Batch norm calculation Returns: `torch.Tensor` of shape `(batch_size, sequence_length, d_model)` rr#)r+ru)rJrwoutputs r3razPatchTSTBatchNorm.forwards@!!!Q''''1%%%r5 rbrcrdrerrBr)rirarkrls@r3rnrnsrM~MMMMMM &el & & & & & & & &r5rnFrw mask_ratiounmasked_channel_indiceschannel_consistent_masking mask_valuec|dks|dkrtd|d|j\}}}}|j} t|d|z z} |r0t j|d|| } | d|d} nt j|||| } t j|||| } d| ddddd| f<t j| d} t j| d}t j | d| } | dddd|} |d| dd|ddddf<| | |}|| d fS) arandom_masking: Mask the input considering the control variables. Args: inputs (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, num_features)`): The input tensor to mask. mask_ratio (`float`): Masking ratio applied to mask the input data during random pretraining. It is the number between 0 and 1. unmasked_channel_indices (list, *optional*): Indices of channels that will not be masked. channel_consistent_masking (bool, *optional*, defaults to `False`): When true, masking will be same across all channels of a timeseries. Otherwise, masking positions will vary across channels. mask_value (int, *optional*, defaults to 0): Define the value of masked patches for pretraining. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as input Tensor and mask tensor of shape [bs x c x n] rrz Mask ratio z has to be between 0 and 1.deviceNr!r$)r%index.r) rDrSrrfr)randrepeatonesargsortgather unsqueeze masked_fillrh)rwr{r|r}r~ batch_size num_channelssequence_length num_featuresrlen_keepnoisemask ids_shuffle ids_restore inputs_masks r3random_maskingrs4A~~qNzNNNOOO>Dl;J o| ]F?a*n566H!U :q/&III Q a00 :|_VTTT :j, O O ODDAAAyy-2...K- 444K <"K 8 8 8D >>"   $ $Q1l ; ;D+23QQQ(!!!QQQ ./$$TYY[[*==K V $$r5num_forecast_mask_patchesct|tr|g}d|D}|j\}}}}tj||||j} g} d} t |} t||D]V\} }| dks| |krtd| dt||z| z }| | ||g| |z } Wt| d} | |kr| dd|| z z| dd<n#| |kr| d d| |z z| d d<d}| D]\}}}||z}d | ||d d | d f<|}tj | jd}| |} | d  d d d |} |d| d d |d d d d f<|| |}|| d fS) aForecast masking that masks the last K patches where K is from the num_forecast_mask_patches. If num_forecast_mask_patches is a list, samples in the batch will be randomly masked by numbers defined in the list. Parameters: inputs (`torch.Tensor`): Input of shape `(bs, num_channels, num_patch, patch_length)` num_forecast_mask_patches (`list`): Number of patches to be masked at the end of each batch sample. e.g. 4 or [3, 5]. unmasked_channel_indices (`list`, *optional*): Indices of channels that are not masked. mask_value (`int`, *optional*, defaults to 0): Values in the masked patches will be filled by `mask_value`. Returns: `tuple(torch.Tensor)`: inputs_mask, masked input, same shape as inputs Tensor and Mask tensor of shape `(bs, num_channels , num_patch)` or `(bs, tsg1, tsg2, num_channels, num_patch)` cg|]}dS)r).0_s r3 z$forecast_masking..sAAA!AAAAr5rrznum_forecast_mask_patches z6 should be greater than 0 and less than total patches.c|dS)Nr#r)xs r3z"forecast_masking..s !A$r5)rr#r!rNr) isinstancerfrSr)zerosrsumziprDappendsortedrandpermrrrrh)rwrr|r~forecast_mask_ratiosrrrrrt_list total_length total_ratio patch_lengthratiotemp_lenbatch1 patch_lenrbatch2permrs r3forecast_maskingrsU0+S11@%>$?!AA'@AAA>Dl;J o| ;z< W W WD FL*++K"#<>RSS!! e 1    ? ?q\qqq zE)K788 |UH5666 F / / /Fj  ay|zL'@Aq !  " "r 1  )BCr 1 F"( 1h("./VF]AAA z{{ *+ >$*Q- ( (D :D >>"   $ $Q1l ; ;D+23QQQ(!!!QQQ ./$$TYY[[*==K V $$r5c>eZdZdZdeffd ZdejfdZxZ S)PatchTSTPatchifyz A class to patchify the time series sequence into different patches Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` r=ct|j|_|j|_|j|_|j|jkr t d|jd|jdt|j|j|jz |jzdz|_|j|j|jdz zz}|j|z |_ dS)NzSequence length (z+) has to be greater than the patch length ()r) rArBcontext_lengthrr patch_striderDmax num_patchessequence_start)rJr=new_sequence_lengthrKs r3rBzPatchTSTPatchify.__init__9s %4"/"/  4#4 4 4yD$8yyeievyyy    4d6GHH4K\\aearruvv"/$2CtGWZ[G[2\\"25HHr5 past_valuesc,|jd}||jkrtd|d|jd|dd|jdddf}|d|j|j}|dd}|S)a! Parameters: past_values (`torch.Tensor` of shape `(batch_size, sequence_length, num_channels)`, *required*): Input for patchification Returns: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)` zInput sequence length (z%) doesn't match model configuration (r?N) dimensionr(step) rSrrDrunfoldrrr+r/)rJrrrys r3razPatchTSTPatchify.forwardJs&+B/ d2 2 2x/xx`d`txxx QQQ 3 5 5qqq89$2C$J[\\!!"b))4466 r5rzrls@r3rr1srI~IIIIII"5<r5rc>eZdZdZdeffd ZdejfdZxZ S)PatchTSTMaskinga Class to perform random or forecast masking. Parameters: config (`PatchTSTConfig`): model config Returns: x_mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points r=ct|j|_|j|_|j|_|j|_|j|_|j|_|jt|j|_dSdSN) rArBrandom_mask_ratior} mask_typerr|r~rrvs r3rBzPatchTSTMasking.__init__ns !'!9*0*K'))/)I&(.(G% +  ( 4,243P,Q,QD ) ) ) 5 4r5 patch_inputc2|jdkr,t||j|j|j|j\}}nI|jdkr&t ||j|j|j\}}ntd|jd| }||fS)a Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input Return: masked_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`) Masked patched input mask (`torch.Tensor` of shape `(batch_size, num_channels, num_patches)`) Bool tensor indicating True on masked points random)rwr{r|r}r~forecast)rwrr|r~zInvalid mask type .) rrrr|r}r~rrrDrh)rJr masked_inputrs r3razPatchTSTMasking.forwardys >X % %!/"1)-)F+/+J? """ L$$^z ) )!1"*.*H)-)F? """ L$$C$.CCCDD Dyy{{T!!r5rzrls@r3rrasr   R~ R R R R R R!"5<!"!"!"!"!"!"!"!"r5rcPeZdZdZdeffd Zddejdee fdZ xZ S) PatchTSTEncoderLayerz PatchTST encoder layer r=c  t|j|_t|j|j|j||_|jdkrtj |jntj |_ |j dkrt||_nH|j dkr&tj|j|j|_nt%|j d|jr|jdkrtj |jntj |_|j dkrt||_nH|j dkr&tj|j|j|_nt%|j dtjtj|j|j|jt3|j|jdkrtj |jntj tj|j|j|j|_|jdkrtj |jntj |_|j dkrt||_nH|j dkr&tj|j|j|_nt%|j d|j|_dS)N)r8r9rr=rru layernormrpz$ is not a supported norm layer type.r@) rArBchannel_attentionr7rsnum_attention_headsattention_dropout self_attn path_dropoutrDropoutIdentity dropout_path1 norm_typernnorm_sublayer1 LayerNormrtrD dropout_path2norm_sublayer2 SequentialrEffn_dimr;r activation_function ff_dropoutff dropout_path3norm_sublayer3pre_normrvs r3rBzPatchTSTEncoderLayer.__init__s !'!9*n0,    AG@SVW@W@WRZ(;<<<]_]h]j]j  { * *"3F";";D     , ,"$,v~6?"S"S"SD   0VVVWW W  ! \DJDWZ[D[D[F,?!@!@!@acalananD ;..&7&?&?##![00&(l6>v&W&W&W## F$4!Z!Z!Z[[[- Ifnfn6; G G G F. / 1 1-3->-B-BBJv( ) ) ) Ifnfn6; G G G   AG@SVW@W@WRZ(;<<<]_]h]j]j  { * *"3F";";D     , ,"$,v~6?"S"S"SD   0VVVWW W r5N hidden_staterOc~|j\}}}}|||z||}|jrG||||\}}} |||z}nF|||\}}} ||||z}|||||}|jr|dd }|||z||}|jrG|| ||\}} } || |z}nF|||\}} } | || |z}|||||}|dd }|||z||}|jr?|| | ||z}n>||| | |z}|||||}|f} |r| |jr|| fn|fz } | S)a Parameters: hidden_state (`torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)`, *required*): Past values of the time series output_attentions (`bool`, *optional*): Whether or not to return the output attention of all layers Return: `torch.Tensor` of shape `(batch_size, num_channels, sequence_length, d_model)` )rLrOr#r)rSr.rrrrrUrr+r/rrrrr) rJrrOrnum_input_channelsrrsr2r1rchannel_attn_weightsoutputss r3razPatchTSTEncoderLayer.forwards%DPCU@ &$((6H)H/[bcc = _+/>>"11,??Sd,:,, (Kq($*<*<[*I*IILL,0>>*>O,:,, (Kq ..|d>P>PQ\>]>]/]^^L$++J8JO]dee   ! E'11!Q77BBDDL',,Z/-IK]_fggL} c7;~~"&"5"5l"C"CWh8F884 11 ,d.@.@.M.MM 8<~~".BS8F884 11 $22<$BTBTU`BaBa3abb (// OM_ahiiL'11!Q77BBDDL$((6H)H/[bcc = i($*<*P>PQUQXQXYeQfQf>g>g/ghhL$++J8JO]dee /  k t?Uj &:;;\h[j jGr5r) rbrcrdrerrBr)rirrhrarkrls@r3rrs0(~0(0(0(0(0(0(dQQELQXd^QQQQQQQQr5rcDeZdZUeed<dZdZdZdej fdZ d dZ dS) PatchTSTPreTrainedModelr=modelrFrct|trt|jj|jj|jjz |jjzdz}|jjr+tj |j d|dz }| |j||_ dSt|tjr?|jj|jjddSt|t(rI|jjj|jjjddSt|tjrR|jj d|jj|j"|jjdSdSdS)z$ Initialize weights rg{Gz?)std?r)meanrN)rPatchTSTPositionalEncodingrr=rrr use_cls_tokenrinitnormal_ cls_token_init_pe position_encrr;datazero_weightfill_rnrurEinit_std)rJrrs r3 _init_weightsz%PatchTSTPreTrainedModel._init_weights/s f8 9 9 )DK. 0HIIDKLdd)*,-.K{( ! 0d;;;q "(//$+{"K"KF     - - ) K  " " $ $ $ M  $ $S ) ) ) ) )  1 2 2 )   ! & , , . . .   # ( . .s 3 3 3 3 3  * * ) M  & &CT[5I & J J J{&  &&((((( ) )&&r5cBt|tr ||_dSdSr)rPatchTSTEncodergradient_checkpointing)rJrrs r3_set_gradient_checkpointingz3PatchTSTPreTrainedModel._set_gradient_checkpointingIs, f 0 0 2,1F ) ) ) 2 2r5N)F) rbrcrdr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointingrModulerrrr5r3rr(sd #O&+#)BI))))4222222r5rc:eZdZdeffd ZdejfdZxZS)PatchTSTEmbeddingr=ct|j|_|j|_|jr&t j|j|j|_dSt j |_t|jD]9}|j t j|j|j:dSr) rArBrshare_embeddingrrErrsinput_embedding ModuleListranger)rJr=rrKs r3rBzPatchTSTEmbedding.__init__Os "(";%5   \#%9V-@&.#Q#QD #%=??D 6455 \ \$++BIf6I6>,Z,Z[[[[ \ \r5rcjd}|jkrtdjd|djr}n2fdt |D}t j|d}|S)a% Parameters: patch_input (`torch.Tensor` of shape `(batch_size, num_channels, num_patches, patch_length)`, *required*): Patch input for embedding return: `torch.Tensor` of shape `(batch_size, num_channels, num_patches, d_model)` rz&The defined number of input channels (zQ) in the config has to be the same as the number of channels in the batch input (rc \g|](}j|dd|ddddf)Sr)r)rirrJs r3rz-PatchTSTEmbedding.forward..msEnnnq1$.q1+aaaAAAqqqj2IJJnnnr5r$)rSrrDrrrr)stack)rJrr embeddingss`` r3razPatchTSTEmbedding.forward[s).q1 !8 8 8j9PjjTfjjj    8--k::JJnnnnnTYZlTmTmnnnJZQ777Jr5 rbrcrdrrBr)rirarkrls@r3r r Nsh \~ \ \ \ \ \ \5<r5r cpeZdZdZdedeffd Zedededej fdZ de j fdZ xZS) rz' Class for positional encoding r=rct|j|_|j|_|jr8t jt jddd|j|_ |dz }| |||_ |j dkrt j |j nt j|_ dS)Nrr)rArBrrr Parameterr)rrsrrrpositional_dropoutrrrJr=rrKs r3rBz#PatchTSTPositionalEncoding.__init__ws #1"(";   \%+aAv~*N*NOODN 1 K MM&+>>6<5NQR5R5RBJv0 1 1 1XZXcXeXe r5rPc|jdkr0tjtj||jd}n:|jdkrtj||j}tjd|d}tj tjd|jdtj d|jz z}tj ||z|dddddf<tj ||z|dddddf<||z }||d zz }tj|d }nt!|jd |S) NrT requires_gradsincosrrr#g@ FzN is not a valid positional encoder. Available types are 'random' and 'sincos'.)positional_encoding_typerrr)randnrsrarangerexpmathlogsincosrrrD)r=rrpositiondiv_terms r3rz#PatchTSTPositionalEncoding._init_pesy  *h 6 6< K(P(P`deeeLL  , 8 8 ;{FNCCL|A{33==a@@Hya!C!CQXHYHY\b\jHjFk!kllH$)Ih.A$B$BLADqD !$)Ih.A$B$BLADqD !',*;*;*=*==L'<+;+;+=+=+BCL< EJJJLL2CCC r5rcX|jr|||jddddfz}|j|jddddfz}||jd|jdd}tj||fd}n|||jz}|S)Nrrr!r#r$) rrrrexpandrSrr)cat)rJrr cls_tokensrs r3raz"PatchTSTPositionalEncoding.forwards   T11+@QRSRTRTVWVWVWRW@X2XYYK):2A2qqq5)AAI"))+*;A*>@WY[]_``J 9j+%>AFFFLL 22;AR3RSSLr5)rbrcrdrerrfrB staticmethodrrrr)rirarkrls@r3rrrs  ~  C       cbl\& 5<        r5rc leZdZdZdedeffd Z d dejde e de e d e fd Z xZ S) rz PatchTST Encoder r=rcBtd|_t|_t ||_tjfdtj D|_ | dS)NFc.g|]}tSr)r)rrr=s r3rz,PatchTSTEncoder.__init__..s"$k$k$ka%9&%A%A$k$k$kr5) rArBrr embedderrpositional_encoderrrrnum_hidden_layerslayers post_initrs ` r3rBzPatchTSTEncoder.__init__s    &+#*&11 "