`ibdZddlZddlmZddlmZddlmZmZddl Z ddl m Z ddl m Z m Z mZdd lmZmZerdd lmZdd lmZdd lmZdd lmZmZmZmZmZmZmZm Z m!Z!ddl"m#Z#ddl$m%Z%ddlm&Z&ddl'm(Z(e&j)e*Z+dZ,dZ-dZ.Gdde j/Z0Gdde j/Z1Gdde j/Z2Gdde j/Z3Gdde j/Z4Gd d!e j/Z5Gd"d#eZ6Gd$d%e j/Z7Gd&d'e j/Z8Gd(d)e j/Z9Gd*d+e j/Z:Gd,d-e j/Z;Gd.d/e j/Z<Gd0d1e j/Z=eGd2d3e#Z>eed45Gd6d7eZ?eGd8d9e>Z@ed:5Gd;dZAeGd=d>e>ZBed?5Gd@dAe>ZCedB5GdCdDe>ZDeGdEdFe>ZEeGdGdHe>ZFeGdIdJe>ZGgdKZHdS)LzPyTorch FNet model.N) dataclass)partial)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)auto_docstringis_scipy_available)linalg)ACT2FN)GradientCheckpointingLayer) BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutput ModelOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward)logging) FNetConfigc|jd}|d|d|f}|tj}tjd|||S)z4Applies 2D matrix multiplication to 3D input arrays.rNzbij,jk,ni->bnk)shapetypetorch complex64einsum)xmatrix_dim_onematrix_dim_two seq_lengths z/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/fnet/modeling_fnet.py_two_dim_matmulr*7sNJ#KZK*$<=N uA <(!^^ L LLc$t|||SN)r*)r%r&r's r)two_dim_matmulr.@s 1nn = ==r+c|}tt|jddD]#}tj||}$|S)z Applies n-dimensional Fast Fourier Transform (FFT) to input array. Args: x: Input n-dimensional array. Returns: n-dimensional Fourier transform of input n-dimensional array. rN)axis)reversedrangendimr"fft)r%outr0s r)fftnr6EsO Cqvqrr*++,,immCdm++ Jr+c*eZdZdZfdZddZxZS)FNetEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|j|_tj|j|_|dt)j|jdd|dt)j|jt(jddS) N) padding_idxeps position_ids)rF) persistenttoken_type_idsdtype)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsLinear projectionDropouthidden_dropout_probdropoutregister_bufferr"arangeexpandzerosr=sizelongselfconfig __class__s r)rDzFNetEmbeddings.__init__XsP !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2H&J\%]%]"f&8f>STTT)F$68JKKz&"<==   EL)GHHOOPWXXej      ek$*;*@*@*B*B%*UUUbg      r+Nc||}n|dd}|d}||jddd|f}|mt|dr2|jddd|f}||d|}|}n+t j|t j|jj}|| |}| |} || z} | |} | | z } | | } | | } || } | S)Nr>rr@rrBdevice)rYr=hasattrr@rWr"rXrZrarIrMrKrNrQrT) r\ input_idsr@r= inputs_embeds input_shaper(buffered_token_type_ids buffered_token_type_ids_expandedrM embeddingsrKs r)forwardzFNetEmbeddings.forwardnse  #..**KK',,..ss3K ^  ,QQQ  ^ J J"%:: "66|DD)) ^^J// __Z00 \\*-- r+)NNNN)__name__ __module__ __qualname____doc__rDri __classcell__r^s@r)r8r8UsRQQ     ,!!!!!!!!r+r8c*eZdZfdZdZdZxZS)FNetBasicFourierTransformcrt||dSr-)rCrD_init_fourier_transformr[s r)rDz"FNetBasicFourierTransform.__init__s3  $$V,,,,,r+cl|js'ttjjd|_dS|jdkrtr|dtj tj |j tj |dtj tj |jtj tt|j|j|_dSt%jdt|_dSt|_dS) N)rdimdft_mat_hiddenrA dft_mat_seq)r&r'zpSciPy is needed for DFT matrix calculation and is not found. Using TPU optimized fast fourier transform instead.)use_tpu_fourier_optimizationsrr"r4r6fourier_transformrJr rUtensorrdftrGr#tpu_short_seq_lengthr.rzryrwarning)r\r]s r)rsz1FNetBasicFourierTransform._init_fourier_transforms*3 *%,UY^%H%H%HD " " "  +t 3 3!## .$$$el6:f>P3Q3QY^Yh&i&i&i$$!5< 6;V0W0W_d_n#o#o#o*1"43CTXTg***&&&**.&&&%)D " " "r+c<||j}|fSr-)r|real)r\ hidden_statesoutputss r)riz!FNetBasicFourierTransform.forwards" ((77<zr+)rjrkrlrDrsrirnros@r)rqrqsV-----***.r+rqc$eZdZfdZdZxZS)FNetBasicOutputcttj|j|j|_dSNr;)rCrDrrNrGrOr[s r)rDzFNetBasicOutput.__init__s9 f&8f>STTTr+c6|||z}|Sr-)rNr\r input_tensors r)rizFNetBasicOutput.forwards|m'CDD r+rjrkrlrDrirnros@r)rrsLUUUUUr+rc$eZdZfdZdZxZS)FNetFourierTransformctt||_t ||_dSr-)rCrDrqr\routputr[s r)rDzFNetFourierTransform.__init__s; -f55 %f-- r+cn||}||d|}|f}|SNr)r\r)r\r self_outputsfourier_outputrs r)rizFNetFourierTransform.forwards7yy// \!_mDD!#r+rros@r)rrsG..... r+rcBeZdZfdZdejdejfdZxZS)FNetIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr-) rCrDrrPrGintermediate_sizedense isinstance hidden_actstrrintermediate_act_fnr[s r)rDzFNetIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r+rreturncZ||}||}|Sr-)rrr\rs r)rizFNetIntermediate.forwards, =11 00?? r+rjrkrlrDr"Tensorrirnros@r)rrs^99999U\elr+rcPeZdZfdZdejdejdejfdZxZS) FNetOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) rCrDrrPrrGrrNrOrRrSrTr[s r)rDzFNetOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r+rrrc||}||}|||z}|Sr-)rrTrNrs r)rizFNetOutput.forwards@ =11  ]33 }|'CDD r+rros@r)rrsi>>>>> U\RWR^r+rc*eZdZfdZdZdZxZS) FNetLayerct|j|_d|_t ||_t ||_t||_ dSNr) rCrDchunk_size_feed_forward seq_len_dimrfourierr intermediaterrr[s r)rDzFNetLayer.__init__s^ '-'E$+F33 ,V44 (( r+c||}|d}t|j|j|j|}|f}|Sr)rrfeed_forward_chunkrr)r\rself_fourier_outputsr layer_outputrs r)rizFNetLayer.forwardsN#||M::-a00  #T%A4CSUc   /r+c\||}|||}|Sr-)rr)r\rintermediate_outputrs r)rzFNetLayer.feed_forward_chunks0"//??{{#6GG r+)rjrkrlrDrirrnros@r)rrsV)))))   r+rc&eZdZfdZddZxZS) FNetEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0_r]s r) z(FNetEncoder.__init__.. s!#_#_#_!If$5$5#_#_#_r+F) rCrDr]r ModuleListr2num_hidden_layerslayergradient_checkpointingr[s `r)rDzFNetEncoder.__init__ s`  ]#_#_#_#_uVE]?^?^#_#_#_`` &+###r+FTc|rdnd}t|jD] \}}|r||fz}||}|d}!|r||fz}|std||fDSt||S)Nrrc3K|]}||V dSr-r)rvs r) z&FNetEncoder.forward..s"XXq!-----XXr+)last_hidden_stater) enumeratertupler)r\routput_hidden_states return_dictall_hidden_statesi layer_module layer_outputss r)rizFNetEncoder.forwards"6@BBD(44 - -OA|# I$58H$H!(L77M)!,MM  E 1]4D D  YXX]4E$FXXXXX XN_````r+)FTrros@r)rr sT,,,,, aaaaaaaar+rcBeZdZfdZdejdejfdZxZS) FNetPoolercttj|j|j|_tj|_dSr-)rCrDrrPrGrTanh activationr[s r)rDzFNetPooler.__init__&sC Yv163EFF '))r+rrcr|dddf}||}||}|Sr)rr)r\rfirst_token_tensor pooled_outputs r)rizFNetPooler.forward+s@+111a40 #566  66 r+rros@r)rr%s^$$$$$ U\elr+rcBeZdZfdZdejdejfdZxZS)FNetPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) rCrDrrPrGrrrrrtransform_act_fnrNrOr[s r)rDz$FNetPredictionHeadTransform.__init__6s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr+rrc||}||}||}|Sr-)rrrNrs r)riz#FNetPredictionHeadTransform.forward?s= =11 --m<< }55 r+rros@r)rr5scUUUUUU\elr+rc,eZdZfdZdZddZxZS)FNetLMPredictionHeadc:tt||_t j|j|j|_t j tj |j|_ |j |j_ dSr-) rCrDr transformrrPrGrFdecoder Parameterr"rXbiasr[s r)rDzFNetLMPredictionHead.__init__Gss 4V<<y!3V5FGG LV->!?!?@@  I r+cZ||}||}|Sr-)rrrs r)rizFNetLMPredictionHead.forwardRs*}55  ]33 r+rNc|jjjjdkr|j|j_dS|jj|_dS)Nmeta)rrrar!r\s r) _tie_weightsz!FNetLMPredictionHead._tie_weightsWs< <  # (F 2 2 $ DL    )DIIIr+)rN)rjrkrlrDrirrnros@r)rrFs[ & & & & & ********r+rc$eZdZfdZdZxZS)FNetOnlyMLMHeadcptt||_dSr-)rCrDr predictionsr[s r)rDzFNetOnlyMLMHead.__init__as/ /77r+c0||}|Sr-)r)r\sequence_outputprediction_scoress r)rizFNetOnlyMLMHead.forwardes ,,_==  r+rros@r)rr`sG88888!!!!!!!r+rc$eZdZfdZdZxZS)FNetOnlyNSPHeadcttj|jd|_dSNru)rCrDrrPrGseq_relationshipr[s r)rDzFNetOnlyNSPHead.__init__ls6  " &*>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r+N)rjrkrlr__annotations__base_model_prefixsupports_gradient_checkpointingrrr+r)rrs= &*#*****r+rz0 Output type of [`FNetForPreTraining`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dS)FNetForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsr) rjrkrlrmr rr" FloatTensorrrrrrrr+r)r r s  )-D(5$ %,,,59x 12999;?Xe&78???8z5You have to specify either input_ids or inputs_embedsrxzThe `tpu_short_seq_length` in FNetConfig should be set equal to the sequence length being passed to the model when using TPU optimizations.r@r`)rcr=r@rd)rrrr)r pooler_outputr)r]ruse_return_dict ValueErrorrYr{rrarbrhr@rWr"rXrZrrrr)r\rcr@r=rdrrre batch_sizer(rarfrgembedding_outputencoder_outputsrrs r)rizFNetModel.forwards&%9$D $+Jj &1%cdd d  "#..**K%0 "J  &',,..ss3K%0 "J TUU U K 5 d"" 0J>>;  &/%:!! @T  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z??%)' +   ,, !5#'   *!,8< 8O O444UY  J#]3oabb6II I)-')7    r+)T)NNNNNN)rjrkrlrmrDrrr rr" LongTensorrboolrrrrirnros@r)rrs$ ///00015593759/3&*C C E,-C !!12C u/0 C   12 C 'tn C d^C  uo% &C C C ^C C C C C r+rz FNet Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. c,eZdZddgZfdZdZdZe ddee j dee j d ee j d ee j d ee j d ee j d ee dee de e effdZxZS)FNetForPreTrainingcls.predictions.decoder.biascls.predictions.decoder.weightct|t||_t ||_|dSr-)rCrDrrrclsrr[s r)rDzFNetForPreTraining.__init__!sQ    f%% '// r+c$|jjjSr-r,rrrs r)get_output_embeddingsz(FNetForPreTraining.get_output_embeddings*x#++r+cT||jj_|j|jj_dSr-r,rrrr\new_embeddingss r)set_output_embeddingsz(FNetForPreTraining.set_output_embeddings-%'5$$2$7!!!r+Nrcr@r=rdlabelsnext_sentence_labelrrrc ||n |jj}|||||||} | dd\} } || | \} } d}||t }|| d|jj|d}|| dd|d}||z}|s| | f| ddz}||f|zn|St|| | | jS)aH labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForPreTraining.from_pretrained("google/fnet-base") >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```Nr@r=rdrrrur>)r rrr) r]r rr,r viewrFr r)r\rcr@r=rdr7r8rrrrrrr total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r)rizFNetForPreTraining.forward1s_J&1%rur logitsr) r]r rr,r r;rFrr)r\rcr@r=rdr7rrrrrr>r=rs r)rizFNetForMaskedLM.forwards"&1%:K[b[pqqqqr+NNNNNNN)rjrkrlrArDr/r5r rr"rr&rrrrirnros@r)rCrCws28:Z[,,,888-115/304)-/3&*'r'rEL)'r!.'ru|, 'r   - 'r & 'r'tn'rd^'r un$ %'r'r'r^'r'r'r'r'rr+rCzT FNet Model with a `next sentence prediction (classification)` head on top. ceZdZfdZe d deejdeejdeejdeejdeejdeed eed e e e ffd Z xZ S) FNetForNextSentencePredictionct|t||_t ||_|dSr-)rCrDrrrr,rr[s r)rDz&FNetForNextSentencePrediction.__init__rEr+Nrcr@r=rdr7rrrc d|vr/tjdt|d}||n |jj}|||||||} | d} || } d} |At} | | dd| d} |s| f| ddz}| | f|zn|St| | | j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring). Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Example: ```python >>> from transformers import AutoTokenizer, FNetForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/fnet-base") >>> model = FNetForNextSentencePrediction.from_pretrained("google/fnet-base") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> logits = outputs.logits >>> assert logits[0, 0] < logits[0, 1] # next sentence was random ```r8zoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nr:rr>rurI) warningswarn FutureWarningpopr]r rr,r r;rr)r\rcr@r=rdr7rrkwargsrrseq_relationship_scoresr?r=rs r)riz%FNetForNextSentencePrediction.forwards:H !F * * M%    ZZ 566F%0% 1` a classification loss is computed (Cross-Entropy). Nr:r regressionsingle_label_classificationmulti_label_classificationr>rurI)r]r rrTr[ problem_typerZrBr"rZintr squeezer r;rrr)r\rcr@r=rdr7rrrrrJr r=rs r)riz%FNetForSequenceClassification.forward"s"&1%r:rurI) r]r r r;rYrrTr[r rr)r\rcr@r=rdr7rr num_choicesrrrJreshaped_logitsr r=rs r)rizFNetForMultipleChoice.forwardksR&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  )) )%'!5#      ]33 // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E(d?ZaZoppppr+rK)rjrkrlrDr rr"rr&rrrrirnros@r)rdrd_s-115/304)-/3&*LqLqEL)Lq!.Lqu|, Lq   - Lq & Lq'tnLqd^Lq u// 0LqLqLq^LqLqLqLqLqr+rdceZdZfdZe d deejdeejdeejdeejdeejdeed eed e e e ffd Z xZ S) FNetForTokenClassificationc6t||j|_t||_t j|j|_t j |j |j|_ | dSr-rYr[s r)rDz#FNetForTokenClassification.__init__sy     +f%% z&"<== )F$68IJJ r+Nrcr@r=rdr7rrrc||n |jj}|||||||}|d} || } || } d} |Ft } | | d|j|d} |s| f|ddz} | | f| zn| St| | |j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nr:rr>rurI) r]r rrTr[r r;rZrr)r\rcr@r=rdr7rrrrrJr r=rs r)riz"FNetForTokenClassification.forwards&1%rv) ignore_indexru)r  start_logits end_logitsr) r]r rrqsplitrb contiguouslenrYclampr rr)r\rcr@r=rdrrrsrrrrrJrvrwr< ignored_indexr= start_lossend_lossrs r)riz FNetForQuestionAnswering.forwards&1%rsX!!!!!!"""""""" AAAAAAAAAA77777777!!!!!!999999                      .-----666666******  H % %MMM>>>    :::::RY:::z##### ###Lbi     29        ry           *6aaaaa")aaa8         ")"*****29***4!!!!!bi!!!&&&&&bi&&& 9 9 9 9 929 9 9 9*****/***.  ====={==   =$b b b b b #b b b J V V V V V ,V V  V r;r;r;r;r;r);r;r;r| R R R R R $7R R  R j FgFgFgFgFg$7FgFg  FgRXqXqXqXqXq/XqXqXqv6d6d6d6d6d!46d6d6drA A A A A 2A A A H   r+