`iJdZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZmZdd lmZmZmZdd lmZdd lmZdd lmZmZmZmZmZm Z m!Z!m"Z"dd l#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+m,Z,ddl-m.Z.ddl/m0Z0e,j1e2Z3dJdZ4Gdde j5Z6Gdde j5Z7Gdde j5Z8de7iZ9Gdde j5Z:Gdde j5Z;Gd d!e j5Z<Gd"d#eZ=Gd$d%e j5Z>Gd&d'e j5Z?Gd(d)e j5Z@e+Gd*d+e$ZAee+d,-Gd.d/e*ZBe+Gd0d1eAZCGd2d3e j5ZDGd4d5e j5ZEe+d6-Gd7d8eAZFe+d9-Gd:d;eAZGe+d<-Gd=d>eAZHe+d?-Gd@dAeAZIe+GdBdCeAZJe+GdDdEeAZKe+dF-GdGdHeAeZLgdIZMdS)KzPyTorch ELECTRA model.N) dataclass)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentions)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging)deprecate_kwarg) ElectraConfig discriminatorc ddl}ddl}ddl}n)#t$rtdwxYwt j|}t d||j |}g} g} |D]j\} } t d| d| |j || } | | | | kt| | D]\} } | } t|t r| dd} |d kr,| d d } | d d } | d d} | dd} | d} t'd| Drt d||}| D]}|d|r|d|}n|g}|ddks |ddkrt+|d}ny|ddks |ddkrt+|d}nP|ddkrt+|d}n3|ddkrt+|d}nt+||d}t-|dkrt/|d }||}|d!rt+|d}n|dkr|| } |j| jkr t7d"|jd#| jd$n/#t6$r"}|xj|j| jfz c_d}~wwxYwt;d%| |t=j| |_ #tB$r}t;d|| |Yd}~d}~wwxYw|S)&z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape zelectra/embeddings/zgenerator/embeddings/ generatorzelectra/zdiscriminator/z generator/dense_1dense_predictionz!generator_predictions/output_biaszgenerator_lm_head/bias/c3K|]}|dvV dS)) global_step temperatureN).0ns /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/electra/modeling_electra.py z-load_tf_weights_in_electra..\s(EE1166EEEEEEz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierr# _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )"renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzip isinstanceElectraForMaskedLMreplacesplitany fullmatchgetattrlenintendswith transposeshape ValueErrorargsprinttorch from_numpydataAttributeError)modelconfigtf_checkpoint_pathdiscriminator_or_generatorr?nptftf_path init_varsnamesarraysnamerYarray original_namepointerm_name scope_namesnumes r1load_tf_weights_in_electrars2s     Q      goo011G KKBBBCCC''00I E F  e BBB5BBCCC&&w55 T e5&))66 e! 3 %!344 T||$9;RSS)[88||J0@AA||L*==<< +=>>D<< CE]^^D::c??DEEEEEEE  7 77888G + +<< 0&99+"$((9f"="=KK#)(Kq>X--Q71J1J%gx88GG ^}44 A&8P8P%gv66GG ^'777%gx88GG ^w..%g|<>G{##q((k!n--C%clG}-- ,!'8448## U++ =EK//$%ngm%n%nV[Va%n%n%nooo0   7=%+66  5t55} E E E +E22GLL    -m--tQ 7 7 7 HHHH  LsL &5CO-E O:0M+*O+ N5NN/O O1O,,O1c eZdZdZfdZ d deejdeejdeejdeejd e d ej f d Z xZ S) ElectraEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj |j|j |_ tj|j|_|dt%j|jddt+|dd|_|d t%j|jt$j ddS) N) padding_idxeps position_ids)r#F) persistentposition_embedding_typeabsolutetoken_type_idsdtype)super__init__r Embedding vocab_sizeembedding_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingstype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsDropouthidden_dropout_probdropoutregister_bufferr]arangeexpandrTr}zerosrzsizelongselfrb __class__s r1rzElectraEmbeddings.__init__sM !|F,=v?Tbhbuvvv#%<0NPVPe#f#f %'\&2H&J_%`%`"f&;AVWWWz&"<==   EL)GHHOOPWXXej    (/v7PR\']']$  ek$*;*@*@*B*B%*UUUbg      r3Nr input_idsrrz inputs_embedspast_key_values_lengthreturnc||}n|dd}|d}||jdd|||zf}|mt|dr2|jddd|f}||d|} | }n+t j|t j|jj}|| |}| |} || z} |j dkr| |} | | z } | | } || } | S)Nr{r#rrrdevicer~)rrzhasattrrrr]rrrrrr}rrr) rrrrzrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expandedr embeddingsrs r1forwardzElectraEmbeddings.forwardsm  #..**KK',,..ss3K ^  ,QQQ0FVlIl0l-lmL  !t-.. m*.*=aaa*n*M'3J3Q3QR]^_R`bl3m3m0!A!&[ SWSdSk!l!l!l   00;;M $ : :> J J"%::  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r3)NNNNr) __name__ __module__ __qualname____doc__rrr] LongTensor FloatTensorrVTensorr __classcell__rs@r1rurusQQ     .15593759&' ''E,-'!!12'u/0 '   12 ' !$ ' ''''''''r3ruceZdZdfd Zeddd ddejd eejd eejd eejdee d ee d eejde ejfdZ xZ S)ElectraSelfAttentionNcRt|j|jzdkr0t |ds t d|jd|jd|j|_t |j|jz |_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj|j|_|pt#|dd|_|jdks |jd kr6|j|_tjd |jzd z |j|_|j|_||_dS) NrrzThe hidden size (z6) is not a multiple of the number of attention heads ()r}r~ relative_keyrelative_key_queryr=r#)rr hidden_sizenum_attention_headsrrZrVattention_head_size all_head_sizerLinearquerykeyvaluerattention_probs_dropout_probrrTr}rrdistance_embedding is_decoder layer_idxrrbr}rrs r1rzElectraSelfAttention.__init__s    : :a ? ?PVXhHiHi ?8F$688 4888  $*#= #&v'9F( ' -zC C $  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # +"r3past_key_valuepast_key_values4.58new_nameversionF hidden_statesattention_mask head_maskencoder_hidden_statesoutput_attentionscache_positionrc|j\}} } ||} | |d|j|jdd} d} |du} |Ht |tr1|j |j } | r|j }n |j }n|}| r|n|}| r3|1| r/|j |j j}|j |j j}n||}||d|j|jdd}||}||d|j|jdd}|N| s|nd}||||j d|i\}}| r$t |trd|j|j <t'j| |dd}|jdks |jd krt| jd|jd}}|>t'j|dz t&j|j dd}n:t'j|t&j|j dd}t'j|t&j|j dd}||z }|||jzdz }|| j }|jdkrt'jd | |}||z}n?|jd kr4t'jd | |}t'jd ||}||z|z}|t?j |jz }|||z}tBj"#|d}|$|}|||z}t'j||}|%dddd&}|'dd|j(fz}||}||fS)Nr{r#r=FrTrrrrzbhld,lrd->bhlrzbhrd,lrd->bhlrdimrr ))rYrviewrrrXrNr is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrupdater]matmulr}tensorrrrrrtoreinsummathsqrtr functionalsoftmaxrpermute contiguousrr)rrrrrrrr batch_sizer_ query_layerris_cross_attentioncurr_past_key_valuecurrent_states key_layer value_layerattention_scores query_length key_lengthposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layernew_context_layer_shapes r1rzElectraSelfAttention.forwards%2$7! Jjj// !&&z2t7OQUQijjtt q   2$>  &/+>?? 6,7;;DNKK %O*9*O''*9*N''&5#2DW..-  F/"=*"=+24>BGI-4T^DKKK00I!z2t7OQUQijjtt1I**^44K%**B 8$:Ri1oo *7I!St)<)C)C{DN=M~<^**& ;&F*_FY*Z*ZFAEO.t~>!< Y5H5HR5P5PQQ  '> 9 9T=Y]q=q=q'2'8';Y_Q=O*L*!&j1nEJWdWk!l!l!l!q!q"""'l%*UbUi!j!j!j!o!oprtu!v!v"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|S"S%**+BCC o--r3NNNNNNFN)rrrrr"r]rrrrbooltuplerrrs@r1rrs######6_%0A6RRR7;15=A+/,115e.e.|e.!!23e.E-. e. ((9: e. "% e.$D>e.!.e. u| e.e.e.SRe.e.e.e.e.r3rcPeZdZfdZdejdejdejfdZxZS)ElectraSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNrx) rrrrrdenserrrrrrs r1rzElectraSelfOutput.__init__Nsf Yv163EFF f&8f>STTTz&"<== r3r input_tensorrc||}||}|||z}|SNrrrrrrs r1rzElectraSelfOutput.forwardT@ =11  ]33 }|'CDD r3rrrrr]rrrrs@r1rrMi>>>>> U\RWR^r3reagerceZdZdfd ZdZeddd dd ejd eej d eej d eej dee d ee deejde ejfdZ xZS)ElectraAttentionNctt|j||||_t ||_t|_dS)Nr}r) rrELECTRA_SELF_ATTENTION_CLASSES_attn_implementationrroutputset pruned_headsrs r1rzElectraAttention.__init__bsc 263NO $;    (// EEr3ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrr#r)rUrrrrrrrrrrrrunion)rheadsindexs r1 prune_headszElectraAttention.prune_headsls u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r3rrrrFrrrrrrrc ||||||||}||d|} | f|ddz} | S)Nrrrrrrrr#)rr) rrrrrrrr self_outputsattention_outputoutputss r1rzElectraAttention.forward~sfyy )"7+/)!   ;;|A FF#% QRR(88r3rr)rrrrrr"r]rrrrrrrrrs@r1rras"""""";;;$_%0A6RRR7;15=A+/,115|!!23E-.  ((9:  "% $D>!. u| SRr3rcBeZdZfdZdejdejfdZxZS)ElectraIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) rrrrrintermediate_sizerrN hidden_actstrr intermediate_act_fnrs r1rzElectraIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r3rrcZ||}||}|Sr)rr()rrs r1rzElectraIntermediate.forwards, =11 00?? r3r rs@r1r#r#s^99999U\elr3r#cPeZdZfdZdejdejdejfdZxZS) ElectraOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) rrrrr%rrrrrrrrs r1rzElectraOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r3rrrc||}||}|||z}|Srr r s r1rzElectraOutput.forwardr r3r rs@r1r+r+r r3r+c0eZdZdfd Zeddd ddejd eejd eejd eejd eejdee d ee deejde ejfdZ dZ xZS) ElectraLayerNct|j|_d|_t |||_|j|_|j|_|jr0|jst|dt |d||_ t||_ t||_ dS)Nr#rz> should be used as a decoder model if cross attention is addedr~r)rrchunk_size_feed_forward seq_len_dimr attentionradd_cross_attentionrZcrossattentionr# intermediater+rrrbrrs r1rzElectraLayer.__init__s '-'E$)&IFFF +#)#=  # t? j D!h!h!hiii"26S]ir"s"s"sD /77#F++ r3rrrrFrrrrencoder_attention_maskrrrc h|||||||} | d} | dd} |jrV|Tt|dstd|d|| ||||||} | d} | | ddz} t |j|j|j| } | f| z} | S)N)rrrrrrr#r6z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`r) r4rrrZr6rfeed_forward_chunkr2r3)rrrrrr9rrrself_attention_outputsr r!cross_attention_outputs layer_outputs r1rzElectraLayer.forwards!"& )/+) "0" " 2!4(, ? <4@4!122  DdDDD '+&9&9 5#&; /"3-':'' # 7q9  7 ;;G0  #T%A4CSUe   /G+r3c\||}|||}|Sr)r7r)rr intermediate_outputr>s r1r;zElectraLayer.feed_forward_chunks2"//0@AA{{#68HII r3r)NNNNNFN)rrrrr"r]rrrrrrrr;rrs@r1r/r/s. , , , , , ,_%0A6RRR7;15=A>B+/,115..|.!!23.E-. . ((9: . !)): ; ."%.$D>.!.. u| ...SR.`r3r/cHeZdZdfd Z ddejdeejdeejdeejd eejd eed ee d ee d ee dee deejde e eje ffdZ xZS)ElectraEncoderNct|_tjfdt jD|_d|_dS)Nc2g|]}t|S)r1)r/)r/irbs r1 z+ElectraEncoder.__init__..s&#o#o#o!L1$E$E$E#o#o#or3F) rrrbr ModuleListrangenum_hidden_layerslayergradient_checkpointingr8s ` r1rzElectraEncoder.__init__sa  ]#o#o#o#ouU[UmOnOn#o#o#opp &+###r3FTrrrrr9r use_cacheroutput_hidden_states return_dictrrc | rdnd} |rdnd} |r|jjrdnd}|jr%|jr|rtdd}|rD|jjr8|6tt|jt|j}|rO|jjrCt|tr.tdtj |}t|j D]Z\}}| r| |fz} |||nd}||||||||| }|d}|r$| |dfz} |jjr ||d fz}[| r| |fz} | std ||| | |fDSt||| | | S) Nr.zZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...F)rbzPassing a tuple of `past_key_values` is deprecated and will be removed in Transformers v4.58.0. You should pass an instance of `EncoderDecoderCache` instead, e.g. `past_key_values=EncoderDecoderCache.from_legacy_cache(past_key_values)`.)r9rrrrr#r=c3K|]}||V dSrr.)r/vs r1r2z)ElectraEncoder.forward..Ds4  =!===  r3)last_hidden_staterr attentionscross_attentions)rbr5rKtrainingrC warning_oncerrrrNrfrom_legacy_cache enumeraterJr)rrrrrr9rrLrrMrNrall_hidden_statesall_self_attentionsall_cross_attentionsrE layer_modulelayer_head_mask layer_outputss r1rzElectraEncoder.forwardsc#7@BBD$5?bb4%6d4;;Zdrr`d  & "4= " "##p"  v/ vO4K1,dk2R2R2RT`hlhsTtTtTtuuO  U/ UJPU4V4V U   \    2COTTO(44 V VOA|# I$58H$H!.7.CillO(L%'= /"3-   M*!,M  V&9]1=MB+/$(,1/4&*15P P |P !!23P E-. P ((9: P !)): ; P "%P D>P $D>P 'tnP d^P !.P  uU\"$MM NP P P P P P P P r3rBc(eZdZdZfdZdZxZS)ElectraDiscriminatorPredictionszEPrediction module for the discriminator, made up of two dense layers.cttj|j|j|_t |j|_tj|jd|_ ||_ dSNr#) rrrrrrr r& activationr)rbrs r1rz(ElectraDiscriminatorPredictions.__init__[sf Yv163EFF ():;; " &*??  66 &&}55==bAA r3rrrrrrrrs@r1r`r`XsMOOr3r`c(eZdZdZfdZdZxZS)ElectraGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cttd|_t j|j|j|_t j|j |j|_ dS)Ngelurx) rrr rcrrrrrrrrs r1rz$ElectraGeneratorPredictions.__init__nsa (00f&;AVWWWYv163HII r3c||}||}||}|Sr)rrcr)rgenerator_hidden_statesrs r1rz#ElectraGeneratorPredictions.forwardus< #:;;  66 }55 r3rhrs@r1rjrjksRKKJJJJJr3rjc,eZdZUeed<eZdZdZdZ dS)ElectraPreTrainedModelrbelectraTct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSdS)zInitialize the weightsg)meanstdNg?)rNrrr6r_normal_rbinitializer_ranger9zero_rrwrfill_)rmodules r1 _init_weightsz$ElectraPreTrainedModel._init_weightss) fbi ( ( * M  & &CT[5R & S S S{&  &&((((('&  - - * M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - * K  " " $ $ $ M  $ $S ) ) ) ) ) * *r3N) rrrr$__annotations__rsload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrzr.r3r1rprp}sB 0O!&*#*****r3rpz3 Output type of [`ElectraForPreTraining`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)ElectraForPreTrainingOutputa+ loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss of the ELECTRA objective. logits (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Prediction scores of the head (scores for each token before SoftMax). NlossrgrrS) rrrrrrr]rr{rgrrrSr.r3r1rrs)-D(5$ %,,,*.FHU& '...8cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T!"  &"/5996"1%+B//$3355 #  !"Z FCCCN  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z"&"B"B>S^"_"_ ; ! 3&;&G=R=W=W=Y=Y :  7$68O#P %-).4HQW)X)X)X&.2.H.HI_.`.` + +.2 +&&y$+2OPP %)'#9 (   4- . . C 33MBBM 2"7#B+/!5#%   r3) NNNNNNNNNNNNN)rrrrrrrr rr]rrrrrrrrrs@r1rrs     ///000CCC-11515/3,0048<9=+/$(,0/3&*\\EL)\!.\!. \ u|, \ EL) \  -\ ( 5\!) 6\"%\D>\$D>\'tn\d^\ uU\"$FF G\\\^\\\\\r3rc(eZdZdZfdZdZxZS)ElectraClassificationHeadz-Head for sentence-level classification tasks.c\ttj|j|j|_|j|jn|j}td|_ tj ||_ tj|j|j |_ dS)Nrl)rrrrrrclassifier_dropoutrr rcrr num_labelsout_projrrbrrs r1rz"ElectraClassificationHead.__init__(s Yv163EFF )/)B)NF % %TZTn )00z"455  &"4f6GHH r3c |dddddf}||}||}||}||}||}|S)Nr)rrrcr)rfeatureskwargsxs r1rz!ElectraClassificationHead.forward2sn QQQ111W  LLOO JJqMM OOA   LLOO MM!  r3rhrs@r1rr%sR77IIIIIr3rcjeZdZdZdeffd Z d dejdeej dejfdZ xZ S) ElectraSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`ElectraConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. rbcVtt|dd|_|jdkrtt j|_t|drW|j rPt|dr|j r|j dkr|j }n|j }t j |j ||_t|dd}|rt|nt j|_t j|_t|dr)|jdkrt j|j|_t j|_t|d r+|jdkr"t j|j|_dSdSdS) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)rrrTrNotImplementedErrorrIdentitysummaryrrrrrrr rc first_dropoutrr last_dropoutr)rrb num_classesactivation_stringrs r1rzElectraSequenceSummary.__init__Ws #FNFCC   & && %{}} 6- . . F63J Fv788 1V=Z 1_e_pst_t_t$/ $0 9V%7EEDL#F,@$GGIZ$mN3D$E$E$E`b`k`m`m[]] 62 3 3 J8TWX8X8X!#F,H!I!ID KMM 61 2 2 Hv7RUV7V7V " 6+F G GD    H H7V7Vr3Nr cls_indexrc:|jdkr|dddf}n-|jdkr|dddf}n|jdkr|d}n|jd kr|=tj|d ddddf|jd dz tj }nl|dd}|d |dz z| dfz}| d | d }n|jdkrt| |}||}||}||}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rNr{firstrrsr#rr.rr)r{r)rrsr] full_likerYr unsqueezerrrgatherrerrrrcr)rrrrs r1rzElectraSequenceSummary.forwardts   & &"111b5)FF  ' ) )"111a4(FF  & ( ("''A'..FF  + - - !O!#rr111*-!'+a/* &//33==bAA %,,Uimmoo6I-JmN`N`acNdNdMf-fgg "))"i88@@DDFF  & ( (% %##F++f%%((""6** r3r) rrrrr$rr]rrrrrrs@r1rr=s2H}HHHHHH<Y]))".);CEDT;U)  ))))))))r3rz ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S) ElectraForSequenceClassificationct||j|_||_t ||_t ||_|dSr) rrrrbrrqrr<rrs r1rz)ElectraForSequenceClassification.__init__sb     + #F++ 3F;; r3NrrrrzrrlabelsrrMrNrc | | n |jj} ||||||||| |  } | d} || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j| d}n*|jjdkrt}|| |}| s| f| ddz}||f|zn|St|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrrzrrrrMrNrr# regressionsingle_label_classificationmulti_label_classificationr{rrgrrS)rbrrqr< problem_typerrr]rrVr rer rrrrrS)rrrrrzrrrrrMrNrfsequence_outputrgrloss_fctrs r1rz(ElectraForSequenceClassification.forwards(&1%D 'tnD d^D  uU\"$<< =D D D ^D D D D D r3rz Electra model with a binary classification head on top as used during pretraining for identifying generated tokens. It is recommended to load the discriminator checkpoint into that model. c\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ElectraForPreTrainingct|t||_t ||_|dSr)rrrrqr`discriminator_predictionsrrs r1rzElectraForPreTraining.__init__sP    #F++ )H)P)P& r3NrrrrzrrrrrMrNrc | | n |jj} ||||||||| |  } | d} || } d}|t j}|s|d| jddk}| d| jd|}||}|||}n=|| d| jd|}| s| f| ddz}||f|zn|St|| | j | j S)am labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the ELECTRA loss. Input should be a sequence of tokens (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates the token is an original token, - 1 indicates the token was replaced. Examples: ```python >>> from transformers import ElectraForPreTraining, AutoTokenizer >>> import torch >>> discriminator = ElectraForPreTraining.from_pretrained("google/electra-base-discriminator") >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-discriminator") >>> sentence = "The quick brown fox jumps over the lazy dog" >>> fake_sentence = "The quick brown fox fake over the lazy dog" >>> fake_tokens = tokenizer.tokenize(fake_sentence, add_special_tokens=True) >>> fake_inputs = tokenizer.encode(fake_sentence, return_tensors="pt") >>> discriminator_outputs = discriminator(fake_inputs) >>> predictions = torch.round((torch.sign(discriminator_outputs[0]) + 1) / 2) >>> fake_tokens ['[CLS]', 'the', 'quick', 'brown', 'fox', 'fake', 'over', 'the', 'lazy', 'dog', '[SEP]'] >>> predictions.squeeze().tolist() [0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0] ```Nrrr{r#r) rbrrqrrrrrYfloatrrrS)rrrrrzrrrrrMrNrfdiscriminator_sequence_outputrgrr active_loss active_logits active_labelsrs r1rzElectraForPreTraining.forward sZ&1%Q 'tnQ d^Q  uU\"$?? @Q Q Q ^Q Q Q Q Q r3rz Electra model with a language modeling head on top. Even though both the discriminator and generator may be loaded into this model, the generator is the only model of the two to have been trained for the masked language modeling task. cneZdZdgZfdZdZdZe ddee j dee j dee j d ee j d ee j d ee j d ee j d ee dee dee de e e j effdZxZS)rOgenerator_lm_head.weightc t|t||_t ||_t j|j|j |_ | dSr) rrrrqrjgenerator_predictionsrrrrgenerator_lm_headrrs r1rzElectraForMaskedLM.__init__isj    #F++ %@%H%H"!#6+@&BS!T!T r3c|jSrrrs r1get_output_embeddingsz(ElectraForMaskedLM.get_output_embeddingss %%r3c||_dSrr)rrs r1set_output_embeddingsz(ElectraForMaskedLM.set_output_embeddingsvs!0r3NrrrrzrrrrrMrNrc | | n |jj} ||||||||| |  } | d} || } || } d}|Pt j}|| d|jj|d}| s| f| ddz}||f|zn|St|| | j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrr{r#r) rbrrqrrrr rrrrrS)rrrrrzrrrrrMrNrngenerator_sequence_outputprediction_scoresrrrs r1rzElectraForMaskedLM.forwardys2(&1%! 667PQQ 223DEE  *,,H8-222t{7MNNPVP[P[\^P_P_``D F'),CABB,GGF)-)9TGf$$v E$1?.9     r3r)rrr_tied_weights_keysrrrr rr]rrrrrrrrs@r1rOrO^s`55&&&111-11515/3,004)-,0/3&*4 4 EL)4 !.4 !. 4 u|, 4 EL) 4   -4 &4 $D>4 'tn4 d^4  uU\"N2 34 4 4 ^4 4 4 4 4 r3rOz Electra model with a token classification head on top. Both the discriminator and generator may be loaded into this model. c\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ElectraForTokenClassificationcVt||j|_t||_|j|jn|j}tj||_ tj |j |j|_ | dSr)rrrrrqrrrrrrrr<rrs r1rz&ElectraForTokenClassification.__init__s     +#F++ )/)B)NF % %TZTn z"455 )F$68IJJ r3NrrrrzrrrrrMrNrc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr{r#r) rbrrqrr<r rrrrrS)rrrrrzrrrrrMrNrfrrgrrrs r1rz%ElectraForTokenClassification.forwards$&1%??  '))H8FKKDO<1 'tn1 d^1  uU\"$99 :1 1 1 ^1 1 1 1 1 r3rceZdZUeed<dZfdZe ddee j dee j dee j dee j d ee j d ee j d ee j d ee j d ee dee dee de e e j effdZxZS)ElectraForQuestionAnsweringrbrqct||j|_t||_t j|j|j|_| dSr) rrrrrqrrr qa_outputsrrs r1rz$ElectraForQuestionAnswering.__init__se     +#F++ )F$68IJJ r3Nrrrrzrrstart_positions end_positionsrrMrNrc | | n |jj} |||||||| | } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) N)rrrzrrrrMrr#r{r) ignore_indexr=)r start_logits end_logitsrrS)rbrrqrrQrerrUrclampr rrrS)rrrrrzrrrrrrMrNrfrrgrr total_loss ignored_indexr start_lossend_lossrs r1rz#ElectraForQuestionAnswering.forward s&1%@ 'tn@ d^@  uU\"$@@ A@ @ @ ^@ @ @ @ @ r3rc\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)ElectraForMultipleChoicect|t||_t ||_t j|jd|_ | dSrb) rrrrqrsequence_summaryrrrr<rrs r1rz!ElectraForMultipleChoice.__init__Psh    #F++ 6v > >)F$6:: r3NrrrrzrrrrrMrNrc | | n |jj} | |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||| |  } | d} || }||}|d| }d}|t}|||}| s|f| ddz}||f|zn|St||| j | j S)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nr#r{rrrr) rbrrYrrrqrr<r rrrS)rrrrrzrrrrrMrN num_choicesrfr pooled_outputrgreshaped_logitsrrrs r1rz ElectraForMultipleChoice.forwardZs1X&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  '+ll ))%'/!5#'3 ' ' #6a8--o>> // ++b+66  '))H8OV44D F%'*Eabb*IIF)-)9TGf$$v E("5C2=     r3r)rrrrr rr]rrrrrrrrs@r1rrNsL-11515/3,004)-,0/3&*X X EL)X !.X !. X u|, X EL) X   -X &X $D>X 'tnX d^X  uU\"$== >X X X ^X X X X X r3rzS ELECTRA Model with a `language modeling` head on top for CLM fine-tuning. c"eZdZdgZfdZdZdZe ddee j dee j dee j d ee j d ee j d ee j d ee j d ee j dee j dee dee dee dee dee de ee j effdZxZS)ElectraForCausalLMrcLt||jstdt ||_t||_tj |j |j |_ |dS)NzOIf you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.`)rrrrCwarningrrqrjrrrrrr init_weightsrs r1rzElectraForCausalLM.__init__s      n NNl m m m#F++ %@%H%H"!#6+@&BS!T!T r3c|jSrrrs r1rz(ElectraForCausalLM.get_output_embeddingsrr3c||_dSrr)rnew_embeddingss r1rz(ElectraForCausalLM.set_output_embeddingss!/r3Nrrrrzrrrr9rrrLrrMrNrc ||n |jj}| d} |||||||||| | | | | }|d}|||}d}| |j|| fd|jji|}|s|f|ddz}||f|zn|St|||j|j |j |j S)a3 labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the left-to-right language modeling loss (next word prediction). Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> from transformers import AutoTokenizer, ElectraForCausalLM, ElectraConfig >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/electra-base-generator") >>> config = ElectraConfig.from_pretrained("google/electra-base-generator") >>> config.is_decoder = True >>> model = ElectraForCausalLM.from_pretrained("google/electra-base-generator", config=config) >>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt") >>> outputs = model(**inputs) >>> prediction_logits = outputs.logits ```NF) rrrzrrrr9rrLrrMrNrrr#)rrgrrrSrT) rbrrqrr loss_functionrrrrrSrT)rrrrrzrrrr9rrrLrrMrNrr!rrlm_lossrs r1rzElectraForCausalLM.forwards>R&1%S $D>S 'tnS d^S " uU\"$EE F#S S S ^S S S S S r3r) rrOrrrrrrrprs)r%)NrrrE dataclassesrtypingrrrr]rtorch.nnrr r activationsr r cache_utilsrrr generationrmodeling_layersrmodeling_outputsrrrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrr r!utils.deprecationr"configuration_electrar$ get_loggerrrCrsModulerurrrrr#r+r/rBr`rjrprrrrrrrOrrrr__all__r.r3r1r sO !!!!!!,,,,,,,,,, AAAAAAAAAA11111111CCCCCCCCCC))))))999999                    .-----llllllllll9999999999000000000000  H % %OOOOd????? ???FB.B.B.B.B.29B.B.B.L         !" 33333ry333n     ")        BI   CCCCC-CCCNW W W W W RYW W W tbi&")$*****_***.  : : : : :+ : :   :xxxxx)xxxv 0`````RY```F P P P P P '=P P  P f[ [ [ [ [ 2[ [ [ |H H H H H /H H H V@ @ @ @ @ $:@ @ @ FO O O O O "8O O O dd d d d d 5d d d N i i i i i /i i  i X   r3