`iZdZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl m Z m Z mZddlmZmZdd lmZdd lmZmZmZmZmZmZdd lmZdd lmZmZm Z dd l!m"Z"m#Z#ddl$m%Z%e#j&e'Z(dZ)Gdde j*Z+e"GddeZ,Gdde j*Z-Gdde j*Z.Gdde j*Z/Gdde j*Z0Gdde j*Z1Gdd e j*Z2Gd!d"e j*Z3Gd#d$eZ4Gd%d&e j*Z5Gd'd(e j*Z6Gd)d*e j*Z7e"Gd+d,e,Z8Gd-d.e j*Z9e"Gd/d0e,Z:Gd1d2e j*Z;e"d34Gd5d6e,Ze"Gd;dzPyTorch ConvBERT model.N) attrgetter)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FNget_activation)GradientCheckpointingLayer)"BaseModelOutputWithCrossAttentionsMaskedLMOutputMultipleChoiceModelOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringlogging)ConvBertConfigc ddl}n)#t$rtdwxYwtj|}td||j |}i}|D]E\}}td|d||j ||} | ||<Fddd d d d d d} |j dkrd} nd} t|j D]:} d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d| d| d<d| d | d| d!<d| d"| d| d#<d| d$| d| d%<d| d&| d| d'<d| d(| d| d)<d| d*| d| d+<d| d,| d| d-<d| d.| d| d/<d| d0| d| d1<d| d2| d| d3<d| d4| d| d5<d| d6| d7| d| d8<d| d6| d9| d| d:<d| d;| d7| d| d<<d| d;| d9| d| d=<d| d>| d| d?<d| d@| d| dA<<|D]7} | d}t|}||}| |}t!j||}tdB|dC|dD|d7r1|dEs|dFs|j}|dGr|ddHd}|dIr|dHdd}|dJr|dK}||_9|S)Lz'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape z"electra/embeddings/word_embeddingsz&electra/embeddings/position_embeddingsz(electra/embeddings/token_type_embeddingsz"electra/embeddings/LayerNorm/gammaz!electra/embeddings/LayerNorm/betaz!electra/embeddings_project/kernelzelectra/embeddings_project/bias)z!embeddings.word_embeddings.weightz%embeddings.position_embeddings.weightz'embeddings.token_type_embeddings.weightzembeddings.LayerNorm.weightzembeddings.LayerNorm.biaszembeddings_project.weightzembeddings_project.biasrg_densedensezelectra/encoder/layer_z/attention/self/query/kernelzencoder.layer.z.attention.self.query.weightz/attention/self/query/biasz.attention.self.query.biasz/attention/self/key/kernelz.attention.self.key.weightz/attention/self/key/biasz.attention.self.key.biasz/attention/self/value/kernelz.attention.self.value.weightz/attention/self/value/biasz.attention.self.value.biasz./attention/self/conv_attn_key/depthwise_kernelz4.attention.self.key_conv_attn_layer.depthwise.weightz./attention/self/conv_attn_key/pointwise_kernelz4.attention.self.key_conv_attn_layer.pointwise.weightz"/attention/self/conv_attn_key/biasz(.attention.self.key_conv_attn_layer.biasz'/attention/self/conv_attn_kernel/kernelz(.attention.self.conv_kernel_layer.weightz%/attention/self/conv_attn_kernel/biasz&.attention.self.conv_kernel_layer.biasz&/attention/self/conv_attn_point/kernelz%.attention.self.conv_out_layer.weightz$/attention/self/conv_attn_point/biasz#.attention.self.conv_out_layer.biasz/attention/output/dense/kernelz.attention.output.dense.weightz!/attention/output/LayerNorm/gammaz".attention.output.LayerNorm.weightz/attention/output/dense/biasz.attention.output.dense.biasz /attention/output/LayerNorm/betaz .attention.output.LayerNorm.biasz/intermediate/z/kernelz.intermediate.dense.weightz/biasz.intermediate.dense.biasz/output/z.output.dense.weightz.output.dense.biasz/output/LayerNorm/gammaz.output.LayerNorm.weightz/output/LayerNorm/betaz.output.LayerNorm.biaszTF: z, PT:  z/intermediate/g_dense/kernelz/output/g_dense/kernelz/depthwise_kernelz/pointwise_kernelz/conv_attn_key/bias) tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variable num_groupsrangenum_hidden_layersnamed_parametersrtorch from_numpyendswithTpermute unsqueezedata)modelconfigtf_checkpoint_pathtftf_path init_varstf_datanameshapearray param_mappinggroup_dense_namejparam param_name retrieverresulttf_namevalues /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/convbert/modeling_convbert.pyload_tf_weights_in_convbertrM0sX   Q      goo011G KKBBBCCC''00IG  e BBB5BBCCC&&w55 .R1Y3]'K%H%H#DM1$" 6+ , ,CwCw DQ D D D FqFFFG CQ B B B DqDDDE CQ B B B DqDDDE AQ @ @ @ BqBBBC EQ D D D FqFFFG CQ B B B DqDDDE WQ V V V ^q^^^_ WQ V V V ^q^^^_ KQ J J J RqRRRS PQ O O O RqRRRS NQ M M M PqPPPQ OQ N N N OqOOOP MQ L L L MqMMMN GQ F F F HqHHHI JQ I I I LqLLLM EQ D D D FqFFFG IQ H H H JqJJJK PQ O O6F O O O DqDDDE NQ M M6F M M M BqBBBC JQ I I0@ I I I >q>>>? HQ G G0@ G G G  J J"%88;PP ^^J// \\*-- rr)NNNN) __name__ __module__ __qualname____doc__rZrr2 LongTensor FloatTensorr __classcell__rqs@rLrOrOsQQ     (15593759 $$E,-$!!12$u/0 $   12 $   $$$$$$$$rrrOc,eZdZUeed<eZdZdZdZ dS)ConvBertPreTrainedModelr:convbertTct|tjtjfrT|jjd|jj|j |j j dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|j j |jjddSt|tr |j j dSt|t rK|jjd|jj|j j dSdS)zInitialize the weightsmeanstdNg?) isinstancerLinearConv1dweightr8normal_r:initializer_rangebiaszero_r[rQrdfill_SeparableConv1DGroupedLinearLayer)rpmodules rL _init_weightsz%ConvBertPreTrainedModel._init_weightss fry")4 5 5 % M  & &CT[5R & S S S{&  &&((((('&  - - % M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - % K  " " $ $ $ M  $ $S ) ) ) ) )  0 0 % K  " " $ $ $ $ $  2 3 3 % M  & &CT[5R & S S S K  " " $ $ $ $ $ % %rrN) rrrr__annotations__rMload_tf_weightsbase_model_prefixsupports_gradient_checkpointingrrrrLrrsB 1O"&*#%%%%%rrrcFeZdZdZfdZdejdejfdZxZS)rzSThis class implements separable convolution, i.e. a depthwise and a pointwise layerc ttj|||||dzd|_tj||dd|_tjtj|d|_ |jj j d|j |jj j d|j dS)Nr!F) kernel_sizegroupspaddingrr)rrrr)rYrZrr depthwise pointwise Parameterr2rlrrr8rr)rpr: input_filtersoutput_filtersrkwargsrqs rLrZzSeparableConv1D.__init__s   # 1$    =.aV[\\\L^Q!?!?@@  "**9Q*RRR "**9Q*RRRRRrr hidden_statesrucn||}||}||jz }|SN)rrr)rprxs rLrzSeparableConv1D.forwards4 NN= ) ) NN1   TYrr rrrrrZr2Tensorrrrs@rLrrsi]]SSSSS U\elrrrceZdZfdZ d dejdeejdeejdeejdeed e ejeejff d Z xZ S) ConvBertSelfAttentioncpt|j|jzdkr0t |ds t d|jd|jd|j|jz}|dkr|j|_d|_n||_|j|_|j|_|j|jzdkrt d|j|jzdz|_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_t||j|j |j|_tj |j |j|jz|_tj |j|j |_tj|jdgt)|jdz dz dg |_tj|j|_dS) Nrr]zThe hidden size (z6) is not a multiple of the number of attention heads ()rz6hidden_size should be divisible by num_attention_headsr!)rr)rYrZ hidden_sizenum_attention_headsry ValueError head_ratioconv_kernel_sizeattention_head_size all_head_sizerrquerykeyrKrkey_conv_attn_layerconv_kernel_layerconv_out_layerUnfoldintunfoldrfattention_probs_dropout_probrh)rpr:new_num_attention_headsrqs rLrZzConvBertSelfAttention.__init__ s&    : :a ? ?PVXhHiHi ?8F$688 4888  #)"<@Q"Q "Q & &$8DO'(D $ $'>D $$/DO & 7   8 8A = =UVV V$*$6$:R$RWX#X !58PPYv143EFF 9V/1CDDYv143EFF #2 F&(:D V u|Xel33 4VVVVVVVVrrrcPeZdZfdZdejdejdejfdZxZS)ConvBertSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNrR) rYrZrrrrrdrerfrgrhros rLrZzConvBertSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== rrr input_tensorruc||}||}|||z}|Srrrhrdrprrs rLrzConvBertSelfOutput.forward@ =11  ]33 }|'CDD rrrrrrZr2rrrrs@rLrrsi>>>>> U\RWR^rrrceZdZfdZdZ d dejdeejdeejdeejd ee d e ejeejff d Z xZ S) ConvBertAttentionctt||_t ||_t |_dSr)rYrZrrproutputset pruned_headsros rLrZzConvBertAttention.__init__sI )&11 (00 EErrct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrr)lenrrprrrrrrrKrrrunion)rpheadsindexs rL prune_headszConvBertAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::rrNFrrrrrruc||||||}||d|}|f|ddz}|S)Nrr)rpr) rprrrrr self_outputsattention_outputrs rLrzConvBertAttention.forwards[yy    !     ;;|A FF#% QRR(88rrr) rrrrZrr2rrrrrrrrs@rLrrs""""" ;;;*7;158<,1 |!!23E-.  ( 5  $D>  u|Xe&788 9rrrcBeZdZfdZdejdejfdZxZS)rct||_||_||_|j|jz|_|j|jz|_tjtj |j|j|j|_ tjtj ||_ dSr) rYrZ input_size output_sizer. group_in_dim group_out_dimrrr2emptyrr)rpr r r.rqs rLrZzGroupedLinearLayer.__init__s $&$ Ot>!-@l5;t@QSWSe#f#fgg L[!9!9:: rrrrucvt|d}tj|d|j|jg}|ddd}tj||j}|ddd}tj||d|j g}||j z}|S)Nrr"rr!) listrmr2rr.r r6rrr r)rprrrs rLrzGroupedLinearLayer.forwards-,,..//2 M-"dot?P)Q R R IIaA   LDK ( ( IIaA   M!j"d.>? @ @  Mrrrrs@rLrrs^;;;;;U\elrrrcBeZdZfdZdejdejfdZxZS)ConvBertIntermediatecrt|jdkr%tj|j|j|_n&t|j|j|j|_t|j trt|j |_ dS|j |_ dS)Nrr r r.)rYrZr.rrrintermediate_sizerrr hidden_actstrr intermediate_act_fnros rLrZzConvBertIntermediate.__init__s    ! !6#5v7OPPDJJ+!-6;S`f`qDJ f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rrrrucZ||}||}|Sr)rrrprs rLrzConvBertIntermediate.forwards, =11 00?? rrrrs@rLrrs^ 9 9 9 9 9U\elrrrcPeZdZfdZdejdejdejfdZxZS)ConvBertOutputczt|jdkr%tj|j|j|_n&t|j|j|j|_tj |j|j |_ tj |j |_ dS)NrrrR)rYrZr.rrrrrrrdrerfrgrhros rLrZzConvBertOutput.__init__s    ! !6#;V=OPPDJJ+!3AS`f`qDJf&8f>STTTz&"<== rrrrruc||}||}|||z}|Srrrs rLrzConvBertOutput.forwardrrrrrs@rLrrsi > > > > >U\RWR^rrrceZdZfdZ d dejdeejdeejdeejdeejd eed e ejeejffd Z d Z xZ S) ConvBertLayerc~t|j|_d|_t ||_|j|_|j|_|jr-|jst|dt ||_ t||_ t||_ dS)Nrz> should be used as a decoder model if cross attention is added)rYrZchunk_size_feed_forward seq_len_dimr attention is_decoderadd_cross_attention TypeErrorcrossattentionr intermediaterrros rLrZzConvBertLayer.__init__s '-'E$*622 +#)#=  # <? i4 g g ghhh"3F";";D 088$V,, rrNFrrrrencoder_attention_maskrruc^|||||}|d}|dd} |jrS|Qt|dstd|d||||||} | d}| | ddz} t |j|j|j|} | f| z} | S)N)rrrr)z'If `encoder_hidden_states` are passed, z` has to be instantiated with cross-attention layers by setting `config.add_cross_attention=True`) r%r&ryAttributeErrorr)rfeed_forward_chunkr#r$) rprrrrr+rself_attention_outputsrrcross_attention_outputs layer_outputs rLrzConvBertLayer.forward's"&   / "0" "  2!4(, ? <4@4!122 $DdDDD'+&9&9 &%! '' # 7q9  7 ;;G0  #T%A4CSUe   /G+rrc\||}|||}|Sr)r*r)rprintermediate_outputr1s rLr.z ConvBertLayer.feed_forward_chunkOs2"//0@AA{{#68HII rr)NNNNF) rrrrZr2rrrrrrr.rrs@rLr!r!s - - - - -"7;158<9=,1&&|&!!23&E-. & ( 5 & !) 6 &$D>& u|Xe&788 9&&&&Prrr!ceZdZfdZ ddejdeejdeejdeejd eejd eed eed eed e e e ffdZ xZ S)ConvBertEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tSr)r!).0rr:s rL z,ConvBertEncoder.__init__..Ys!#c#c#caM&$9$9#c#c#crrF) rYrZr:r ModuleListr/r0layergradient_checkpointingros `rLrZzConvBertEncoder.__init__Vs`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###rrNFTrrrrr+routput_hidden_states return_dictruc |rdnd} |rdnd} |r|jjrdnd} t|jD]W\} } |r| |fz} ||| nd}| ||||||}|d}|r$| |dfz} |jjr | |dfz} X|r| |fz} |st d|| | | fDSt || | | S)Nrrrr!c3K|]}||V dSrr)r8vs rL z*ConvBertEncoder.forward..s0= ===rr)last_hidden_stater attentionscross_attentions)r:r' enumerater;rr)rprrrrr+rr=r>all_hidden_statesall_self_attentionsall_cross_attentionsi layer_modulelayer_head_mask layer_outputss rLrzConvBertEncoder.forward\sp#7@BBD$5?bb4%6d4;;Zdrr`d(44 V VOA|# I$58H$H!.7.CillO(L%&! M*!,M  V&9]1=M0 'tn0 d^0  u88 90 0 0 0 0 0 0 0 rrr5cBeZdZfdZdejdejfdZxZS)ConvBertPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) rYrZrrrrrrrr transform_act_fnrdreros rLrZz(ConvBertPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTrrrruc||}||}||}|Sr)rrQrdrs rLrz'ConvBertPredictionHeadTransform.forwards= =11 --m<< }55 rrrrs@rLrOrOscUUUUUU\elrrrOcjeZdZdZdeffd Z d dejdeej dejfdZ xZ S) ConvBertSequenceSummarya Compute a single vector summary of a sequence hidden states. Args: config ([`ConvBertConfig`]): The config used by the model. Relevant arguments in the config class of the model are (refer to the actual config class of your model for the default values it uses): - **summary_type** (`str`) -- The method to use to make this summary. Accepted values are: - `"last"` -- Take the last token hidden state (like XLNet) - `"first"` -- Take the first token hidden state (like Bert) - `"mean"` -- Take the mean of all tokens hidden states - `"cls_index"` -- Supply a Tensor of classification token position (GPT/GPT-2) - `"attn"` -- Not implemented now, use multi-head attention - **summary_use_proj** (`bool`) -- Add a projection after the vector extraction. - **summary_proj_to_labels** (`bool`) -- If `True`, the projection outputs to `config.num_labels` classes (otherwise to `config.hidden_size`). - **summary_activation** (`Optional[str]`) -- Set to `"tanh"` to add a tanh activation to the output, another string or `None` will add no activation. - **summary_first_dropout** (`float`) -- Optional dropout probability before the projection and activation. - **summary_last_dropout** (`float`)-- Optional dropout probability after the projection and activation. r:cVtt|dd|_|jdkrtt j|_t|drW|j rPt|dr|j r|j dkr|j }n|j }t j |j ||_t|dd}|rt|nt j|_t j|_t|dr)|jdkrt j|j|_t j|_t|d r+|jdkr"t j|j|_dSdSdS) N summary_typelastattnsummary_use_projsummary_proj_to_labelsrsummary_activationsummary_first_dropoutsummary_last_dropout)rYrZgetattrrVNotImplementedErrorrIdentitysummaryryrYrZ num_labelsrrr activation first_dropoutr\rf last_dropoutr])rpr: num_classesactivation_stringrqs rLrZz ConvBertSequenceSummary.__init__s #FNFCC   & && %{}} 6- . . F63J Fv788 1V=Z 1_e_pst_t_t$/ $0 9V%7EEDL#F,@$GGIZ$mN3D$E$E$E`b`k`m`m[]] 62 3 3 J8TWX8X8X!#F,H!I!ID KMM 61 2 2 Hv7RUV7V7V " 6+F G GD    H H7V7VrrNr cls_indexruc:|jdkr|dddf}n-|jdkr|dddf}n|jdkr|d}n|jd kr|=tj|d ddddf|jd dz tj }nl|dd}|d |dz z| dfz}| d | d }n|jdkrt| |}||}||}||}|S)ak Compute a single vector summary of a sequence hidden states. Args: hidden_states (`torch.FloatTensor` of shape `[batch_size, seq_len, hidden_size]`): The hidden states of the last layer. cls_index (`torch.LongTensor` of shape `[batch_size]` or `[batch_size, ...]` where ... are optional leading dimensions of `hidden_states`, *optional*): Used if `summary_type == "cls_index"` and takes the last token of the sequence as classification token. Returns: `torch.FloatTensor`: The summary of the sequence hidden states. rWNr"firstrrrrrh.rrW)r"rX)rVrr2 full_likerArnr7rkrrmgathersqueezer_rdrarcre)rprrhrs rLrzConvBertSequenceSummary.forwards   & &"111b5)FF  ' ) )"111a4(FF  & ( ("''A'..FF  + - - !O!#rr111*-!'+a/* &//33==bAA %,,Uimmoo6I-JmN`N`acNdNdMf-fgg "))"i88@@DDFF  & ( (% %##F++f%%((""6** rrr) rrrrrrZr2rrrrrrs@rLrTrTs2H~HHHHHH<Y]))".);CEDT;U)  ))))))))rrrTc<eZdZfdZdZdZdZe ddee j dee j dee j d ee j d ee j d ee j d ee d ee dee de eeffdZxZS) ConvBertModelc8t|t||_|j|jkr$t j|j|j|_t||_ ||_ | dSr) rYrZrOr~r]rrrembeddings_projectr5encoderr: post_initros rLrZzConvBertModel.__init__s    ,V44  F$6 6 6&(i0EvGY&Z&ZD #&v..   rrc|jjSrr~r_rps rLget_input_embeddingsz"ConvBertModel.get_input_embeddingss ..rrc||j_dSrru)rprKs rLset_input_embeddingsz"ConvBertModel.set_input_embeddingss*/'''rrc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrrr;r%r)rpheads_to_pruner;rs rL _prune_headszConvBertModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C CrrNrsrrVrTrrtrr=r>ruc ||n |jj}||n |jj}| | n |jj} ||t d|+||||} n.||dd} nt d| \} } ||jn|j} |tj | | }|gt|j dr1|j j ddd| f}| | | }|}n!tj| tj| }||| }|||jj}| ||||}t|dr||}||||||| }|S) NzDYou cannot specify both input_ids and inputs_embeds at the same timer"z5You have to specify either input_ids or inputs_embeds)rxrVrw)rsrTrVrtrq)rrrr=r>)r:rr=use_return_dictr%warn_if_padding_and_no_attention_maskrmrxr2onesryr~rVrkrlrnget_extended_attention_mask get_head_maskr0rqrr)rprsrrVrTrrtrr=r>rzrr{rxr|r}extended_attention_maskrs rLrzConvBertModel.forward s2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T  !"Z FCCCN  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z"&"B"B>S^"_"_&&y$+2OPP l>iv(   4- . . C 33MBBM 2/!5# %  rr) NNNNNNNNN)rrrrZrwryr}rrr2rrrrrrrrrs@rLrorosS     ///000CCC156:59371559,0/3&*<<E,-<!!23<!!12 < u/0 < E-. <  12<$D><'tn<d^< u88 9<<<^<<<<<rrrocFeZdZdZfdZdejdejfdZxZS)ConvBertGeneratorPredictionszAPrediction module for the generator, made up of two dense layers.cttd|_t j|j|j|_t j|j |j|_ dS)NgelurR) rYrZr rcrrdr]rerrrros rLrZz%ConvBertGeneratorPredictions.__init__csa (00f&;AVWWWYv163HII rrgenerator_hidden_statesruc||}||}||}|Sr)rrcrd)rprrs rLrz$ConvBertGeneratorPredictions.forwardjs< #:;;  66 }55 rr) rrrrrZr2rrrrs@rLrr`skKKJJJJJu/@UEVrrrcXeZdZdgZfdZdZdZe ddee j dee j dee j d ee j d ee j d ee j d ee j d ee dee dee de eeffdZxZS)ConvBertForMaskedLMzgenerator.lm_head.weightc t|t||_t ||_t j|j|j |_ | dSr) rYrZrorrgenerator_predictionsrrr]r\generator_lm_headrsros rLrZzConvBertForMaskedLM.__init__vsj    %f-- %A&%I%I"!#6+@&BS!T!T rrc|jSrrrvs rLget_output_embeddingsz)ConvBertForMaskedLM.get_output_embeddingss %%rrc||_dSrr)rpr_s rLset_output_embeddingsz)ConvBertForMaskedLM.set_output_embeddingss!0rrNrsrrVrTrrtlabelsrr=r>ruc | | n |jj} ||||||||| | } | d} || } || } d}|Pt j}|| d|jj|d}| s| f| ddz}||f|zn|St|| | j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrr"rlosslogitsrrD) r:rrrrrr rr\rrrD)rprsrrVrTrrtrrr=r>rgenerator_sequence_outputprediction_scoresrloss_fctrs rLrzConvBertForMaskedLM.forwards-(&1%! 667PQQ 223DEE  *,,H8-222t{7MNNPVP[P[\^P_P_``D F'),CABB,GGF)-)9TGf$$v E$1?.9     rr NNNNNNNNNN)rrr_tied_weights_keysrZrrrrr2rrrrrrrrrs@rLrrrs^45&&&111156:59371559-1,0/3&*4 4 E,-4 !!234 !!12 4 u/0 4 E-. 4   124 )*4 $D>4 'tn4 d^4  un$ %4 4 4 ^4 4 4 4 4 rrrcFeZdZdZfdZdejdejfdZxZS)ConvBertClassificationHeadz-Head for sentence-level classification tasks.cBttj|j|j|_|j|jn|j}tj||_ tj|j|j |_ ||_ dSr) rYrZrrrrclassifier_dropoutrgrfrhrbout_projr:rpr:rrqs rLrZz#ConvBertClassificationHead.__init__s Yv163EFF )/)B)NF % %TZTn z"455  &"4f6GHH  rrrruc |dddddf}||}||}t|jj|}||}||}|S)Nr)rhrr r:rr)rprrrs rLrz"ConvBertClassificationHead.forwardst !!!Q' " LLOO JJqMM 4;) *1 - - LLOO MM!  rrrrs@rLrrsd77     U\ rrrz ConvBERT Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ) custom_introcFeZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ffdZ xZS)!ConvBertForSequenceClassificationct||j|_||_t ||_t ||_|dSr) rYrZrbr:rorr classifierrsros rLrZz*ConvBertForSequenceClassification.__init__sb     + %f-- 4V<< rrNrsrrVrTrrtrrr=r>ruc | | n |jj} ||||||||| |  } | d} || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtjks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j| d}n*|jjdkrt}|| |}| s| f| ddz}||f|zn|St|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NrrVrTrrtrr=r>rr regressionsingle_label_classificationmulti_label_classificationr"r)r:rrr problem_typerbrXr2rnrr rmr rrrrrDrprsrrVrTrrtrrr=r>rsequence_outputrrrrs rLrz)ConvBertForSequenceClassification.forwards (&1%D 'tnD d^D  u.. /D D D ^D D D D D rrrcFeZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ffdZ xZS)ConvBertForMultipleChoicect|t||_t ||_t j|jd|_ | dS)Nr) rYrZrorrTsequence_summaryrrrrrsros rLrZz"ConvBertForMultipleChoice.__init__1sh    %f-- 7 ? ?)F$6:: rrNrsrrVrTrrtrrr=r>ruc | | n |jj} | |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||| |  } | d} || }||}|d| }d}|t}|||}| s|f| ddz}||f|zn|St||| j | j S)a\ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert *input_ids* indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrr"rrrr) r:rrArrmrrrr rrrD)rprsrrVrTrrtrrr=r> num_choicesrr pooled_outputrreshaped_logitsrrrs rLrz!ConvBertForMultipleChoice.forward;s-Z&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  -- ))%'/!5#   "!*--o>> // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     rrr)rrrrZrrr2rrrrrrrrrs@rLrr/sL156:59371559-1,0/3&*Y Y E,-Y !!23Y !!12 Y u/0 Y E-. Y   12Y )*Y $D>Y 'tnY d^Y  u// 0Y Y Y ^Y Y Y Y Y rrrcFeZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd ee d ee d ee d e e e ffdZ xZS)ConvBertForTokenClassificationcVt||j|_t||_|j|jn|j}tj||_ tj |j |j|_ | dSr)rYrZrbrorrrgrrfrhrrrrsrs rLrZz'ConvBertForTokenClassification.__init__s     +%f-- )/)B)NF % %TZTn z"455 )F$68IJJ rrNrsrrVrTrrtrrr=r>ruc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrr"rr) r:rrrhrr rrbrrrDrs rLrz&ConvBertForTokenClassification.forwards$&1%2 'tn2 d^2  u++ ,2 2 2 ^2 2 2 2 2 rrrcbeZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd ee d ee d ee de e e ffdZ xZS)ConvBertForQuestionAnsweringct||j|_t||_t j|j|j|_| dSr) rYrZrbrorrrr qa_outputsrsros rLrZz%ConvBertForQuestionAnswering.__init__se     +%f-- )F$68IJJ rrNrsrrVrTrrtstart_positions end_positionsrr=r>ruc | | n |jj} |||||||| | |  } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) Nrrrr"r) ignore_indexr!)r start_logits end_logitsrrD)r:rrrsplitrmrrrmclampr rrrD)rprsrrVrTrrtrrrr=r>rrrrr total_loss ignored_indexr start_lossend_lossrs rLrz$ConvBertForQuestionAnswering.forwards&1% > E,-> !!23> !!12 > u/0 > E-. >   12> "%"23>   01> $D>> 'tn> d^>  u22 3> > > ^> > > > > rrr) rrrrrr!rorrM)Arrr'operatorrtypingrrrr2rtorch.nnrr r activationsr r modeling_layersrmodeling_outputsrrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrconfiguration_convbertr get_loggerrr%rMModulerOrrrrrrrrr!r5rOrTrorrrrrrr__all__rrrrLrs ,,,,,,,,,, AAAAAAAAAA11111111999999.-----llllllllll322222  H % %yyyx99999999x%%%%%o%%%8bi4~~~~~BI~~~B        ***** ***Z,29(RY&:::::.:::z7 7 7 7 7 bi7 7 7 tbi$`````bi```FXXXXX+XXXv29$H H H H H 1H H H V0 P P P P P (?P P  P fe e e e e 7e e e PB B B B B %<B B B JJ J J J J #:J J J Z   rr