`i$ddlZddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z m Z m Z ddlmZddlmZmZmZmZmZmZmZmZdd lmZdd lmZmZdd lmZm Z m!Z!d d l"m#Z#e!j$e%Z&dZ'Gdde j(Z)e j*e)dZ+Gdde j(Z,Gdde j(Z-Gdde j(Z.Gdde j(Z/Gdde j(Z0Gdde j(Z1Gdde j(Z2Gd d!e j(Z3Gd"d#e j(Z4Gd$d%e j(Z5Gd&d'e j(Z6Gd(d)e j(Z7Gd*d+e j(Z8Gd,d-e j(Z9Gd.d/e j(Z:Gd0d1e j(Z;Gd2d3e j(Z<Gd4d5e j(Z=e Gd6d7eZ>ee d89Gd:d;eZ?e Gd<d=e>Z@e d>9Gd?d@e>ZAe GdAdBe>ZBGdCdDe j(ZCe dE9GdFdGe>ZDe dH9GdIdJe>ZEe GdKdLe>ZFe GdMdNe>ZGe GdOdPe>ZHgdQZIdS)RN) dataclass)OptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputMultipleChoiceModelOutputNextSentencePredictorOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging)MobileBertConfigc  ddl}ddl}ddl}n)#t$rtdwxYwt j|}t d||j |}g}g} |D]j\} } t d| d| |j || } | | | | kt|| D]\} } | dd} | d d } | d d } | d d} | d} t#d| Dr1t dd| |} | D]H}|d|r|d|}n|g}|ddks |ddkrt)| d} n|ddks |ddkrt)| d} n|ddkrt)| d} nv|ddkrt)| d} nY t)| |d} nA#t*$r4t dd| YwxYwt-|dkrt/|d}| |} J|ddd krt)| d} n|dkr|| } | j| jksJd!| jd"| jd#n/#t4$r"}|xj| j| jfz c_d}~wwxYwt d$| t9j| | _|S)%z'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape ffn_layerffn FakeLayerNorm LayerNormextra_output_weightsz dense/kernelbert mobilebert/c3K|]}|dvV dS))adam_vadam_mAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepN).0ns /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/mobilebert/modeling_mobilebert.py z0load_tf_weights_in_mobilebert..Vs<   n n      z Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammaweight output_biasbetabiasoutput_weightssquad classifierri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipreplacesplitanyjoin fullmatchgetattrAttributeErrorlenint transposeshapeAssertionErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr<nptftf_path init_varsnamesarraysnamerUarraypointerm_name scope_namesnumes r.load_tf_weights_in_mobilebertrk5sc     Q      goo011G KKBBBCCC''00I E F  e BBB5BBCCC&&w55 T e5&))1/1/ e||K//||O[99||2NCC||FL11zz#          KK4CHHTNN44 5 5 5  ' 'F||,f55 ' hhy&99 %h 1~))[^w-F-F!'844Q=00KNf4L4L!'622Q#333!'844Q7**!'<88%g{1~>>GG%KK J J"%88;PP ^^J// \\*-- r0)NNNN) r~rr__doc__rqrrX LongTensor FloatTensorrr|rrs@r.rrsQQ     015593759 00E,-0!!120u/0 0   12 0  00000000r0rceZdZfdZ d dejdejdejdeejdeejdeed e ejfd Z xZ S) MobileBertSelfAttentionct|j|_t|j|jz |_|j|jz|_tj|j|j|_ tj|j|j|_ tj|j r|jn|j |j|_ tj|j|_dSro)rprqnum_attention_headsrStrue_hidden_sizeattention_head_size all_head_sizerrquerykeyuse_bottleneck_attentionrrrattention_probs_dropout_probrrur\rxs r.rqz MobileBertSelfAttention.__init__s #)#= #&v'>A['[#\#\ !58PPYv68JKK 9V4d6HIIY'-'F ^F # #FL^`d`r   z&"EFF r0N query_tensor key_tensor value_tensorattention_mask head_maskoutput_attentionsrzc|j\}}} |||d|j|jdd} |||d|j|jdd} |||d|j|jdd} tj | | dd} | tj |jz } || |z} tj | d}||}|||z}tj || }|dddd}|dd|jfz}||}|r||fn|f}|S)Nrrr:rrr )rUrviewrrrTrrrXmatmulmathsqrtrrsoftmaxrpermute contiguousrr)rurrrrrr batch_sizer_ query_layer key_layer value_layerattention_scoresattention_probs context_layernew_context_layer_shapeoutputss r.r|zMobileBertSelfAttention.forwards%1$6! J JJ| $ $ T*b$":DS"S%**+BCC 6G]=/22mM]r0NNN r~rrrqrXrrrbooltupler|rrs@r.rrs G G G G G$7;15,0--l-L-l - !!23 - E-. -$D>- u| --------r0rcPeZdZfdZdejdejdejfdZxZS)MobileBertSelfOutputcLt|j|_tj|j|j|_t|j|j|j |_ |js tj |j |_ dSdSNrw)rprquse_bottleneckrrrdenserrlayer_norm_epsr rrrrs r.rqzMobileBertSelfOutput.__init__s $3Yv68OPP  !:;F u| r0rcBeZdZfdZdejdejfdZxZS)MobileBertIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSro) rprqrrrintermediate_sizer isinstance hidden_actstrr intermediate_act_fnrs r.rqzMobileBertIntermediate.__init__^sn Yv68PQQ f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r0rrzcZ||}||}|Sro)rrrurs r.r|zMobileBertIntermediate.forwardfs, =11 00?? r0r}rs@r.rr]s^99999U\elr0rcPeZdZfdZdejdejdejfdZxZS)OutputBottleneckc"ttj|j|j|_t|j|j|j |_ tj |j |_ dSr)rprqrrrrrrrrr rrrrs r.rqzOutputBottleneck.__init__msm Yv68JKK  !:;F>>>> U\ELUZUar0r c^eZdZfdZdejdejdejdejfdZxZS)MobileBertOutputcft|j|_tj|j|j|_t|j |j|_ |js tj |j |_ dSt||_dSro)rprqrrrrrrrrr rrrr  bottleneckrs r.rqzMobileBertOutput.__init__{s $3Yv79PQQ  !:;F>,9J*JKKLL>>,9J*JKKL??<9JKKLr0r}rs@r.rrzsu77777 #(< DIL ejeq          r0rcBeZdZfdZdejdejfdZxZS)BottleneckLayercttj|j|j|_t|j|j|j |_ dSr) rprqrrrintra_bottleneck_sizerrrrr rs r.rqzBottleneckLayer.__init__sY Yv163OPP  !:;F%>"  ( ].014 4  - ]%)^^M%B%B "*,BMSmn n!=-A[\ \r0 r~rrrqrXrrr|rrs@r.rrsm55555]U\]eEL6I]]]]]]]]r0rcPeZdZfdZdejdejdejfdZxZS) FFNOutputcttj|j|j|_t|j|j|j |_ dSr) rprqrrrrrrrrr rs r.rqzFFNOutput.__init__sY Yv79PQQ  !:;F> $7GG r0r}rs@r.r,r,s^((((( U\elr0r,c eZdZfdZ d dejdeejdeejdeede ejf dZ xZ S) MobileBertLayerctj|_j|_t |_t |_t|_ |jrt|_ jdkrz,MobileBertLayer.__init__..s!%k%k%k1hv&6&6%k%k%kr0)rprqrnum_feedforward_networksrr!rr.rrrrr ModuleListrangerrs `r.rqzMobileBertLayer.__init__s $3(.(G%,V4426::&v..   1(00DO  *Q . .}%k%k%k%kfFehiFi@j@j%k%k%kllDHHH / .r0Nrrrrrzc |jr||\}}}}n |gdz\}}}}||||||||} | d} | f} | dd} |jdkr+t |jD]\} }|| } | | fz } || }||| |}|f| ztj d||||| |fz| z} | S)Nr#)rrri) rrr!r7 enumeraterr.rrXtensor)rurrrrrrrrself_attention_outputsrsri ffn_moduler0rs r.r|zMobileBertLayer.forwardsZ   VBF//R_B`B` ?L*lKKCP/TUBU ?L*lK!%      /"0" " 2!4  (,  (A - -!*48!4!4 ) ) :#-:.>#?#? &(("//0@AA{{#68H-XX O  T"" #   r0rrrs@r.r2r2s m m m m m 7;15,0 ..|.!!23.E-. . $D> . u|  ........r0r2ceZdZfdZ d dejdeejdeejdeed eed eed e e e ffd Z xZ S)MobileBertEncodercttjfdt jD|_dS)Nc.g|]}tSr+)r2r5s r.r6z.MobileBertEncoder.__init__..s!#e#e#eOF$;$;#e#e#er0)rprqrr8r9num_hidden_layerslayerrs `r.rqzMobileBertEncoder.__init__sO ]#e#e#e#eU6KcEdEd#e#e#eff r0NFTrrrroutput_hidden_states return_dictrzc"|rdnd}|rdnd}t|jD]7\} } |r||fz}| |||| |} | d}|r || dfz}8|r||fz}|std|||fDSt|||S)Nr+rrc3K|]}||V dSror+)r,vs r.r/z,MobileBertEncoder.forward..=s(hhqZ[ZgZgZgZgZghhr0)last_hidden_stater attentions)r;rFrr ) rurrrrrGrHall_hidden_statesall_attentionsr? layer_modulers r.r|zMobileBertEncoder.forwards#7@BBD0:d(44 F FOA|# I$58H$H!(L! ! M *!,M  F!/=3C2E!E  E 1]4D D  ihh]4E~$Vhhhhh h+;LYg    r0)NNFFT)r~rrrqrXrrrrrrr r|rrs@r.rBrBsggggg7;15,1/4&*" " |" !!23" E-. " $D> " 'tn " d^"  uo% &" " " " " " " " r0rBcBeZdZfdZdejdejfdZxZS)MobileBertPoolerct|j|_|jr&t j|j|j|_dSdSro)rprqclassifier_activation do_activaterrrrrs r.rqzMobileBertPooler.__init__DsY !7   K6#5v7IJJDJJJ K Kr0rrzc|dddf}|js|S||}tj|}|S)Nr)rUrrXtanh)rurfirst_token_tensor pooled_outputs r.r|zMobileBertPooler.forwardJsO+111a40 !% % JJ'9::M!J}55M r0r}rs@r.rRrRCscKKKKK !U\ !el ! ! ! ! ! ! ! !r0rRcBeZdZfdZdejdejfdZxZS)!MobileBertPredictionHeadTransformcXttj|j|j|_t |jtrt|j|_ n |j|_ td|j|j |_ dS)Nrr)rprqrrrrrrrr transform_act_fnrrr rs r.rqz*MobileBertPredictionHeadTransform.__init__Ws Yv163EFF f' - - 6$*6+<$=D ! !$*$5D ! .v/AvG\]]]r0rrzc||}||}||}|Sro)rr]r rs r.r|z)MobileBertPredictionHeadTransform.forward`s= =11 --m<< }55 r0r}rs@r.r[r[Vsc^^^^^U\elr0r[cJeZdZfdZddZdejdejfdZxZS)MobileBertLMPredictionHeadctt||_t j|j|j|jz d|_ t j|j|jd|_ t j tj |j|_|j|j _dS)NF)r6)rprqr[ transformrrrrrrdecoderrrrXrsr6rs r.rqz#MobileBertLMPredictionHead.__init__hs :6BBYv0&2DvG\2\chiii y!68IPUVVV LV->!?!?@@  I r0rzNc(|j|j_dSro)r6rcrus r. _tie_weightsz'MobileBertLMPredictionHead._tie_weightsss I r0rc||}|tj|jj|jjgd}||jjz }|S)Nrr) rbrrXrrcr3trr6rs r.r|z"MobileBertLMPredictionHead.forwardvsh}55 %,,UY 8K8M8M8O8OQUQ[Qb7cij-k-k-kll ** r0)rzN) r~rrrqrfrXrr|rrs@r.r`r`gsr & & & & &&&&&U\elr0r`cBeZdZfdZdejdejfdZxZS)MobileBertOnlyMLMHeadcptt||_dSro)rprqr` predictionsrs r.rqzMobileBertOnlyMLMHead.__init__~s/ 5f==r0sequence_outputrzc0||}|Sro)rl)rurmprediction_scoress r.r|zMobileBertOnlyMLMHead.forwards ,,_==  r0r}rs@r.rjrj}s^>>>>>!u|! !!!!!!!!r0rjc\eZdZfdZdejdejdeejfdZxZS)MobileBertPreTrainingHeadsctt||_t j|jd|_dSNr:)rprqr`rlrrrseq_relationshiprs r.rqz#MobileBertPreTrainingHeads.__init__sF 5f== " &*>>>>.- v 6 7 7 % K  " " $ $ $ M  $ $S ) ) ) ) )  : ; ; % K  " " $ $ $ $ $ % %r0N) r~rrr__annotations__rkload_tf_weightsbase_model_prefixrr+r0r.rxrxs< 3O$%%%%%r0rxz6 Output type of [`MobileBertForPreTraining`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)MobileBertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). seq_relationship_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logitsseq_relationship_logitsrrM)r~rrrrrrXrrrrrrrMr+r0r.rrs  )-D(5$ %,,,59x 12999;?Xe&78???8cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN150P0PQ_al0m0m&&y$+2OPP ??l>iv+  ,, 2/!5# '  *!,8< 8O O444UY  J#]3oabb6II I)-')7&1     r0)T) NNNNNNNNN)r~rrrrqrrrrrrXrrrrrr r|rrs@r.rrsu      ///000CCC156:59371559/3,0&*D D E,-D !!23D !!12 D u/0 D E-. D   12D 'tnD $D>D d^D  u00 1D D D ^D D D D D r0rz MobileBert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `next sentence prediction (classification)` head. ceZdZddgZfdZdZdZddeede j ffd Z e dd ee jd ee jd ee jd ee jdee jdee jdee jdee jdee jdee jdee jdeeeffdZxZS)MobileBertForPreTrainingcls.predictions.decoder.weightcls.predictions.decoder.biasct|t||_t ||_|dSro)rprqrr#rqclsrrs r.rqz!MobileBertForPreTraining.__init__9sQ    )&11-f55 r0c$|jjjSrorrlrcres r.get_output_embeddingsz.MobileBertForPreTraining.get_output_embeddingsAx#++r0cT||jj_|j|jj_dSrorrlrcr6runew_embeddingss r.set_output_embeddingsz.MobileBertForPreTraining.set_output_embeddingsD%'5$$2$7!!!r0Nnew_num_tokensrzc||jjj|d|jj_t |SNT)r transposed)r_get_resized_lm_headrrlrrpresize_token_embeddingsrurrxs r.rz0MobileBertForPreTraining.resize_token_embeddingsHsR%)%>%> H &~RV&?& & "ww..n.MMMr0rrrrrrlabelsnext_sentence_labelrrGrHc .| | n |jj} |||||||| | |  } | dd\} }|| |\}}d}||t }||d|jj|d}||dd|d}||z}| s||f| ddz}||f|zn|St|||| j| j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` next_sentence_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`: - 0 indicates sequence B is a continuation of sequence A, - 1 indicates sequence B is a random sequence. Examples: ```python >>> from transformers import AutoTokenizer, MobileBertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForPreTraining.from_pretrained("google/mobilebert-uncased") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> seq_relationship_logits = outputs.seq_relationship_logits ```NrrrrrrrGrHr:r)rrrrrM) r\rr#rrrrrrrM)rurrrrrrrrrrGrHrrmrYrorv total_lossloss_fctmasked_lm_lossnext_sentence_lossrs r.r|z MobileBertForPreTraining.forwardPsnV&1%9=<@37K K E,-K !!23K !!12 K u/0 K E-. K   12K )*K &e&67K $E$56K 'u'89K e/0K  u44 5K K K ^K K K K K r0rceZdZddgZfdZdZdZddeede j ffd Z e dd ee jd ee jd ee jd ee jdee jdee jdee jdeedeedeedeeeffdZxZS)MobileBertForMaskedLMrrct|t|d|_t ||_||_|dSNF)r)rprqrr#rjrr\rrs r.rqzMobileBertForMaskedLM.__init__s]    )&EJJJ(00  r0c$|jjjSrorres r.rz+MobileBertForMaskedLM.get_output_embeddingsrr0cT||jj_|j|jj_dSrorrs r.rz+MobileBertForMaskedLM.set_output_embeddingsrr0Nrrzc||jjj|d|jj_t |Srrrs r.rz-MobileBertForMaskedLM.resize_token_embeddingssR%)%>%> H &~RV&?& & "ww..n.MMMr0rrrrrrrrrGrHc | | n |jj} ||||||||| |  } | d} || } d}|Kt }|| d|jj|d}| s| f| ddz}||f|zn|St|| | j| j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Nrrrr:rlogitsrrM) r\rr#rrrrrrrM)rurrrrrrrrrGrHrrmrorrrs r.r|zMobileBertForMaskedLM.forwards(&1%2 'tn2 d^2  un$ %2 2 2 ^2 2 2 2 2 r0rcBeZdZfdZdejdejfdZxZS)MobileBertOnlyNSPHeadcttj|jd|_dSrs)rprqrrrrtrs r.rqzMobileBertOnlyNSPHead.__init__s6  " &*>> from transformers import AutoTokenizer, MobileBertForNextSentencePrediction >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("google/mobilebert-uncased") >>> model = MobileBertForNextSentencePrediction.from_pretrained("google/mobilebert-uncased") >>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced." >>> next_sentence = "The sky is blue due to the shorter wavelength of blue light." >>> encoding = tokenizer(prompt, next_sentence, return_tensors="pt") >>> outputs = model(**encoding, labels=torch.LongTensor([1])) >>> loss = outputs.loss >>> logits = outputs.logits ```rzoThe `next_sentence_label` argument is deprecated and will be removed in a future version, use `labels` instead.Nrrrr:r) warningswarn FutureWarningpopr\rr#rrrrrrM)rurrrrrrrrrGrHkwargsrrYrvrrrs r.r|z+MobileBertForNextSentencePrediction.forward sIR !F * * M%    ZZ 566F%0%O 'tnO d^O  u11 2O O O ^O O O O O r0rz MobileBert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)#MobileBertForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSro)rprq num_labelsr\rr#classifier_dropoutrrrrrrr9rrur\rrxs r.rqz,MobileBertForSequenceClassification.__init__ds     + )&11)/)B)NF % %TZTn z"455 )F$68IJJ r0NrrrrrrrrrGrHrzc | | n |jj} ||||||||| |  } | d} || } || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j|d}n*|jjdkrt}|| |}| s| f| ddz}||f|zn|St!|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationrr:r)r\rr#rr9 problem_typerrrXrrSr squeezerrrrrrM)rurrrrrrrrrGrHrrYrrrrs r.r|z+MobileBertForSequenceClassification.forwardss(&1%E 'tnE d^E  uU\"$<< =E E E ^E E E E E r0rcxeZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eejd eed eed eede e eje ffdZ xZ S)MobileBertForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSr) rprqrrr#rrr qa_outputsrrs r.rqz'MobileBertForQuestionAnswering.__init__sj     +)&EJJJ)F$68IJJ r0Nrrrrrrstart_positions end_positionsrrGrHrzc | | n |jj} |||||||| | |  } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) Nrrrrr) ignore_indexr:)r start_logits end_logitsrrM)r\rr#rrLrrrRrclamprrrrM)rurrrrrrrrrrGrHrrmrrrr ignored_indexr start_lossend_lossrs r.r|z&MobileBertForQuestionAnswering.forwards&1% > EL)> !.> !. > u|, > EL) >   -> "%,/>   -> $D>> 'tn> d^>  uU\"$@@ A> > > ^> > > > > r0rc\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S)MobileBertForMultipleChoicec4t|t||_|j|jn|j}t j||_t j |j d|_ | dS)Nr) rprqrr#rrrrrrrr9rrs r.rqz$MobileBertForMultipleChoice.__init__s    )&11)/)B)NF % %TZTn z"455 )F$6:: r0NrrrrrrrrrGrHrzc | | n |jj} | |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||| |  } | d} || } || }|d| }d}|t}|||}| s|f| ddz}||f|zn|St||| j | j S)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where `num_choices` is the size of the second dimension of the input tensors. (See `input_ids` above) Nrrrrr:r) r\rrUrrr#rr9rrrrM)rurrrrrrrrrGrH num_choicesrrYrreshaped_logitsrrrs r.r|z#MobileBertForMultipleChoice.forwards+X&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  // ))%'/!5#"      ]33 // ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     r0r)r~rrrqrrrXrrrrrr|rrs@r.rr sL     -11515/3,004)-,0/3&*X X EL)X !.X !. X u|, X EL) X   -X &X $D>X 'tnX d^X  uU\"$== >X X X ^X X X X X r0rc\eZdZfdZe ddeejdeejdeejdeejdeejdeejd eejd eed eed eed e e eje ffdZ xZ S) MobileBertForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr)rprqrrr#rrrrrrrr9rrs r.rqz)MobileBertForTokenClassification.__init__zs     +)&EJJJ)/)B)NF % %TZTn z"455 )F$68IJJ r0NrrrrrrrrrGrHrzc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrr:r) r\rr#rr9rrrrrrM)rurrrrrrrrrGrHrrmrrrrs r.r|z(MobileBertForTokenClassification.forwards$&1%2 'tn2 d^2  uU\"$99 :2 2 2 ^2 2 2 2 2 r0r) rrrrrrrr2rrxrk)JrrBr dataclassesrtypingrrrXrtorch.nnrrr activationsr modeling_outputsr r rrrrrrmodeling_utilsr pytorch_utilsrrutilsrrrconfiguration_mobilebertr get_loggerr~r@rkModulermr rrrrrrr rrrr(r,r2rBrRr[r`rjrqrxrrrrrrrrrr__all__r+r0r.rs. !!!!!!"""""""" AAAAAAAAAA!!!!!!                    .-----QQQQQQQQ9999999999666666  H % %KKK\66666RY666& 9 9IIIII29IIIX;;;;;bi;;;|29"/////")///d     RY        ry   ry0     bi   !]!]!]!]!]!]!]!]H             ry   <<<<<bi<<<~' ' ' ' '  ' ' ' T!!!!!ry!!!& ",!!!!!BI!!! 9 9 9 9 9 9 9 9%%%%%%%%0  :::::[::   :&g g g g g /g g g T f f f f f 8f f  f RM M M M M 5M M M `&&&&&BI&&& Z Z Z Z Z *CZ Z  Z zV V V V V *CV V V rJ J J J J %>J J J Zg g g g g ";g g g TB B B B B '@B B B J   r0