`iQdZddlZddlZddlmZddlmZmZddlZddlm Z ddl m Z m Z m Z ddlmZdd lmZdd lmZmZmZmZmZmZmZdd lmZdd lmZmZmZdd l m!Z!m"Z"m#Z#ddl$m%Z%e#j&e'Z(dZ)Gdde j*Z+Gdde j*Z,Gdde,Z-e,e-dZ.Gdde j*Z/Gdde j*Z0Gdde j*Z1e"GddeZ2ee"d !Gd"d#e!Z3e"Gd$d%e2Z4e"d&!Gd'd(e2Z5Gd)d*e j*Z6Gd+d,e j*Z7e"Gd-d.e2Z8e"d/!Gd0d1e2Z9e"Gd2d3e2Z:e"Gd4d5e2Z;e"Gd6d7e2Zdd?krt+|d.}n|d7kr|| } |j| jkr t5d@|jdA| jdBn/#t4$r"}|xj|j| jfz c_d}~wwxYwtdC| dD| t9j| |_k|S)Ez'Load tf checkpoints in a pytorch model.rNzLoading a TensorFlow model in PyTorch, requires TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.z&Converting TensorFlow checkpoint from zLoading TF weight z with shape zmodule/ffn_1ffnzbert/zalbert/ attention_1 attentionz transform/ LayerNorm_1full_layer_layer_norm LayerNormzattention/LayerNormz transformer/zintermediate/dense/zffn/intermediate/output/dense/z ffn_output/z/output//z/self/z pooler/densepoolerzcls/predictions predictionszpredictions/attentionzembeddings/attention embeddings inner_group_zalbert_layers/group_zalbert_layer_groups/r output_biasoutput_weightsz classifier/seq_relationshipzseq_relationship/output_zsop_classifier/classifier/weightsweightadam_madam_vAdamWeightDecayOptimizerAdamWeightDecayOptimizer_1 global_stepz Skipping z [A-Za-z]+_\d+z_(\d+)kernelgammabetabiassquad classifieri _embeddingszPointer shape z and array shape z mismatchedzInitialize PyTorch weight z from )renumpy tensorflow ImportErrorloggererrorospathabspathinfotrainlist_variables load_variableappendzipprintreplacelensplitjoin fullmatchgetattrAttributeErrorint transposeshape ValueErrorargstorch from_numpydata)modelconfigtf_checkpoint_pathr>nptftf_path init_varsnamesarraysnamerWarray original_namepointerm_name scope_namesnumes ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/albert/modeling_albert.pyload_tf_weights_in_albertro2s3     Q      goo011G KKBBBCCC''00I E F  e BBB5BBCCC&&w55 T e5&)) e d 5&))]/]/ e ||Ir**||GU++||GY//||M;77||L"--||M+BCC||K)>??||NB//||1266||>GG%KK J J"%::  ': 5 5"&":":<"H"H  - -J^^J// \\*-- r)NNNNr)__name__ __module__ __qualname____doc__rrrrZ LongTensor FloatTensorrUTensorr __classcell__rs@rnrqrqs |      .15593759&' ''E,-'!!12'u/0 '   12 ' !$ ' ''''''''rrqceZdZdeffd ZdeeddfdZ d dej d e ej d e ej d e de eej eej ej fff d ZxZS)AlbertAttentionr^ct|j|jzdkr/t |dst d|jd|j|j|_|j|_|j|jz|_|j|jz|_tj |j|j|_ tj |j|j|_ tj |j|j|_ tj |j|_tj |j|_tj |j|j|_tj|j|j|_t+|_t/|dd|_|jdks |jd kr8|j|_tjd |jzd z |j|_dSdS) NrrzThe hidden size (z6) is not a multiple of the number of attention heads (rtryrz relative_keyrelative_key_queryr<r)r~r hidden_sizenum_attention_headsrrXattention_head_size all_head_sizerLinearquerykeyvaluerattention_probs_dropout_probattention_dropoutroutput_dropoutdenser%rset pruned_headsrSryrrdistance_embeddingrs rnrzAlbertAttention.__init__s    : :a ? ?PVXhHiHi ?7F$677 477  $*#= !-#)#59S#S !58PPYv143EFF 9V/1CDDYv143EFF !#F,O!P!P j)CDDYv163EFF f&8f>STTTEE'.v7PR\']']$  '> 9 9T=Y]q=q=q+1+ID (&(l1v7U3UXY3Y[_[s&t&tD # # #>r=qrheadsrNct|dkrdSt||j|j|j\}}t |j||_t |j||_t |j||_t |j |d|_ |jt|z |_|j|jz|_ |j ||_dS)Nrrdim) rOrrrrrrrrrrunion)rrindexs rn prune_headszAlbertAttention.prune_headss u::?? F7 4+T-EtGX  u ( E:: %dh66' E:: ' EqAAA $(#;c%jj#H !58PP -33E::rF hidden_statesattention_mask head_maskoutput_attentionsc|j\}}}||}||} ||} ||d|j|jdd}| |d|j|jdd} | |d|j|jdd} tj || dd} | tj |jz } || |z} |j dks |j dkr4| d}tj|tj|jdd} tj|tj|jdd} | | z }|||jzdz }||j}|j dkrtjd ||}| |z} n?|j dkr4tjd ||}tjd | |}| |z|z} t,j| d }||}|||z}tj || }|ddd}||}||}|||z}|r||fn|fS) Nrwrr<rrrr|zbhld,lrd->bhlrzbhrd,lrd->bhlrr)rWrrrviewrrrVrZmatmulmathsqrtryrrrrrrtor}einsumr functionalsoftmaxrflattenrrr%)rrrrr batch_sizer_ query_layer key_layer value_layerattention_scoresposition_ids_lposition_ids_rdistancepositional_embeddingrelative_position_scoresrelative_position_scores_queryrelative_position_scores_keyattention_probs context_layerprojected_context_layerprojected_context_layer_dropoutlayernormed_context_layers rnrzAlbertAttention.forward#sR%2$7! Jjj// HH]++ jj// !&&z2t7OQUQijjtt q  NN:r43KTMeffppqrtuvv !&&z2t7OQUQijjtt q  !< Y5H5HR5P5PQQ+di8P.Q.QQ  %/.@   '> 9 9T=Y]q=q=q&++--a0J"\*EJ}OcdddiijlnoppN"\*EJ}OcdddiijkmoppN%6H#'#:#:8dFb;bef;f#g#g #7#:#:AR#:#S#S +~==+0<8H+Wk+l+l(#36N#N  -1EEE16>NP[]q1r1r./4|U*V*V'$(NN=Cb3b$c$c!?Pr)?;;WpVrrrNNF)rrrrrlistrUrrZrrrboolrtuplerrrs@rnrrsu|uuuuuu8;c;t;;;;*7;15"' <s<s|<s!!23<sE-. <s  <s uU\"E%, *D$EE F <s<s<s<s<s<s<s<srrceZdZfdZ d dejdeejdeejdede e eje ejejfff fd Z xZ S) AlbertSdpaAttentioncbt||j|_dSN)r~rr dropout_probrs rnrzAlbertSdpaAttention.__init__cs,    "?rNFrrrrrc|jdks|r>tdt|||S|\}}}|||d|j|j  dd}| ||d|j|j  dd} | ||d|j|j  dd} tjj|| | ||jr|jndd } | dd} | |||j} || } || } ||| z}|fS) NrzaAlbertSdpaAttention is used but `torch.nn.functional.scaled_dot_product_attention` does not support non-absolute `position_embedding_type` or `output_attentions=True` . Falling back to the eager attention implementation, but specifying the eager implementation will be required from Transformers version v5.0.0 onwards. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.)rrwrr<F)rrr attn_mask dropout_p is_causal)ryrBwarningr~rrrrrrrVrrrZrrscaled_dot_product_attentiontrainingrreshaperrrr%)rrrrrrseq_lenrrrrattention_outputrrrrs rnrzAlbertSdpaAttention.forwardgs  ': 5 59J 5 NNH   77??=.Te?ff f!.!3!3!5!5 GQ JJ} % % T*b$":D*.*=*=>U*V*V'$(NN=Cb3b$c$c!)++rr) rrrrrZrrrrrrrrrs@rnrrbs@@@@@7;15"' 1,1,|1,!!231,E-. 1,  1, uU\"E%, *D$EE F 1,1,1,1,1,1,1,1,1,1,rr)eagersdpaceZdZdeffd Z ddejdeejdeejde d e d e ejejff d Z d ejd ejfd Z xZ S) AlbertLayerr^ct||_|j|_d|_t j|j|j|_ t|j ||_ t j |j|j|_t j |j|j|_t"|j|_t j|j|_dS)Nrrt)r~rr^chunk_size_feed_forward seq_len_dimrr%rrr$ALBERT_ATTENTION_CLASSES_attn_implementationr"rintermediate_sizer ffn_outputr hidden_act activationrrrrs rnrzAlbertLayer.__init__s  '-'E$%'\&2D&J_%`%`%`"1&2MNvVV9V/1IJJ)F$PQQ !23z&"<== rNFrrrroutput_hidden_statesrc|||||}t|j|j|j|d}|||dz}|f|ddzS)Nrr)r"rff_chunkrrr$)rrrrrr rrs rnrzAlbertLayer.forwards~ >>-Teff. M  (   Q    22:@PQR@S3STT "2122"666rrc||}||}||}|Sr)r r r)rrrs rnr zAlbertLayer.ff_chunks<XX.// __Z00 __Z00 rNNFF)rrrrrrZrrrrrrr rrs@rnrrs >| > > > > > > 7;15"'%* 77|7!!237E-. 7  7 # 7 u|U\) *7777(%,rrceZdZdeffd Z d dejdeejdeejde d e d e e eje ejfd ff d Z xZ S)AlbertLayerGroupr^cttjfdt jD|_dS)Nc.g|]}tS)r.0rr^s rn z-AlbertLayerGroup.__init__..s!+g+g+gAK,?,?+g+g+gr)r~rr ModuleListrangeinner_group_num albert_layersrs `rnrzAlbertLayerGroup.__init__sR ]+g+g+g+gvOeIfIf+g+g+ghhrNFrrrrr r.cd}d}t|jD]7\}} | |||||} | d}|r || dfz}|r||fz}8|f} |r| |fz} |r| |fz} | S)Nrrr) enumerater) rrrrrr layer_hidden_stateslayer_attentions layer_index albert_layer layer_outputoutputss rnrzAlbertLayerGroup.forwards!)243E)F)F M M %K'< ~yQ\G]_pqqL(OM  I#3|A6H#H # M&9].s"1t1t1tq2B62J2J1t1t1tr) r~rr^rrrrembedding_hidden_mapping_inrrnum_hidden_groupsalbert_layer_groupsrs `rnrzAlbertTransformer.__init__su  +-9V5JFL^+_+_(#%=1t1t1t1tTYZ`ZrTsTs1t1t1t#u#u   rNFTrrrrr  return_dictrc 4||}|r|fnd}|rdnd}|dg|jjzn|}t|jjD]} t |jj|jjz } t | |jj|jjz z } |j| |||| | z| dz| z||} | d}|r || dz}|r||fz}|std|||fDSt|||S)Nrrrrwc3K|]}||V dSrr)rvs rn z,AlbertTransformer.forward..s(hhqZ[ZgZgZgZgZghhr)last_hidden_stater attentions) r'r^num_hidden_layersrrUr(r)rr ) rrrrrr r*all_hidden_statesall_attentionsilayers_per_group group_idxlayer_group_outputs rnrzAlbertTransformer.forwards88GG 0DN],,$0:d>G>OTFT[:::U^ t{455 I IA"4;#@4;C`#`aa A!>A^!^_``I!D!9)!D)&66)a-K[9[[\!$ "" /q1M  I!/2DR2H!H# I$58H$H! ihh]4E~$Vhhhhh h+;LYg    r)NNFFT)rrrrrrZrrrrrr rrrrs@rnr$r$sv|vvvvvv7;15"'%* * * |* !!23* E-. *  * # * *  % &* * * * * * * * rr$c,eZdZUeed<eZdZdZdZ dS)AlbertPreTrainedModelr^albertTct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSt|tr |jj dSdS)zInitialize the weights.r)meanstdN?) isinstancerrr0r\normal_r^initializer_ranger9zero_rrsr%fill_ AlbertMLMHead)rmodules rn _init_weightsz#AlbertPreTrainedModel._init_weights)sX fbi ( ( % M  & &CT[5R & S S S{&  &&((((('&  - - % M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - % K  " " $ $ $ M  $ $S ) ) ) ) )  . . % K  " " $ $ $ $ $ % %rN) rrrr__annotations__roload_tf_weightsbase_model_prefix_supports_sdparFrrrnr9r9"sA /O N%%%%%rr9z2 Output type of [`AlbertForPreTraining`]. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ ee ejed<dZ ee ejed<dS)AlbertForPreTrainingOutputa loss (*optional*, returned when `labels` is provided, `torch.FloatTensor` of shape `(1,)`): Total loss as the sum of the masked language modeling loss and the next sequence prediction (classification) loss. prediction_logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). sop_logits (`torch.FloatTensor` of shape `(batch_size, 2)`): Prediction scores of the next sequence prediction (classification) head (scores of True/False continuation before SoftMax). Nlossprediction_logits sop_logitsrr0)rrrrrNrrZrrGrOrPrrr0rrrnrMrM<s  )-D(5$ %,,,59x 12999.2J*+2228 '-'E$ rrc|jjSrr)rrs rnget_input_embeddingsz AlbertModel.get_input_embeddingsqs ..rrNc||j_dSrr[)rrs rnset_input_embeddingsz AlbertModel.set_input_embeddingsts*/'''rheads_to_prunec|D]u\}}t||jjz }t|||jjzz }|jj|j|j|vdS)a Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} ALBERT has a different architecture in that its layers are shared across groups, which then has inner groups. If an ALBERT model has 12 hidden layers and 2 hidden groups, with two inner groups, there is a total of 4 different layers. These layers are flattened: the indices [0,1] correspond to the two inner groups of the first hidden layer, while [2,3] correspond to the two inner groups of the second hidden layer. Any layer with in index other than [0,1,2,3] will result in an error. See base class PreTrainedModel for more information about head pruning N) itemsrUr^rrUr)rr"r)rr`layerrr6inner_group_idxs rn _prune_headszAlbertModel._prune_headsws+0022 t tLE5EDK$??@@I!%)dk6Q*Q"QRRO L ,Y 7 Eo V ` l lmr s s s s t trrrr{rvrrrr r*c 0||n |jj}||n |jj}| | n |jj} ||t d|+||||} n.||dd} nt d| \} } ||jn|j} |tj | | }|gt|j dr1|j j ddd| f}| | | }|}n!tj| tj| }| ||||}|jdko|jd ko|duo| }|rt%||j| }ng|d d }||j }d|z tj|jjz}|||jj}||||||| }|d}|j2|||dddfnd}| s||f|d dzSt;|||j|jS)NzDYou cannot specify both input_ids and inputs_embeds at the same timerwz5You have to specify either input_ids or inputs_embeds)rr{r)rvr{rrrz)tgt_lenrr<r|r>)rrr r*r)r/ pooler_outputrr0) r^rr use_return_dictrX%warn_if_padding_and_no_attention_maskrrrZonesrr)r{rrrrXryr r} unsqueezerfinfomin get_head_maskr1rUr'rWrrr0)rrrr{rvrrrr r*rrrrrrembedding_outputuse_sdpa_attention_maskextended_attention_maskencoder_outputssequence_output pooled_outputs rnrzAlbertModel.forwards,2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U!, J%.%:!! @T  !"Z FCCCN  !t(899 [*./*HKZK*X'3J3Q3QR\^h3i3i0!A!&[ SY!Z!Z!Z?? L_l+    $ . &, : &T! &&%  # d&I 0 6 ''' # #'5&>&>q&A&A&K&KA&N&N #&=&@&@tz&@&R&R #'*-D'D TXT^H_H_Hc&c #&&y$+2OPP ,,  #/!5# '  *!,VZVaVm..t{{?111a4;P/Q/QRRRsw  J#]3oabb6II I)-')7&1     r)T) NNNNNNNNN)rrrrrGrIrrrrr]r_dictrUrrerrrZrrrrrrrrs@rnrRrRUs  |./bl////0",040000t4T#Y+?tDtttt"156:59371559,0/3&*T T E,-T !!23T !!12 T u/0 T E-. T   12T $D>T 'tnT d^T  )50 1T T T ^T T T T T rrRz Albert Model with two heads on top as done during the pretraining: a `masked language modeling` head and a `sentence order prediction (classification)` head. ceZdZddgZdeffd ZdejfdZdejddfd Z dej fd Z e dd e ejd e ejd e ejde ejde ejde ejde ejde ejde ede ede edeeeffdZxZS)AlbertForPreTrainingpredictions.decoder.biaspredictions.decoder.weightr^ct|t||_t ||_t ||_|dSr) r~rrRr:rDr( AlbertSOPHeadsop_classifierrYrs rnrzAlbertForPreTraining.__init__sb    !&)) (00+F33 rrc|jjSrr(decoderr\s rnget_output_embeddingsz*AlbertForPreTraining.get_output_embeddings''rnew_embeddingsNc||j_dSrrrrs rnset_output_embeddingsz*AlbertForPreTraining.set_output_embeddingss#1   rc$|jjjSrr:r)rr\s rnr]z)AlbertForPreTraining.get_input_embeddings{%55rrrr{rvrrlabelssentence_order_labelrr r*c P| | n |jj} |||||||| | |  } | dd\} }|| }||}d}||t }||d|jj|d}||dd|d}||z}| s||f| ddz}||f|zn|St|||| j | j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` sentence_order_label (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the next sequence prediction (classification) loss. Input should be a sequence pair (see `input_ids` docstring) Indices should be in `[0, 1]`. `0` indicates original order (sequence A, then sequence B), `1` indicates switched order (sequence B, then sequence A). Example: ```python >>> from transformers import AutoTokenizer, AlbertForPreTraining >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForPreTraining.from_pretrained("albert/albert-base-v2") >>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0) >>> # Batch size 1 >>> outputs = model(input_ids) >>> prediction_logits = outputs.prediction_logits >>> sop_logits = outputs.sop_logits ```Nrr{rvrrrr r*r<rw)rNrOrPrr0) r^rir:r(r}rrrrMrr0)rrrr{rvrrrrrr r*r"rtruprediction_scores sop_scores total_lossloss_fctmasked_lm_losssentence_order_lossoutputs rnrzAlbertForPreTraining.forwardswR&1%L 'tnL d^L  )50 1L L L ^L L L L L rrxcPeZdZdeffd ZdejdejfdZddZxZ S) rDr^cttj|j|j|_tjtj|j |_ tj |j |j|_ tj |j|j |_t|j|_|j |j_ dS)Nrt)r~rrr%rr ParameterrZrrr9rrrrr rr rs rnrzAlbertMLMHead.__init__Ms f&;AVWWWLV->!?!?@@ Yv163HII y!68IJJ  !23 I rrrc||}||}||}||}|}|Sr)rr r%r)rrrs rnrzAlbertMLMHead.forwardWsR =11  66 }55  ]33 )  rNc|jjjjdkr|j|j_dS|jj|_dS)Nmeta)rr9rtyper\s rn _tie_weightszAlbertMLMHead._tie_weightsas< <  # (F 2 2 $ DL    )DIIIr)rN) rrrrrrZrrrrrs@rnrDrDLs~&|&&&&&&!U\!el!!!!********rrDcHeZdZdeffd ZdejdejfdZxZS)r|r^cttj|j|_tj|j|j|_ dSr) r~rrrclassifier_dropout_probrrr num_labelsr;rs rnrzAlbertSOPHead.__init__ksJ z&"@AA )F$68IJJrrurcZ||}||}|Sr)rr;)rrudropout_pooled_outputlogitss rnrzAlbertSOPHead.forwardqs+ $ ] ; ;!677 r) rrrrrrZrrrrs@rnr|r|jsqK|KKKKKK U\elrr|ceZdZddgZfdZdejfdZdejddfdZdej fd Z e dd e e jd e e jd e e jd e e jde e jde e jde e jde ede ede edeeeffdZxZS)AlbertForMaskedLMryrzct|t|d|_t ||_|dSNF)rS)r~rrRr:rDr(rYrs rnrzAlbertForMaskedLM.__init__{sW    !&EBBB (00 rrc|jjSrrr\s rnrz'AlbertForMaskedLM.get_output_embeddingsrrrNc@||j_|j|j_dSr)r(rr9rs rnrz'AlbertForMaskedLM.set_output_embeddingss!#1 . 3rc$|jjjSrrr\s rnr]z&AlbertForMaskedLM.get_input_embeddingsrrrrr{rvrrrrr r*c | | n |jj} ||||||||| |  } | d} || } d}|Kt }|| d|jj|d}| s| f| ddz}||f|zn|St|| | j| j S)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Example: ```python >>> import torch >>> from transformers import AutoTokenizer, AlbertForMaskedLM >>> tokenizer = AutoTokenizer.from_pretrained("albert/albert-base-v2") >>> model = AlbertForMaskedLM.from_pretrained("albert/albert-base-v2") >>> # add mask_token >>> inputs = tokenizer("The capital of [MASK] is Paris.", return_tensors="pt") >>> with torch.no_grad(): ... logits = model(**inputs).logits >>> # retrieve index of [MASK] >>> mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0] >>> predicted_token_id = logits[0, mask_token_index].argmax(axis=-1) >>> tokenizer.decode(predicted_token_id) 'france' ``` ```python >>> labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"] >>> labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100) >>> outputs = model(**inputs, labels=labels) >>> round(outputs.loss.item(), 2) 0.81 ``` N rrr{rvrrrr r*rrwr<rNrrr0) r^rir:r(rrrrrr0)rrrr{rvrrrrr r*r"sequence_outputsrrrrs rnrzAlbertForMaskedLM.forwardsb&1%>  '))H%X&7&<&O 'tnO d^O  ~u$ %O O O ^O O O O O rrz Albert Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. cLeZdZdeffd Ze ddeejdeej deejdeejdeej d eej d eejd ee d ee d ee de e e ffdZxZS)AlbertForSequenceClassificationr^cNt||j|_||_t ||_t j|j|_ t j |j |jj|_ | dSr)r~rrr^rRr:rrrrrrr;rYrs rnrz(AlbertForSequenceClassification.__init__s     + !&)) z&"@AA )F$6 8NOO rNrrr{rvrrrrr r*rc | | n |jj} ||||||||| |  } | d} || } || } d}|Z|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|| |}n|jjdkrGt}|| d|j|d}n*|jjdkrt}|| |}| s| f| ddz}||f|zn|St!|| | j| j S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Nrr regressionsingle_label_classificationmulti_label_classificationrwr<r)r^rir:rr; problem_typerr}rZrrUr squeezerrrrrr0)rrrr{rvrrrrr r*r"rurrNrrs rnrz'AlbertForSequenceClassification.forwards(&1%F 'tnF d^F  '. /F F F ^F F F F F rrcLeZdZdeffd Ze ddeejdeej deejdeejdeej d eej d eejd ee d ee d ee de e e ffdZxZS)AlbertForTokenClassificationr^cdt||j|_t|d|_|j|jn|j}tj||_ tj |j |j j|_ |dSr)r~rrrRr:rrrrrrrr^r;rY)rr^rrs rnrz%AlbertForTokenClassification.__init__@s     +!&EBBB -9  * *+ z"9:: )F$6 8NOO rNrrr{rvrrrrr r*rc | | n |jj} ||||||||| |  } | d} || } || } d}|Ft }|| d|j|d}| s| f| ddz}||f|zn|St|| | j | j S)z labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Nrrrwr<r) r^rir:rr;rrrrrr0)rrrr{rvrrrrr r*r"rtrrNrrs rnrz$AlbertForTokenClassification.forwardPs$&1%sD| 156:59371559-1,0/3&*2 2 E,-2 !!232 !!12 2 u/0 2 E-. 2   122 )*2 $D>2 'tn2 d^2  $e+ ,2 2 2 ^2 2 2 2 2 rrcheZdZdeffd Ze ddeejdeej deejdeejdeej d eej d eejd eejd ee d ee dee de e e ffdZxZS)AlbertForQuestionAnsweringr^ct||j|_t|d|_t j|j|j|_| dSr) r~rrrRr:rrr qa_outputsrYrs rnrz#AlbertForQuestionAnswering.__init__sj     +!&EBBB )F$68IJJ rNrrr{rvrrstart_positions end_positionsrr r*rc | | n |jj} |||||||| | |  } | d} || }|dd\}}|d}|d}d}||t|dkr|d}t|dkr|d}|d}| d|}| d|}t|}|||}|||}||zdz }| s||f| ddz}||f|zn|St|||| j | j S) Nrrrrwr) ignore_indexr<)rN start_logits end_logitsrr0)r^rir:rrPr contiguousrOrclamprrrr0)rrrr{rvrrrrrr r*r"rtrrrr ignored_indexr start_lossend_lossrs rnrz"AlbertForQuestionAnswering.forwards&1% > E,-> !!23> !!12 > u/0 > E-. >   12> "%"23>   01> $D>> 'tn> d^>  )50 1> > > ^> > > > > rrcLeZdZdeffd Ze ddeejdeej deejdeejdeej d eej d eejd ee d ee d ee de e e ffdZxZS)AlbertForMultipleChoicer^ct|t||_t j|j|_t j|j d|_ | dS)Nr) r~rrRr:rrrrrrr;rYrs rnrz AlbertForMultipleChoice.__init__sl    !&)) z&"@AA )F$6:: rNrrr{rvrrrrr r*rc | | n |jj} | |jdn |jd} |)|d|dnd}|)|d|dnd}|)|d|dnd}|)|d|dnd}|=|d|d|dnd}||||||||| |  } | d} || } || }|d| }d}|t}|||}| s|f| ddz}||f|zn|St||| j | j S)a[ input_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.__call__`] and [`PreTrainedTokenizer.encode`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) position_ids (`torch.LongTensor` of shape `(batch_size, num_choices, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.max_position_embeddings - 1]`. [What are position IDs?](../glossary#position-ids) inputs_embeds (`torch.FloatTensor` of shape `(batch_size, num_choices, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the multiple choice classification loss. Indices should be in `[0, ..., num_choices-1]` where *num_choices* is the size of the second dimension of the input tensors. (see *input_ids* above) Nrrwrrr<r) r^rirWrrr:rr;rrrr0)rrrr{rvrrrrr r* num_choicesr"rurreshaped_logitsrNrrs rnrzAlbertForMultipleChoice.forwards+X&1%G>SINN2y~~b'9'9:::Y] M[Mg,,R1D1DR1H1HIIImqM[Mg,,R1D1DR1H1HIIImqGSG_|((\->->r-B-BCCCei (   r=#5#5b#9#9=;M;Mb;Q;Q R R R  ++ ))%'/!5#      ]33 #}== ++b+66  '))H8OV44D F%''!""+5F)-)9TGf$$v E("!/)     rrrrs@rnrrsX|156:59371559-1,0/3&*W W E,-W !!23W !!12 W u/0 W E-. W   12W )*W $D>W 'tnW d^W  )50 1W W W ^W W W W W rr) ror9rRrxrrrrr)>rrrD dataclassesrtypingrrrZrtorch.nnrrr activationsr modeling_attn_mask_utilsr modeling_outputsr rrrrrrmodeling_utilsr pytorch_utilsrrrutilsrrrconfiguration_albertr get_loggerrrBroModulerqrrrrrr$r9rMrRrxrDr|rrrrr__all__rrrnrs !!!!!!"""""""" AAAAAAAAAA!!!!!!KKKKKK.----- :999999999......  H % %{{{|AAAAAryAAAHksksksksksbiksksks\6,6,6,6,6,/6,6,6,t  &&&&&")&&&R     ry   F2 2 2 2 2  2 2 2 j%%%%%O%%%2  :::::::   :&G G G G G 'G G G T c c c c c 0c c  c L*****BI***<     BI   f f f f f -f f f R T T T T T &;T T  T nD D D D D #8D D D NJ J J J J !6J J J Zc c c c c 3c c c L   r