`ioddZddlZddlmZmZmZddlZddlmZddlm Z m Z m Z ddl m Z ddlmZdd lmZmZmZmZmZmZdd lmZmZdd lmZmZmZdd lmZm Z m!Z!d dl"m#Z#e!j$e%Z&Gddej'Z(dBdZ)Gddej'Z*Gddej'Z+Gddej'Z,Gddej'Z-Gddej'Z.Gddej'Z/Gddej'Z0Gd d!ej'Z1 dCd#ej'd$ej2d%ej2d&ej2d'eej2d(e3d)e3d*eej2fd+Z4Gd,d-ej'Z5Gd.d/ej'Z6Gd0d1eZ7Gd2d3ej'Z8eGd4d5eZ9eGd6d7e9Z:eGd8d9e9Z;ed:;Gd<d=e9Z<ed>;Gd?d@e9Z=gdAZ>dS)DzPyTorch MarkupLM model.N)CallableOptionalUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringcan_return_tuplelogging)MarkupLMConfigc*eZdZdZfdZddZxZS)XPathEmbeddingszConstruct the embeddings from xpath tags and subscripts. We drop tree-id in this version, as its info can be covered by xpath. ctj|_tjj|jzj|_tjj |_ tj |_ tjj|jzdjz|_ tjdjzj|_tjfdt!|jD|_tjfdt!|jD|_dS)NcNg|]!}tjjj"S)r Embeddingmax_xpath_tag_unit_embeddingsxpath_unit_hidden_size.0_configs /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/markuplm/modeling_markuplm.py z,XPathEmbeddings.__init__..>s;    VA6C`aa   cNg|]!}tjjj"Sr")rr#max_xpath_subs_unit_embeddingsr%r&s r*r+z,XPathEmbeddings.__init__..Es;    VBFDabb   r,)super__init__ max_depthrLinearr% hidden_sizexpath_unitseq2_embeddingsDropouthidden_dropout_probdropoutReLU activationxpath_unitseq2_inner inner2emb ModuleListrangexpath_tag_sub_embeddingsxpath_subs_sub_embeddingsselfr) __class__s `r*r0zXPathEmbeddings.__init__1s? ))+63PSWSa3acicu)v)v&z&"<== '))$&If.Kdn.\^_bhbt^t$u$u!1v'9#96;MNN(*     t~..   ) ) %*,    t~..   * * &&&r,Nc g}g}t|jD]n}||j||dddd|f||j||dddd|fot j|d}t j|d}||z}||| | |}|S)Ndim) r=r1appendr>r?torchcatr;r7r9r:)rAxpath_tags_seqxpath_subs_seqxpath_tags_embeddingsxpath_subs_embeddingsixpath_embeddingss r*forwardzXPathEmbeddings.forwardKs' " "t~&& e eA ! ( ()I)Fq)I.YZYZYZ\]\]\]_`Y`Ja)b)b c c c ! ( ()J)G)J>Z[Z[Z[]^]^]^`aZaKb)c)c d d d d % *?R H H H % *?R H H H03HH>>$,,ttG`G`aqGrGr7s7s*t*tuur,)NN)__name__ __module__ __qualname____doc__r0rP __classcell__rBs@r*rr+sV      4        r,rc||}tj|d||z|z}||zS)a Replace non-padding symbols with their position numbers. Position numbers begin at padding_idx+1. Padding symbols are ignored. This is modified from fairseq's `utils.make_positions`. Args: x: torch.Tensor x: Returns: torch.Tensor rrE)neintrHcumsumtype_aslong) input_ids padding_idxpast_key_values_lengthmaskincremental_indicess r*"create_position_ids_from_input_idsrb^sg << $ $ ( ( * *D <!444<eZdZdZfdZdZ ddZxZS)MarkupLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.ct||_tj|j|j|j|_tj|j |j|_ |j |_ t||_ tj|j|j|_tj|j|j|_tj|j|_|dt-j|j dd|j|_tj|j |j|j|_ dS)N)r^eps position_ids)rrDF) persistent)r/r0r)rr# vocab_sizer3 pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsr1rrOtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsr5r6r7register_bufferrHarangeexpandr^r@s r*r0zMarkupLMEmbeddings.__init__qs?  !|F,=v?Q_e_rsss#%<0NPVPb#c#c ) / 7 7%'\&2H&J\%]%]"f&8f>STTTz&"<==   EL)GHHOOPWXXej    ".#%<  *F,>DL\$ $ $    r,c|dd}|d}tj|jdz||jzdztj|j}|d|S)z We are provided embeddings directly. We cannot infer which are padded so just generate sequential position ids. Args: inputs_embeds: torch.Tensor Returns: torch.Tensor NrDrdtypedevicer)sizerHrtr^r\ry unsqueezeru)rA inputs_embeds input_shapesequence_lengthrhs r*&create_position_ids_from_inputs_embedsz9MarkupLMEmbeddings.create_position_ids_from_inputs_embedss$((**3B3/ %a.|  q /D4D"Dq"HPUPZcpcw   %%a((// <<*::;;5:V\777N  !![4uzd;''4>*::;;5:V\888N)"66|DD $ : :> J J00PP%(;;>SSVff ^^J// \\*-- r,)NNNNNNr)rQrRrSrTr0rrPrUrVs@r*rdrdnsxQQ     2===& 22222222r,rdcPeZdZfdZdejdejdejfdZxZS)MarkupLMSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNrf) r/r0rr2r3denserqrrr5r6r7r@s r*r0zMarkupLMSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== r, hidden_states input_tensorreturnc||}||}|||z}|SNrr7rqrArrs r*rPzMarkupLMSelfOutput.forward@ =11  ]33 }|'CDD r,rQrRrSr0rHTensorrPrUrVs@r*rri>>>>> U\RWR^r,rcBeZdZfdZdejdejfdZxZS)MarkupLMIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r/r0rr2r3intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr@s r*r0zMarkupLMIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $r,rrcZ||}||}|Sr)rrrArs r*rPzMarkupLMIntermediate.forwards, =11 00?? r,rrVs@r*rrs^99999U\elr,rcPeZdZfdZdejdejdejfdZxZS)MarkupLMOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r/r0rr2rr3rrqrrr5r6r7r@s r*r0zMarkupLMOutput.__init__sf Yv79KLL f&8f>STTTz&"<== r,rrrc||}||}|||z}|Srrrs r*rPzMarkupLMOutput.forwardrr,rrVs@r*rrrr,rcBeZdZfdZdejdejfdZxZS)MarkupLMPoolercttj|j|j|_tj|_dSr)r/r0rr2r3rTanhr9r@s r*r0zMarkupLMPooler.__init__sC Yv163EFF '))r,rrcr|dddf}||}||}|S)Nr)rr9)rArfirst_token_tensor pooled_outputs r*rPzMarkupLMPooler.forwards@+111a40 #566  66 r,rrVs@r*rrs^$$$$$ U\elr,rcBeZdZfdZdejdejfdZxZS)MarkupLMPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) r/r0rr2r3rrrrr transform_act_fnrqrrr@s r*r0z(MarkupLMPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTr,rrc||}||}||}|Sr)rrrqrs r*rPz'MarkupLMPredictionHeadTransform.forwards= =11 --m<< }55 r,rrVs@r*rrscUUUUUU\elr,rc*eZdZfdZdZdZxZS)MarkupLMLMPredictionHeadc>tt||_t j|j|jd|_t j tj |j|_ |j |j_ dS)NF)bias) r/r0r transformrr2r3rjdecoder ParameterrHrrr@s r*r0z!MarkupLMLMPredictionHead.__init__"sz 8@@y!3V5FUSSS LV->!?!?@@ !I r,c(|j|j_dSr)rrrAs r* _tie_weightsz%MarkupLMLMPredictionHead._tie_weights/s I r,cZ||}||}|Sr)rrrs r*rPz MarkupLMLMPredictionHead.forward2s*}55  ]33 r,)rQrRrSr0rrPrUrVs@r*rr!sV & & & & &&&&r,rcBeZdZfdZdejdejfdZxZS)MarkupLMOnlyMLMHeadcptt||_dSr)r/r0r predictionsr@s r*r0zMarkupLMOnlyMLMHead.__init__:s/ 3F;;r,sequence_outputrc0||}|Sr)r)rArprediction_scoress r*rPzMarkupLMOnlyMLMHead.forward>s ,,_==  r,rrVs@r*rr9s^<<<<<!u|! !!!!!!!!r,rmodulequerykeyvalueattention_maskscalingr7 head_maskc 8tj||dd|z} |$|ddddddd|jdf} | | z} tj| dtj|j } tj | ||j } || | ddddz} tj| |} | dd } | | fS)Nr rD)rFrx)ptrainingr)rHmatmul transposeshaper functionalsoftmaxfloat32torxr7rview contiguous) rrrrrrr7rkwargs attn_weights causal_mask attn_outputs r*eager_attention_forwardrDs<s}}Q':':;;gEL!$QQQ111o " o%=> #k1 =((2U](SSVVW\WbccL=((6?([[L#innQAq&A&AA ,|U33K''1--88::K  $$r,c eZdZfdZ d dejdeejdeejdeede ejf d Z xZ S) MarkupLMSelfAttentionct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_tj|j|_|j|_|jdz|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r/r0r3num_attention_headshasattr ValueErrorr)rYattention_head_size all_head_sizerr2rrrr5attention_probs_dropout_probr7attention_dropoutrr@s r*r0zMarkupLMSelfAttention.__init__as:    : :a ? ?PVXhHiHi ?8F$688 4888   #)#= #&v'9F ! u| !!!!!!!!r,rc eZdZfdZdZ d dejdeejdeejdee d e ejf d Z xZ S) MarkupLMAttentionctt||_t ||_t |_dSr)r/r0rrAroutputset pruned_headsr@s r*r0zMarkupLMAttention.__init__sI )&11 (00 EEr,ct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)NrrrE)lenrrArrrrrrrrrrunion)rAheadsindexs r* prune_headszMarkupLMAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::r,NFrrrrrc ~|j|f|||d|}||d|}|f|ddz}|SN)rrrrr)rAr) rArrrrr self_outputsattention_outputrs r*rPzMarkupLMAttention.forwardsl!ty  )/       ;;|A FF#% QRR(88r,r) rQrRrSr0rrHrrrrrrPrUrVs@r*rrs""""" ;;;*7;15,1 |!!23E-.  $D>  u| r,rc eZdZfdZ d dejdeejdeejdeede ejf d Z d Z xZ S) MarkupLMLayerct|j|_d|_t ||_t ||_t||_ dS)Nr) r/r0chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr@s r*r0zMarkupLMLayer.__init__s^ '-'E$*622088$V,, r,NFrrrrrc |j|f|||d|}|d}|dd}t|j|j|j|} | f|z}|Sr)rrfeed_forward_chunkrr) rArrrrrself_attention_outputsr r layer_outputs r*rPzMarkupLMLayer.forwards"0 " )/ " "  " " 2!4(,0  #T%A4CSUe   /G+r,c\||}|||}|Sr)rr)rAr intermediate_outputrs r*rz MarkupLMLayer.feed_forward_chunks2"//0@AA{{#68HII r,r) rQrRrSr0rHrrrrrrPrrUrVs@r*r r s-----7;15,1 |!!23E-.  $D>  u| 2r,r ceZdZfdZe d dejdeejdeejdee d ee d ee d e e eje ffd Z xZS)MarkupLMEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tSr")r )r'rNr)s r*r+z,MarkupLMEncoder.__init__..s!#c#c#caM&$9$9#c#c#cr,F) r/r0r)rr<r=num_hidden_layerslayergradient_checkpointingr@s `r*r0zMarkupLMEncoder.__init__s`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###r,NFTrrrroutput_hidden_states return_dictrc |rdnd}|rdnd} t|jD]<\} } |r||fz}||| nd} | d||| |d|} | d}|r | | dfz} =|r||fz}t||| S)Nr")rrrrrr)last_hidden_stater attentions) enumeraterr )rArrrrrr rall_hidden_statesall_self_attentionsrN layer_modulelayer_head_mask layer_outputss r*rPzMarkupLMEncoder.forwards#7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO(L+-)"3   M*!,M  P&9]1=M & 'tn & d^&  uU\"O3 4& & & & & & & & r,rcpeZdZUeed<dZdZedee e e j fffd Z xZS)MarkupLMPreTrainedModelr)markuplmct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tj r?|jj |jj ddSt|tr |jj dSdS)zInitialize the weightsr)meanstdN?)rrr2weightdatanormal_r)initializer_rangerzero_r#r^rqfill_r)rArs r* _init_weightsz%MarkupLMPreTrainedModel._init_weights)sY fbi ( ( % M  & &CT[5R & S S S{&  &&((((('&  - - % M  & &CT[5R & S S S!- "6#56<<>>>>>.-  - - % K  " " $ $ $ M  $ $S ) ) ) ) )  8 9 9 % K  " " $ $ $ $ $ % %r,pretrained_model_name_or_pathc>tj|g|Ri|Sr)r/from_pretrained)clsr8 model_argsrrBs r*r:z'MarkupLMPreTrainedModel.from_pretrained;s,&uww&'D\z\\\U[\\\r,)rQrRrSr__annotations__base_model_prefixr7 classmethodrrrosPathLiker:rUrVs@r*r+r+#s "%%%$]HU3PRP[K[E\<]]]]]][]]]]]r,r+ceZdZdfd ZdZdZdZee dde e j de e j d e e j d e e j d e e j d e e j d e e j de e j de e de e de e deeeffdZxZS) MarkupLMModelTct|||_t||_t ||_|rt|nd|_| dS)zv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer N) r/r0r)rdrrencoderrpooler post_init)rAr)add_pooling_layerrBs r*r0zMarkupLMModel.__init__Css     ,V44&v.. 0AKnV,,,t  r,c|jjSrrrlrs r*get_input_embeddingsz"MarkupLMModel.get_input_embeddingsSs ..r,c||j_dSrrJ)rArs r*set_input_embeddingsz"MarkupLMModel.set_input_embeddingsVs*/'''r,c|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsrErrr)rAheads_to_prunerrs r* _prune_headszMarkupLMModel._prune_headsYsU +0022 C CLE5 L u % / ; ;E B B B B C Cr,Nr]rJrKrrrhrr|rrr rc | | n |jj} | | n |jj} | | n |jj} ||t d|+||||} n.||dd} nt d||jn|j} |tj | | }|!tj | tj | }| d d}| |j }d |z d z}||dkrr| d  d  d d}||jjdddd}nS|dkr;| d d d}| t%|j }ndg|jjz}||||||| }||||| | d}|d }|j||nd}t/|||j|jS)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMModel >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = MarkupLMModel.from_pretrained("microsoft/markuplm-base") >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> outputs = model(**encoding) >>> last_hidden_states = outputs.last_hidden_state >>> list(last_hidden_states.shape) [1, 4, 768] ```NzDYou cannot specify both input_ids and inputs_embeds at the same timerDz5You have to specify either input_ids or inputs_embeds)ryrwrr)rxr0gr)r]rJrKrhrr|T)rrrr )r" pooler_outputrr#)r)rruse_return_dictr%warn_if_padding_and_no_attention_maskrzryrHrrr\r{rrxrFrurnext parametersrrErFrrr#)rAr]rJrKrrrhrr|rrr r}ryextended_attention_maskembedding_outputencoder_outputsrrs r*rPzMarkupLMModel.forwardasN2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN"0":":1"="="G"G"J"J"9"<"<4:"<"N"N#&)@#@H"L  }}!##%//22<>I??))%)' +  ,,  #/!5 '  *!,8< 8O O444UY )-')7&1     r,)T) NNNNNNNNNNN)rQrRrSr0rKrMrQrrrrH LongTensorrrrrrrPrUrVs@r*rCrC@s ///000CCC1559596:59371559,0/3&*c c E,-c !!12c !!12 c !!23 c !!12 c u/0c E-.c   12c $D>c 'tnc d^c  u00 1c c c ^c c c c c r,rCc!eZdZfdZee ddeejdeejdeejdeejdeejdeejd eejd eejd eejd eejd ee dee dee de e eje ffdZ xZS)MarkupLMForQuestionAnsweringct||j|_t|d|_t j|j|j|_| dSNF)rH) r/r0 num_labelsrCr,rr2r3 qa_outputsrGr@s r*r0z%MarkupLMForQuestionAnswering.__init__sj     +%fFFF )F$68IJJ r,Nr]rJrKrrrhrr|start_positions end_positionsrrr rc ~| | n |jj} |||||||||| | d }|d}||}|dd\}}|d}|d}d}| | t| dkr| d} t| dkr| d} |d}| d|| d|t|}||| }||| }||zd z }t||||j |j S) ae xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. Examples: ```python >>> from transformers import AutoProcessor, MarkupLMForQuestionAnswering >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> model = MarkupLMForQuestionAnswering.from_pretrained("microsoft/markuplm-base-finetuned-websrc") >>> html_string = " My name is Niels " >>> question = "What's his name?" >>> encoding = processor(html_string, questions=question, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> answer_start_index = outputs.start_logits.argmax() >>> answer_end_index = outputs.end_logits.argmax() >>> predict_answer_tokens = encoding.input_ids[0, answer_start_index : answer_end_index + 1] >>> processor.decode(predict_answer_tokens).strip() 'Niels' ```NT rJrKrrrhrr|rrr rrrDrE) ignore_indexr)loss start_logits end_logitsrr#)r)rTr,rasplitsqueezerrrzclamp_rrrr#)rAr]rJrKrrrhrr|rbrcrrr rrlogitsrhri total_loss ignored_indexloss_fct start_lossend_losss r*rPz$MarkupLMForQuestionAnswering.forwards`&1%\ 'tn\ d^\  uU\"$@@ A\ \ \ ^\ \ \ \ \ r,r]zC MarkupLM Model with a `token_classification` head on top. ) custom_introceZdZfdZee ddeejdeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e eje ffdZ xZS)MarkupLMForTokenClassificationcZt||j|_t|d|_|j|jn|j}tj||_ tj |j |j|_ | dSr_)r/r0r`rCr,classifier_dropoutr6rr5r7r2r3 classifierrGrAr)rwrBs r*r0z'MarkupLMForTokenClassification.__init__>s     +%fFFF )/)B)NF % %TZTn z"455 )F$68IJJ r,Nr]rJrKrrrhrr|labelsrrr rc t| | n |jj} |||||||||| | d } | d}||}d}| Kt }||d|jj| d}t||| j| j S)a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoProcessor, AutoModelForTokenClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> processor.parse_html = False >>> model = AutoModelForTokenClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> nodes = ["hello", "world"] >>> xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] >>> node_labels = [1, 2] >>> encoding = processor(nodes=nodes, xpaths=xpaths, node_labels=node_labels, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTrerrDrgrmrr#) r)rTr,rxrrr`rrr#)rAr]rJrKrrrhrr|rzrrr rrrrgrps r*rPz&MarkupLMForTokenClassification.forwardLs\&1%L 'tnL d^L  uU\"N2 3L L L ^L L L L L r,ruz MarkupLM Model transformer with a sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. ceZdZfdZee ddeejdeejdeejdeejdeejdeejd eejd eejd eejd ee d ee dee de e eje ffdZ xZS)!MarkupLMForSequenceClassificationcdt||j|_||_t ||_|j|jn|j}tj ||_ tj |j |j|_ |dSr)r/r0r`r)rCr,rwr6rr5r7r2r3rxrGrys r*r0z*MarkupLMForSequenceClassification.__init__s     + %f-- )/)B)NF % %TZTn z"455 )F$68IJJ r,Nr]rJrKrrrhrr|rzrrr rc | | n |jj} |||||||||| | d } | d}||}||}d}| Z|jjf|jdkr d|j_nN|jdkr7| jtj ks| jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n||| }n|jjdkrGt}||d|j| d}n*|jjdkrt}||| }t!||| j| j S) a xpath_tags_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Tag IDs for each token in the input sequence, padded up to config.max_depth. xpath_subs_seq (`torch.LongTensor` of shape `(batch_size, sequence_length, config.max_depth)`, *optional*): Subscript IDs for each token in the input sequence, padded up to config.max_depth. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoProcessor, AutoModelForSequenceClassification >>> import torch >>> processor = AutoProcessor.from_pretrained("microsoft/markuplm-base") >>> model = AutoModelForSequenceClassification.from_pretrained("microsoft/markuplm-base", num_labels=7) >>> html_string = " Page Title " >>> encoding = processor(html_string, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**encoding) >>> loss = outputs.loss >>> logits = outputs.logits ```NTrer regressionsingle_label_classificationmulti_label_classificationrDr|)r)rTr,r7rx problem_typer`rxrHr\rYr rkrrrrrr#)rAr]rJrKrrrhrr|rzrrr rrrmrgrps r*rPz)MarkupLMForSequenceClassification.forwardsZ&1%\ 'tn\ d^\  uU\"$<< =\ \ \ ^\ \ \ \ \ r,r)r]rrurCr+)r)rN)?rTr@typingrrrrHrtorch.nnrrr activationsr modeling_layersr modeling_outputsr rrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrconfiguration_markuplmr get_loggerrQloggerModulerrbrdrrrrrrrrfloatrrrr rr+rCr]rur__all__r"r,r*rs ,,,,,,,,,, AAAAAAAAAA!!!!!!999999GFFFFFFFllllllllll>>>>>>>>>>222222  H % %/ / / / / bi/ / / f 4 4 4 4 ________F             29        RY        RY    bi$ry0!!!!!")!!!$(,%% I% <% % < % U\* %  %% %%%%%877777BI777v***** ***\%%%%%.%%%R. . . . . bi. . . b]]]]]o]]]8E E E E E +E E E Pj j j j j #:j j j Z ^ ^ ^ ^ ^ %<^ ^  ^ B o o o o o (?o o  o d   r,