`ihdZddlmZmZmZddlZddlmZddlmZm Z m Z ddl m Z ddl mZdd lmZmZmZmZmZmZdd lmZmZdd lmZmZmZdd lmZmZm Z d dl!m"Z"e j#e$Z%ej&Z'Gddej(Z) dAdej(dej*dej*dej*deej*de+de+deej*fdZ,Gddej(Z-Gddej(Z.Gdd ej(Z/Gd!d"ej(Z0Gd#d$ej(Z1Gd%d&eZ2Gd'd(ej(Z3Gd)d*ej(Z4Gd+d,ej(Z5Gd-d.ej(Z6Gd/d0ej(Z7eGd1d2eZ8eGd3d4e8Z9eGd5d6e8Z:ed78Gd9d:e8Z;ed;8Gd<d=e8Zd?e8Z=gd@Z>dS)BzPyTorch LayoutLM model.)CallableOptionalUnionN)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingMaskedLMOutputQuestionAnsweringModelOutputSequenceClassifierOutputTokenClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)apply_chunking_to_forward find_pruneable_heads_and_indicesprune_linear_layer)auto_docstringcan_return_tuplelogging)LayoutLMConfigc4eZdZdZfdZ ddZxZS)LayoutLMEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.cNttj|j|j|j|_tj|j|j|_ tj|j |j|_ tj|j |j|_ tj|j |j|_ tj|j |j|_tj|j|j|_t#|j|j|_tj|j|_|dt1j|jdddS)N) padding_idxeps position_ids)rF) persistent)super__init__r Embedding vocab_size hidden_size pad_token_idword_embeddingsmax_position_embeddingsposition_embeddingsmax_2d_position_embeddingsx_position_embeddingsy_position_embeddingsh_position_embeddingsw_position_embeddingstype_vocab_sizetoken_type_embeddingsLayoutLMLayerNormlayer_norm_eps LayerNormDropouthidden_dropout_probdropoutregister_buffertorcharangeexpandselfconfig __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/layoutlm/modeling_layoutlm.pyr'zLayoutLMEmbeddings.__init__0sV !|F,=v?Q_e_rsss#%<0NPVPb#c#c %'\&2SU[Ug%h%h"%'\&2SU[Ug%h%h"%'\&2SU[Ug%h%h"%'\&2SU[Ug%h%h"%'\&2H&J\%]%]"*6+=6CXYYYz&"<==   EL)GHHOOPWXXej      Nc:||}n|dd}|d}||jn|j}||jddd|f}|!tj|tj|}|||}|} ||} ||dddddf} | |dddddf} ||dddddf} | |dddddf}n"#t$r}td|d}~wwxYw| |dddddf|dddddfz }| |dddddf|dddddfz }| |}| | z| z| z| z|z|z|z|z}||}||}|S)Nr$rdtypedevicerr z:The `bbox`coordinate values should be within 0-1000 range.)sizerIr#r=zeroslongr,r.r0r1 IndexErrorr2r3r5r8r;)rA input_idsbboxtoken_type_idsr# inputs_embeds input_shape seq_lengthrIwords_embeddingsr.left_position_embeddingsupper_position_embeddingsright_position_embeddingslower_position_embeddingser2r3r5 embeddingss rDforwardzLayoutLMEmbeddings.forwardAs  #..**KK',,..ss3K ^ %.%:!! @T  ,QQQ  ^ J J ! "& '( ((  ( (  ( $  $$ $$ $ ^^J// \\*-- s3BD<< EEE)NNNNN)__name__ __module__ __qualname____doc__r'r\ __classcell__rCs@rDrr-scQQ     &  55555555rErmodulequerykeyvalueattention_maskscalingr; head_maskc 8tj||dd|z} |$|ddddddd|jdf} | | z} tj| dtj|j } tj | ||j } || | ddddz} tj| |} | dd } | | fS)NrJr r$)dimrH)ptrainingr)r=matmul transposeshaper functionalsoftmaxfloat32torHr;roview contiguous) rdrerfrgrhrir;rjkwargs attn_weights causal_mask attn_outputs rDeager_attention_forwardr}zs<s}}Q':':;;gEL!$QQQ111o " o%=> #k1 =((2U](SSVVW\WbccL=((6?([[L#innQAq&A&AA ,|U33K''1--88::K  $$rEc eZdZfdZ d dejdeejdeejdeede ejf d Z xZ S) LayoutLMSelfAttentionct|j|jzdkr0t |ds t d|jd|jd||_|j|_t|j|jz |_|j|jz|_ tj |j|j |_ tj |j|j |_ tj |j|j |_tj|j|_|j|_|jdz|_dS)Nrembedding_sizezThe hidden size (z6) is not a multiple of the number of attention heads ()g)r&r'r*num_attention_headshasattr ValueErrorrBintattention_head_size all_head_sizerLinearrerfrgr9attention_probs_dropout_probr;attention_dropoutrir@s rDr'zLayoutLMSelfAttention.__init__s:    : :a ? ?PVXhHiHi ?8F$688 4888   #)#= #&v'9F ! u| !!!!!!!!rErcPeZdZfdZdejdejdejfdZxZS)LayoutLMSelfOutputcttj|j|j|_tj|j|j|_tj|j |_ dSNr!) r&r'rrr*denser8r7r9r:r;r@s rDr'zLayoutLMSelfOutput.__init__sf Yv163EFF f&8f>STTTz&"<== rEr input_tensorrc||}||}|||z}|SNrr;r8rArrs rDr\zLayoutLMSelfOutput.forward@ =11  ]33 }|'CDD rEr]r^r_r'r=rr\rarbs@rDrri>>>>> U\RWR^rErc eZdZfdZdZ d dejdeejdeejdee d e ejf d Z xZ S) LayoutLMAttentionctt||_t ||_t |_dSr)r&r'rrAroutputset pruned_headsr@s rDr'zLayoutLMAttention.__init__sI )&11 (00 EErEct|dkrdSt||jj|jj|j\}}t |jj||j_t |jj||j_t |jj ||j_ t |j j |d|j _ |jjt|z |j_|jj|jjz|j_ |j ||_dS)Nrrrm)lenrrArrrrrerfrgrrrunion)rAheadsindexs rD prune_headszLayoutLMAttention.prune_headss u::?? F7 490$)2OQUQb  u -TY_eDD *49=%@@ ,TY_eDD .t{/@%QOOO )- (EE (R %"&)"?$)B_"_  -33E::rENFrrhrjrrc ~|j|f|||d|}||d|}|f|ddz}|SN)rhrjrrr)rAr) rArrhrjrry self_outputsattention_outputrs rDr\zLayoutLMAttention.forwardsl!ty  )/       ;;|A FF#% QRR(88rEr) r]r^r_r'rr=rrrrrr\rarbs@rDrrs""""" ;;;*7;15,1 |!!23E-.  $D>  u| rErcBeZdZfdZdejdejfdZxZS)LayoutLMIntermediatecttj|j|j|_t|jtrt|j|_ dS|j|_ dSr) r&r'rrr*intermediate_sizer isinstance hidden_actstrr intermediate_act_fnr@s rDr'zLayoutLMIntermediate.__init__sn Yv163KLL f' - - 9'-f.?'@D $ $ $'-'8D $ $ $rErrcZ||}||}|Sr)rrrArs rDr\zLayoutLMIntermediate.forwards, =11 00?? rErrbs@rDrrs^99999U\elrErcPeZdZfdZdejdejdejfdZxZS)LayoutLMOutputcttj|j|j|_tj|j|j|_tj |j |_ dSr) r&r'rrrr*rr8r7r9r:r;r@s rDr'zLayoutLMOutput.__init__sf Yv79KLL f&8f>STTTz&"<== rErrrc||}||}|||z}|Srrrs rDr\zLayoutLMOutput.forward%rrErrbs@rDrrrrErc eZdZfdZ d dejdeejdeejdeede ejf d Z d Z xZ S) LayoutLMLayerct|j|_d|_t ||_t ||_t||_ dS)Nr) r&r'chunk_size_feed_forward seq_len_dimr attentionr intermediaterrr@s rDr'zLayoutLMLayer.__init__.s^ '-'E$*622088$V,, rENFrrhrjrrc |j|f|||d|}|d}|dd}t|j|j|j|} | f|z}|Sr)rrfeed_forward_chunkrr) rArrhrjrryself_attention_outputsrr layer_outputs rDr\zLayoutLMLayer.forward6s"0 " )/ " "  " " 2!4(,0  #T%A4CSUe   /G+rEc\||}|||}|Sr)rr)rArintermediate_outputrs rDrz LayoutLMLayer.feed_forward_chunkOs2"//0@AA{{#68HII rEr) r]r^r_r'r=rrrrrr\rrarbs@rDrr-s-----7;15,1 |!!23E-.  $D>  u| 2rErceZdZfdZe d dejdeejdeejdee d ee d ee d e e eje ffd Z xZS)LayoutLMEncoderct|_tjfdt jD|_d|_dS)Nc.g|]}tS)r).0irBs rD z,LayoutLMEncoder.__init__..Zs!#c#c#caM&$9$9#c#c#crEF) r&r'rBr ModuleListrangenum_hidden_layerslayergradient_checkpointingr@s `rDr'zLayoutLMEncoder.__init__Ws`  ]#c#c#c#c5IaCbCb#c#c#cdd &+###rENFTrrhrjroutput_hidden_states return_dictrc |rdnd}|rdnd} t|jD]<\} } |r||fz}||| nd} | d||| |d|} | d}|r | | dfz} =|r||fz}t||| S)Nr)rrhrjrrr)last_hidden_stater attentions) enumeraterr )rArrhrjrrrryall_hidden_statesall_self_attentionsr layer_modulelayer_head_mask layer_outputss rDr\zLayoutLMEncoder.forward]s#7@BBD$5?bb4(44 P POA|# I$58H$H!.7.CillO(L+-)"3   M*!,M  P&9]1=M & 'tn & d^&  uU\"O3 4& & & & & & & & rErcBeZdZfdZdejdejfdZxZS)LayoutLMPoolercttj|j|j|_tj|_dSr)r&r'rrr*rTanh activationr@s rDr'zLayoutLMPooler.__init__sC Yv163EFF '))rErrcr|dddf}||}||}|S)Nr)rr)rArfirst_token_tensor pooled_outputs rDr\zLayoutLMPooler.forwards@+111a40 #566  66 rErrbs@rDrrs^$$$$$ U\elrErcBeZdZfdZdejdejfdZxZS)LayoutLMPredictionHeadTransformcVttj|j|j|_t |jtrt|j|_ n |j|_ tj |j|j |_ dSr) r&r'rrr*rrrrr transform_act_fnr8r7r@s rDr'z(LayoutLMPredictionHeadTransform.__init__s Yv163EFF f' - - 6$*6+<$=D ! !$*$5D !f&8f>STTTrErrc||}||}||}|Sr)rrr8rs rDr\z'LayoutLMPredictionHeadTransform.forwards= =11 --m<< }55 rErrbs@rDrrscUUUUUU\elrErc*eZdZfdZdZdZxZS)LayoutLMLMPredictionHeadc>tt||_t j|j|jd|_t j tj |j|_ |j |j_ dS)NF)bias) r&r'r transformrrr*r)decoder Parameterr=rLrr@s rDr'z!LayoutLMLMPredictionHead.__init__sz 8@@y!3V5FUSSS LV->!?!?@@ !I rEc(|j|j_dSr)rrrAs rD _tie_weightsz%LayoutLMLMPredictionHead._tie_weightss I rEcZ||}||}|Sr)rrrs rDr\z LayoutLMLMPredictionHead.forwards*}55  ]33 rE)r]r^r_r'rr\rarbs@rDrrsV & & & & &&&&rErcBeZdZfdZdejdejfdZxZS)LayoutLMOnlyMLMHeadcptt||_dSr)r&r'r predictionsr@s rDr'zLayoutLMOnlyMLMHead.__init__s/ 3F;;rEsequence_outputrc0||}|Sr)r )rAr prediction_scoress rDr\zLayoutLMOnlyMLMHead.forwards ,,_==  rErrbs@rDrrs^<<<<<!u|! !!!!!!!!rErc(eZdZUeed<dZdZdZdS)LayoutLMPreTrainedModelrBlayoutlmTct|tjrT|jjd|jj|j |jj dSdSt|tj r_|jjd|jj|j +|jj|j  dSdSt|tr?|jj |jj ddSt|tr |jj dSdS)zInitialize the weightsrc)meanstdN?)rrrweightdatanormal_rBinitializer_rangerzero_r(r r6fill_r)rArds rD _init_weightsz%LayoutLMPreTrainedModel._init_weightssX fbi ( ( % M  & &CT[5R & S S S{&  &&((((('&  - - % M  & &CT[5R & S S S!- "6#56<<>>>>>.-  1 2 2 % K  " " $ $ $ M  $ $S ) ) ) ) )  8 9 9 % K  " " $ $ $ $ $ % %rEN)r]r^r_r__annotations__base_model_prefixsupports_gradient_checkpointingrrrErDrrs= "&*#%%%%%rErcheZdZfdZdZdZdZee dde e j de e j de e j d e e j d e e j d e e j d e e j d e e de e de e deeeffdZxZS) LayoutLMModelct|||_t||_t ||_t||_| dSr) r&r'rBrr[rencoderrpooler post_initr@s rDr'zLayoutLMModel.__init__sg     ,V44&v.. $V,,  rEc|jjSrr[r,rs rDget_input_embeddingsz"LayoutLMModel.get_input_embeddingss ..rEc||j_dSrr&)rArgs rDset_input_embeddingsz"LayoutLMModel.set_input_embeddingss*/'''rEc|D]/\}}|jj|j|0dS)z Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel N)itemsr"rrr)rAheads_to_prunerrs rD _prune_headszLayoutLMModel._prune_headssU +0022 C CLE5 L u % / ; ;E B B B B C CrENrOrPrhrQr#rjrRrrrrc "||n |jj}| | n |jj} | | n |jj} ||t d|+||||} n.||dd} nt d||jn|j} |tj | | }|!tj | tj | }|$tj | dztj | }| d d } | |j } d | z tj|jjz} ||dkrr| d  d  d d}||jjdddd}nS|d kr;| d d d}| t)|j }ndg|jjz}|||||| }||| ||| d}|d }||}t3|||j|jS)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMModel >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMModel.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "world"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> outputs = model( ... input_ids=input_ids, bbox=bbox, attention_mask=attention_mask, token_type_ids=token_type_ids ... ) >>> last_hidden_states = outputs.last_hidden_state ```NzDYou cannot specify both input_ids and inputs_embeds at the same timer$z5You have to specify either input_ids or inputs_embeds)rIrG)rrJ)rHrr)rOrPr#rQrRT)rjrrr)r pooler_outputrr)rBrruse_return_dictr%warn_if_padding_and_no_attention_maskrKrIr=onesrLrM unsqueezervrHfinfominrmr?rnext parametersr[r"r#rrr)rArOrPrhrQr#rjrRrrrrSrIextended_attention_maskembedding_outputencoder_outputsr rs rDr\zLayoutLMModel.forwards"j2C1N--TXT_Tq$8$D $+Jj &1%cdd d  "  6 6y. Q Q Q#..**KK  &',,..ss3KKTUU U%.%:!! @T  !"Z FCCCN  !"[EJvVVVN <;{T1FSSSD"0":":1"="="G"G"J"J"9"<"<4:"<"N"N#&)@#@EKPTPZD[D[D_"_  }}!##%//22<>I??%)' +  ,,  #/!5 '  *!, O44 )-')7&1     rE) NNNNNNNNNN)r]r^r_r'r'r)r-rrrr= LongTensorrrrrrr\rarbs@rDr r s     ///000CCC15+/6:59371559,0/3&*s s E,-s u'(s !!23 s !!12 s u/0 s E-.s   12s $D>s 'tns d^s  u00 1s s s ^s s s s s rEr ceZdZddgZfdZdZdZdZee dde e j d e e j d e e j d e e j d e e j d e e j de e j de e j de ede ede edeeeffdZxZS)LayoutLMForMaskedLMzcls.predictions.decoder.biaszcls.predictions.decoder.weightct|t||_t ||_|dSr)r&r'r rrclsr$r@s rDr'zLayoutLMForMaskedLM.__init__|sQ    %f-- &v.. rEc$|jjjSrrr[r,rs rDr'z(LayoutLMForMaskedLM.get_input_embeddings}'77rEc$|jjjSr)r@r rrs rDget_output_embeddingsz)LayoutLMForMaskedLM.get_output_embeddingssx#++rEcT||jj_|j|jj_dSr)r@r rr)rAnew_embeddingss rDset_output_embeddingsz)LayoutLMForMaskedLM.set_output_embeddingss%'5$$2$7!!!rENrOrPrhrQr#rjrRlabelsrrrrc r| | n |jj} ||||||||| | d } | d} || }d}|Kt }||d|jj|d}t||| j| j S)a2 bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ..., config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMForMaskedLM >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMForMaskedLM.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "[MASK]"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> labels = tokenizer("Hello world", return_tensors="pt")["input_ids"] >>> outputs = model( ... input_ids=input_ids, ... bbox=bbox, ... attention_mask=attention_mask, ... token_type_ids=token_type_ids, ... labels=labels, ... ) >>> loss = outputs.loss ```NT)rhrQr#rjrRrrrrr$losslogitsrr) rBr1rr@rrwr)rrr)rArOrPrhrQr#rjrRrIrrrrr r masked_lm_lossloss_fcts rDr\zLayoutLMForMaskedLM.forwards@&1%r>xs8:Z[888,,,88815+/6:59371559-1,0/3&*] ] E,-] u'(] !!23 ] !!12 ] u/0 ] E-.]   12] )*] $D>] 'tn] d^]  un$ %] ] ] ^] ] ] ] ] rEr>z LayoutLM Model with a sequence classification head on top (a linear layer on top of the pooled output) e.g. for document image classification tasks such as the [RVL-CDIP](https://www.cs.cmu.edu/~aharley/rvl-cdip/) dataset. ) custom_introcxeZdZfdZdZee ddeej deej deej deej deej d eej d eej d eej d ee d ee dee de e effdZxZS)!LayoutLMForSequenceClassificationc6t||j|_t||_t j|j|_t j |j |j|_ | dSr r&r' num_labelsr rrr9r:r;rr* classifierr$r@s rDr'z*LayoutLMForSequenceClassification.__init__y     +%f-- z&"<== )F$68IJJ rEc$|jjjSrrBrs rDr'z6LayoutLMForSequenceClassification.get_input_embeddingsrCrENrOrPrhrQr#rjrRrIrrrrc | | n |jj} ||||||||| | d } | d} || } || }d}|Z|jjf|jdkr d|j_nN|jdkr7|jtj ks|jtj kr d|j_n d|j_|jjdkrWt}|jdkr1|| | }n|||}n|jjdkrGt}||d|j|d}n*|jjdkrt}|||}t!||| j| j S) aB bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMForSequenceClassification >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMForSequenceClassification.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "world"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> sequence_label = torch.tensor([1]) >>> outputs = model( ... input_ids=input_ids, ... bbox=bbox, ... attention_mask=attention_mask, ... token_type_ids=token_type_ids, ... labels=sequence_label, ... ) >>> loss = outputs.loss >>> logits = outputs.logits ```NT rOrPrhrQr#rjrRrrrr regressionsingle_label_classificationmulti_label_classificationr$rK)rBr1rr;rX problem_typerWrHr=rMrr squeezerrwrrrr)rArOrPrhrQr#rjrRrIrrrrrrMrLrOs rDr\z)LayoutLMForSequenceClassification.forwards@&1%n 'tnn d^n  u.. /n n n ^n n n n n rErTa3 LayoutLM Model with a token classification head on top (a linear layer on top of the hidden-states output) e.g. for sequence labeling (information extraction) tasks such as the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset and the [SROIE](https://rrc.cvc.uab.es/?ch=13) dataset. cxeZdZfdZdZee ddeej deej deej deej deej d eej d eej d eej d ee d ee dee de e effdZxZS)LayoutLMForTokenClassificationc6t||j|_t||_t j|j|_t j |j |j|_ | dSrrVr@s rDr'z'LayoutLMForTokenClassification.__init__rYrEc$|jjjSrrBrs rDr'z3LayoutLMForTokenClassification.get_input_embeddingsrCrENrOrPrhrQr#rjrRrIrrrrc | | n |jj} ||||||||| | d } | d} || } || }d}|Ft }||d|j|d}t||| j | j S)a bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the token classification loss. Indices should be in `[0, ..., config.num_labels - 1]`. Examples: ```python >>> from transformers import AutoTokenizer, LayoutLMForTokenClassification >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased") >>> model = LayoutLMForTokenClassification.from_pretrained("microsoft/layoutlm-base-uncased") >>> words = ["Hello", "world"] >>> normalized_word_boxes = [637, 773, 693, 782], [698, 773, 733, 782] >>> token_boxes = [] >>> for word, box in zip(words, normalized_word_boxes): ... word_tokens = tokenizer.tokenize(word) ... token_boxes.extend([box] * len(word_tokens)) >>> # add bounding boxes of cls + sep tokens >>> token_boxes = [[0, 0, 0, 0]] + token_boxes + [[1000, 1000, 1000, 1000]] >>> encoding = tokenizer(" ".join(words), return_tensors="pt") >>> input_ids = encoding["input_ids"] >>> attention_mask = encoding["attention_mask"] >>> token_type_ids = encoding["token_type_ids"] >>> bbox = torch.tensor([token_boxes]) >>> token_labels = torch.tensor([1, 1, 0, 0]).unsqueeze(0) # batch size of 1 >>> outputs = model( ... input_ids=input_ids, ... bbox=bbox, ... attention_mask=attention_mask, ... token_type_ids=token_type_ids, ... labels=token_labels, ... ) >>> loss = outputs.loss >>> logits = outputs.logits ```NTr\rr$rK) rBr1rr;rXrrwrWrrr)rArOrPrhrQr#rjrRrIrrrrr rMrLrOs rDr\z&LayoutLMForTokenClassification.forwards|&1%Z 'tnZ d^Z  u++ ,Z Z Z ^Z Z Z Z Z rErcceZdZdfd ZdZee ddeej deej deej deej d eej d eej d eej d eej d eej dee dee dee de e effdZxZS)LayoutLMForQuestionAnsweringTct||j|_t||_t j|j|j|_| dS)z has_visual_segment_embedding (`bool`, *optional*, defaults to `True`): Whether or not to add visual segment embeddings. N) r&r'rWr rrrr* qa_outputsr$)rArBhas_visual_segment_embeddingrCs rDr'z%LayoutLMForQuestionAnswering.__init__sg     +%f-- )F$68IJJ rEc$|jjjSrrBrs rDr'z1LayoutLMForQuestionAnswering.get_input_embeddingsrCrENrOrPrhrQr#rjrRstart_positions end_positionsrrrrc || | n |jj} ||||||||| | d } | d}||}|dd\}}|d}|d}d}|| t|dkr|d}t| dkr| d} |d}| d|}| d|} t|}|||}||| }||zd z }t|||| j | j S) a4 bbox (`torch.LongTensor` of shape `(batch_size, sequence_length, 4)`, *optional*): Bounding boxes of each input sequence tokens. Selected in the range `[0, config.max_2d_position_embeddings-1]`. Each bounding box should be a normalized version in (x0, y0, x1, y1) format, where (x0, y0) corresponds to the position of the upper left corner in the bounding box, and (x1, y1) represents the position of the lower right corner. See [Overview](#Overview) for normalization. Example: In the example below, we prepare a question + context pair for the LayoutLM model. It will give us a prediction of what it thinks the answer is (the span of the answer within the texts parsed from the image). ```python >>> from transformers import AutoTokenizer, LayoutLMForQuestionAnswering >>> from datasets import load_dataset >>> import torch >>> tokenizer = AutoTokenizer.from_pretrained("impira/layoutlm-document-qa", add_prefix_space=True) >>> model = LayoutLMForQuestionAnswering.from_pretrained("impira/layoutlm-document-qa", revision="1e3ebac") >>> dataset = load_dataset("nielsr/funsd", split="train") >>> example = dataset[0] >>> question = "what's his name?" >>> words = example["words"] >>> boxes = example["bboxes"] >>> encoding = tokenizer( ... question.split(), words, is_split_into_words=True, return_token_type_ids=True, return_tensors="pt" ... ) >>> bbox = [] >>> for i, s, w in zip(encoding.input_ids[0], encoding.sequence_ids(0), encoding.word_ids(0)): ... if s == 1: ... bbox.append(boxes[w]) ... elif i == tokenizer.sep_token_id: ... bbox.append([1000] * 4) ... else: ... bbox.append([0] * 4) >>> encoding["bbox"] = torch.tensor([bbox]) >>> word_ids = encoding.word_ids(0) >>> outputs = model(**encoding) >>> loss = outputs.loss >>> start_scores = outputs.start_logits >>> end_scores = outputs.end_logits >>> start, end = word_ids[start_scores.argmax(-1)], word_ids[end_scores.argmax(-1)] >>> print(" ".join(words[start : end + 1])) M. Hamann P. Harper, P. Martinez ```NTr\rrr$r) ignore_indexrJ)rL start_logits end_logitsrr)rBr1rrjsplitrarxrrKclamprrrr)rArOrPrhrQr#rjrRrmrnrrrrr rMrqrr total_loss ignored_indexrO start_lossend_losss rDr\z$LayoutLMForQuestionAnswering.forwardsD&1%m 'tnm d^m  u22 3m m m ^m m m m m rErh)r>rTrcrhr r)rcN)?r`typingrrrr=rtorch.nnrrr activationsr modeling_layersr modeling_outputsr rrrrrmodeling_utilsrr pytorch_utilsrrrutilsrrrconfiguration_layoutlmr get_loggerr]loggerr8r6Modulerrfloatr}rrrrrrrrrrrrr r>rTrcrh__all__rrErDrs,,,,,,,,,, AAAAAAAAAA!!!!!!999999GFFFFFFFllllllllll>>>>>>>>>>222222  H % %LIIIIIIIIh(,%% I% <% % < % U\* %  %% %%%%%877777BI777v        ***** ***\     29        RY   %%%%%.%%%R. . . . . bi. . . d     RY    bi$ry0!!!!!")!!!%%%%%o%%%0O O O O O +O O O du u u u u 1u u u p ~ ~ ~ ~ ~ (?~ ~  ~ Bj j j j j %<j j j ZA A A A A #:A A A H   rE