`i! d Z ddlZddlZddlmZ ddlmZmZ ddlm Z ddl
m Z m Z m
Z
ddlmZmZ dd lmZ G d
d e d
Z ej e Z G d de ZdgZdS )z
Processor class for Donut.
N)contextmanager)OptionalUnion )
ImageInput)ProcessingKwargsProcessorMixinUnpack)PreTokenizedInput TextInput)loggingc e Zd Zi ZdS )DonutProcessorKwargsN)__name__
__module__ __qualname__ _defaults ~/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/donut/processing_donut.pyr r s IIIr r F)totalc
e Zd ZdZddgZdZdZd fd Z ddee d ee
e e e e
ef d
ee fd Zed Zed
ZddZed Zed Z xZS )DonutProcessora
Constructs a Donut processor which wraps a Donut image processor and an XLMRoBERTa tokenizer into a single
processor.
[`DonutProcessor`] offers all the functionalities of [`DonutImageProcessor`] and
[`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. See the [`~DonutProcessor.__call__`] and
[`~DonutProcessor.decode`] for more information.
Args:
image_processor ([`DonutImageProcessor`], *optional*):
An instance of [`DonutImageProcessor`]. The image processor is a required input.
tokenizer ([`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`], *optional*):
An instance of [`XLMRobertaTokenizer`/`XLMRobertaTokenizerFast`]. The tokenizer is a required input.
image_processor tokenizerAutoImageProcessor
AutoTokenizerNc d }d|v r/t j dt | d }||n|}t || | j | _ d| _ d S )Nfeature_extractorzhThe `feature_extractor` argument is deprecated and will be removed in v5, use `image_processor` instead.F) warningswarn
FutureWarningpopsuper__init__r current_processor_in_target_context_manager)selfr r kwargsr __class__s r r% zDonutProcessor.__init__9 s & ( ( M
!'
+> ? ? -<-H//N_
)444!%!5 */'''r imagestextr) c f | j r | j ||fi |S ||t d | j t fd| j j i|}| | j |fi |d }|2| |d dd | j |fi |d }||S ||S |d |d < |d |d< |S )
a
When used in normal mode, this method forwards all its arguments to AutoImageProcessor's
[`~AutoImageProcessor.__call__`] and returns its output. If used in the context
[`~DonutProcessor.as_target_processor`] this method forwards all its arguments to DonutTokenizer's
[`~DonutTokenizer.__call__`]. Please refer to the docstring of the above two methods for more information.
NzBYou need to specify either an `images` or `text` input to process.tokenizer_init_kwargs
images_kwargs text_kwargsadd_special_tokensF input_idslabels) r' r&
ValueError
_merge_kwargsr r init_kwargsr
setdefault) r( r+ r, audiovideosr)
output_kwargsinputs encodingss r __call__zDonutProcessor.__call__I s * B)4)&$AA&AA A >dl abb b**
"&."<
)T)&SSM/4RSSF ! m,778LeTTT&tLL}]/KLLI <M
^ ( 5F8 "+K"8F; Mr c B | j j }t |ddgz S )Nr2 r3 )r model_input_nameslist)r( image_processor_input_namess r r? z DonutProcessor.model_input_namess s' &*&:&L#/;2IIJJJr c # K t j d d| _ | j | _ dV | j | _ d| _ dS )z
Temporarily sets the tokenizer for processing the input. Useful for encoding the labels when fine-tuning TrOCR.
z`as_target_processor` is deprecated and will be removed in v5 of Transformers. You can process your labels by using the argument `text` of the regular `__call__` method (either in the same call as your images inputs, or in a separate call.TNF)r r! r' r r& r r( s r as_target_processorz"DonutProcessor.as_target_processory sW
9
+/'!%
!%!5 */'''r Fc || j }i }|rt j d|t j }|n|| d }d|vrn|d| d dz }|t d t d }t j | }t j d| d|t j } | | |d }n|
} t j | }
t j | } t j |
d| |t j t j z } | |
d } d| v r?d| v r;|
| d| }
|
r t |
dk r|
d
}
|
||< ng ||< | d D ]Y}| }||v r$|d
d k r|d
d dk r
|dd
}|| | Zt || dk r|| d
||< || | t | z d }|dd d k r$|g|
|dd d| z S ||r|r|gn|S |rg nd|iS )zS
Convert a (generated) token sequence into an ordered JSON format.
Nz z
text_sequence)r get_added_vocabresearch
IGNORECASEstartindexlenescapereplacegroupDOTALLstrip
token2jsonsplitappendfind)r( tokensrI rJ outputpotential_start start_tokenkey key_escaped end_tokenstart_token_escapedend_token_escapedcontentvalueleafs r r[ zDonutProcessor.token2json s0 .88::K * p i
F FO & !6!6!8!8!:!:;K+%%%&B (9(9#(>(>(B&BCK c%jjCHH9 45C )C..K "8+"8"8"8&"-PPI R88 %OO - - &(i &<&<#$&Ii$8$8! )*DD1BDDfbm^`^gNg & %mmA . . 4 4 6 6G ((W-?-? $Ze f f 0 "5zzQ(-a*/F3K&(s $+MM)$<$< 5 5D#'::<r~ s.
% % % % % % " " " " " " " " % % % % % % H H H H H H H H H H C C C C C C C C +5
H % %l$ l$ l$ l$ l$^ l$ l$ l$^
r