`i+dZddlZddlmZmZddlmZmZmZer ddl Z ddl m Z ej e Z GddeZdgZdS) z' Feature extractor class for MarkupLM. N) BatchFeatureFeatureExtractionMixin)is_bs4_availableloggingrequires_backends) BeautifulSoupc@eZdZdZfdZdZdZdZdefdZ xZ S)MarkupLMFeatureExtractorao Constructs a MarkupLM feature extractor. This can be used to get a list of nodes and corresponding xpaths from HTML strings. This feature extractor inherits from [`~feature_extraction_utils.PreTrainedFeatureExtractor`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. c \t|dgtjdi|dS)Nbs4)rsuper__init__)selfkwargs __class__s /home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/transformers/models/markuplm/feature_extraction_markuplm.pyrz!MarkupLMFeatureExtractor.__init__+s8$(((""6"""""c g}g}|jr|n|jjD]}|jd}|j|dt |krdn(t fdt|dD|||||fS)NF) recursiverc3*K|] \}}|u |VdS)Nr).0ischilds r z6MarkupLMFeatureExtractor.xpath_soup..7s31e1e1Z[_dZdZd!ZdZdZdZd1e1er) nameparentparentsfind_allappendlennext enumeratereverse)relement xpath_tagsxpath_subscriptsr siblingsrs @r xpath_soupz#MarkupLMFeatureExtractor.xpath_soup/s "<;W^m  FuzUCCH   ej ) ) )  # ##h--''T1e1e1e1e (TU@V@V1e1e1e-e-e   EE  """+++rct|d}g}g}g}|jD]}t|tjjrt |jtjjurGtj | }|sp| || |\}} | || | t|t|krtdt|t|krtd|||fS)Nz html.parserz3Number of doc strings and xtags does not correspondz3Number of doc strings and xsubs does not correspond)r descendants isinstancer r(NavigableStringtyper Taghtmlunescapestripr#r,r$ ValueError) r html_string html_codeall_doc_stringsstring2xtag_seqstring2xsubs_seqr(text_in_this_tagr)r*s rget_three_from_singlez.MarkupLMFeatureExtractor.get_three_from_single>sF!+}==  , : :G'3;#>?? :''s{>>#'=#9#9#?#?#A#A '&&'7888/3w/G/G, ,&&z222 ''(8999   3#7#7 7 7RSS S   3'7#8#8 8 8RSS S1AAArcdd}t||D]\}}|d|z }|dkr |d|dz }|S)N/r[])zip)rr)r*xpathtagnamesubss rconstruct_xpathz(MarkupLMFeatureExtractor.construct_xpath[sV -=>> % %MGT ]]] "EqyyT$ rreturncd}t|trd}nLt|ttfr0t |dkst|dtrd}|s t dt |dt|ttfot|dt}|s|g}g}g}|D]}||\}}} ||g} t||| D]1\} } } | | | }| |2|| ||d}t|d}|S) a\ Main method to prepare for the model one or several HTML strings. Args: html_strings (`str`, `list[str]`): The HTML string or batch of HTML strings from which to extract nodes and corresponding xpaths. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **nodes** -- Nodes. - **xpaths** -- Corresponding xpaths. Examples: ```python >>> from transformers import MarkupLMFeatureExtractor >>> page_name_1 = "page1.html" >>> page_name_2 = "page2.html" >>> page_name_3 = "page3.html" >>> with open(page_name_1) as f: ... single_html_string = f.read() >>> feature_extractor = MarkupLMFeatureExtractor() >>> # single example >>> encoding = feature_extractor(single_html_string) >>> print(encoding.keys()) >>> # dict_keys(['nodes', 'xpaths']) >>> # batched example >>> multi_html_strings = [] >>> with open(page_name_2) as f: ... multi_html_strings.append(f.read()) >>> with open(page_name_3) as f: ... multi_html_strings.append(f.read()) >>> encoding = feature_extractor(multi_html_strings) >>> print(encoding.keys()) >>> # dict_keys(['nodes', 'xpaths']) ```FTrzQHTML strings must of type `str`, `list[str]` (batch of examples), but is of type .)nodesxpathsN)data tensor_type) r/strlisttupler$r6r1r=r#rCrGr)r html_strings valid_strings is_batchedrKrLr7r9r:r; xpath_stringsnodetag_listsub_list xpath_stringrMencoded_inputss r__call__z!MarkupLMFeatureExtractor.__call__cs`  lC ( ( % MM  tUm 4 4 %<  A%%LOS)I)I% $  8"&|"4"4888   tUm<<c*\Z[_^aBbBb  *(>L' ) )KAEA[A[\gAhAh >O_.> LL ) ) )M,/Rb,c,c 3 3(h#33HhGG $$\2222 MM- ( ( ( (&11%4TBBBr) __name__ __module__ __qualname____doc__rr,r=rGrr[ __classcell__)rs@rr r !s##### , , ,BBB:T TTTTTTTTrr )r_r3feature_extraction_utilsrrutilsrrrr r get_loggerr\loggerr __all__rrrrfs LLLLLLLLAAAAAAAAAA"JJJ!!!!!!  H % %VVVVV5VVVr & &r