`i. d Z ddlZddlmZ ddlmZ ddlmZ ddlm Z m
Z
e rdd l m Z ndZ e
j
e Zd
d d Z G d
de ZdgZdS )z Tokenization classes for XGLM. N)copyfile)Optional )PreTrainedTokenizerFast)is_sentencepiece_availablelogging )
XGLMTokenizerzsentencepiece.bpe.modelztokenizer.json)
vocab_filetokenizer_filec e Zd ZdZeZddgZeZ d fd Z dd
e
e d e e
e d e
e fd
Z
dd
e
e d e e
e d e
e fdZddede e d ee fdZ xZS )XGLMTokenizerFasta{
Construct a "fast" XGLM tokenizer (backed by HuggingFace's *tokenizers* library). Adapted from [`RobertaTokenizer`]
and [`XLNetTokenizer`]. Based on
[BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
additional_special_tokens (`list[str]`, *optional*, defaults to `["NOTUSED", "NOTUSED"]`):
Additional special tokens used by the tokenizer.
input_idsattention_maskN X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr. r1 r2 seps r build_inputs_with_special_tokensz2XGLMTokenizerFast.build_inputs_with_special_tokens{ s@ ( %&4 4 ![ 3&,{::r c | j g}|t ||z dgz S t ||z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
Nr )r5 lenr6 s r $create_token_type_ids_from_sequencesz6XGLMTokenizerFast.create_token_type_ids_from_sequences sX $ ! s[())QC/ /3$s*S0;>??1#EEr save_directoryfilename_prefixc | j st d t j | s t
d| d d S t j ||r|dz ndt d z }t j | j
t j | k rt | j
| |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory.- r ) can_save_slow_tokenizer
ValueErrorospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr r )r. r< r= out_vocab_files r save_vocabularyz!XGLMTokenizerFast.save_vocabulary s +
w}}^,, LLU^UUU V V V F o M_s 2 22QbcoQp p
7??4? + +rw~/N/N N N T_n 5 5 5 r )NNr r r r r r )N)__name__
__module__ __qualname____doc__rI vocab_files_namesmodel_input_namesr
slow_tokenizer_classr- listintr r8 r; strtuplerL
__classcell__)r0 s @r r r $ sP - -^ *$&67( !% !% !% !% !% !%H JN; ; 9;3;DI3F;
c; ; ; ;4 JNF F 9F3;DI3FF
cF F F F0! !c !HSM !]bcf]g ! ! ! ! ! ! ! !r r )rP rC shutilr typingr tokenization_utils_fastr utilsr r tokenization_xglmr
get_loggerrM rF rI r __all__r r r r` s % $ > > > > > > 8 8 8 8 8 8 8 8 0000000M
H % %#