PiBddlmZmZmZddlmZGddeZdS))AnyDictList)SentencePieceBaseTokenizerc eZdZdZddededeffd Zded eeffd Z dd e ee fd ed e ee ffdZ xZ S) T5Tokenizerz Text tokenizer for T5. Args: path (str): the path to the T5 sentencepiece tokenizer file max_seq_len (int): the context length truncate (bool): whether to truncate the token sequence when longer than max_seq_len Tpath max_seq_lentruncatecft|||_||_dS)N)super__init__r r )selfr r r __class__s r/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/torchtune/models/t5/_tokenizer.pyrzT5Tokenizer.__init__s. &  textreturnct|dddd}t||jkr*|js Jd|d|j}|j|d<|S)z Given a string, return the encoded list of token ids. Args: text (str): The text to encode. Returns: List[int]: The encoded list of token ids. FTN)add_bosadd_eostrim_leading_whitespaceprefixzWTokenized text is larger than the maximum sequence length but truncate is set to False.)rencodelenr r eos_id)rrtokensrs rrzT5Tokenizer.encodes $)     v;;) ) )=  ,  =.d../FF2J rFsample inferencec`|d}|||d<|S)aW Tokenize the "text" field in the sample. Args: sample (Dict[str, Any]): A sample with a "text" field containing a string to tokenize inference (bool): Unused by this tokenizer Returns: Dict[str, Any]: The sample with added "tokens" field and the "messages" field removed. rr)popr)rr r!rs r__call__zT5Tokenizer.__call__6s0zz&!!;;t,,x r)r T)F)__name__ __module__ __qualname____doc__strintboolrrrrrr$ __classcell__)rs@rrr s!!S!s!D!!!!!! 34969>38n15 c3hrrN)typingrrr6torchtune.modules.transforms.tokenizers._sentencepiecerrrrr0sx #""""""""" 88888,88888r