.`i:ddlmZdeedzfdZdedeededed ef d Zd Z dded eeded e eeeeffdZ dedeed eefdZ ddedeedeedzdedededed e eeeeeffdZ dS)) TokenizerLiketokensNc>t|D] \}}|d||< dS)N) enumerate)ritokens u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/vllm/tokenizers/detokenizer_utils.py_replace_none_with_emptyr s6f%%5 =F1I tokenizer output_tokensskip_special_tokensspaces_between_special_tokensreturncg}g}|j}t|}|rt|jnd}|D]j} | |vr| |vrJ|r2|||||| U|| k|r||||rd|Sd|S)N r)convert_tokens_to_stringsetget_added_vocaball_special_tokensappendclearjoin) r rrr sub_textscurrent_sub_textradded_vocab_setrr s r -_convert_tokens_to_string_with_added_encodersrs8I"$(A)335566O-@HI ()))b + + & & &  O # # )  !9!9:J!K!KLLL &&(((   U # # # #  # #E * * * *E112BCCDDD$#xx """ 779  r F prompt_idsc||t dz d|}t|}t|tz d}t ||||fS)aConverts the prompt ids to tokens and returns the tokens and offsets for incremental detokenization. Note that not all tokens are converted to strings. Only the tokens that are necessary for incremental detokenization are converted to strings. Nrr)convert_ids_to_tokens)INITIAL_INCREMENTAL_DETOKENIZATION_OFFSETlenmaxr )r r!r new_tokens read_offset prefix_offsets r convert_prompt_ids_to_tokensr,;sv00==ACCD/1Jj//K &OOQRSSMZ((( }k 11r token_idscrg}|D]1}||g}|d}||2|S)zDetokenize the input ids individually. Args: tokenizer: tokenizer used by model under test token_ids: convert these tokens (Python list form) Returns: Python list of token string representations Nr)decoder)r r- token_str_lsttoken_id token_strs r convert_ids_list_to_tokensr3SsUM(($$hZ00  IY'''' r T all_input_ids prev_tokensr+r*c|d}|du}|rt||dd|\}}}|Jd|cxkrt|kr4nn1||g|} t| tr| g} ndg} || z} |r| } |js|s;|| ||} || |d} n6t|| ||||} t|| |d||} t| t| ks| dr| d||fS| t| d} | | |t| fS)a(Detokenizes the input ids incrementally and returns the new tokens and the new text. If `prev_tokens` is None, this function will convert the input ids to tokens and return the tokens and the new text. Otherwise, it will return the new tokens and the new text. This function will also return the new prefix offset and the new read offset to be used in the next iteration. The offsets are necessary to defeat cleanup algorithms in the decode which decide to add a space or not depending on the surrounding ids. Args: tokenizer: The tokenizer to use. all_input_ids: The input ids. The last id is the new token id. prev_tokens: The previous tokens. If None, this function will convert the input ids to tokens and return the tokens and the new text. prefix_offset: The prefix offset. read_offset: The read offset. skip_special_tokens: Whether to skip special tokens. spaces_between_special_tokens: Whether to add spaces between special tokens. Nr$rr)rru�) r,r'r% isinstancestris_fastrrrendswith) r r4r5r+r*rr new_token_id is_first_iterr)r prefix_textnew_texts r detokenize_incrementallyr@ns B!$L4'M 4P }SbS)?R5 5 5 1m[  " " " L))))3y>>)))))44 N0C5   j# & & &$JT *,M#"  9 9 ; ; 88 - 3 4  55mMNN6STTC  - 3 4 3*G    A  -.. ) 3*G     8}}K((((H,=,=e,D,D( 2}k99K((**+H xc-.@.@ @@r )F)FT) vllm.tokenizersrlistr9r boolrr&inttupler,r3r@rr r rFs *)))))T#*%5 %%9%%$( %  %%%%T-.) !&222S 22 49c3  22220Cy #YB!&*.XAXAXA9XAcT!XA XA  XA  XA$(XA 49c3 #$XAXAXAXAXAXAr