*`i= ddlZddlmZddlmZmZddlmZmZddl m Z m Z m Z Gdde ZGdd eZd eed ed eeeeeed fffdZdeede d eefdZdeede d efdZdeede d efdZdeeede d eefdZdS)N)Sequence)_split_integer_list_by_value,_split_tokens_by_one_occurence_control_token) FunctionCallToolCall)SpecialTokenPolicy TokenizerTokenizerVersionceZdZdS)InvalidToolCallErrorN__name__ __module__ __qualname__u/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/mistral_common/experimental/tools.pyr r Drr ceZdZdS)InvalidArgsToolCallErrorNr rrrrrrrrtokenstool_call_token_idreturn.c|sgdfSt||}|dd|k}|r|d}|dd}ng}|}||fS)aSplit the content and tool calls from a list of tokens. The content is the first sequence of tokens that does not start with the tool call token ID. The tool calls are the remaining sequences of tokens that start with the tool call token ID. Args: tokens: The list of tokens. tool_call_token_id: The token ID that indicates the start of a tool call. Returns: A tuple containing the content and tool calls. rrN)r)rrmaybe_content_and_tools_calls has_contentcontent_tokenstools_calls_tokenss r_split_content_and_tool_callsr sv 2v $@I[$\$\!/2159KKK;6q9:122>: - --rtool_call_tokens tokenizerc||tj} tj|}|D]0}d|vst |dt std1n'#tj$r}td|d}~wwxYwt |tstdd|DS)aDecode a list of tool call tokens into a list of tool calls for tokenizer versions v2 to v7. Note: Expects the tool call tokens to be in the format: `[TOOL_CALLS][{"id": "call_id", "name": "name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ...]` or `[TOOL_CALLS][{"name": "name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ...]` special_token_policy argumentsz:Invalid tool call arguments tokenization. Expected a dict.zCInvalid tool call tokenization. Expected a JSON list of tool calls.Nz>Invalid tool call tokenization. Expected a list of tool calls.c g|]B}t|ddt|d|dCS)idnullnamer&r*r&r(function)rgetr).0 tool_calls r z2_decode_tool_calls_v2_up_to_v7..Msd     }}T6**!y'8IkDZ[[[      r) decoderIGNOREjsonloads isinstancedictrJSONDecodeErrorr list)r!r"tool_calls_list_stringtool_calls_decoded_listr0es r_decode_tool_calls_v2_up_to_v7r=3s'--.>UgUn-oo "&*-C"D"D0 m mI)++:i >TVZ3[3[+./klll, m  " Q    -t 4 4e"#cddd   1    sAA++B:B  Bc t||d\}}t||d\}} t||t||t j||t j}n'#tj$r}td|d}~wwxYw|S)zDecode a list of tool call tokens into a tool call for tokenizer version v11 with call ID. Note: Expects the tool call tokens to be in the format: `[TOOL_CALLS]name[CALL_ID]call_id[ARGS]{"arg1": "value1", "arg2": "value2"}` [CALL_ID][ARGS]r$r+r,&Invalid tokenized tool call arguments.N) rrr2rr4r5rr3r8r)r!r"r*call_id_and_argscall_idargsr0r<s r"_decode_tool_call_v11_with_call_idrEVsJJZ\egrssD @AQS\^fggMGT X((!%%d++*Y%5%5dQcQj%5%k%kll     XXX&'OPPVWWX sA7B""C1CCc ^t||d\}} tt||tjt j||tj}n'#t j$r}td|d}~wwxYw|S)zDecode a list of tool call tokens into a tool call for tokenizer version v11 without call ID. Note: Expects the tool call tokens to be in the format: `[TOOL_CALLS]name[ARGS]{"arg1": "value1", "arg2": "value2"}` r@r$r+)r-rAN) rrrr2rr3r4r5r8r)r!r"r*rDr0r<s r_decode_tool_call_v11rGps>>NPY[cddJD$X!%%dASAZ%[[*Y%5%5dQcQj%5%k%kll    XXX&'OPPVWWX sA/BB*B%%B*cg}|D]}|jtjkrtd|jtjkr$|t ||_|jtjkr;|d|vr$| t||| t|||S)atDecode a list of tool call tokens into a list of tool calls. Note: Each list of tool call tokens are expected to be in the format: - v2 to v7: `[TOOL_CALLS][{"name": "name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ...]` - v11+ without call ID: `[TOOL_CALLS]name[ARGS]{"arg1": "value1", "arg2": "value2"}` - v11+ with call ID: `[TOOL_CALLS]name[CALL_ID]call_id[ARGS]{"arg1": "value1", "arg2": "value2"}` Args: tool_call_tokens: A list of lists of tokens. tokenizer: The tokenizer to use for decoding. Returns: The list of decoded tool calls. z6Tool calls are not supported for tokenizer version v1.r?) versionr v1 ValueErrorv7extendr=v11get_special_tokenappendrErG)r!r" tools_callsr0s r_decode_tool_callsrRs K%LL   0 3 3 3UVV V  "2"5 5 5   =iSS T T T T  "2"6 6 69;V;VWb;c;cgp;p;p   A)YWW X X X X   4Y JJ K K K K r)r4typingr!mistral_common.experimental.utilsrr+mistral_common.protocol.instruct.tool_callsrr%mistral_common.tokens.tokenizers.baserr r rKr rr9inttupler r=rErGrRrrrrYs ONNNNNNNaaaaaaaaaa     :        3   . I.+.. 49eDIsN+ +,....> T#Y 9 Y]^fYg    Fcy]e4DI)PX*$s))<W[\dWer