*`i->.dZddlZddlmZmZmZmZmZmZm Z m Z ddl m Z ddl mZddlmZmZddlmZmZd e eeeefefd efd Zd e eee eeeffd efd Zdeedeeefd efdZGddeZdS)z3This module provides classes representing grammars.N)AnyDictListOptionalTupleTypeUnionoverload) BaseModel) deprecated) XGRObject_core) StructuralTagStructuralTagIteminstancereturnct|trtj|St|tr|St|t r|Std)a)Convert a instance to a string representation. It returns the schema in string format because it's faster to send to C++. This function handles different instance input types and converts them to a JSON string: - StructuralTag. - String inputs are returned as-is (assumed to be valid JSON) - Dictionary inputs are converted to JSON strings Parameters ---------- instance : Union[str, StructuralTag, Dict[str, Any]] The instance to convert, which can be a StructuralTag, a JSON schema string, or a dictionary representing a JSON schema. Returns ------- str The JSON schema as a string. Raises ------ ValueError When the instance type is not supported. TypeError When he dictionary is not serializable. zInvalid instance type) isinstancedictjsondumpsstrrmodel_dump_json ValueError)rs d/home/jaya/work/projects/VOICE-AGENT/VIET/agent-env/lib/python3.11/site-packages/xgrammar/grammar.py_convert_instance_to_strr sp6(D!!2z(### Hc " "2 Hm , ,2'')))0111schemact|trt|tr{t |dr&t j|St |dr&t j|Stdt|tr|St|trt j|Std)aSConvert a schema to a string representation. It returns the schema in string format because it's faster to send to C++. This function handles different schema input types and converts them to a JSON string: - Pydantic models are converted using their schema methods - String inputs are returned as-is (assumed to be valid JSON) - Dictionary inputs are converted to JSON strings Parameters ---------- schema : Union[str, Type[BaseModel], Dict[str, Any]] The schema to convert, which can be a Pydantic model class, a JSON schema string, or a dictionary representing a JSON schema. Returns ------- str The JSON schema as a string. Raises ------ ValueError When the schema type is not supported. TypeError When the dictionary is not serializable. model_json_schema schema_jsonzAThe schema should have a model_json_schema or json_schema method.z2The schema should be a string or a Pydantic model.) rtype issubclassr hasattrrrr!r"rrr)rs r_convert_schema_to_strr&2s6&$ OJvy$A$A O 6. / / ::f668899 9 6= ) ) b:f002233 3`aa a FC O FD ! !Oz&!!!MNNNrargskwargsct|dkrLt|dtttfrt |dSt dt|dkrkt|dtrPt|dtr5t j|d|d dSd|vrt |dSd|vr9d |vr5t j|d|d  dSt d ) aGet the structural tag string from the arguments. It returns the structural tag in string format because it's faster to send to C++. Parameters ---------- args : List[Any] The positional arguments. kwargs : Dict[str, Any] The keyword arguments. Returns ------- str The structural tag string. Raises ------ TypeError When the arguments are invalid. r rz-Invalid argument type for from_structural_tagN)indentstructural_tagtagstriggersz)Invalid arguments for from_structural_tag) lenrrrrr TypeErrorlistfrom_legacy_structural_tagr)r'r(s r!_get_structural_tag_str_from_argsr3\s4* 4yyA~~ d1gT=9 : : M+DG44 4KLL L TaJtAw55*T!Wd:S:S7QaIIYYZ    V # #'/?(@AAA 6  jF227 6NF:.  // & & 'CDDDrcheZdZdZdefdZedddededdfdZed d d d d d d d eee e e ee ffde deedeeeefde deede ddfdZed ddede ddfdZeedeeee ee ffddfdZeeeddeedeeddfdZed&dZed&dZed'd!Zed'd"Zdefd#Zed$eddfd%Zd S)(GrammaraThis class represents a grammar object in XGrammar, and can be used later in the grammar-guided generation. The Grammar object supports context-free grammar (CFG). EBNF (extended Backus-Naur Form) is used as the format of the grammar. There are many specifications for EBNF in the literature, and we follow the specification of GBNF (GGML BNF) in https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md. When printed, the grammar will be converted to GBNF format. rc4|jS)zPrint the BNF grammar to a string, in EBNF format. Returns ------- grammar_string : str The BNF grammar string. )_handle to_stringselfs r__str__zGrammar.__str__s|%%'''rroot)root_rule_name ebnf_stringr=crttj||S)a Construct a grammar from EBNF string. The EBNF string should follow the format in https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md. Parameters ---------- ebnf_string : str The grammar string in EBNF format. root_rule_name : str, default: "root" The name of the root rule in the grammar. Raises ------ RuntimeError When converting the regex pattern fails, with details about the parsing error. )r5_create_from_handler from_ebnf)r>r=s rrAzGrammar.from_ebnfs+$**5=+B+B;P^+_+_```rTNF)any_whitespacer+ separators strict_modemax_whitespace_cntprint_converted_ebnfrrBr+rCrDrErFc t|}ttj|||||||S)a Construct a grammar from JSON schema. Pydantic model or JSON schema string can be used to specify the schema. It allows any whitespace by default. If user want to specify the format of the JSON, set `any_whitespace` to False and use the `indent` and `separators` parameters. The meaning and the default values of the parameters follows the convention in json.dumps(). It internally converts the JSON schema to a EBNF grammar. Parameters ---------- schema : Union[str, Type[BaseModel], Dict[str, Any]] The schema string or Pydantic model or JSON schema dict. any_whitespace : bool, default: True Whether to use any whitespace. If True, the generated grammar will ignore the indent and separators parameters, and allow any whitespace. indent : Optional[int], default: None The number of spaces for indentation. If None, the output will be in one line. Note that specifying the indentation means forcing the LLM to generate JSON strings strictly formatted. However, some models may tend to generate JSON strings that are not strictly formatted. In this case, forcing the LLM to generate strictly formatted JSON strings may degrade the generation quality. See for more details. separators : Optional[Tuple[str, str]], default: None Two separators used in the schema: comma and colon. Examples: (",", ":"), (", ", ": "). If None, the default separators will be used: (",", ": ") when the indent is not None, and (", ", ": ") otherwise. strict_mode : bool, default: True Whether to use strict mode. In strict mode, the generated grammar will not allow properties and items that is not specified in the schema. This is equivalent to setting unevaluatedProperties and unevaluatedItems to false. This helps LLM to generate accurate output in the grammar-guided generation with JSON schema. max_whitespace_cnt : Optional[int], default: None The maximum number of whitespace characters allowed between elements, such like keys, values, separators and so on. If None, there is no limit on the number of whitespace characters. If specified, it will limit the number of whitespace characters to at most max_whitespace_cnt. It should be a positive integer. print_converted_ebnf : bool, default: False If True, the converted EBNF string will be printed. For debugging purposes. Returns ------- grammar : Grammar The constructed grammar. Raises ------ RuntimeError When converting the json schema fails, with details about the parsing error. )r&r5r@rfrom_json_schema)rrBr+rCrDrErF schema_strs rrHzGrammar.from_json_schemasTN,F33 ** M * *"$     r)rF regex_stringcrttj||S)aCreate a grammar from a regular expression string. Parameters ---------- regex_string : str The regular expression pattern to create the grammar from. print_converted_ebnf : bool, default: False This method will convert the regex pattern to EBNF first. If this is true, the converted EBNF string will be printed. For debugging purposes. Default: False. Returns ------- grammar : Grammar The constructed grammar from the regex pattern. Raises ------ RuntimeError When parsing the regex pattern fails, with details about the parsing error. )r5r@r from_regex)rJrFs rrLzGrammar.from_regexs2.** M $ $\3G H H   rr,cdSN)r,s rfrom_structural_tagzGrammar.from_structural_tags Crzqfrom_structural_tag(tags, triggers) is deprecated. Construct structural tag with the StructuralTag class instead.r-r.cdSrNrO)r-r.s rrPzGrammar.from_structural_tag#s _b^arct||}ttj|S)agCreate a grammar from a structural tag. See the Structural Tag Usage in XGrammar documentation for its usage. This method supports two calling patterns: 1. Single structural tag parameter: from_structural_tag(structural_tag) 2. Legacy pattern (deprecated): from_structural_tag(tags, triggers) Parameters ---------- structural_tag : Union[StructuralTag, str, Dict[str, Any]] The structural tag either as a StructuralTag object, or a JSON string or a dictionary. tags : List[StructuralTagItem] (Deprecated) The structural tags. Use StructuralTag class instead. triggers : List[str] (Deprecated) The triggers. Use StructuralTag class instead. Returns ------- grammar : Grammar The constructed grammar from the structural tag. Raises ------ InvalidJSONError When the structural tag is not a valid JSON string. InvalidStructuralTagError When the structural tag is not valid. TypeError When the arguments are invalid. Notes ----- The legacy pattern from_structural_tag(tags, triggers) is deprecated. Use the StructuralTag class to construct structural tags instead. For the deprecated pattern: The structural tag handles the dispatching of different grammars based on the tags and triggers: it initially allows any output, until a trigger is encountered, then dispatch to the corresponding tag; when the end tag is encountered, the grammar will allow any following output, until the next trigger is encountered. See the Advanced Topics of the Structural Tag in XGrammar documentation for its semantic. Structural Tag in XGrammar documentation for its semantic. )r3r5r@rrP)r'r(structural_tag_strs rrPzGrammar.from_structural_tag+s:d?tVLL**5=+L+LM_+`+`aaarcnttjS)zGet the grammar of standard JSON. This is compatible with the official JSON grammar specification in https://www.json.org/json-en.html. Returns ------- grammar : Grammar The JSON grammar. )r5r@rbuiltin_json_grammarrOrrrUzGrammar.builtin_json_grammar`s&**5=+M+M+O+OPPPrgrammarscd|D}ttj|S)aCreate a grammar that matches the concatenation of the grammars in the list. That is equivalent to using the `+` operator to concatenate the grammars in the list. Parameters ---------- grammars : List[Grammar] The grammars to create the concatenation of. Returns ------- grammar : Grammar The concatenation of the grammars. cg|] }|j SrOr7.0grammars r z"Grammar.concat..{CCCw7?CCCr)r5r@rconcatrVgrammar_handless rr_zGrammar.concatls<DC(CCC**5=+?+?+P+PQQQrcd|D}ttj|S)aCreate a grammar that matches any of the grammars in the list. That is equivalent to using the `|` operator to concatenate the grammars in the list. Parameters ---------- grammars : List[Grammar] The grammars to create the union of. Returns ------- grammar : Grammar The union of the grammars. cg|] }|j SrOrYrZs rr]z!Grammar.union..r^r)r5r@runionr`s rrdz Grammar.union~s<DC(CCC**5=+>+>+O+OPPPrc4|jS)zSerialize the grammar to a JSON string. Returns ------- json_string : str The JSON string. )r7serialize_jsonr9s rrfzGrammar.serialize_jsons|**,,,r json_stringcpttj|S)aQDeserialize a grammar from a JSON string. Parameters ---------- json_string : str The JSON string. Returns ------- grammar : Grammar The deserialized grammar. Raises ------ InvalidJSONError When the JSON string is invalid. DeserializeFormatError When the JSON string does not follow the serialization format of the grammar. DeserializeVersionError When the __VERSION__ field in the JSON string is not the same as the current version. )r5r@rdeserialize_json)rgs rrizGrammar.deserialize_jsons(.**5=+I+I++V+VWWWr)rr5)rVr5rr5)__name__ __module__ __qualname____doc__rr; staticmethodrAr rr rrboolrintrrHrLr rrPr rrrUr_rdrfrirOrrr5r5s  (((((=Caaasasa aaa\a& $ $04 ,0%*Q Q Q c4 ?DcN:;Q Q  Q U38_- Q  Q %SMQ #Q  Q Q Q \Q fFK    t PY   \ 4mS$sCx.@A \XZ 'b$'8"9aT#YaS\aaa \X b2b2b2b\2bh Q Q Q\ QRRR\R"QQQ\Q"-----XcXiXXX\XXXrr5)rmrtypingrrrrrrr r pydanticr typing_extensionsr baserrr,rrrrr&r3r5rOrrrus99 JJJJJJJJJJJJJJJJJJJJ((((((""""""""<<<<<<<<"2uS$sCx.--O'P"2UX"2"2"2"2J'O5d9otCH~)M#N'OSV'O'O'O'OT%EDI%EtCH~%ERU%E%E%E%EPmXmXmXmXmXimXmXmXmXmXr