---
title: 'XGrammar 2: High-Performance Grammar Systems'
url: https://www.emergentmind.com/topics/xgrammar-2
type: topic
---

# XGrammar 2: High-Performance Grammar Systems

XGrammar 2 refers to two independent systems: (1) a dynamic, high-efficiency structured generation engine (“Contour”) targeting agentic LLMs [2601.04426], and (2) a declarative, streaming grammar language for XML domain-specific languages [1506.03366]. Both employ advanced formal machinery to optimize for expressivity, performance, and low overhead in structured text generation and parsing.

## 1. TagDispatch and Dynamic Grammar Dispatch

XGrammar 2 introduces TagDispatch, a dispatch mechanism optimized for dynamic structured generation in LLM-based agents [2601.04426]. In agentic tasks (e.g., tool-calling, stepwise conditional reasoning), sequence generation often begins with a dedicated “tag” (e.g., `<function=foo>`, `〈|channel|〉msg`), after which generation must proceed under a specialized context-free grammar (CFG).

### TagDispatch Structure

Formally, a TagDispatch instance is a triple:
$$
TD = (T,\, G,\, S_{\text{stop}})
$$
where $T$ is a finite set of tag strings, $G = \{G_i\}$ are corresponding grammars, and $S_{\text{stop}}$ is a set of stop-strings. Decoding alternates between:

- **Dispatching Mode**: Uses an Aho–Corasick automaton to scan for any tag in $T$ amidst generated tokens; only very lightweight token masking is required.
- **Dispatched Mode**: On matching a tag $t_i$, control passes to $G_i$, enforcing its CFG via mask generation until completed, then returns to dispatching.

A crucial performance gain derives from deferring expensive mask construction and cache occupancy for $G_i$ until its tag $t_i$ is seen, drastically reducing memory and computational overhead versus static approaches where all tool schemas are unioned and tracked simultaneously.

## 2. Just-in-Time Mask Compilation

Grammar-constrained decoding necessitates a mapping from token-prefixes to allowable next-token sets (“masks”). Precomputing this adaptive mask-cache for large, highly adaptive grammars is prohibitive: compilation cost can reach several seconds per grammar [2601.04426]. 

XGrammar 2 adopts a partial-JIT (just-in-time) strategy:

- In a prefilling phase, the states are sorted by estimated compile cost; the top $K$ are precompiled within a user-specified time budget, the rest are marked compile-on-demand.
- When decoding reaches an uncached state, mask computation is performed only as needed; runtime spikes are often hidden by overlapping with LLM steps.

Reported speed-ups are extrema: on the JSONSchemaBench, preprocessing drops from 4960 ms (static) to 612 ms (JIT), with further reductions from auxiliary techniques reducing to single-digit milliseconds.

## 3. Cross-Grammar Caching via FSM Hashing

Mask computation for CFGs is expensive, but sub-grammars (e.g., “string”, “number”, common object patterns) are ubiquitous across many schemas. XGrammar 2 minimizes redundant computation by hashing minimized FSM representations of grammar fragments, thereby recognizing and reusing structurally identical subgraphs [2601.04426].

### FSM Hashing and Cache Lookup

- Each production rule is converted to a minimized FSM and a canonical 64-bit hash is computed, resolving cycles to guarantee uniqueness.
- The token-mask cache is keyed by (fsm-hash, lookahead signature); upon a cache miss, partial results can be reused if only lookahead differs, reducing recomputation.
- This mechanism is critical for dynamic settings where grammars composed at runtime share repeated patterns.

## 4. Efficient Mask Generation Algorithms

### PDA and Earley-Parser-Based Masking

Push-down automaton (PDA) approaches underlie many CFG-constrained decoders. XGrammar 2 extends this with an Earley parser backend, enabling efficient coverage of non-LL(1) grammars, where PDA states can explode combinatorially.

- Earley’s dynamic programming table $E[j]$ encodes “Earley items”; only states that can accept a terminal at the next token, called “scannable,” require full caching.
- For each scannable state $s$, three disjoint subsets of tokens are identified: accepted ($A_s$), rejected ($R_s$), and uncertain ($U_s$)—with uncertain entries resolved via further lookahead.
- This yields linear growth in cache size with grammar complexity and amortizes cache lookups to $O(1 + |U_s|\cdot \text{lookahead cost})$.

The worst-case time is $O(n^3)$, but practical grammars are nearly deterministic, yielding $O(n)$–$O(n^2)$.

### Repetition Compression

High-arity repetition rules (e.g., $A \to B^{0,65536}$) can generate tens of thousands of distinct PDA/Earley states with negligible practical difference in masks. XGrammar 2 compresses repetition by expanding only up to threshold $T$ explicit copies, summarizing the intervening states with compact repetition operators. This reduces state space to $O(T)$ and drastically improves both cache hits and mask-inference sharpness for large, variadic structures.

## 5. Performance Analysis and Empirical Results

XGrammar 2 (“Contour”) has been empirically evaluated on multiple structured-generation and function-calling workloads [2601.04426].

| Engine      | Per-token overhead | Grammar compile time |
|-------------|-------------------|---------------------|
| XGrammar    | 200–400 μs        | 1,000–1,200 ms      |
| llguidance  | 250–1,000 μs      | –                   |
| Contour     | 30–80 μs          | 10–15 ms            |

- In function-calling tasks (BFCL-v3 and SGLang), XGrammar 2 delivered end-to-end overheads of <6% compared to unconstrained decoding, corresponding to a 7x speed-up over XGrammar.
- Ablation studies on JSONSchemaBench show compounding reductions: Earley-only (4960 ms, 45 μs/mask), +JIT (612 ms, 722 μs), +cross-grammar (535 ms, 334 μs), +repetition compression (5.4 ms, 126 μs).

End-to-end, XGrammar 2 achieves 6–10× per-token speedup and >100× compilation-time reduction over XGrammar.

## 6. Integration with LLM Inference and Engine Architecture

XGrammar 2 is designed for tight coupling with modern LLM inference pipelines (e.g., SGLang, vLLM). The typical processing pipeline is:

- Tokenizer encodes prompt and history to prefix IDs.
- LLM encoder computes logits for possible next tokens.
- XGrammar 2’s mask module intercepts logits, applies the relevant mask depending on dispatching mode (TagDispatch vs. specific subgrammar), and returns masked logits to the sampler.
- Minimal hooks are required—engine-agnostic integration is enabled via a `mask_callback()` interface.

CPU+GPU evaluation on RTX 5090 and Xeon Platinum yields <10 ms per mask generation step, netting <6% total overhead due to overlap with GPU LLM execution.

## 7. Declarative Grammar Processing for XML (Clark XGrammar 2)

A separate “XGrammar 2” system, as described by Clark [1506.03366], is a declarative grammar formalism designed for streaming, table-driven, high-performance XML parsing atop SAX. Key features include:

- Grammars specified in BNF/EBNF-like style, supporting element patterns, bindings, repetition, disjunction, and semantic actions.
- Parsing specified via inference rules matching input sequences to synthesized values, with the grammar’s operational semantics defined by a prediction-table-driven abstract machine.
- All parsing occurs with LL(1) predictiveness, enabling constant-time dispatch per event and eliminating the need to materialize full XML trees in memory (unlike DOM).
- The machine state consists of code pointer, environment, value stack, SAX event queue, and return stack, all manipulated via a fixed set of transition rules.

This grammar engine achieves $O(n)$ parse time (with $n$ SAX events), constant per-event cost, and is suitable for large or streaming XML inputs, rivaling hand-tuned efficiency while being substantially more maintainable.

## Summary Table: Principal Features of XGrammar 2 Systems

| System          | Domain          | Core Techniques                    | Evaluated Speedup           |
|-----------------|----------------|------------------------------------|-----------------------------|
| Contour (2026)  | LLM Structured Gen. | TagDispatch, JIT, FSM caching, Earley, repetition compression | 6–10x per-token, 100x compile [2601.04426] |
| Clark XGrammar  | XML Parsing     | Declarative LL(1) grammars, table-driven stack machine | Comparable to tuned SAX [1506.03366]       |

Both strands of XGrammar 2 offer formal, efficient mechanisms for syntax-driven generation or recognition in their respective domains, with demonstrated low overhead, high reusability, and empirical scalability.

Source: https://www.emergentmind.com/topics/xgrammar-2