CFG-Constrained Decoding

• CFG-constrained decoding is a method that guides language models to generate outputs strictly conforming to specified context-free grammars using token-level masking and parsing techniques.
• It integrates classical parsing with modern decoding methods, employing pushdown automata and finite-state transducers to overcome challenges like tokenizer misalignment and ambiguity.
• Empirical results demonstrate significant improvements in syntactic correctness, with applications in code completion and structured data extraction despite challenges in deeply nested or ambiguous grammars.

CFG-constrained decoding is the process of steering the output of a LLM, including both autoregressive and diffusion-based systems, to ensure that generated sequences strictly conform to a specified context-free grammar (CFG). This paradigm is essential for domains requiring formal correctness, such as code generation, structured data extraction, and scientific expressions, where outputs must adhere to nontrivial syntactic structures. Techniques for CFG-constrained decoding extend classical parsing and constraint-propagation machinery into the token sampling loop of modern LLMs, handling the complexities introduced by tokenizer misalignment, parallel masked sampling, and multi-region infilling.

1. Foundations: Context-Free Grammars and Model Integration

CFG-constrained decoding enforces that all generated output lies within the language $\mathcal{L}(G)$ of a context-free grammar $G = (N, \Sigma, P, S)$, where $N$ is the set of nonterminals, $\Sigma$ is the terminal alphabet, $P$ the production rules, and $S$ the start symbol. In the context of LLM decoding, the admissible next-token set after a generated prefix $u \in \Sigma^*$ is defined by $$A(u) = \{ a \in \Sigma : \exists v \text{ such that } ua v \in \mathcal{L}(G)\}$$.

For autoregressive LLMs, every generation step must ensure that extended prefixes remain completable to a full string in $\mathcal{L}(G)$. In diffusion-based LLMs (dLLMs), which generate outputs via iterative masked infilling, the decoding method must verify that every intermediate partially-filled string can still be completed to a valid member of the CFG language. Crucially, decoding methods must account for potential misalignment between model tokens and grammar terminals—particularly important for subword vocabularies—by composing tokenization and parsing transducers (Park et al., 7 Feb 2025, Alpay et al., 4 Mar 2026).

2. Algorithmic Techniques for CFG-Constrained Decoding

CFG-constrained decoding bifurcates into methodologies tailored for autoregressive models and those adapted for diffusion-based or infilling models.

2.1 Autoregressive Masking and Reachability

In autoregressive decoding, the standard technique involves online masking, where at each position, only those vocabulary tokens are permitted which are reachable by the parser given the prefix so far. This reachability is computed by compiling the CFG into a pushdown automaton (PDA) and maintaining at each decoding step the set of parser states (and stacks) that can still lead to acceptance (Alpay et al., 4 Mar 2026).

The oracle invariance theorem establishes that language-equivalent grammars yield the same admissibility sets for each prefix, although their PDAs may differ significantly in state complexity and ambiguity cost (Alpay et al., 4 Mar 2026).

2.2 Masking under Tokenizer Misalignment

CFG-constrained decoding must bridge the gap between model tokenization—often subword units—and grammar terminals. This is achieved by constructing a composition automaton (FST) that maps possible token sequences to the terminal alphabet, and by maintaining lookup tables for "always-accepted" and context-dependent token sequences. The precomputed lookup enables efficient per-step admissible-token masking, with empirical overheads as low as 7 ms per token (Park et al., 7 Feb 2025).

2.3 Diffusion LLMs and Parallel Infilling

For dLLMs generating text in parallel or with arbitrary infilling regions, several challenges arise:

• Completable Prefix Verification: At each step, when proposing to fill a masked token, the decoder verifies that the new partial output can be completed (possibly through multiple infilling regions) into a valid word in $\mathcal{L}(G)$. The underlying decision problem reduces to checking non-emptiness of the intersection between $G = (N, \Sigma, P, S)$0 and a regular language representing the positions of fixed and unfixed substrings (Mündler et al., 13 Aug 2025).
• Lookahead Sampling: Methods such as LAVE sample possible fillings for remaining masked positions using the model's parallel token distributions, then use an Earley or LR parser to verify that at least one sampled completion can be extended to a valid CFG sentence, avoiding uncompletable prefixes (Zhang et al., 31 Jan 2026).
• Efficient Checking: The key computational step is to construct, at each intermediate step, the intersection grammar and perform an optimized bottom-up search for non-emptiness, which can be done in $G = (N, \Sigma, P, S)$1 time, where $G = (N, \Sigma, P, S)$2 is the number of grammar rules and $G = (N, \Sigma, P, S)$3 DFA states for the regular language (Mündler et al., 13 Aug 2025).

3. Efficiency and Structural Complexity

The computational efficiency of CFG-constrained decoding is dictated by the structure of the grammar, the alignment complexity of the tokenizer, and the architecture of the decoder.

• Preprocessing: Offline construction of token-level finite-state machines, parser lookup tables, and inverse mappings between tokens and terminal sequences amortize substantial cost, with best-in-class tools preprocessing Python- or Java-like grammars (hundreds of rules, 100k-token vocabularies) in 25–170 seconds (Park et al., 7 Feb 2025).
• Online Overhead: With precomputed data, per-token decoding latency drops to 5–30 ms (subword tokens), supporting practical deployment.
• Ambiguity Cost: The structural ambiguity cost (SAC)—incremental parse forest growth per token—can be made $G = (N, \Sigma, P, S)$4 per step for right-recursive, unambiguous grammars, but reaches $G = (N, \Sigma, P, S)$5 per token for highly ambiguous forms. Engine-independent lower bounds constrain any sound, retrieval-efficient masking engine to incur these costs on adversarial grammars (Alpay et al., 4 Mar 2026).

Grammar Form	Per-Token SAC	Cumulative Cost $G = (N, \Sigma, P, S)$6 tokens
Right-recursive	$G = (N, \Sigma, P, S)$7	$G = (N, \Sigma, P, S)$8
Concatenative	$G = (N, \Sigma, P, S)$9	$N$0

4. Soundness, Completeness, and Correctness Guarantees

CFG-constrained decoding methods guarantee:

• Soundness: No invalid output (non-completable prefixes) can be produced. This is enforced by exact masking and parser checks (Park et al., 7 Feb 2025).
• Completeness: No valid output is erroneously excluded, provided the lookup tables comprehensively capture the alignment between tokens and grammar terminals (Park et al., 7 Feb 2025).
• Semantic Guarantees: For settings with enriched grammars (e.g., variable scoping, types, or API properties), dynamic parsing trees propagate context, and next-step regular expressions enforce "non-extensible" matches, supporting strong semantic and even runtime correctness properties (Li et al., 20 Aug 2025).

For weighted or probabilistic CFGs, cubic-time chart parsing (e.g., weighted CYK) can be used "in the loop," and constraint-propagation frameworks ensure domain-consistency across all assignment variables (0909.4456).

5. Empirical Performance and Limitations

Experimental benchmarks demonstrate that state-of-the-art CFG-constrained decoding frameworks deliver near-perfect syntactic correctness across domains such as code completion, structured output, and formal language generation. On C++ multi-region code infilling, for example, syntax correctness jumps from 32% (vanilla dLLM) to 90% under constrained decoding, and functional correctness improves modestly as well (Mündler et al., 13 Aug 2025).

In diffusion LLMs, the LAVE approach attains syntactic@1 rates above 96% (Dream-7B on CPP-Bench) and functional correctness improvements up to +13 points over the best previous baselines, with end-to-end overheads under 10% (Zhang et al., 31 Jan 2026). These results confirm the practical feasibility of hard constraint enforcement at scale.

Limitations remain:

• For highly ambiguous or deeply nested grammars, per-prefix parsing cost scales cubically in prefix length, and parallel lookahead is subject to the limitations of finite-sample witness search, potentially causing rare false rejections (Zhang et al., 31 Jan 2026).
• CFGs do not encode semantic constraints unless extended with context-sensitive side conditions or augmented dynamic parsing trees (Li et al., 20 Aug 2025).
• Token-budget/length constraints and open-vocabulary support require further integration of regular constraints into the decision automata (Mündler et al., 13 Aug 2025).

6. Extensions and Open Directions

Recent work targets several open extensions:

• Semantic Integration: Augmenting CFG-based methods with side-condition checkers or SMT-based semantic verifiers, enabling type and API conformance (Li et al., 20 Aug 2025).
• Incremental and Streaming Approaches: Efficient incremental emptiness checks and streaming-aligned character-level parsing for open-vocabulary use cases (Mündler et al., 13 Aug 2025).
• Automated Grammar Optimization: Using structural cost models and equality-saturation rewrites to select low-SAC or tokenizer-aligned grammar representations, further reducing runtime and memory (Alpay et al., 4 Mar 2026).
• Applications Beyond Text: Adaptation to structured diffusion models for images, molecular graphs, or 3D flows, leveraging the same intersection emptiness and masked decision frameworks (Saini et al., 9 Oct 2025).

In sum, CFG-constrained decoding is now a mature algorithmic field, enabling provably correct, efficient, and scalable enforcement of complex syntactic—and, increasingly, semantic—constraints across a broad range of LLM architectures (Park et al., 7 Feb 2025, Mündler et al., 13 Aug 2025, Zhang et al., 31 Jan 2026, Alpay et al., 4 Mar 2026, Li et al., 20 Aug 2025).