Globally Constrained Decoding (GCD)

• Globally Constrained Decoding (GCD) is a technique that restricts autoregressive model outputs to tokens that guarantee a syntactically valid and completable sequence.
• It employs reachability oracles and formal grammar constraints, masking tokens that cannot lead to valid completions during decoding.
• GCD is applied in structured NLP, code generation, and safety tasks to improve output validity and maintain alignment with strict grammatical rules.

Globally Constrained Decoding (GCD) is a decoding procedure in which an autoregressive LLM is restricted, at each step, to next tokens that preserve the existence of a complete output satisfying a global constraint. In the large-language-model literature covered here, the constraint is most often formal-language membership: a context-free grammar, a finite automaton language, a JSON schema, a SQL grammar, a keyword obligation, or a code grammar. In the most explicit formulation, GCD is “attention meets reachability”: a Transformer-style next-token distribution is coupled online with a reachability oracle over a pushdown system compiled from a context-free grammar, and logits are hard-masked so that the realized output remains completable under the target language (Alpay et al., 4 Mar 2026). Closely related work treats grammar-constrained decoding as a special case of globally constrained decoding in which the global property is enforced incrementally through valid-prefix computation and masking (Park et al., 7 Feb 2025).

1. Conceptual Scope

GCD operates on the same autoregressive backbone as unconstrained decoding, but it changes the support of the next-token distribution. In a code-generation formulation, standard decoding samples each next token from the full vocabulary $\mathcal V$, whereas GCD computes the set of valid next tokens $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$, then renormalizes over that set only. In context-free settings, the same idea is expressed as masking out tokens that would provably lead to outputs that do not belong to a specified context-free grammar; in finite-automaton settings, the same idea is expressed as masking out tokens that cannot lead to any valid completion within the remaining budget (Zhang et al., 10 Jun 2026, Park et al., 7 Feb 2025, Dang et al., 1 Jun 2026).

This literature treats grammar-constrained decoding as a unified framework for structured generation. Outputs may be constrained by a fixed grammar, or by an input-dependent grammar whose rule set or terminal vocabulary depends on the input instance. The latter is used, for example, when entity candidates depend on a mention, or when a constituency parse must contain exactly the words of the input sentence in order (Geng et al., 2023).

2. Reachability Semantics

A central formalization starts from an autoregressive model over a vocabulary $V$,

$$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$

together with a context-free grammar $G=(N,\Sigma,P,S)$ and a tokenizer homomorphism $\tau:V\to\Sigma^*$ extended to sequences by concatenation. A sequence is acceptable iff $\tau(y_{1:T})\in L(G)$. For a language $L\subseteq\Sigma^*$, the relevant prefix notion is

$$Pref(L)=\{u\in\Sigma^*:\exists v\in\Sigma^*,\, uv\in L\},$$

and the allowed next terminal after a prefix $u$ is

$$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$0

GCD is then defined by coupling the model with a reachability oracle over a pushdown system compiled from the grammar (Alpay et al., 4 Mar 2026).

In that formulation, the grammar is compiled into a pushdown system or NPDA, and the oracle returns

$$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$1

where $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$2 denotes the live configurations reachable after consuming exactly $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$3. The masked decoding distribution is

$$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$4

This makes the global condition “$$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$5” operative as an online reachability problem over a compiled pushdown automaton (Alpay et al., 4 Mar 2026).

A notable theorem is oracle invariance: if $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$6, then for all prefixes $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$7, $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$8. Language-equivalent grammars therefore induce identical admissible next-token sets for every prefix, hence identical logit masks, even though they may have different compiled state spaces, different $$\mathcal V_G(y_{<t}) = \{ v \in \mathcal V \mid \exists\, y_{>t}\ \text{s.t.}\ (y_{<t}, v, y_{>t}) \in \mathcal L(G) \}$$9-closure graphs, and different ambiguity or parse-forest structures. The paper identifies this as a key conceptual separation: semantics depend only on the language, whereas cost depends on how the grammar is compiled and how reachability is implemented (Alpay et al., 4 Mar 2026).

3. Conditional Sampling, Bias, and Alignment

Hard masking guarantees validity, but it does not in general sample from the true conditional distribution $V$0. The exact conditional process can be written through a Doob $V$1-transform. Let $V$2 be the event that the run terminates with $V$3 and $V$4, and define the survival function $V$5. Then, whenever $V$6,

$V$7

Hard-masked decoding equals this true conditional sampler iff $V$8 is constant over all admissible $V$9. The one-step distortion admits sharp bounds: $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$0 with $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$1, and the corresponding total-variation bound follows by Pinsker’s inequality (Alpay et al., 4 Mar 2026).

A parallel line of work makes the same point from the perspective of expected future grammaticality. Standard GCD uses a binary feasibility approximation,

$$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$2

whereas grammar-aligned decoding requires weighting next tokens by the probability that the prefix will actually complete to a grammatical string under the model. “Grammar-Aligned Decoding” defines the correct target as $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$3 and proposes ASAp, described as the first algorithm that provably converges to the correct distribution while still enforcing the grammar. “AdapTrack” makes an analogous argument for constrained decoding in code generation, introducing backtracking so that the distribution produced aligns with the model’s distribution given the generated tokens, thereby ensuring that the model’s output intent is not distorted (Park et al., 2024, Li et al., 20 Oct 2025).

A related finite-horizon automaton formulation distinguishes locally constrained decoding from globally constrained decoding. LCD masks tokens that preserve extendability at some length, whereas GCD masks tokens that preserve extendability to length exactly $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$4: $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$5 This guarantees constraint satisfaction at termination, but the resulting process remains locally normalized and therefore biased relative to $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$6. The same work introduces probabilistic GCD (P-GCD) and uses it as a proposal and potential inside sequential Monte Carlo to mitigate that bias (Dang et al., 1 Jun 2026).

4. Structural Complexity and Systems

Oracle invariance implies that masks are a language-level invariant, but it does not imply that equivalent grammars have comparable decoding cost. To quantify the intrinsic online complexity of parse-preserving engines, “Attention Meets Reachability” introduces the left-to-right structural ambiguity cost (SAC), defined from the incremental growth of the packed parse structure: $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$7 For two equivalent grammars for $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$8, SAC separates sharply: a binary-concatenative ambiguous grammar has $$p_\theta(y_{1:T})=\prod_{t=1}^{T} p_\theta(y_t \mid y_{<t}),\quad y_t\in V,$$9 per-token SAC and $G=(N,\Sigma,P,S)$0 cumulative growth, whereas a right-recursive regular grammar admits $G=(N,\Sigma,P,S)$1 per-token SAC and $G=(N,\Sigma,P,S)$2 cumulative growth. The same paper gives an exact control-state blowup for $G=(N,\Sigma,P,S)$3: a minimal grammar compiles to $G=(N,\Sigma,P,S)$4 control states, while a language-equivalent grammar with redundant delegation compiles to $G=(N,\Sigma,P,S)$5, a factor $G=(N,\Sigma,P,S)$6. It also proves an engine-independent lower bound: for a constant-size CFG family, any sound, parse-preserving, retrieval-efficient online masking engine must incur $G=(N,\Sigma,P,S)$7 work per token and $G=(N,\Sigma,P,S)$8 cumulatively on some prefixes (Alpay et al., 4 Mar 2026).

Automaton-based GCD for fixed-length constraints makes the cost model more explicit. Constraints are compiled into tensorized NFAs or DFAs represented by sparse binary matrices, and forward–backward messages over the automaton yield the mask

$G=(N,\Sigma,P,S)$9

In experiments on H100 GPUs, GCD incurred about $\tau:V\to\Sigma^*$0 latency over unconstrained decoding, very close to LCD at $\tau:V\to\Sigma^*$1, whereas P-GCD incurred $\tau:V\to\Sigma^*$2, $\tau:V\to\Sigma^*$3, and $\tau:V\to\Sigma^*$4 overhead for HMM sizes $\tau:V\to\Sigma^*$5, $\tau:V\to\Sigma^*$6, and $\tau:V\to\Sigma^*$7, respectively. The same work emphasizes that NFA support is critical because the NFA can be exponentially more succinct than the DFA; for Spider SQL, the character-level NFA had 8k states versus 57k for the DFA, and the token-level DFA exhausted memory (Dang et al., 1 Jun 2026).

System design has therefore focused on shifting work offline. “Flexible and Efficient Grammar-Constrained Decoding” constructs a token-level lexing transducer, a set of realizable terminal sequences, and an inverse token spanner table, then preprocesses parser states into always-accepted tokens and context-dependent sequences. Its implementation, GreatGramma, reports $\tau:V\to\Sigma^*$8 faster offline preprocessing than existing approaches while preserving state-of-the-art efficiency in online mask computation. Across Go, Java, and Python grammars and LLaMA-, LLaMA-3-, and Qwen-family tokenizers, GreatGramma reports 5–32 ms per token online overhead, whereas Outlines required more than 3 seconds per token on LLaMA and more than 12 seconds on LLaMA-3 and Qwen-2, and SynCode was $\tau:V\to\Sigma^*$9 slower on average where it was sound (Park et al., 7 Feb 2025).

5. Applications and Architectural Integration

The most direct application area is structured NLP without finetuning. “Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning” treats grammars as declarative specifications of the output space and uses Grammatical Framework to separate abstract syntax from tokenizer-specific concrete syntax. The paper introduces input-dependent grammars, where the grammar depends on the input instance, and demonstrates the approach on information extraction, entity disambiguation, and constituency parsing. In closed information extraction, LLaMA-33B + GCD reaches about 36 micro-F1, compared with about 17.5 for unconstrained decoding; in entity disambiguation, LLaMA-33B + GCD with an input-dependent grammar reaches 80.3 average accuracy, versus 68.2 with an input-independent grammar and 54.1 unconstrained; in constituency parsing, the input-dependent grammar yields 100% valid trees by construction and about 54.6 F1 for LLaMA-33B on shorter sentences (Geng et al., 2023).

Finite-automaton GCD has been evaluated on function calling, keyword-based generation, and text-to-SQL. In xLAM function calling, GCD and P-GCD achieve 100% constraint satisfaction in the tested settings, whereas LCD remained around 95–99% at low particle counts. In CommonGen, GCD proposals improve over LCD proposals, and adding the P-GCD potential yields further BLEU gains and faster convergence in particle count. In Spider SQL, GCD and P-GCD both improve over LCD at small particle counts (Dang et al., 1 Jun 2026).

The reachability formulation also admits direct architectural integration. Grammar-modulated logits may be obtained by adding a learned embedding of the live configuration set $\tau(y_{1:T})\in L(G)$0 to the model logits, and Mixture-of-Experts routing can be modulated by a grammar-state embedding so that expert selection depends on syntactic state while hard masking preserves soundness. The same paper derives latency envelopes in terms of vocabulary size, active state sets, and beam width, and connects SAC to instrumentation-based predictive performance models and automated grammar optimization (Alpay et al., 4 Mar 2026).

6. Limitations, Security, and Terminological Ambiguity

Several limitations recur across the literature. Grammar structure—not just language—drives NPDA control-state universe size, online packed-structure growth, active configuration counts, and mask-computation latency; more restrictive grammars can reduce search space but may themselves become large; and GCD requires token-level access to logits, so closed APIs that do not expose distributions cannot implement it directly (Alpay et al., 4 Mar 2026, Geng et al., 2023). A recurrent decoding pathology is empty-string dominance: when the grammar allows empty outputs, the globally most likely output under the unconstrained model, subject to grammar, is often EOS immediately. The reported mitigation is length normalization in beam search, with score $\tau(y_{1:T})\in L(G)$1, and instruction-tuned models exhibit a reduced tendency to output empty strings (Geng et al., 2023).

A distinct criticism is that globally valid decoding can alter the semantics of refusal and safety. “Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code” shows that simply applying a benign code grammar constraint can effectively jailbreak LLMs, because typical code grammars eliminate natural-language refusals from the support. On 10 popular LLMs across 4 benchmarks, CodeSpear increases the attack success rate by more than 30 percentage points on average, and on API-served models the average ASR rises from 22.00% to 67.39%. The same paper proposes CodeShield, which aligns the model in the code modality by teaching it to generate honeypot code under GCD; under deepred, deepgreen reduces ASR and malicious rate to single-digit percentages while largely preserving code-generation utility (Zhang et al., 10 Jun 2026).

A final source of confusion is terminological. In the LLM and structured-generation papers discussed above, GCD denotes grammar-constrained or globally constrained decoding. In coding-theory papers, however, GCD is explicitly “Guessing Codeword Decoding,” not “Globally Constrained Decoding,” and refers to a soft-input forward error correction decoder for arbitrary binary linear codes (Griffin et al., 2024, Feng et al., 20 Mar 2025). This suggests that the acronym is domain-sensitive: in large-language-model decoding it names a family of global support-restriction procedures, whereas in channel coding it names a distinct guessing-based decoder.