---
title: Candidate-Constrained Decoding
url: https://www.emergentmind.com/topics/candidate-constrained-decoding
type: topic
---

# Candidate-Constrained Decoding

to=arxiv_search.search  大发快三是什么query":"candidate-constrained decoding draft-conditioned constrained decoding structured generation LLMs", "max_results": 5}
to=arxiv_search.search  彩神争霸 ＿国产?
to=arxiv_search.search  彩神争霸输钱json_string
{"query":"candidate-constrained decoding", "max_results": 10}
to=arxiv_search ՞նչ
{"query":"candidate-constrained decoding", "max_results": 10}
to=arxiv_search.search  北京赛车女json
{"query":"Draft-Conditioned Constrained Decoding for Structured Generation in LLMs", "max_results": 5}
Candidate-constrained decoding denotes a family of inference procedures in which generation is restricted to, selected from, or conditioned on an explicit candidate set rather than left to unconstrained autoregressive decoding. In the cited literature, this idea appears in several closely related forms: selecting the highest-probability feasible labeling under a hard constraint function, choosing from a finite candidate pool by expected utility, conditioning constrained generation on an unconstrained draft, evaluating top-\(k\) next-token candidates by speculative lookahead, and scoring candidates directly from logits without full decoding [2312.03367] [2512.01316] [2603.03305] [2412.10418] [2501.17338]. *This suggests* that “candidate-constrained decoding” is best treated as a unifying perspective on constrained inference, not as a single standardized algorithm.

## 1. Formal scope and recurring formulations

A canonical formulation appears in structured prediction, where the decoder searches over candidate outputs but retains only those satisfying a hard global constraint function. For token classification, the objective is
\[
\hat{\mathbf{y}} = \arg\max_{\mathbf{y} \in \mathcal{Y}^n} p(\mathbf{y} \mid \mathbf{x}) \cdot \mathcal{C}(\mathbf{x}, \mathbf{y}),
\]
with \(\mathcal{C} : \mathcal{X}^n \times \mathcal{Y}^n \to \{0,1\}\), so infeasible candidates are assigned probability \(0\) and excluded from selection [2312.03367].

A second formulation constrains decoding to a finite candidate set \(\mathcal{Y} = \{y_1,\dots,y_N\}\), then ranks candidates by a decision rule rather than by token-level next-step probabilities. In minimum Bayes risk decoding, the chosen output is
\[
y_{\text{MBR}} = \argmax_{y_i \in \mathcal{Y}} \frac{1}{M} \sum_{j=1}^{M} O_{ij},
\qquad O_{ij} = u(y_i,\hat{y}_j),
\]
so decoding is explicitly candidate-constrained because the search space is \(\mathcal{Y}\), not \(\mathcal{T}\) [2512.01316].

A third formulation preserves token-by-token constrained decoding, but changes the conditional distribution on which constraints are enforced. In Draft-Conditioned Constrained Decoding, a free-form draft \(y\) is generated first, and the final structured output \(z\) is then decoded under hard constraints from
\[
\tilde q(z_t \mid \tilde h_t)
=
\frac{p_2(z_t\mid \tilde h_t)\,\mathbb{I}[z_t\in A(h_t)]}{\tilde\alpha(\tilde h_t)},
\qquad
\tilde h_t=(x,y,z_{<t}),
\]
with the same valid-next-token set \(A(h_t)\) as standard constrained decoding, but a draft-conditioned feasible mass \(\tilde\alpha(\tilde h_t)\) [2603.03305].

A fourth formulation avoids autoregressive decoding entirely. In decoding-free generative candidate selection, the answer must belong to a candidate pool \(C = \{c^1,\dots,c^{|C|}\}\), and a function \(f_{\mathrm{est}}\) estimates candidate probabilities from prior-decoding logits without discrete decoding:
\[
P(c \mid t^{in}_{1:|seq_{in}|}) = f_{\mathrm{est}}(f_{\mathrm{encode}}(seq_{in})).
\]
Here the constraint is that the final prediction must be an element of \(C\), but inference occurs in candidate space rather than token space [2501.17338].

## 2. Principal inference patterns

Across the literature, candidate-constrained decoding is implemented through a small number of recurring mechanisms. The differences concern what counts as a “candidate,” when constraints are evaluated, and whether feasibility is enforced as hard masking, as best-first selection, or as reweighting.

| Pattern | Candidate object | Representative mechanism |
|---|---|---|
| Feasible-set decoding | Full output or labeling | Maximize model score subject to \(\mathcal{C}(\mathbf{x},\mathbf{y})=1\) |
| Candidate-set reranking | Finite set \(\mathcal{Y}\) | Rank by expected utility or approximate MBR score |
| Draft-conditioned decoding | Free-form draft \(y\) | Condition constrained decoding on \(y\) |
| Lookahead-constrained decoding | Top-\(k\) next tokens or short continuations | Roll out lookaheads and score them with \(\mathcal{R}\) |
| Decoding-free selection | Candidate pool \(C\) | Estimate \(P(c\mid x)\) directly from logits |

In standard constrained decoding for structured generation, hard constraints are enforced token-by-token by masking and renormalization:
\[
q(z_t \mid h_t)
=
\frac{\pi_{\theta}(z_t\mid h_t)\,\mathbb{I}[z_t\in A(h_t)]}{\alpha(h_t)},
\qquad
\alpha(h_t)=\sum_{a\in A(h_t)} \pi_\theta(a\mid h_t).
\]
When \(\alpha(h_t)\) is small, renormalization strongly distorts the distribution, especially under strict schemas that require early formatting tokens such as braces, quotes, commas, or field names [2603.03305].

Lazy-\(k\) represents a different pattern: it enumerates full label assignments in exact decreasing order of probability and stops when it finds a candidate satisfying \(\mathcal{C}\), or when it reaches a limit \(k\). Its cost function is
\[
g(\mathbf{y}) = - \sum_{i=1}^n \log p(y_i \mid \mathbf{x}),
\]
and the procedure maintains a min-heap over next-best neighbors at edit distance \(1\) in the paper’s ranked-label sense [2312.03367].

In candidate-set MBR, the expensive step is evaluating all pairwise utilities over the candidate set. Probabilistic MBR samples only a subset of matrix entries and reconstructs the remainder by low-rank factorization, while agreement-constrained PMBR adds an auxiliary distilled metric and an agreement term
\[
AC(\mathbf{U}, \mathbf{V}, \mathbf{U}', \mathbf{V}')
=
\sum_{i=1}^{N} \|\mathbf{u}_i - \mathbf{u}'_i\|^2
+
\sum_{j=1}^{M} \|\mathbf{v}_j - \mathbf{v}'_j\|^2
\]
to guide completion of the target score matrix [2512.01316].

Lookahead-constrained decoding uses candidates that are only locally explicit. In constrained decoding with lookahead heuristics, the decoder takes top-\(k\) next-token candidates, greedily rolls out a length-\(d\) continuation for each, scores each sequence with a reward \(\mathcal{R}\), and emits only the one-step candidate whose projected future score is highest. Constrained decoding with speculative lookaheads keeps that search pattern but drafts lookaheads with a smaller model and verifies them with the target model [2412.10418].

## 3. Constraint semantics, distortion, and guarantees

A central theoretical theme is that constraints do not merely filter outputs after the fact; they reshape the effective decoding distribution. In standard structured generation, constrained decoding at each step is the reverse-KL projection of the base distribution onto the feasible simplex
\[
q(\cdot\mid h_t)
=
\arg\min_{p\in \Delta_{A(h_t)}} \mathrm{KL}\!\left(p\,\|\,\pi_\theta(\cdot\mid h_t)\right),
\]
and the per-step distortion is
\[
\mathrm{KL}\!\left(q(\cdot\mid h_t)\,\|\,\pi_\theta(\cdot\mid h_t)\right) = \log\frac{1}{\alpha(h_t)}.
\]
Aggregated over a trajectory, this yields the cumulative “projection tax,”
\[
\sum_t \log\frac{1}{\alpha(h_t)},
\]
which biases decoding toward prefixes that are easier to keep valid, not necessarily semantically correct [2603.03305].

Draft-conditioned constrained decoding attacks exactly this mechanism. Because the valid set \(A(h_t)\) is unchanged but the context becomes \((x,y,z_{<t})\), feasible mass can increase from \(\alpha(h_t)\) to \(\tilde\alpha(\tilde h_t)\). The paper’s KL-projection view therefore interprets draft selection as choosing a context that lowers projection tax and preserves semantics under hard structural constraints. With multiple drafts, the selected candidate is
\[
k^\star \in \arg\max_k S^{(k)},
\qquad
S^{(k)}=\sum_{t=1}^{T}\log \tilde\alpha_t^{(k)},
\]
so the chosen draft is the one whose constrained realization is most “projection-friendly” [2603.03305].

Other lines of work emphasize exactness or formal safety rather than distributional distortion. Lazy-\(k\) guarantees optimality within the explored candidate ordering under the independence assumption
\[
p(\mathbf{y} \mid \mathbf{x}) = \prod_{i=1}^n p(y_i \mid \mathbf{x}),
\]
because it enumerates complete labelings in exact probability order; the paper gives space complexity \(O(kn)\) and time complexity \(O\bigl(k(\log k + n \log n)\bigr)\) [2312.03367].

Speculative decoding introduces a different constraint variable: the number of in-flight candidate tokens before verification. SpecDec++ formulates the choice of candidate length as an MDP and proves that the optimal policy takes the form of a threshold policy: speculation should stop when the probability that at least one token gets rejected exceeds a threshold. Its operational signal is
\[
1-\prod_{i=1}^k \hat{\mathbb{P}}(\text{accept}_i),
\]
which estimates the probability that at least one candidate token in the current segment will be rejected [2405.19715].

In robotics, candidate-constrained decoding is defined over action sequences rather than text strings. The ideal constrained distribution is
\[
Q_{\pi,\varphi}(a_{1:T}) =
\frac{\pi(a_{1:T}) \cdot \mathbf{1}[(x_{1:T}, a_{1:T}) \models \varphi]}
{\sum_{a_{1:T}'} \pi(a_{1:T}') \cdot \mathbf{1}[(x_{1:T}', a_{1:T}') \models \varphi]},
\]
where feasibility is determined by signal temporal logic and a dynamics model. Hard Constrained Decoding masks actions whose predicted next states violate \(\varphi\), while Robustness Constrained Decoding reweights logits using STL robustness. The hard variant comes with a theorem: if the dynamics model \(f\) is deterministic and exact, then the generated trajectory satisfies the specification [2509.01728].

Correctness-guaranteed code generation pushes constraint semantics further still. There the decoder is constrained by a context-sensitive parser whose `next_regex()` must satisfy a non-extensible property: if a string matches the regex, then no strict extension of that string may also match it. This allows each segment to be decoded under a DFA whose final states correspond to maximal semantically valid next segments. The resulting guarantees are syntactic correctness, semantic correctness with respect to typing and scoping, and—under the paper’s sLua and API restrictions—termination and runtime error freedom for completed scripts [2508.15866].

## 4. Domains and representative constraint classes

Candidate-constrained decoding spans a wide range of tasks because the candidate object can be a label sequence, a structured string, a translation hypothesis, a speculative token segment, a robot action trajectory, or a program. The constraint language changes accordingly.

| Domain | Candidates | Constraint class |
|---|---|---|
| Structured reasoning | JSON objects, symbolic expressions, FOL strings | JSON schemas, expression grammars, FOL grammars |
| Information extraction | Token labelings | BIO legality, numeric parsing, invoice arithmetic |
| Machine translation | Sampled translation set | Utility-defined candidate-set selection |
| Robotics | Action sequences | STL satisfaction or robustness |
| Code generation | Next code segments / programs | CFG, typing, scoping, API, runtime-safety restrictions |

In structured reasoning, Draft-Conditioned Constrained Decoding enforces several classes of hard constraints: JSON schemas for GSM8K and MATH500; an arithmetic expression grammar enclosed in `<<` and `>>` for GSM-Symbolic; and a first-order logic grammar with `Predicates:`, `Premises:`, and `Conclusion:` sections for FOLIO, verified by a Prover9 theorem prover. These constraints are implemented via XGrammar as finite-state or CFG-like objects exposing the valid-next-token set \(A(h_t)\) [2603.03305].

In invoice information extraction, the constraint function can combine BIO legality with domain arithmetic. The paper lists concrete invoice semantics such as
\[
\text{total\_value} = \text{subtotal\_value} + \text{tax\_value}
\]
for WildReceipt and
\[
\text{amount\_total\_gross} = \text{amount\_total\_net} + \text{amount\_total\_tax}
\]
for DocILE, alongside float-parsing constraints for numeric spans [2312.03367].

In translation, candidate-constrained decoding is utility-driven rather than grammar-driven. The candidate set consists of \(1{,}024\) sampled translations, reused as both candidates and pseudo-references, and utilities are computed by BLEURT-20. The constraint is therefore not syntactic validity, but restriction to a fixed candidate pool whose members are selected by expected utility over the score matrix [2512.01316].

In robotics, the candidates are actions in a discrete vocabulary \(\mathbb{A}\), but constraint checking is state-based. For each candidate action, the decoder predicts a next state \(\hat{x}_{t+1}^{(i)} = f(x_t,a^{(i)})\) and then either masks the action if it violates \(\varphi\), or computes robustness
\[
\rho(s_t,\mathbf{G}_{[a,b]}(\varphi)) = \min_{t' \in [t+a, t+b]} \rho(s_{t'}, \varphi)
\]
and uses it to reweight the logits. The paper’s experiments emphasize geofencing and obstacle avoidance [2509.01728].

In code generation, the constraints are simultaneously syntactic and semantic. The parser tracks modular CFG state, variable scopes, types, available fields, effect IDs, talent IDs, and API signatures. The generated language, sLua, excludes `nil`, dynamic arrays, recursive functions, and unrestricted while loops; these design choices are part of the decoding-time guarantee that completed scripts terminate and execute without runtime errors in the game engine [2508.15866].

## 5. Empirical behavior and computational trade-offs

Empirical work consistently reports that candidate-constrained decoding can improve strict validity or task performance, but usually by paying extra search, verification, or candidate-scoring cost. The shape of that trade-off depends on whether the method changes the conditional distribution, enumerates more candidates, or amortizes verification with a smaller auxiliary model.

Draft-Conditioned Constrained Decoding reports large gains in strict structured accuracy. On GSM8K with a 1B model, strict structured accuracy rises from \(15.24\%\) under standard constrained decoding to \(39.04\%\) under DCCD; with a 1.5B model it rises from \(49.36\%\) to \(73.92\%\); and with a 14B model from \(86.43\%\) to \(95.15\%\). The paper also reports that on MATH500 a \(1.5\text{B}+1.5\text{B}\) DCCD composition achieves \(12.7\) accuracy per billion parameters, versus \(3.6\) for an 8B constrained-decoding baseline [2603.03305].

Lazy-\(k\) shows the classical search-depth trade-off. On CORD, \(F_1^s\) rises from \(81.2\) for Argmax to \(92.2\) at \(k=2^6\), \(93.9\) at \(k=2^{11}\), and \(93.9\) at \(k=2^{16}\), while decoding time rises from approximately \(0\) s to \(0.002\) s, \(0.016\) s, and \(0.333\) s. The paper states that, for the same \(F_1^s\) as Lazy-ILP (\(k=1\)), Lazy-\(k\) is about \(38\times\) faster on CORD, about \(144\times\) faster on WildReceipt, and about \(182\times\) faster on DocILE [2312.03367].

Agreement-Constrained PMBR addresses a different bottleneck: pairwise candidate scoring cost. In the high-reduction setting, PMBR degrades sharply, whereas AC-PMBR “improved approximation errors of matrix completion by up to 3 times” and remained ahead on every metric. The paper gives an En→De example with D6 distilled where BLEU is approximately \(18.23\) versus approximately \(16.8\) for PMBR, chrF approximately \(45.75\) versus approximately \(44.05\), and MSE approximately \(11.16\) versus \(26.06\) [2512.01316].

For reward-guided lookahead methods, the main variable is inference speed. Constrained decoding with speculative lookaheads achieves \(2.2\times\) to \(12.15\times\) speedup over constrained decoding with lookahead heuristics without significant performance reduction. On CommonGen with OPT-13B, greedy decoding has hard constraint satisfaction \(60.80\%\), CDLH \(93.10\%\), and CDSL \(81.70\%\), while CDSL obtains a \(4.97\times\) speedup over CDLH. On harmless text generation with OPT-13B, harmlessness improves from \(80.3\%\) under greedy decoding to \(87.85\%\) under CDSL, with a \(12.15\times\) speedup over CDLH [2412.10418].

Adaptive candidate-length control in speculative decoding yields another speed-quality profile. SpecDec++ reports a \(2.04\times\) speedup on Alpaca, a \(2.26\times\) speedup on GSM8K, and a \(2.23\times\) speedup on HumanEval, corresponding to additional improvements over baseline speculative decoding of \(7.2\%\), \(9.4\%\), and \(11.1\%\), respectively [2405.19715].

Decoding-free candidate selection behaves differently because it avoids generation entirely. On massive candidate pools it can be much faster than full decoding: for Diagnoses with \(94{,}739\) candidates, decoding takes \(424.16\) s per instance, while “First” takes \(17.38\) s and “Last” \(7.33\) s; for Prescriptions with \(24{,}785\) candidates, decoding takes \(397.69\) s while “First” takes \(14.20\) s. The paper states that logits-based estimation can outperform full decoding when full decoding is weak, particularly for non-instruction-tuned models or very large candidate pools [2501.17338].

## 6. Limitations, failure modes, and directions

A recurring limitation is that candidate-constrained decoding often shifts complexity from training to inference. Draft-conditioned decoding is training-free, but depends heavily on draft quality; if drafts are wrong or misleading, conditioning may amplify incorrect reasoning, and for extremely strict schemas the draft-conditioned feasible mass may still remain low [2603.03305].

Search-based methods inherit assumptions from their search space. Lazy-\(k\) relies on the independence of token labels to justify its neighbor-ordering property, is tailored to fixed-length token classification, and maintains all visited full sequences in memory, so very long sequences or very large \(k\) can be problematic [2312.03367].

Candidate-set MBR methods inherit the weaknesses of their auxiliary structure. AC-PMBR assumes access to a good distilled version of the target metric, uses BLEURT-20 as the target metric, and is evaluated only on WMT’23 En↔De. The paper identifies dependence on distilled-metric quality and limited task and language coverage as explicit limitations [2512.01316].

Safety-guaranteed decoding in robotics is only as strong as its world model. The HCD theorem is explicitly “modulo dynamics”: guarantees hold when the dynamics model is deterministic and exact. Robustness Constrained Decoding has no absolute guarantee because unsafe actions retain nonzero probability, even though STL satisfaction rates improve substantially [2509.01728].

Correctness-guaranteed code generation achieves its strongest results by restricting the language. The guarantees rely on sLua design choices such as no `nil`, no dynamic data structures, bounded while loops, and no recursive functions. The paper also reports failure modes from distribution distortion, including non-termination via repetition under tight constraints [2508.15866].

Decoding-free candidate scoring is fast, but the estimators are deliberately crude. The paper emphasizes that current methods use only initial logits and simple aggregations such as “First,” “Last,” “Average,” and “Sum,” so they ignore token dependencies, multi-step reasoning, and context-specific token importance [2501.17338].

Speculative candidate methods require alignment between auxiliary and target models. CDSL assumes draft and target models from the same family with shared tokenizers, while SpecDec++ introduces an acceptance-prediction head and still tunes a stopping threshold \(h\) empirically because the theoretical threshold involves a hard-to-compute constant \(\Delta\) [2412.10418] [2405.19715].

*This suggests* a broad research direction: candidate-constrained decoding becomes most powerful when the candidate structure, the verifier or constraint oracle, and the scoring rule are matched to the semantics of the task. The existing literature already spans structured outputs, information extraction, translation, robotics, and code, but it does so through different notions of what a candidate is and what it means for that candidate to be feasible.

Source: https://www.emergentmind.com/topics/candidate-constrained-decoding