Decoding-Constrained Beam Search

• Decoding-constrained beam search is a family of inference methods that modify standard beam search to enforce explicit lexical and structural constraints.
• It integrates various constraint families—lexical, length, surprisal regularization, and more—to mitigate issues like beam search curse and off-target generations in applications such as NMT and ASR.
• Innovative algorithmic patterns including grid-based, banked, and priority-queue methods optimize search-state management, balancing computational cost with constraint satisfaction.

Decoding-constrained beam search denotes a family of inference procedures for autoregressive sequence models in which beam expansion, pruning, stopping, or scoring is modified so that generated sequences satisfy explicit constraints or optimize augmented decoding objectives. In the left-to-right setting, a model factorizes as $$p_\theta(y\mid x)=\prod_t p_\theta(y_t\mid y_{<t},x)$$, and standard beam search approximately maximizes cumulative log-probability. The constrained variants surveyed in the literature extend this setup with lexical inclusion requirements, length and progress controls, surprisal-regularized objectives, target-language constraints, conformal coverage guarantees, near-verbatim distance bounds, or bounded-emission constraints for transducer lattices (Hokamp et al., 2017, Post et al., 2018, Meister et al., 2020, Lu et al., 2021, Deutschmann et al., 2023, Yang et al., 2024, Cooper et al., 26 Mar 2026, Kim et al., 2020, Kang et al., 2022).

1. Formal setting and decoding objectives

For sequence generation, the unconstrained decoding target is commonly written as

$$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$

This is the baseline used in NMT, controllable generation, and many transducer decoders (Meister et al., 2020, Hokamp et al., 2017).

Decoding-constrained variants modify either the feasible set or the score. Hard-constrained formulations retain the model score but require the output to satisfy structural properties. Examples include lexical constraints that must appear contiguously in the output, EOS gating until all constraints are satisfied, or the target-language requirement $L(y)=T$ in multilingual translation (Hokamp et al., 2017, Post et al., 2018, Yang et al., 2024). Soft-constrained formulations instead add rewards or penalties to the score. Representative forms include UID regularization,

$J(y)=\log p_\theta(y\mid x)-\lambda R(y),$

single-queue decoding with length normalization, progress penalty, and learned length matching,

$$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$

and language-informed reranking,

$$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$

all of which preserve beam-style search but alter the ranking criterion (Meister et al., 2020, Shu et al., 2017, Yang et al., 2024).

A further generalization changes the output object from a single sequence to a set of sequences. In conformal autoregressive generation, beam search is filtered or constructed so that the final set $C(x)$ satisfies marginal coverage bounds under exchangeability; in probabilistic extraction auditing, beam search is used to lower-bound the total probability mass of an $\epsilon$-ball of near-verbatim continuations rather than to select a single MAP sequence (Deutschmann et al., 2023, Cooper et al., 26 Mar 2026).

2. Constraint families

The literature spans several distinct but overlapping constraint families.

Constraint family	Core mechanism	Representative methods
Lexical and phrase inclusion	Coverage tracking, start/continue transitions, EOS only after satisfaction	GBS (Hokamp et al., 2017), DBA (Post et al., 2018), NeuroLogic A*esque (Lu et al., 2021)
Length and progress control	Universal score with length normalization, progress penalty, learned length matching	SQD (Shu et al., 2017)
Surprisal regularization	Penalize non-uniform local surprisal via $R(y)$	UID-aware decoding (Meister et al., 2020)
Auxiliary prefix scoring	Add prefix-level classifier or attention reward to beam ranking	LiBS (Yang et al., 2024), BSDAR (Ni'mah et al., 2019)
Coverage-guaranteed set prediction	Conformal thresholds and variable beam width	Conformal autoregressive generation (Deutschmann et al., 2023)
Distance or monotonicity constraints	Edit-distance viability pruning or bounded emissions per frame	DCBS (Cooper et al., 26 Mar 2026), OSC and constrained transducer decoding (Kim et al., 2020, Kang et al., 2022)

Lexical constraints are the historically canonical case. Grid Beam Search, introduced by Hokamp and Liu, treats decoding as search over a 2D grid indexed by time and constraint-token coverage, with hypotheses marked as open or closed depending on whether they are currently emitting a multi-token constraint; finished hypotheses must end with EOS after all constraint tokens have been covered (Hokamp et al., 2017). Dynamic Beam Allocation preserves a fixed beam size while partitioning candidates into banks indexed by the number of satisfied constraints, explicitly considering global top-$k$, per-row best, and start/continue tokens for unmet constraints (Post et al., 2018). NeuroLogic extends lexical constraints to CNF formulas with positive and negative literals, plus prefix tracking for multi-token constraints (Lu et al., 2021).

Several methods constrain beam search through score shaping rather than feasibility. UID-aware decoding penalizes non-uniform surprisal profiles, thereby converting beam search into the limiting solution of a regularized objective (Meister et al., 2020). SQD uses a universal score over mixed-length hypotheses together with a learned length matching penalty, which biases search toward plausible final lengths without hard exclusion (Shu et al., 2017). LiBS adds a target-language log-probability from an external LiD model at every step, while BSDAR adds word-level and n-gram-level attention rewards for keyphrase generation (Yang et al., 2024, Ni'mah et al., 2019).

A separate line uses constraints to obtain uncertainty or risk guarantees. Conformal beam search produces dynamically sized sets with marginal coverage guarantees, and DCBS for memorization auditing constrains beam search to a decoding scheme and a distance predicate so that the explored sequences yield a deterministic lower bound on near-verbatim extraction risk (Deutschmann et al., 2023, Cooper et al., 26 Mar 2026).

3. Search-state organization and algorithmic patterns

The principal algorithmic distinction is how constraint state is represented during beam maintenance.

Grid-based methods make constraint progress explicit in the beam topology. GBS allocates a beam to each pair $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$0, where $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$1 is decoding step and $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$2 is the number of covered constraint tokens. From an open state, a hypothesis may either generate arbitrarily, start an unmet constraint, or—if closed—continue the active constraint. Phrase contiguity is enforced because a closed hypothesis can emit only the next token of the active constraint; EOS is accepted only at coverage level $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$3 (Hokamp et al., 2017).

Banked methods preserve the standard fixed-width beam but alter candidate selection. DBA groups candidates by the number of satisfied constraints and dynamically redistributes beam slots across banks. Its candidate set is the union of the global top-$$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$4 items, the best local continuation from each row, and all constraint-directed start or continue actions. The base allocation is $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$5 per bank with the remainder initially assigned to the most constrained banks, followed by reallocation from under-populated to over-populated banks (Post et al., 2018).

Priority-queue methods abandon the fixed-length frontier. SQD stores unfinished and finished hypotheses of different lengths in a single queue $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$6, pops the best unfinished hypotheses under a universal score, expands them, merges the best scored successors back into $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$7, and stops once $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$8 contains at least $$\hat y=\arg\max_y \log p_\theta(y\mid x) =\arg\max_y \sum_{t=1}^{T}\log p_\theta(y_t\mid y_{<t},x).$$9 finished hypotheses. This mechanism permits revisiting previously discarded short hypotheses when longer continuations become unattractive under the combined score (Shu et al., 2017).

Lookahead methods retain ordinary beam mechanics but augment the score with future-cost surrogates. NeuroLogic A*esque scores a prefix by its LM likelihood, a partial-match reward for unsatisfied multi-token constraints, and a lookahead estimate of future constraint satisfaction computed from greedy, soft, beam, or sampling continuations of horizon $L(y)=T$0. It also prunes candidates that violate negative clauses or make a clause impossible to satisfy, and groups beams to promote diversity (Lu et al., 2021).

Auxiliary-scorer methods interleave beam search with an external or internal judge. LiBS pre-selects the top-$L(y)=T$1 next-token expansions from each live beam item, detokenizes the resulting $L(y)=T$2 prefixes, evaluates a FastText LiD model on each, and reranks them with $L(y)=T$3. Only the NMT score $L(y)=T$4 is stored in the beam state, specifically to avoid double-counting LiD evidence across steps (Yang et al., 2024). BSDAR similarly perturbs the local score using attention-derived word and n-gram rewards, plus sibling reranking heuristics (Ni'mah et al., 2019).

Specialized variants arise in other domains. DCBS for extraction-risk estimation performs beam search under the renormalized distribution induced by the chosen decoding scheme, such as top-$L(y)=T$5, tracks Hamming or Levenshtein viability online, removes EOS children before across-beam pruning for $L(y)=T$6, and on the final step returns all $L(y)=T$7 children with exact top-$L(y)=T$8 probabilities (Cooper et al., 26 Mar 2026). In RNN-T, OSC limits each hypothesis to at most one non-blank emission per encoder frame, eliminating the usual while-loop and enabling vectorized batched scoring; the later constrained-transducer work similarly imposes a per-frame emission cap $L(y)=T$9, with $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$0 as the recommended fully parallel regime (Kim et al., 2020, Kang et al., 2022).

4. Objective reinterpretation, guarantees, and complexity

A central theoretical result is that beam search can be recast as exact optimization of a different objective. The UID analysis defines surprisal $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$1 and shows that greedy search is the $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$2 limit of

$J(y)=\log p_\theta(y\mid x)-\lambda R(y),$3

while standard beam search with width $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$4 is the $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$5 optimizer of a set-valued beam regularizer $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$6. In this formulation, beam pruning enforces a UID-like preference for locally low-surprisal continuations rather than merely approximating unconstrained MAP decoding (Meister et al., 2020).

Conformal methods provide explicit coverage statements. Post-hoc conformal filtering over a fixed beam yields

$J(y)=\log p_\theta(y\mid x)-\lambda R(y),$7

with probability at least $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$8 over the calibration sample, where $J(y)=\log p_\theta(y\mid x)-\lambda R(y),$9 is a Clopper–Pearson lower bound on beam inclusion probability. The in-decoding dynamic conformal beam uses calibrated prefix thresholds and guarantees

$$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$0

The first guarantee is capped by beam recall; the second is selected a priori by the per-step miscoverage but may require large beam widths on long sequences (Deutschmann et al., 2023).

DCBS supplies a different kind of guarantee: a deterministic lower bound on near-verbatim extraction risk. For the returned final set $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$1, the lower bound is

$$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$2

and $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$3 because the sum ranges over a subset of the admissible $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$4-ball. The bound is non-decreasing in beam width, and early stopping is possible if the maximal beam probability at step $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$5 falls below $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$6 (Cooper et al., 26 Mar 2026).

Complexity varies sharply across formulations. GBS multiplies runtime and memory by approximately the total number of constraint tokens $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$7, yielding a naive upper bound $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$8, whereas DBA keeps the dominant complexity at $$S(y)=|y|^{-\lambda}\log p(y\mid X)+\mathrm{PG}(y)+\mathrm{LMP}(y),$$9 and is reported as $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$0 in the number of constraints because it reuses the standard $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$1 model computation (Hokamp et al., 2017, Post et al., 2018). NeuroLogic A*esque adds lookahead cost: greedy and soft lookahead are $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$2, sampling lookahead is $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$3, and beam lookahead can reach $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$4 in the worst case (Lu et al., 2021). Exact UID search over prefix trees uses Dijkstra’s algorithm for monotone regularizers and admissible upper bounds for non-monotone ones, but still explores an exponentially large space in the worst case; the beam version remains $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$5 with negligible overhead for simple penalties (Meister et al., 2020). In RNN-T with per-frame emission cap $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$6, the decoding cost becomes $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$7, and for $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$8 all successors land at the next frame, which is what enables batched GPU and FSA implementations (Kang et al., 2022).

5. Empirical behavior across application domains

The empirical literature shows that constraints are often introduced not merely to enforce side conditions, but to repair systematic pathologies of unconstrained beam search.

Domain	Method	Reported outcome
NMT beam curse	UID-aware decoding (Meister et al., 2020)	On WMT’14 Newstest’14 first 1000 samples, BLEU dropped 36.42 $$S(y)=\log p_\theta(y\mid x)+\alpha\log p_\gamma(T\mid y),$$9 14.66 without regularization when $C(x)$0 increased 5 $C(x)$1 500, but stayed 36.92 $C(x)$2 35.96 with the squared regularizer
Interactive and terminology-constrained MT	GBS (Hokamp et al., 2017)	Autodesk EN-PT BLEU 15.41 baseline, 29.15 with GBS
Mixed-length NMT decoding	SQD (Shu et al., 2017)	+1.14 BLEU over beam search with length normalization on ASPEC En–Ja at beam size 5
Constrained text generation and MT	NeuroLogic A*esque (Lu et al., 2021)	CommonGen zero-shot ROUGE-L 44.3, BLEU-4 28.6, Coverage 97.1; constrained MT up to BLEU 26.7 and Term% 95.8
Zero-shot multilingual NMT	LiBS (Yang et al., 2024)	Off-target 22.9% $C(x)$3 7.7% on WMT and 65.8% $C(x)$4 25.3% on OPUS; average BLEU gains +1.1 and +0.9
Memorization-risk estimation	DCBS (Cooper et al., 26 Mar 2026)	Cost comparable to roughly 20 MC samples; OLMo 2 32B near-verbatim rate 2.57% vs verbatim 1.42% and greedy verbatim 0.61%
RNN-T ASR	OSC and constrained transducer decoding (Kim et al., 2020, Kang et al., 2022)	Up to about 20× decoding speedup with neutral or improved PER/WER in reported settings

In NMT, the UID paper directly addresses the “beam search blessing.” Exact MAP search can be degenerate in poorly calibrated systems, and Stahlberg and Byrne are cited there as having observed exact search returning the empty string in more than 50% of cases for some models. On IWSLT’14 De→En and WMT’14 En→Fr, the average per-sentence standard deviation of surprisals showed a strong inverse relationship with BLEU; the reported sub-graph gives $C(x)$5 between surprisal standard deviation and BLEU (Meister et al., 2020).

Lexically constrained MT exhibits both substantial gains and characteristic trade-offs. GBS produced large BLEU improvements in simulated interactive translation and strong domain-adaptation gains without retraining, especially in EN-PT on the Autodesk domain (Hokamp et al., 2017). DBA kept runtime roughly flat in the number of constraints, about $C(x)$6 sec/sentence on a Tesla V100 for $C(x)$7, roughly three times slower than unconstrained decoding, and was reported to meet all constraints reliably even when the number of constraints exceeded the beam size, although BLEU degraded when the beam was too small relative to many independent constraints (Post et al., 2018).

In controllable generation, NeuroLogic A*esque consistently outperformed GBS, CBS, DBA, and NeuroLogic without lookahead on CommonGen, constrained MT, few-shot E2ENLG, and zero-shot interrogative question generation. The strongest improvements were reported on tasks requiring complex constraint satisfaction or operating in few-shot and zero-shot regimes (Lu et al., 2021). Conformal beam search addressed a different target—coverage of the correct sequence rather than lexical satisfaction—and empirically met or exceeded its guarantees on both integer addition and chemical reaction prediction, with dynamic beam sizes becoming large on long sequences and small per-step miscoverage (Deutschmann et al., 2023).

In multilingual NMT, LiBS diagnosed off-target branches as early-emerging prefixes that initially score poorly but later accumulate strong local probabilities. By applying the LiD penalty from the first step to pre-selected candidates, LiBS suppressed those branches before they dominated the beam. The paper reports that this “breaks the beam search curse” for De→Fr and Cs→De, where baseline off-target rates rose sharply with beam size while LiBS held them near 1–2% (Yang et al., 2024).

The extraction-risk application uses constraints for auditability rather than generation quality. DCBS surfaces many “unlocked” sequences whose exact target suffix has zero or sub-threshold verbatim mass but whose near-verbatim mass exceeds threshold $C(x)$8. On Llama 2 70B for The Great Gatsby, 1,606 sequences were reported as unlocked, of which 1,289, or 80.3%, had zero verbatim mass (Cooper et al., 26 Mar 2026).

In speech recognition, decoding constraints are primarily computational. OSC found that the number of non-blank expansions applied to a hypothesis at a given encoder time was one in 97.73% of cases, which motivated the one-step constraint; the later constrained-transducer work aligned training and decoding under a forced-blank recursion and reported batched greedy and beam-search speedups of approximately 8.7× and 18.4× respectively for $C(x)$9, with an additional FSA-based batched beam-search speedup to an RTF of about $\epsilon$0 on a V100 (Kim et al., 2020, Kang et al., 2022).

6. Limitations, misconceptions, and open directions

A common misconception is that beam search is merely an approximation to MAP decoding and that larger beams should therefore improve output quality. The UID analysis disputes this directly: standard beam search has high search error yet often outperforms exact MAP because the MAP objective alone does not encode desirable textual properties such as uniform information density (Meister et al., 2020). Parallel observations appear in multilingual NMT, where larger beams can worsen off-target drift, and in constrained MT, where increasing model score can correlate negatively with BLEU. DBA explicitly describes this as a “shaky relationship” between model and BLEU scores, and reports that forcing reference-derived constraints can decrease model score while increasing BLEU (Post et al., 2018, Yang et al., 2024).

Constraint satisfaction does not remove beam-search approximation error. GBS remains heuristic because the globally best constrained sequence can still be pruned at small $\epsilon$1, and infeasible or conflicting constraint sets can yield no finished hypotheses (Hokamp et al., 2017). DBA alleviates the linear-in-constraints cost of GBS, but too-small beams still degrade quality when many independent constraints must be permuted into fluent output (Post et al., 2018). NeuroLogic A*esque is heuristic rather than admissible A*, and its performance degrades when lookahead horizon $\epsilon$2 becomes too large or heuristic weights are too aggressive (Lu et al., 2021).

Soft-constraint approaches introduce their own sensitivities. SQD adds memory overhead on the order of $\epsilon$3 for retained decoder states and is vulnerable to mis-specified progress or length-matching penalties (Shu et al., 2017). LiBS incurs substantial auxiliary-scoring overhead: 7.5× slower than baseline beam search on a single CPU, reduced to 3.5× with parallel LiD scoring across 20 CPUs, and short-prefix LiD can be noisy for related languages or shared scripts (Yang et al., 2024). BSDAR depends on attention quality and on the utility of n-grams extracted from the initial attention map (Ni'mah et al., 2019).

Guarantee-oriented methods have structurally different limitations. Post-hoc conformal filtering cannot achieve coverage higher than beam inclusion probability, while dynamic conformal beams may need to grow very large for long sequences because the sequence-level guarantee scales as $\epsilon$4 (Deutschmann et al., 2023). DCBS is only a lower bound, can undercount viable mass pruned earlier, and is sensitive to tokenization because small character-level changes may expand into many token edits; its own upper bound from pruned-ancestor mass is reported as often too loose to be informative (Cooper et al., 26 Mar 2026). In RNN-T, hard caps such as $\epsilon$5 or one-step expansion can miss cases where multiple labels should be emitted at a single frame, even if such cases are empirically rare (Kim et al., 2020, Kang et al., 2022).

The literature collectively suggests that decoding-constrained beam search has become a general design pattern rather than a single algorithmic template. It serves control, calibration, coverage, robustness, efficiency, and auditing objectives, and its future extensions are already visible within the cited work: additional soft penalties in single-queue decoding, intersection of conformal and structural constraints, hybrid hard-and-soft lexical control, A*-style lookahead for richer constraint languages, and direct integration of decoding-aware constraints with training objectives (Shu et al., 2017, Deutschmann et al., 2023, Lu et al., 2021, Kang et al., 2022).