Misalignment in Constrained Decoding

• Misalignment in constrained decoding is a phenomenon where the intended conditional distribution differs from the actual distribution produced by practical decoders.
• It encompasses issues like support mismatch, distributional bias, and constrainer mismatches that lead to deviations between modeled and valid outputs.
• Mitigation strategies include improved proposal designs, probabilistic lookahead, and backtracking techniques to preserve semantic intent and structural reliability.

Misalignment in constrained decoding denotes a family of discrepancies between the distribution one intends to sample and the distribution a decoding procedure actually induces once formal constraints are imposed. In the narrow autoregressive sense, the target is often the LLM conditioned on a constraint, $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}$$, whereas practical decoders frequently implement only token-local masking and renormalization, yielding a different distribution with altered support or altered relative probabilities among valid outputs (Dang et al., 1 Jun 2026). In a broader systems sense, misalignment also includes mismatches among the constrainer, the model, and the target specification language; architectural mismatches between a decoder and the model’s generation process; and search procedures whose local scoring rules do not recover the desired conditional distribution even when the underlying model is Bayes-optimal (Biagiola et al., 19 Jun 2026).

1. Formal definitions and scope

A canonical formalization starts from a base LLM

$$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$

together with a constraint $C$ defining a valid set $\mathcal{X}_n$. The target constrained distribution is then

$$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$

Locally constrained decoding instead uses a proposal

$$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$

where $\mathcal{X}_{t,*}$ contains prefixes extendable to some valid continuation at some length, not necessarily to the target length $n$. This already separates the desired conditional distribution from the actual decoding distribution (Dang et al., 1 Jun 2026).

A second formalization treats constrained decoding as a three-way relation among the target language $L_T$, the model language $L_M$, and the constrained language $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$0. In that setting, a constrainer is sound if $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$1, complete if $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$2, unsound if $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$3, and incomplete if $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$4. One proposed alignment score is the three-way Jaccard similarity

$$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$5

This formulation makes explicit that misalignment can arise even when every emitted sequence is constraint-compliant, because the constrainer may still reject target-valid programs or admit target-invalid ones (Biagiola et al., 19 Jun 2026).

Misalignment form	Characteristic mechanism	Representative settings
Support mismatch	Invalid full outputs or dead-end intermediate states remain reachable	LCD, diffusion CFG decoding
Distributional bias	Relative probabilities among valid outputs differ from the true conditional	LCD, projection-tax analyses
Constrainer mismatch	$$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$6 is incomplete or unsound relative to $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$7	TypeScript, TOML, code generation
Search mismatch	Marginal-based search does not recover the desired joint objective	Generative retrieval, beam search

This suggests that “constrained decoding” is not a single failure mode but a layered problem: support, distribution, formal specification, and search may each be aligned or misaligned independently.

2. Local masking, probability distortion, and intent drift

The standard local-masking failure is distributional. A simple example uses a uniform binary LLM over length-3 strings with the constraint “exactly one 1.” The true constrained distribution is uniform over $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$8, $$p_{\mathrm{lm}}(x_{1:n})=\prod_{t=1}^n p_{\mathrm{lm}}(x_t\mid x_{<t}),$$9, and $C$0, each with probability $C$1. Under locally constrained decoding, however, the model assigns $C$2 and $C$3. The paper’s explanation is that local renormalization lacks probabilistic lookahead: it knows only whether a prefix can be extended, not how much valid probability mass lies behind each prefix. The exact next-token conditional should depend on $C$4, whereas LCD replaces that quantity with a binary indicator (Dang et al., 1 Jun 2026).

A closely related account appears in the KL-projection view of constrained decoding. If $C$5 is the valid-next-token set at history $C$6, constrained decoding uses

$C$7

This is the reverse-KL projection of the model distribution onto the constrained simplex, and the per-step distortion is

$C$8

The sequence-level distortion is the expected cumulative projection tax

$C$9

When feasible mass is small at many steps, locally valid but semantically poor trajectories can be strongly upweighted relative to the model’s unconstrained preferences (Reddy et al., 8 Feb 2026).

Recent work on code completion states this point more directly as distortion of output intent. The ideal constrained distribution is $\mathcal{X}_n$0, so for valid complete sequences $\mathcal{X}_n$1,

$\mathcal{X}_n$2

Standard constrained decoding violates this fairness property because the sequence ratio acquires path-dependent normalization factors. The paper’s PyTorch example shows the mechanism: after torch., the model strongly prefers an invalid removed API matrix_rank; greedy constrained decoding commits to the locally valid prefix matrix, later blocks rank, and then forces continuation to matrix_power, even though linalg.matrix_rank is the higher-probability valid completion (Li et al., 20 Oct 2025).

Hard syntax masking can nevertheless be useful when the main problem is protocol conformance rather than semantic choice. Finite-state masking for tool use can guarantee that a tool call refers to an existent tool name and passes type-conforming arguments, but the same work explicitly notes that this does not ensure “semantic accuracy and contextual relevance of tool calls” (Zhang et al., 2023). A common misconception follows: syntactic validity is neither sufficient for conditional-distribution alignment nor sufficient for semantic correctness.

3. Structural reliability across decoding paradigms

In diffusion LLMs, misalignment is often architectural rather than merely probabilistic. The critical notion is reliability: every intermediate output must remain extendable to at least one valid sentence in the target language. For autoregressive models, grammar filters operate on complete prefixes, so prefix validity and generation order coincide. For diffusion models, however, intermediate states are partial strings with holes, and autoregressive grammar filters do not apply directly. Existing diffusion-specific CFG methods can therefore accept tokens that appear locally plausible under a coarse over-approximation but actually lead to dead-end states with no valid completion (Zhang et al., 31 Jan 2026).

The proposed remedy in that setting is a “lookahead-then-verify” scheme. Whenever a diffusion model proposes a token at some masked position, the decoder samples concrete fillings of the remaining masked positions from the model’s current per-position distributions, obtains complete candidate prefixes, and calls a parser primitive $\mathcal{X}_n$3. A token is accepted only if at least one sampled completion yields an extendable complete prefix. This enforces the paper’s reliability criterion and empirically raises syntactic@1 substantially; for example, on CPP-Bench with Dream-7B, syntactic@1 moves from 76.2% for unconstrained decoding to 87.6% for IG-CD and 96.0% for LAVE (Zhang et al., 31 Jan 2026).

A related line reformulates constrained decoding for diffusion models as additive infilling. A partial output $\mathcal{X}_n$4 is extendable iff the intersection $\mathcal{X}_n$5 is non-empty, where $\mathcal{X}_n$6 is the CFG language and $\mathcal{X}_n$7 is the regular language of all completions of the partial output. The constrained infilling problem is therefore reduced to CFG–regular-language intersection emptiness, handled through an efficient construction and emptiness check for the intersection grammar (Mündler et al., 13 Aug 2025). Empirically, this yields near-perfect syntactic correctness for JSON, C++, and SMILES, while consistently preserving or improving functional correctness.

These diffusion results clarify a broader point. Support alignment cannot be defined only at the end of generation; for non-autoregressive models it must often be maintained as an invariant over partially filled states. This suggests that “valid final output” and “aligned decoding trajectory” are distinct requirements.

4. Proposal design, probabilistic lookahead, and projection-aware mitigation

For autoregressive decoding, one approach is to keep exact conditional sampling as the target and improve the proposal. Sequential Monte Carlo maintains particles and weights to approximate $\mathcal{X}_n$8, but the quality of the proposal $\mathcal{X}_n$9 and potential $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$0 dominates efficiency. Replacing LCD with Globally Constrained Decoding changes the mask from $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$1 to $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$2, so only prefixes that admit completion within the remaining token budget are allowed: $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$3 This fixes support misalignment. Probabilistic GCD then adds probabilistic lookahead via an HMM distilled from the LLM and multiplied with the finite automaton of the constraint, yielding a proposal

$$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$4

Under the same SMC setup, GCD and P-GCD converge faster than LCD with significantly fewer particles; for function calling and JSON-like schemas, GCD and P-GCD also ensure 100% constraint satisfaction within the length budget, whereas LCD remains below 100% (Dang et al., 1 Jun 2026).

A separate mitigation strategy is to decouple semantic planning from structural enforcement. Draft-Conditioned Constrained Decoding first generates an unconstrained draft $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$5, then performs constrained decoding conditioned on that draft: $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$6 The analysis again uses feasible mass and KL projection, but now the draft raises feasible mass and reduces cumulative projection tax. Across GSM8K, MATH500, GSM-Symbolic, and FOLIO, this improves strict structured accuracy by up to +24 percentage points over standard constrained decoding; one reported example is GSM8K with a 1B model, where strict structured accuracy rises from 15.2% to 39.0% (Reddy et al., 8 Feb 2026).

These results indicate that proposal design is not a peripheral engineering choice. It determines whether the constrainer acts like a faithful posterior filter or like a semantic perturbation that reorients generation toward whatever remains locally feasible.

5. Backtracking, approximate posteriors, and ensemble correction

Another family of methods targets the conditional distribution more directly by treating constraint handling as a posterior-sampling problem. AdapTrack formalizes the desired outcome as preserving relative LM probabilities among valid complete sequences. Its mechanism maintains estimates $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$7 over prefixes and sampled next-token choices $$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$8, then uses rejection sampling over prefixes to backtrack when updated validity estimates sharply reduce the viability of the current branch. The resulting decoding distribution is proved to align with the model’s distribution given the generated tokens, thereby avoiding output-intent distortion on the explored sequences. Empirically, the method yields up to 360.87% improvement over constrained decoding on a synthetic API completion dataset, up to 38.93% on a real-world API completion dataset, up to 7.84% on HumanEval, and up to 6.42% on MBPP (Li et al., 20 Oct 2025).

Approximately Aligned Decoding addresses the same problem from the opposite direction: exact posterior conditioning is available through rejection sampling or ASAp, but becomes prohibitively expensive when the error set is dense. The target is the error-free posterior

$$p_{\mathrm{lm}}(x_{1:n}\mid C)\propto p_{\mathrm{lm}}(x_{1:n})\cdot 1\{x_{1:n}\in\mathcal{X}_n\}.$$9

and the main empirical question is how closely a practical decoder approximates it while keeping generation ratio manageable. On toy simulations, AprAD consistently lands between exact ASAp and hard local masking in KL divergence to the ideal conditional. On lipograms, it combines human quality 4.52 and constraint-intent 2.84 with generation ratio 4.2, versus quality 3.56 and ratio 1.0 for local constrained generation and quality 1.72 with ratio 321 for ASAp. On BigCodeBench with StarCoder2-15B, AprAD reaches pass@1 0.316 and !NameErr@1 0.980 at ratio 1.016, closely matching ASAp’s task performance while staying much cheaper (Melcer et al., 2024).

A third mitigation strategy does not attempt to approximate the posterior in one decoder at all. BoostCD first runs a base model twice, once unconstrained and once constrained, obtaining two weak predictions $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$0 and $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$1. A second autoregressive model is then trained on $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$2. In closed information extraction, this exploits complementary errors: unconstrained decoding may produce semantically correct but KB-invalid entities, while constrained decoding may snap to similarly named KB entities or unrelated triples. The boosted model learns that disagreement between $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$3 and $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$4 often signals such distortions, and it largely mitigates the resulting misclassified entities and unrelated triplets (Šakota et al., 17 Jun 2025).

Taken together, these methods imply that preserving alignment frequently requires either posterior-style global correction, explicit backtracking, or a second-stage model that reasons about how constraints have deformed the base model’s outputs.

6. Incomplete constrainers, alignment tax, and unresolved tensions

The strongest recent warning is that constrained decoding can remain harmful even when it enforces exactly the property it was designed to enforce. In constrained code generation, incompleteness of the constrainer creates an incompleteness bias: valid, high-probability programs in $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$5 are excluded, so decoding is pushed into low-probability regions of program space. Across seven LLMs, two target languages, and multiple constrainers, the paper reports that when the constrainer is incomplete, unconstrained decoding significantly outperforms constrained decoding in functional correctness, and incompleteness can reduce functional correctness by up to 97% (Biagiola et al., 19 Jun 2026). This directly contradicts the intuition that “more validity constraints” monotonically improve downstream correctness.

A complementary failure appears in constrained reflection for agentic reasoning. Outlines-based schema constraints produce near-perfect superficial syntactic alignment, but on HotpotQA they reduce accuracy from 0.50 to 0.38 and yield 96% first reflections labeled as FORMATTING_MISMATCH. The paper names the resulting loop structure snowballing: reflection becomes trapped in formatting diagnoses and correction rules, while deeper retrieval or inference errors remain unaddressed. The same study describes this as an alignment tax, namely the performance and cognitive cost of enforcing strict structure at decode time without external critics or additional training (Zhou, 7 Apr 2026).

Misalignment also persists at the search level even under idealized assumptions. In generative retrieval, a Bayes-optimal autoregressive model over the complete document universe still incurs a non-zero KL gap once corpus-specific constraints are applied only during decoding, because the decoder uses $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$6 rather than the true downstream conditional $$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$7. The paper derives lower bounds on that KL error and separately shows that beam search based on marginal branch probabilities can achieve perfect top-1 precision yet poor top-$$q_{\mathrm{LCD}}(x_t\mid x_{<t})\propto p_{\mathrm{lm}}(x_t\mid x_{<t})\cdot 1\{x_{1:t}\in\mathcal{X}_{t,*}\},$$8 recall in sparse, heavy-tailed relevance regimes (Wu et al., 14 Apr 2025). This makes the limitation intrinsic to the decoding procedure rather than to model misspecification.

Three misconceptions therefore require rejection. First, formal validity is not equivalent to sampling from the intended constrained distribution. Second, exact posterior alignment can be computationally impractical in dense-constraint regimes, so approximation quality matters. Third, stronger structure does not automatically produce better semantics; under limited model capacity or incomplete formalizations, stronger structure can exacerbate semantic error. A plausible implication is that future constrained decoding systems will need to be evaluated simultaneously along four axes: support correctness, distributional fidelity, search faithfulness, and constrainer completeness.