Future-Validity Function in Speculative Decoding

• Future-Validity Function is a prefix-dependent statistic that estimates the probability a token sequence can be completed to form a fully grammatical sentence.
• It reweights locally valid tokens using a Doob transform, thereby correcting standard masking pipelines to match the intended grammar-conditional law.
• Approximate methods such as OneStep and MC rollout mitigate the #P-hardness of exact computation, ensuring practical decoding with explicit total-variation bounds.

The future-validity function is the prefix-dependent statistic that corrects grammar-constrained generation from “currently valid” tokens to the intended distribution conditioned on eventual grammatical completion. In the formulation of grammar-faithful speculative decoding, it is defined for a base autoregressive LLM $p$, a prefix-checkable grammar $\mathcal C$, and a candidate next token $y$ after prefix $x_{<t}$ as

$$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$

The central claim is that standard serving pipelines combining local vocabulary masking with speculative decoding omit exactly this statistic, and therefore sample from the locally projected law $\mu^{\mathrm{proj}}$ rather than the intended grammar-conditional law $\mu^\star$. With exact $\Phi$, the target distribution becomes a Doob transform of the base model; with approximate $\Phi$, one obtains explicit total-variation guarantees; and for general context-free grammars, exact computation is $\#P$-hard, so practical use depends on estimator hierarchies and tractable grammar families (Nie et al., 8 May 2026).

1. Formal definition and probabilistic setting

The setup assumes an autoregressive base LLM $\mathcal C$0 over a finite vocabulary $\mathcal C$1, with per-step conditional

$\mathcal C$2

together with a prefix-checkable constraint or grammar $\mathcal C$3, and valid-next-token sets

$\mathcal C$4

Within this setup, the future-validity function is

$\mathcal C$5

The shorthand $\mathcal C$6 suppresses the conditioning on the committed prefix, but the paper emphasizes that the quantity is really prefix-dependent. A “valid completion” means completion into a full terminal string in $\mathcal C$7, including EOS as needed (Nie et al., 8 May 2026).

This definition makes $\mathcal C$8 a survival probability under the base model $\mathcal C$9: after appending $y$0, one samples the suffix from $y$1, and $y$2 is the probability that the resulting full sequence remains in the grammar language. In principle this depends on the full committed history through $y$3; if the language-model state is not Markov-compressible into grammar state, then grammar state alone is not sufficient. Operationally, for finite languages and regularized settings, the paper computes the grammar side from token tries or automaton states, while the model side still depends on prefix-conditioned probabilities (Nie et al., 8 May 2026).

2. From local masking to the grammar-conditional law

The paper distinguishes sharply between the deployed one-step masking law and the intended full-sequence conditional law. Local masking retains only $y$4 and renormalizes: $y$5 The intended target, however, is the base-model sequence law restricted to the language and renormalized: $y$6 These are different objects: $y$7 is the product of masked-and-renormalized one-step conditionals, whereas $y$8 is the original sequence distribution conditioned on eventual grammaticality (Nie et al., 8 May 2026).

The future-validity function is the missing correction statistic because the exact per-step grammar-conditional kernel is

$y$9

Relative to local masking, each locally valid token is reweighted by its probability of surviving to a full valid completion. Local masking is recovered by the degenerate approximation $x_{<t}$0, i.e. by pretending that every locally valid token has identical future validity. The paper gives the exact condition under which this omission is harmless: $x_{<t}$1 This identifies non-uniformity of $x_{<t}$2 over $x_{<t}$3 as the sole source of distortion (Nie et al., 8 May 2026).

The same point appears in the KL identity. If

$x_{<t}$4

then

$x_{<t}$5

Thus the divergence caused by ignoring future validity is controlled exactly by how non-uniform $x_{<t}$6 is across the locally valid token set (Nie et al., 8 May 2026).

3. Doob transform, Bellman recursion, and survival semantics

The paper characterizes $x_{<t}$7 as a Doob $x_{<t}$8-transform of the base model with $x_{<t}$9. This gives a precise probabilistic interpretation: future validity is not an auxiliary heuristic, but the harmonic function that converts the unconstrained next-token kernel into the sequence law conditioned on eventual membership in $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$0 (Nie et al., 8 May 2026).

The appendix gives the recursive identity

$$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$1

which is a Bellman or backward recursion. Future validity at the current decision equals the model-weighted sum of future validity at the next decision over valid next tokens. This makes the quantity directly analogous to inside or backward completion probabilities in constrained language modeling and parsing, a connection stated explicitly in the paper (Nie et al., 8 May 2026).

The same appendix shows why shallow lookahead is biased. For the “true one-step estimator,”

$$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$2

and therefore

$$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$3

One-step lookahead therefore overestimates true future validity by the expected future invalidity after the next step. This clarifies why local admissibility is insufficient: a token can be locally legal yet lead almost surely to future dead ends (Nie et al., 8 May 2026).

4. Speculative decoding, LMS impossibility, and oracle correction

A central negative result is that speculative decoding does not repair the mismatch between $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$4 and $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$5. The paper defines the LMS class by three axioms: B1 Local Mask, B2 Leviathan Rejection, and B3 Rollback Soundness. Under these assumptions, every decoder in LMS has per-step sampling kernel $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$6, hence marginal law $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$7. The corollary is an impossibility statement: whenever $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$8, no LMS method samples $$\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].$$9 (Nie et al., 8 May 2026).

The appendix makes this explicit using masked verifier and draft distributions

$\mu^{\mathrm{proj}}$0

together with the standard Leviathan accept/reject kernel: draw $\mu^{\mathrm{proj}}$1, accept with probability

$\mu^{\mathrm{proj}}$2

and on rejection resample from $\mu^{\mathrm{proj}}$3. Because Leviathan rejection samples exactly from $\mu^{\mathrm{proj}}$4, and $\mu^{\mathrm{proj}}$5, speculative decoding faithfully preserves the wrong law when only local masks are visible (Nie et al., 8 May 2026).

The positive result is the oracle decoder FVO-Spec. Its modification is conceptually minimal: keep the speculative-decoding machinery, but replace the masked verifier target by the $\mu^{\mathrm{proj}}$6-reweighted target $\mu^{\mathrm{proj}}$7. With exact $\mu^{\mathrm{proj}}$8, the resulting speculative decoder samples exactly from $\mu^{\mathrm{proj}}$9 at every step. In pseudocode terms: compute $\mu^\star$0, compute corrected weights $\mu^\star$1 for $\mu^\star$2, normalize to obtain $\mu^\star$3, and run the usual speculative accept/reject step against this corrected target rather than $\mu^\star$4 (Nie et al., 8 May 2026).

5. Approximation, hardness, tractable regimes, and empirical behavior

Exact computation of $\mu^\star$5 is generally difficult. The paper proves that computing $\mu^\star$6 exactly is $\mu^\star$7-hard for general context-free grammars, even under a unigram base LLM. This motivates an estimator hierarchy. Uniform uses $\mu^\star$8, exactly recovering $\mu^\star$9. OneStep uses

$\Phi$0

reusing $\Phi$1 as a proxy for $\Phi$2, with no extra neural forwards and $\Phi$3 trie operations. MC rollout estimates

$\Phi$4

with Hoeffding and union-bound guarantees but cost $\Phi$5 target forwards. Exact methods use dynamic programming or enumeration on tractable subclasses: bounded Dyck grammars, finite languages via token tries, and regular languages via automaton-state backward recursion (Nie et al., 8 May 2026).

Approximate $\Phi$6 leads to an explicit perturbed target

$\Phi$7

If

$\Phi$8

then

$\Phi$9

The appendix also gives a telescoping sequence-level bound and a multiplicative certificate

$\Phi$0

when $\Phi$1 uniformly on positive-mass candidates. The paper emphasizes that additive guarantees become hard when $\Phi$2 is small, precisely the recursive-grammar regime where future validity matters most (Nie et al., 8 May 2026).

Empirically, the distortion is often dominant rather than marginal. For Qwen3-8B on Dyck $\Phi$3, the paper reports

$\Phi$4

and in a bounded-Dyck analytic confirmation on the same 988-string support,

$\Phi$5

In the Dyck estimator hierarchy, Uniform, OneStep-Cheap, and Exact achieve TV $\Phi$6 to $\Phi$7; thus OneStep reduces Dyck TV by $\Phi$8 with under $\Phi$9 throughput overhead, and exact dynamic programming reduces it by $\#P$0. The speed model reports AR baseline $\#P$1 tok/s, SD with no $\#P$2 $\#P$3 tok/s, SD + OneStep-Cheap $\#P$4 tok/s, and SD + exact DP $\#P$5 tok/s. On Dyck, local projection also overproduces deeper and longer strings, with mean nesting depth $\#P$6 versus $\#P$7 under $\#P$8, and mean length $\#P$9 versus $\mathcal C$00 (Nie et al., 8 May 2026).

For finite canonical JSON under Qwen3-8B, exact-token-trie experiments report

$\mathcal C$01

across schemas with 3 to 2000 valid strings, while exact $\mathcal C$02-correction reduces the residual to numerical zero,

$\mathcal C$03

In one 3-string status schema, $\mathcal C$04 gives $\mathcal C$05 mass to "error", local projection emits it $\mathcal C$06 of the time, and exact $\mathcal C$07 brings it to $\mathcal C$08 in the verifier-path pilot. The online finite-trie FVO-Spec loop samples within TV $\mathcal C$09 of $\mathcal C$10 on the 24-string schema and mean TV $\mathcal C$11 across five finite schemas, with mean acceptance $\mathcal C$12. In a production-like DFlash finite-trie pilot, exact $\mathcal C$13 reduces mean terminal TV on schemas A–D from $\mathcal C$14 to $\mathcal C$15, a $\mathcal C$16 pooled reduction, while throughput changes from $\mathcal C$17 tok/s to $\mathcal C$18 tok/s (Nie et al., 8 May 2026).

The paper is explicit about scope. Its strongest fidelity claims are limited to enumerable grammars and token tries, where $\mathcal C$19 and $\mathcal C$20 can be computed exactly. Exact $\mathcal C$21 is tractable for finite languages, regular languages, and bounded-depth or bounded-length grammars small enough for dynamic programming, but not for arbitrary open grammars such as general CFG-style JSON with arbitrary nesting or free-text fields (Nie et al., 8 May 2026).

6. Relation to other “future validity” notions

The phrase “future validity” appears elsewhere in arXiv literature, but it refers to different objects. In conformal prediction for future insurance claims, the closest analogue is the conformal plausibility function

$\mathcal C$22

which calibrates whether a candidate future claim value is retained in a prediction set, with finite-sample guarantee

$\mathcal C$23

under exchangeability (Hong, 5 Mar 2025). In consonant conformal prediction and related inferential-model work, the central object is a plausibility contour $\mathcal C$24 satisfying

$\mathcal C$25

or, more generally, an IM validity property of the form

$\mathcal C$26

for assertions $\mathcal C$27 (Cella et al., 2020, Cella et al., 2021, Martin, 2021).

By contrast, the future-validity function in grammar-faithful speculative decoding is not a coverage-calibration device for future observations. It is a base-model completion probability: $\mathcal C$28 used to transform one-step local admissibility into the correct law conditioned on eventual grammaticality. The paper explicitly relates this $\mathcal C$29 to expected futures in Grammar-Aligned Decoding, to a Doob harmonic function, and to inside or backward completion probabilities in constrained language modeling and parsing (Nie et al., 8 May 2026). This suggests that the shared vocabulary of “future validity” spans two quite different traditions: one centered on calibration of uncertainty about future observations, the other on model-based survival mass over future constrained completions.