---
title: Future-Validity Function in Speculative Decoding
url: https://www.emergentmind.com/topics/future-validity-function
type: topic
---

# Future-Validity Function in Speculative Decoding

The future-validity function is the prefix-dependent statistic that corrects grammar-constrained generation from “currently valid” tokens to the intended distribution conditioned on eventual grammatical completion. In the formulation of grammar-faithful speculative decoding, it is defined for a base autoregressive language model \(p\), a prefix-checkable grammar \(\mathcal C\), and a candidate next token \(y\) after prefix \(x_{<t}\) as
\[
\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].
\]
The central claim is that standard serving pipelines combining local vocabulary masking with speculative decoding omit exactly this statistic, and therefore sample from the locally projected law \(\mu^{\mathrm{proj}}\) rather than the intended grammar-conditional law \(\mu^\star\). With exact \(\Phi\), the target distribution becomes a Doob transform of the base model; with approximate \(\Phi\), one obtains explicit total-variation guarantees; and for general context-free grammars, exact computation is \(\#P\)-hard, so practical use depends on estimator hierarchies and tractable grammar families [2605.07698].

## 1. Formal definition and probabilistic setting

The setup assumes an autoregressive base language model \(p\) over a finite vocabulary \(V\), with per-step conditional
\[
p_t(y)=p(y\mid x_{<t}),
\]
together with a prefix-checkable constraint or grammar \(\mathcal C\), and valid-next-token sets
\[
A_t=A_C(x_{<t})\subseteq V.
\]
Within this setup, the future-validity function is
\[
\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}\!\big[x_{<t}yz_{t+1:T}\in \mathcal{L}(\mathcal{C})\big].
\]
The shorthand \(\Phi_t(y)=\Pr_p[\text{valid completion}\mid y]\) suppresses the conditioning on the committed prefix, but the paper emphasizes that the quantity is really prefix-dependent. A “valid completion” means completion into a full terminal string in \(L(C)\), including EOS as needed [2605.07698].

This definition makes \(\Phi_t(y)\) a survival probability under the base model \(p\): after appending \(y\), one samples the suffix from \(p\), and \(\Phi_t(y)\) is the probability that the resulting full sequence remains in the grammar language. In principle this depends on the full committed history through \(p(\cdot\mid x_{<t})\); if the language-model state is not Markov-compressible into grammar state, then grammar state alone is not sufficient. Operationally, for finite languages and regularized settings, the paper computes the grammar side from token tries or automaton states, while the model side still depends on prefix-conditioned probabilities [2605.07698].

## 2. From local masking to the grammar-conditional law

The paper distinguishes sharply between the deployed one-step masking law and the intended full-sequence conditional law. Local masking retains only \(A_t\) and renormalizes:
\[
\mu^{\mathrm{proj}}_t(y)=\frac{p_t(y)\mathbf{1}[y\in A_t]}{Z_p},
\qquad
Z_p=\sum_{y'\in A_t}p_t(y').
\]
The intended target, however, is the base-model sequence law restricted to the language and renormalized:
\[
\mu^\star(x_{1:T})=\frac{p(x_{1:T})\mathbf{1}[x_{1:T}\in L(C)]}{Z_C},
\qquad
Z_C=\sum_{y\in L(C)}p(y).
\]
These are different objects: \(\mu^{\mathrm{proj}}\) is the product of masked-and-renormalized one-step conditionals, whereas \(\mu^\star\) is the original sequence distribution conditioned on eventual grammaticality [2605.07698].

The future-validity function is the missing correction statistic because the exact per-step grammar-conditional kernel is
\[
\mu^\star_t(y\mid x_{<t})=
\frac{p_t(y)\Phi_t(y\mid x_{<t})}
{\sum_{y'\in A_t}p_t(y')\Phi_t(y'\mid x_{<t})}.
\]
Relative to local masking, each locally valid token is reweighted by its probability of surviving to a full valid completion. Local masking is recovered by the degenerate approximation \(h\equiv 1\), i.e. by pretending that every locally valid token has identical future validity. The paper gives the exact condition under which this omission is harmless:
\[
\mu^{\mathrm{proj}}_t=\mu^\star_t
\iff
\Phi_t \text{ is constant over locally valid tokens.}
\]
This identifies non-uniformity of \(\Phi_t\) over \(A_t\) as the sole source of distortion [2605.07698].

The same point appears in the KL identity. If
\[
\bar{\Phi}_t=\mathbb{E}_{\mu^{\mathrm{proj}}_t}[\Phi_t]=Z^\star/Z_p,
\]
then
\[
D\!\left(\mu^\star_t\middle\|\mu^{\mathrm{proj}}_t\right)
=
\mathbb{E}_{Y\sim \mu^\star_t}
\left[
\log\frac{\Phi_t(Y)}{\bar{\Phi}_t}
\right].
\]
Thus the divergence caused by ignoring future validity is controlled exactly by how non-uniform \(\Phi_t\) is across the locally valid token set [2605.07698].

## 3. Doob transform, Bellman recursion, and survival semantics

The paper characterizes \(\mu^\star\) as a Doob \(h\)-transform of the base model with \(h=\Phi\). This gives a precise probabilistic interpretation: future validity is not an auxiliary heuristic, but the harmonic function that converts the unconstrained next-token kernel into the sequence law conditioned on eventual membership in \(L(C)\) [2605.07698].

The appendix gives the recursive identity
\[
\Phi_t(y)
=
\sum_{u\in A_{t+1}(x_{<t}y)}
p(u\mid x_{<t}y)\,
\Phi_{t+1}(u\mid x_{<t}yu),
\]
which is a Bellman or backward recursion. Future validity at the current decision equals the model-weighted sum of future validity at the next decision over valid next tokens. This makes the quantity directly analogous to inside or backward completion probabilities in constrained language modeling and parsing, a connection stated explicitly in the paper [2605.07698].

The same appendix shows why shallow lookahead is biased. For the “true one-step estimator,”
\[
\hat{\Phi}^{\text{true}}_t(y)
=
\sum_{u\in A_{t+1}(x_{<t}y)} p(u\mid x_{<t}y),
\]
and therefore
\[
\hat{\Phi}^{\text{true}}_t(y)-\Phi_t(y)
=
\sum_{u\in A_{t+1}(x_{<t}y)}
p(u\mid x_{<t}y)\bigl(1-\Phi_{t+1}(u\mid x_{<t}yu)\bigr).
\]
One-step lookahead therefore overestimates true future validity by the expected future invalidity after the next step. This clarifies why local admissibility is insufficient: a token can be locally legal yet lead almost surely to future dead ends [2605.07698].

## 4. Speculative decoding, LMS impossibility, and oracle correction

A central negative result is that speculative decoding does not repair the mismatch between \(\mu^{\mathrm{proj}}\) and \(\mu^\star\). The paper defines the LMS class by three axioms: **B1 Local Mask**, **B2 Leviathan Rejection**, and **B3 Rollback Soundness**. Under these assumptions, every decoder in LMS has per-step sampling kernel \(\mu^{\mathrm{proj}}_t\), hence marginal law \(\mu^{\mathrm{proj}}\). The corollary is an impossibility statement: whenever \(\mu^{\mathrm{proj}}\neq\mu^\star\), no LMS method samples \(\mu^\star\) [2605.07698].

The appendix makes this explicit using masked verifier and draft distributions
\[
\tilde p_t(y)=p_t(y)\mathbf{1}[y\in A_t]/Z_{p,t},
\qquad
\tilde q_t(y)=q_t(y)\mathbf{1}[y\in A_t]/Z_{q,t},
\]
together with the standard Leviathan accept/reject kernel: draw \(d_t\sim \tilde q_t\), accept with probability
\[
\min\!\left(1,\frac{\tilde p_t(d_t)}{\tilde q_t(d_t)}\right),
\]
and on rejection resample from \((\tilde p_t-\tilde q_t)^+/Z^+\). Because Leviathan rejection samples exactly from \(\tilde p_t\), and \(\tilde p_t=\mu^{\mathrm{proj}}_t\), speculative decoding faithfully preserves the wrong law when only local masks are visible [2605.07698].

The positive result is the oracle decoder FVO-Spec. Its modification is conceptually minimal: keep the speculative-decoding machinery, but replace the masked verifier target by the \(\Phi\)-reweighted target \(\mu_t^\star\). With exact \(\Phi\), the resulting speculative decoder samples exactly from \(\mu^\star_t\) at every step. In pseudocode terms: compute \(A_t\), compute corrected weights \(p_t(y)\Phi_t(y)\) for \(y\in A_t\), normalize to obtain \(\mu_t^\star\), and run the usual speculative accept/reject step against this corrected target rather than \(\mu_t^{\mathrm{proj}}\) [2605.07698].

## 5. Approximation, hardness, tractable regimes, and empirical behavior

Exact computation of \(\Phi_t(y)\) is generally difficult. The paper proves that computing \(\Phi_t(y)\) exactly is \(\#P\)-hard for general context-free grammars, even under a unigram base language model. This motivates an estimator hierarchy. **Uniform** uses \(\hat\Phi_t\equiv 1\), exactly recovering \(\mu^{\mathrm{proj}}\). **OneStep** uses
\[
\hat{\Phi}^{\mathrm{1s}}_t(y)
=
\sum_{u\in A_{t+1}(x_{<t}y)} p_t(u),
\]
reusing \(p_t(\cdot)=p(\cdot\mid x_{<t})\) as a proxy for \(p(\cdot\mid x_{<t}y)\), with no extra neural forwards and \(O(|A_t|\cdot|V|)\) trie operations. **MC rollout** estimates
\[
\hat{\Phi}^{\mathrm{mc}}_t(y)
=
\frac1k\sum_{j=1}^k
\mathbf{1}\!\big[\text{rollout}_j(x_{<t}y)\in L(C)\big],
\]
with Hoeffding and union-bound guarantees but cost \(O(kh|A_t|)\) target forwards. **Exact** methods use dynamic programming or enumeration on tractable subclasses: bounded Dyck grammars, finite languages via token tries, and regular languages via automaton-state backward recursion [2605.07698].

Approximate \(\Phi\) leads to an explicit perturbed target
\[
\hat{\mu}_t(y)
=
\frac{p_t(y)\hat{\Phi}_t(y)}
{\sum_{y'\in A_t}p_t(y')\hat{\Phi}_t(y')}.
\]
If
\[
|\hat{\Phi}_t(y)-\Phi_t(y)|\le \delta
\quad\text{for all } y\in A_t,
\qquad
\delta<\bar{\Phi}_t,
\]
then
\[
TV(\hat{\mu}_t,\mu^\star_t)\le \frac{\delta}{\bar{\Phi}_t-\delta}.
\]
The appendix also gives a telescoping sequence-level bound and a multiplicative certificate
\[
TV(\hat{\mu}_t,\mu^\star_t)\le \frac{\epsilon}{1-\epsilon}
\]
when \(\hat{\Phi}_t(y)/\Phi_t(y)\in[1-\epsilon,1+\epsilon]\) uniformly on positive-mass candidates. The paper emphasizes that additive guarantees become hard when \(\bar{\Phi}_t\) is small, precisely the recursive-grammar regime where future validity matters most [2605.07698].

Empirically, the distortion is often dominant rather than marginal. For Qwen3-8B on Dyck \(D_{3,16}\), the paper reports
\[
TV(\mu^{\mathrm{proj}},\mu^\star)=0.996\pm 0.002,
\]
and in a bounded-Dyck analytic confirmation on the same 988-string support,
\[
0.999948\pm 0.000002.
\]
In the Dyck estimator hierarchy, Uniform, OneStep-Cheap, and Exact achieve TV \(0.418/0.359/0.014\) to \(\mu^\star\); thus OneStep reduces Dyck TV by \(14\%\) with under \(1\%\) throughput overhead, and exact dynamic programming reduces it by \(97\%\). The speed model reports AR baseline \(16.0\) tok/s, SD with no \(\Phi\) \(76.8\) tok/s, SD + OneStep-Cheap \(76.2\) tok/s, and SD + exact DP \(76.8\) tok/s. On Dyck, local projection also overproduces deeper and longer strings, with mean nesting depth \(1.72\) versus \(1.09\) under \(\mu^\star\), and mean length \(5.85\) versus \(2.77\) [2605.07698].

For finite canonical JSON under Qwen3-8B, exact-token-trie experiments report
\[
TV(\mu^{\mathrm{proj}},\mu^\star)\in[0.174,0.681]
\]
across schemas with 3 to 2000 valid strings, while exact \(\Phi\)-correction reduces the residual to numerical zero,
\[
TV(\mu^\Phi,\mu^\star)\lesssim 10^{-15}.
\]
In one 3-string status schema, \(\mu^\star\) gives \(8.8\%\) mass to `"error"`, local projection emits it \(26.8\%\) of the time, and exact \(\Phi\) brings it to \(7.9\%\) in the verifier-path pilot. The online finite-trie FVO-Spec loop samples within TV \(0.0052\) of \(\mu^\star\) on the 24-string schema and mean TV \(0.0051\) across five finite schemas, with mean acceptance \(0.951\). In a production-like DFlash finite-trie pilot, exact \(\Phi\) reduces mean terminal TV on schemas A–D from \(0.350\) to \(0.017\), a \(95\%\) pooled reduction, while throughput changes from \(91.7\) tok/s to \(81.1\) tok/s [2605.07698].

The paper is explicit about scope. Its strongest fidelity claims are limited to enumerable grammars and token tries, where \(\mu^\star\) and \(\Phi\) can be computed exactly. Exact \(\Phi\) is tractable for finite languages, regular languages, and bounded-depth or bounded-length grammars small enough for dynamic programming, but not for arbitrary open grammars such as general CFG-style JSON with arbitrary nesting or free-text fields [2605.07698].

## 6. Relation to other “future validity” notions

The phrase “future validity” appears elsewhere in arXiv literature, but it refers to different objects. In conformal prediction for future insurance claims, the closest analogue is the conformal plausibility function
\[
pl_{Z^n}(x_{n+1},y),
\]
which calibrates whether a candidate future claim value is retained in a prediction set, with finite-sample guarantee
\[
P^{n+1}\{Y_{n+1}\in C_\alpha(X_{n+1};Z^n)\}\ge 1-\alpha
\]
under exchangeability [2503.03659]. In consonant conformal prediction and related inferential-model work, the central object is a plausibility contour \(\pi(y;y^n)\) satisfying
\[
P\{\pi(Y_{n+1};Y^n)\le \alpha\}\le \alpha,
\]
or, more generally, an IM validity property of the form
\[
\sup_{\theta\in A}P_{Y\mid \theta}\{\overline{\Pi}_Y(A)\le \alpha\}\le \alpha
\]
for assertions \(A\) [2001.09225; 2112.10234; 2112.13247].

By contrast, the future-validity function in grammar-faithful speculative decoding is not a coverage-calibration device for future observations. It is a base-model completion probability:
\[
\Phi_t(y\mid x_{<t})=\Pr_{z\sim p}[x_{<t}yz_{t+1:T}\in \mathcal L(\mathcal C)],
\]
used to transform one-step local admissibility into the correct law conditioned on eventual grammaticality. The paper explicitly relates this \(\Phi\) to expected futures in Grammar-Aligned Decoding, to a Doob harmonic function, and to inside or backward completion probabilities in constrained language modeling and parsing [2605.07698]. This suggests that the shared vocabulary of “future validity” spans two quite different traditions: one centered on calibration of uncertainty about future observations, the other on model-based survival mass over future constrained completions.

Source: https://www.emergentmind.com/topics/future-validity-function