Value-Filtered Decoding

• Value-Filtered Decoding is a method that uses a hard threshold on estimated continuation values to filter out low-value token candidates during inference.
• The approach calibrates the threshold to control false interventions, balancing improved safety with preservation of base model behavior.
• It offers practical insights into decoding trade-offs and compares favorably with soft reweighting methods in terms of computational efficiency and safety guarantees.

Searching arXiv for papers on value-filtered decoding and closely related guided/value-guided decoding. Value-filtered decoding denotes an inference-time control strategy in which a value estimate of a partial continuation is used to exclude low-value next-token candidates before generation proceeds. In the safety-steering formulation, the value attached to a prefix is the probability that the eventual completion is safe under the base autoregressive policy, and decoding applies a hard threshold $c$ that filters out tokens whose continuation value falls below that threshold (Einbinder et al., 14 May 2026). In the broader guided-decoding literature, closely related methods use value estimates to rerank, reweight, or search over continuations without retraining the generator, positioning value-filtered decoding within a larger family of test-time alignment procedures (Liu et al., 4 Mar 2025, Liu et al., 2023).

1. Formal definition

The canonical formulation assumes a base autoregressive LLM

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$

where $x\sim\mathcal D$ is a prompt and $Y=(Y_1,\dots,Y_T)$ is the generated sequence. In the safety setting, the terminal reward is binary,

$r(x,Y)\in\{0,1\},$

with $r(x,Y)=1$ meaning safe. The corresponding value of a prefix is

$$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$

For a candidate next token $y_t$, one writes

$V_t = V(x,Y_{1:t-1},y_t).$

This quantity is a continuation-level safety probability under the base policy, not merely a token-local classifier score (Einbinder et al., 14 May 2026).

Value-filtered decoding introduces a threshold $c\in(0,1)$ and restricts sampling to candidates whose value exceeds that threshold. The oracle admissible set is

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$0

with normalizer

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$1

The oracle filtered policy is therefore

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$2

Because the true value is unavailable in practice, one replaces $$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$3 by an estimator $$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$4 and obtains the empirical policy

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$5

The distinguishing feature of this construction is its hard truncation. Unlike soft exponential tilts or additive rerankers, it does not continuously deform all token probabilities; it removes candidates below a specified value threshold and renormalizes the remainder. This makes the intervention explicitly selective and gives the threshold $$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$6 a direct operational meaning (Einbinder et al., 14 May 2026).

2. Statistical control of unnecessary interventions

A central contribution of the safety-steering formulation is an explicit account of false interventions. An unnecessary intervention, or type-I error, occurs when the base model would have produced a safe completion, $$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$7, but at some intermediate step the filter rejects the token actually drawn by the base policy. Under

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$8

this event is

$$\pi(y_{1},\dots,y_T\mid x)=\prod_{t=1}^{T}\pi(y_t\mid x,y_{1:t-1}),$$9

The analysis shows that the oracle value process $x\sim\mathcal D$0 is a martingale under $x\sim\mathcal D$1 conditioned on $x\sim\mathcal D$2, so Ville’s inequality yields the bound

$x\sim\mathcal D$3

where

$x\sim\mathcal D$4

In particular, smaller $x\sim\mathcal D$5 implies smaller $x\sim\mathcal D$6 (Einbinder et al., 14 May 2026).

This result makes the threshold a control parameter for selective intervention. Raising $x\sim\mathcal D$7 tightens the filter and improves safety of completions, but it also increases the rate at which already-safe base trajectories are altered. Lowering $x\sim\mathcal D$8 preserves more of the base model’s behavior, at the price of weaker safety gains. The paper characterizes this as a trade-off between safety and fidelity.

Because deployed systems use $x\sim\mathcal D$9 rather than $Y=(Y_1,\dots,Y_T)$0, the oracle guarantee need not hold directly. To restore finite-sample control, the method uses conformal risk control on a held-out set of safe completions. For each safe example $Y=(Y_1,\dots,Y_T)$1, one computes

$Y=(Y_1,\dots,Y_T)$2

The calibrated threshold is the largest $Y=(Y_1,\dots,Y_T)$3 such that

$Y=(Y_1,\dots,Y_T)$4

The resulting guarantee is

$Y=(Y_1,\dots,Y_T)$5

so the unnecessary intervention rate is at most $Y=(Y_1,\dots,Y_T)$6 in expectation (Einbinder et al., 14 May 2026).

The practical significance is narrow but important: the calibration target is not generic classification accuracy, but the intervention rate on completions that were already safe under the base model. This separates value-filtered decoding from decoding-time steering methods that improve safety but provide no direct control over how often they perturb safe generations.

3. Decoding procedure and implementation patterns

The online decoding algorithm is operationally simple. Given prompt $Y=(Y_1,\dots,Y_T)$7, base policy $Y=(Y_1,\dots,Y_T)$8, value estimator $Y=(Y_1,\dots,Y_T)$9, threshold $r(x,Y)\in\{0,1\},$0, and candidate budget $r(x,Y)\in\{0,1\},$1, decoding repeatedly samples up to $r(x,Y)\in\{0,1\},$2 candidate tokens iid from $r(x,Y)\in\{0,1\},$3. For each candidate $r(x,Y)\in\{0,1\},$4, it computes

$r(x,Y)\in\{0,1\},$5

accepts the first candidate with $r(x,Y)\in\{0,1\},$6, and, if none is accepted, falls back to the candidate with maximal estimated value. The accepted token is appended and the procedure repeats until end-of-sequence (Einbinder et al., 14 May 2026).

This implementation exposes the main computational trade-off. The method avoids retraining the generator, but each decoding step incurs value-model evaluation over multiple sampled candidates. In the reported experiments, its computational cost is comparable to CARDS and ARGS with $r(x,Y)\in\{0,1\},$7 candidates, and faster than Controlled decoding, which recalculates $r(x,Y)\in\{0,1\},$8 each step (Einbinder et al., 14 May 2026).

Within the broader value-guided decoding literature, related formulations replace hard filtering with scoring or reranking. A representative objective is

$r(x,Y)\in\{0,1\},$9

equivalently

$r(x,Y)=1$0

Inference then proceeds over a top-$r(x,Y)=1$1 candidate set, with either hard filtering by a threshold $r(x,Y)=1$2 or soft top-$r(x,Y)=1$3 reranking (Liu et al., 4 Mar 2025). This suggests a continuum of value-guided policies: strict truncation, as in safety-oriented VFD, at one end; continuous reweighting at the other.

Training the value model is correspondingly central. In Iterative Value Function Optimization, $r(x,Y)=1$4 is fit by Monte Carlo regression on multiple rollouts per prompt and then improved by iterative on-policy data collection from the value-guided policy itself. The stated rationale is that single-trajectory estimates have high variance and poor coverage, whereas multiple rollouts reduce estimator variance and expose the critic to a wider state distribution (Liu et al., 4 Mar 2025). In this perspective, value-filtered decoding is only as reliable as its continuation-value estimator.

4. Relation to adjacent decoding-time steering methods

The modern literature distinguishes several inference-time alignment mechanisms that all use value or reward information but intervene in different ways.

Method	Mechanism	Stated property
Value-filtered decoding	Hard threshold on $r(x,Y)=1$5 or $r(x,Y)=1$6, then truncate and renormalize	Explicit bound on false interventions; hard threshold gives exact control of intervention rate
Controlled decoding	Gibbs policy $r(x,Y)=1$7	No guarantee on how often safe completions are changed
ARGS	Top-$r(x,Y)=1$8 additive scoring $r(x,Y)=1$9	Sensitive to scale and estimation error
CARDS	Rejection sampling targeting cumulative reward $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$0	Controls expected number of resamples, not safe-safe fidelity
IVO	Value-guided policy $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$1 with iterative on-policy optimization	Intended as a cost-effective alternative to RLHF
PPO-MCTS	Monte-Carlo Tree Search using PPO value estimates during inference	Reduces mismatch between training and test scoring of partial outputs

In this comparison, the salient distinction is between hard admissibility criteria and soft preference shaping. Value-filtered decoding rejects candidates below a threshold and therefore frames intervention as a hypothesis-testing problem over safe trajectories. Controlled decoding and ARGS instead reshape token scores continuously; the VFD paper argues that these alternatives offer no false-intervention control, and that exponentiation can amplify value-estimation errors (Einbinder et al., 14 May 2026).

PPO-MCTS occupies a different point in the design space. Rather than filtering one-step candidates, it performs look-ahead search from the current state using a PPO value model. Selection uses

$$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$2

expanded nodes are evaluated with $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$3, and final next-token choice is based on root visit counts (Liu et al., 2023). The paper’s explicit motivation is that the value network should not be discarded after PPO training. Value-guided search and value-filtered truncation therefore share an emphasis on continuation values, but differ sharply in computational structure and in the form of control they provide.

5. Empirical behavior, trade-offs, and limitations

The main safety-steering evaluation uses mistralai/Mistral-7B-Instruct-v0.3 as the base model and a 3-layer MLP value-head over transformer hidden states producing scalar $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$4. Training uses focal loss plus a temporal smoothness regularizer. The reported datasets are HH-RLHF, BeaverTails, and PKU-SafeRLHF; evaluation metrics are harmlessness, helpfulness, and cosine similarity to the base-model response, judged by meta-llama/Llama-3.1-8B-Instruct (Einbinder et al., 14 May 2026).

The reported empirical pattern is a Pareto frontier between harmlessness and fidelity. Raising $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$5, or equivalently the target $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$6, increases safety but also increases intervention frequency on safe generations; lowering it preserves similarity and helpfulness. Across HH-RLHF and BeaverTails, value-filtered decoding achieves uniformly higher harmlessness for the same drop in helpfulness or similarity. On safe prompts, it intervenes at rate approximately $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$7 by design, whereas the cited baselines intervene arbitrarily often. On unsafe prompts, it matches or exceeds baseline safety gains (Einbinder et al., 14 May 2026).

Broader value-guided decoding results reinforce the claim that inference-time value estimates can materially alter output quality. Iterative Value Function Optimization reports higher average reward than ARGS, FUDGE, and VAS at fixed KL budgets across text summarization, multi-turn dialogue, and instruction following; it also reports dialogue win-rates of approximately $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$8 versus $$V(x,Y_{1:t})=\mathbb{E}_{Y_{t+1:T}\sim \pi(\cdot\mid x,Y_{1:t})}\!\bigl[r(x,(Y_{1:t},Y_{t+1:T}))\bigr] = P_\pi\bigl(r(x,Y)=1\mid x,Y_{1:t}\bigr).$$9 for the next best method, instruction-following improvements of about $y_t$0–$y_t$1 absolute points over the base model, and a practical regime in which two IVO iterations suffice and $y_t$2 rollouts is a sweet spot (Liu et al., 4 Mar 2025). PPO-MCTS reports gains over PPO-only decoding on sentiment steering, toxicity reduction, knowledge introspection, and HH-RLHF, including 86.7% and 91.1% success rates on positive and negative sentiment steering, reduction of average max toxicity from 0.188 to 0.124, and a reward-model increase from 1.7176 to 1.7532 on HH-RLHF (Liu et al., 2023).

These results do not eliminate the standard limitations. Safety-oriented VFD depends on a reliable value estimator and a safe hold-out calibration set; its reward assumption is binary, $y_t$3; the presented method is single-objective, centered on safety; and it assumes delayed reward at completion rather than intermediate feedback (Einbinder et al., 14 May 2026). In adjacent guided-decoding settings, value evaluation also adds inference overhead: IVO reports approximately $y_t$4–$y_t$5 slower decoding than pure base sampling because of extra forward passes through $y_t$6, even though it is still far cheaper than RLHF training (Liu et al., 4 Mar 2025).

6. Distinct usage outside language generation

The phrase “value-filtered decoding” also appears in a separate systems context: fused columnar scan kernels that decompress, filter, and aggregate compressed data. There it does not refer to language-model control, but to the interaction between decode throughput, predicate evaluation, and memory bandwidth in analytical query processing (Mandarapu et al., 21 Jun 2026).

In that setting, a compressed column contains $y_t$7 values encoded in $y_t$8 bits each, sustainable memory bandwidth is $y_t$9 bytes/s, and decoder throughput is $V_t = V(x,Y_{1:t-1},y_t).$0 values/s. The achieved bandwidth fraction is modeled by

$V_t = V(x,Y_{1:t-1},y_t).$1

with compute-to-bandwidth ridge

$V_t = V(x,Y_{1:t-1},y_t).$2

The reported empirical regularity is that, once the decode layout and inner-loop strategy are fixed, $V_t = V(x,Y_{1:t-1},y_t).$3 is essentially independent of bit-width $V_t = V(x,Y_{1:t-1},y_t).$4; across $V_t = V(x,Y_{1:t-1},y_t).$5, the measured $V_t = V(x,Y_{1:t-1},y_t).$6 curve is described as essentially horizontal. Fitting one $V_t = V(x,Y_{1:t-1},y_t).$7 per strategy yields median absolute error 0.027 on x86/AVX2 and 0.003 on held-out Apple M4/NEON data. The same study also validates a branch-vs-branch-free crossover governed by the $V_t = V(x,Y_{1:t-1},y_t).$8 misprediction parabola and argues that zone-map skipping is clustering-gated rather than selectivity-gated (Mandarapu et al., 21 Jun 2026).

This cross-domain usage is terminologically significant. In language modeling, value-filtered decoding denotes thresholded control of token generation via a continuation-value estimate. In columnar systems, the same phrase refers to a fused decode–filter–aggregate pipeline whose bottleneck can be predicted from decoder value throughput and bandwidth. The shared vocabulary is therefore metaphorical rather than methodological: both settings filter after decoding, but they operate on different objects, optimize different criteria, and rely on unrelated analytical frameworks.