Reverse Speculative Decoding Explained

• Reverse Speculative Decoding is defined as a training-free relaxed speculative decoding method where the drafter q is preferred and only defers to the verifier p upon divergence or low confidence.
• It introduces a relaxed target distribution π that blends q and p, with variants like r-fuzzy, spec-casc-opt, and ens offering distinct trade-offs in bias and acceptance rate.
• The method highlights critical performance and cost trade-offs, emphasizing that improved acceptance may come with longer generation spans and relies heavily on the alignment between q and p.

Searching arXiv for papers and terminology around “Reverse Speculative Decoding” and related RSD usages. arXiv search: "Reverse Speculative Decoding" Reverse speculative decoding is not a standardized formal name in the recent speculative-decoding literature. In the most technically precise usage available in current arXiv sources, it denotes a family of training-free relaxed speculative decoding methods that make the drafter $q$ the default source of tokens and only defer to the verifier $p$ under divergence- or confidence-based criteria, thereby replacing the lossless target $p$ with a relaxed target $\pi$ that is equal to, or closer to, $q$ unless a trigger says otherwise (Xia et al., 9 Jul 2026). At the same time, the acronym RSD is used in other papers for Reward-Guided Speculative Decoding, Recursive Speculative Decoding, and Retrieval-based Speculative Decoding, none of which the authors describe as “Reverse Speculative Decoding” (Liao et al., 31 Jan 2025, Jeon et al., 2024, Chen et al., 14 Apr 2026). This suggests that the topic is best understood as a terminological ambiguity centered on a specific relaxed-decoding design pattern rather than as a single universally named algorithm.

1. Terminology and scope

The central terminological fact is that papers do not agree on “RSD.” In the unified study of training-free relaxed speculative decoding, the paper does not use the name “Reverse Speculative Decoding,” but explicitly states that several methods implement the behavior most people mean by “reverse”: rely on the drafter $q$ for generation and replace the verifier target $p$ by a relaxed target $\pi$ that privileges $q$ unless a trigger says otherwise (Xia et al., 9 Jul 2026). By contrast, the paper titled “Reward-Guided Speculative Decoding” states that its method is Reward-Guided, not Reverse; the paper titled “Recursive Speculative Decoding” states that its method is Recursive, not Reverse; and the SENSE paper states that throughout that work, RSD = Retrieval-based Speculative Decoding, with no methodological connection to outside uses of “Reverse Speculative Decoding” (Liao et al., 31 Jan 2025, Jeon et al., 2024, Chen et al., 14 Apr 2026).

Source	Meaning of “RSD”	Relation to “reverse”
(Xia et al., 9 Jul 2026)	Informal “RSD-style” relaxed SD	Closest match
(Liao et al., 31 Jan 2025)	Reward-Guided Speculative Decoding	Not reverse
(Jeon et al., 2024)	Recursive Speculative Decoding	Not reverse
(Chen et al., 14 Apr 2026)	Retrieval-based Speculative Decoding	Not reverse

This ambiguity matters because speculative decoding papers use very different objects as the locus of acceleration: a draft model, a draft tree, a datastore, or a reward model. Conflating these under a single “reverse” label obscures whether the method remains lossless, whether it preserves the verifier distribution exactly, and whether its acceptance rule is ratio-based, reward-gated, or retrieval-conditioned.

2. From lossless speculative decoding to “reverse” behavior

Standard lossless speculative decoding targets the verifier distribution $p$ and uses rejection/resampling to preserve $p$0 exactly. In the formulation summarized by the relaxed-SD survey, drafted tokens are proposed from $p$1, accepted with

$p$2

and, when rejected, corrected by residual sampling from

$p$3

This makes the procedure lossless with respect to $p$4 (Xia et al., 9 Jul 2026).

“Reverse” behavior, in that same framework, changes the target of correction from $p$5 to a relaxed distribution $p$6. Drafting is still done by $p$7, and verification still evaluates $p$8 in parallel, but the acceptance rule becomes

$p$9

with correction from

$p$0

The reversal lies in the stance toward $p$1: strict speculative decoding insists on exact correction to $p$2, whereas reverse-style relaxed decoding accepts more of $p$3’s tokens and uses $p$4 only when a criterion indicates that trust in $p$5 should be withdrawn (Xia et al., 9 Jul 2026).

The paper’s terminology is explicit about which methods fit this pattern. Reducible Fuzzy Speculative Decoding (r-fuzzy) and Speculative Cascades (spec-casc-opt) are identified as the clearest RSD-style methods because they “accept $p$6 as the target when a criterion holds; otherwise they use $p$7.” The simple Ensemble (ens) method is also RSD-style because it sets $p$8, moving the effective target closer to $p$9 than to $\pi$0 (Xia et al., 9 Jul 2026).

3. Principal reverse-style formulations

The divergence-triggered form is r-fuzzy. Its relaxed targets are

$\pi$1

The paper uses Jensen–Shannon Divergence for $\pi$2 to avoid divide-by-zero issues with top-$\pi$3/$\pi$4 supports. Operationally, this means that when the drafter and verifier are sufficiently close, the system simply treats the drafter as the target; otherwise it falls back to verifier-centric correction (Xia et al., 9 Jul 2026).

The confidence-triggered form is spec-casc-opt. It is defined through the per-token loss

$\pi$5

with deferral rule

$\pi$6

Its relaxed target is

$\pi$7

This formulation makes “reverse” trust depend not on a generic divergence threshold but on whether the drafter’s confidence is high enough relative to the verifier after accounting for a deferral cost (Xia et al., 9 Jul 2026).

The softest form is ens, which uses

$\pi$8

The paper states that this is pareto-optimal for trading off deviation from $\pi$9, measured by $q$0, against expected rejection rate $q$1. In reverse-style terms, ens does not switch discretely between trusting $q$2 and trusting $q$3; it continuously interpolates between them (Xia et al., 9 Jul 2026).

At runtime, all three variants share the same high-level procedure. The drafter samples $q$4 tokens from $q$5; the verifier evaluates $q$6 in parallel over the drafted span; method-specific $q$7, $q$8, and $q$9 are constructed; drafted tokens are accepted against $q$0; the process stops at the first rejection; and the correction token is drawn from $q$1. If no rejection occurs, a bonus token is sampled, with $q$2 for these RSD-style methods (Xia et al., 9 Jul 2026).

4. Theoretical properties and distributional consequences

The defining theoretical distinction is between lossless and relaxed decoding. Reverse-style speculative decoding is lossless only when $q$3 everywhere. The relaxed-SD paper gives concrete examples: r-fuzzy is lossless when $q$4 is so small that $q$5 always, spec-casc-opt is lossless as $q$6, and ens is lossless when $q$7 (Xia et al., 9 Jul 2026). Otherwise the method is biased away from $q$8 by construction.

Acceptance behavior can be written in total-variation form. For strict speculative decoding, the paper states

$q$9

Under reverse-style relaxed decoding, $p$0 is replaced by $p$1, so the acceptance probability becomes $p$2. This is the formal expression of the central mechanism: by moving the target closer to $p$3, acceptance increases (Xia et al., 9 Jul 2026).

For r-fuzzy, the paper proves an equivalence result. If

$p$4

with $p$5, then Algorithm 2 with $p$6 and $p$7 yields

$p$8

Moreover, when the fuzzy predicate fails and the method defers to $p$9, the acceptance probability equals

$\pi$0

This connects reverse-style correction directly to classical rejection-sampling structure: acceptance ratio $\pi$1, residual correction to $\pi$2, and $\pi$3 depending on the predicate (Xia et al., 9 Jul 2026).

A related but distinct form of controlled bias appears in Reward-Guided Speculative Decoding, whose authors note that “Reverse Speculative Decoding” is a misnomer for their method. There, the draft model is favored when a process reward model assigns high reward to an intermediate step. The method defines a dynamic mixture

$\pi$4

with threshold policy $\pi$5 if $\pi$6 and $\pi$7 otherwise. The paper proves that a threshold rule is optimal under a compute-budget constraint and characterizes the compute objective

$\pi$8

Although this mechanism also relaxes strict unbiasedness, its accurate name is Reward-Guided Speculative Decoding, not Reverse Speculative Decoding (Liao et al., 31 Jan 2025).

5. Performance, complexity, and practical constraints

The practical behavior of reverse-style relaxed decoding depends heavily on drafter quality and serving overhead. The 2026 practical study reports that many relaxed approaches rely on a drafter that is a good LLM, making them unsuited for lightweight dedicated multi-token-prediction drafters (Xia et al., 9 Jul 2026). With a weak drafter, RSD-style methods such as r-fuzzy, spec-casc-opt, and ens show clear capability degradation and, in many cases, slowdowns compared to strict speculative decoding, even though acceptance and token throughput increase. The slowdown is attributed to longer generation lengths, described in the paper as “rambling” or repetition in thinking traces (Xia et al., 9 Jul 2026).

The paper proposes two speed proxies. The token-throughput proxy is

$\pi$9

and the response-length-aware relaxed proxy is

$q$0

The second expression is crucial for reverse-style methods because acceptance improvements can be offset by longer outputs. The same study emphasizes that the optimal $q$1 depends on $q$2, so draft-length tuning should precede relaxation tuning (Xia et al., 9 Jul 2026).

Serving overhead can be substantial. Using vLLM 0.20.1 on NVIDIA GH200, the paper gives the diagnostic example

$q$3

$q$4

$q$5

The conclusion is that stock separate-drafter speculative decoding in vLLM is under-optimized and can erase theoretical gains if $q$6 is too high (Xia et al., 9 Jul 2026).

When the drafter is strong, the picture changes. The same study reports that reverse-style methods can perform much better, sometimes close to lossless capability with meaningful speed-ups, because stronger $q$7 unlocks longer accepted spans. But this improvement is constrained by the increased cost of running a strong drafter, so capability gains and speed gains do not necessarily rise together (Xia et al., 9 Jul 2026). This trade-off aligns with the reward-guided literature, which also accepts controlled bias in exchange for efficiency and reports up to 4.4x fewer FLOPs together with average gains of up to +3.5 over SD/BoN baselines, but under a different mechanism based on a process reward model rather than a relaxed target $q$8 (Liao et al., 31 Jan 2025).

6. Relation to other methods and recurring misconceptions

One recurring misconception is to treat every method that departs from strict verifier matching as “reverse speculative decoding.” The recent literature distinguishes at least three unrelated uses of the acronym RSD.

Reward-Guided Speculative Decoding uses a lightweight draft model, a target model, and a process reward model $q$9 that scores partial trajectories $p$0. High-reward steps are accepted from the draft, low-reward steps invoke the target, and the method departs from standard speculative decoding by admitting controlled bias toward high-reward intermediate tokens or segments. The authors explicitly state that “Reverse Speculative Decoding” is a misnomer for this method (Liao et al., 31 Jan 2025).

Recursive Speculative Decoding is a tree-based, unbiased extension of speculative decoding. It builds draft trees by sampling without replacement, either through Gumbel-Top-$p$1 or Stochastic Beam Search, and verifies them with recursive rejection sampling. Its defining property is that the final distribution exactly matches the target LLM’s distribution. The authors explicitly state that the method is Recursive, not Reverse (Jeon et al., 2024).

Retrieval-based Speculative Decoding in the SENSE paper replaces a draft model with retrieved suffixes from a datastore. That paper emphasizes that conventional RSD means retrieval-based speculative decoding, not reverse, and that some outside sources use “Reverse Speculative Decoding” for different ideas such as target-guided drafting or reverse verification, with no methodological connection to SENSE (Chen et al., 14 Apr 2026). SENSE itself is training-free and accelerates inference by anchoring retrieval on target-model hidden states and verifying semantic equivalence rather than surface forms, reporting mean acceptance length up to 4.09, speedup up to 3.26x, and generation quality preserved at approximately 98.02% relative to vanilla decoding (Chen et al., 14 Apr 2026).

The other recurring misconception is to assume that “reverse” necessarily means a capability improvement. The practical relaxed-SD study shows that RSD-style methods can degrade capability and even slow down with weak drafters, despite higher acceptance. Conversely, strong capability–speed trade-offs are more reliably obtained when $p$2 is itself a strong standalone LLM and the serving stack supports a separate drafter with low $p$3 (Xia et al., 9 Jul 2026). A plausible implication is that the “reverse” idea is not a universally superior substitute for strict speculative decoding, but a relaxation regime whose value depends on verifier–drafter alignment, response-length effects, and systems-level implementation costs.