SpecDiff-2: Diffusion-Based Decoding

• The paper introduces SpecDiff-2, which couples non-autoregressive diffusion drafting with alignment mechanisms to optimize accepted token streaks while preserving lossless decoding.
• It employs streak-distillation and self-selection acceptance to calibrate the diffusion drafter with the autoregressive verifier, addressing misalignment issues and boosting throughput.
• Empirical results demonstrate up to a +55% speedup in tokens-per-second across reasoning, code, and mathematical benchmarks without compromising output accuracy.

Searching arXiv for the primary paper and immediate predecessor to ground the article. SpecDiff-2 is a speculative decoding framework for LLM inference that combines a discrete diffusion drafter with alignment mechanisms designed for autoregressive verifiers. It targets two bottlenecks in conventional speculative decoding: the autoregressive dependency of the drafter, which limits token-level parallelism, and the rejection of drafted tokens caused by drafter–verifier misalignment. Its defining contribution is to couple non-autoregressive diffusion drafting with train-time and test-time calibration procedures—streak-distillation and self-selection acceptance—while retaining lossless decoding, so that the final output distribution remains identical to that of the verifier alone (Sandler et al., 1 Nov 2025).

1. Historical position and conceptual scope

SpecDiff-2 belongs to the speculative decoding literature for autoregressive LLMs. In the standard setting, a smaller drafter proposes a block of future tokens and a larger verifier accepts or rejects them under a correction rule that preserves the verifier’s distribution. Earlier diffusion-based work, "Speculative Diffusion Decoding: Accelerating Language Generation through Diffusion" (Christopher et al., 2024), replaced the autoregressive drafter with a discrete diffusion model, thereby parallelizing drafting as well as verification. That earlier method established the central idea that drafting cost can depend on the number of diffusion steps rather than the draft length, enabling substantially larger draft windows than autoregressive drafters.

SpecDiff-2 is framed as a direct response to a limitation left open by that first generation. The 2024 SpecDiff paper showed that diffusion drafters can accelerate generation but also noted that diffusion models are typically worse aligned with the autoregressive verifier, depressing the acceptance rate. SpecDiff-2 treats this not as a marginal implementation issue but as the second core bottleneck of diffusion-based speculation: a non-autoregressive drafter is useful only if its joint block proposals are calibrated to the verifier well enough to sustain long accepted streaks (Sandler et al., 1 Nov 2025).

A common misconception is that SpecDiff-2 is simply a renamed or lightly tuned version of the 2024 method. The paper defines it more narrowly: it is a diffusion-drafter speculative decoding framework with explicit alignment machinery. Conversely, the earlier paper does not define any numbered variant; it introduces only "SpecDiff" (Christopher et al., 2024).

2. Formal decoding objective

SpecDiff-2 assumes a verifier model $P$, a drafter model $Q$, a prefix $\mathbf{s}$, and a draft block length $\gamma$. In classical speculative decoding, a proposed token $x_i$ at position $i$ is accepted with probability

$$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$

If rejection occurs at the first failed position, the remaining draft suffix is discarded and sampling resumes from the verifier’s residual distribution. The resulting process is lossless: the final output distribution is identical to sampling from the verifier alone (Sandler et al., 1 Nov 2025).

The paper characterizes verifier–drafter agreement through the acceptance rate

$$\alpha(\mathbf{s}) \triangleq 1 - \tfrac{1}{2}\sum_{x \in \mathcal V} \left| P(x \mid \mathbf{s}) - Q(x \mid \mathbf{s}) \right|,$$

so alignment corresponds directly to low total-variation distance. For blockwise drafting, the more relevant quantity is the expected committed streak length,

$$\tau(\gamma,\mathbf{s}) \triangleq \mathbb{E}_{\mathbf{x}_{1:\gamma}\sim Q(\cdot\mid\mathbf{s})}\left[ \sum_{m=1}^{\gamma} \prod_{j=1}^{m} \alpha_j(\mathbf{s} \circ \mathbf{x}_{1:j-1}) \right].$$

This definition is central to SpecDiff-2. It makes clear that throughput is governed not merely by the first accepted token but by the full product structure of acceptances across the draft window. The paper argues that this distinction matters especially for diffusion drafters, because later positions in the block are not exchangeable with earlier ones, and naïve alignment objectives that effectively optimize only the first position underperform on the tail of the window (Sandler et al., 1 Nov 2025).

3. Diffusion drafter architecture

SpecDiff-2 uses masked discrete diffusion LLMs as drafters. The forward process randomly replaces tokens by $[\text{MASK}]$ according to a time-dependent masking probability, producing a corrupted sequence $Q$0. The reverse model $Q$1 predicts token distributions for masked positions, and training uses masked cross-entropy:

$Q$2

At inference time, given a prefix $Q$3 and a draft window of length $Q$4, the drafter appends $Q$5, performs diffusion denoising, and outputs per-position marginals $Q$6. A concrete draft is then sampled independently from these marginals. The crucial systems property is that all $Q$7 positions are produced in parallel, so draft latency scales with the number of diffusion steps rather than with $Q$8 itself (Sandler et al., 1 Nov 2025).

This non-autoregressive drafting mechanism addresses the first speculative decoding bottleneck directly. Whereas classical speculative sampling and tree-based variants still pay autoregressive cost in the drafter, SpecDiff-2 makes large draft windows practical without incurring $Q$9 sequential drafter passes. The paper reports that, with strong 7B diffusion drafters, one diffusion step is empirically the best trade-off: additional steps improve draft quality slightly but degrade net speedup because drafting cost grows linearly in the number of steps (Sandler et al., 1 Nov 2025).

4. Alignment mechanisms: streak-distillation and self-selection

The paper’s principal novelty lies in its calibration of the diffusion drafter to the autoregressive verifier. It attributes the acceptance bottleneck to a mismatch in modeling objectives: the diffusion drafter learns joint denoising distributions over masked blocks, whereas the verifier defines prefix-conditional next-token distributions. As a result, a draft may look globally coherent yet still be poorly aligned with verifier probabilities at the token level.

For train-time alignment, SpecDiff-2 introduces streak-distillation. The construction starts from a greedy acceptance surrogate and defines, for position $\mathbf{s}$0,

$\mathbf{s}$1

This yields the streak-distillation objective

$\mathbf{s}$2

The objective explicitly optimizes the expected length of accepted token streaks under verifier trajectories rather than a local next-token divergence. In the paper’s interpretation, this is the appropriate training target for a diffusion drafter because later positions in the draft window materially affect throughput and are not well served by first-position distillation alone (Sandler et al., 1 Nov 2025).

For test-time alignment, SpecDiff-2 introduces self-selection acceptance. A single denoising pass produces the marginals $\mathbf{s}$3, after which the system samples $\mathbf{s}$4 candidate drafts almost for free. Each candidate $\mathbf{s}$5 is scored by the verifier using

$\mathbf{s}$6

and the best candidate is selected before verification. The paper emphasizes that this exploits a structural advantage unique to diffusion drafters: once the block marginals are available, drawing many joint candidates requires negligible extra drafting compute. It also notes that the subsequent greedy acceptance rule uses only verifier probabilities, which simplifies support for cross-tokenizer or otherwise non-calibrated diffusion drafters (Sandler et al., 1 Nov 2025).

5. Inference pipeline and design regime

The inference loop consists of four stages. First, the diffusion drafter performs one denoising step over a masked block of length $\mathbf{s}$7. Second, the system samples $\mathbf{s}$8 candidate drafts from the resulting marginals. Third, the verifier scores all candidates and selects the one with maximal throughput score. Fourth, the verifier runs the acceptance loop on that selected draft, advancing the cache and repeating until termination (Sandler et al., 1 Nov 2025).

The paper studies this regime with DiffuCoder-7B and DiffuLLaMA-7B as drafters and with Qwen2.5-72B-Instruct, LLaMA-2-70B-chat, Qwen2.5-14B-Instruct, and LLaMA-2-13B-chat as verifiers. The preferred draft window is $\mathbf{s}$9 for code and $\gamma$0 for text. Drafter temperature is treated as an important control knob for self-selection: low temperatures produce higher-quality but less diverse candidates, while temperatures around $\gamma$1 improve candidate diversity and therefore the chance that self-selection finds a well-aligned draft. The paper reports that speedup increases smoothly with $\gamma$2, with up to roughly $\gamma$3 extra throughput at $\gamma$4 (Sandler et al., 1 Nov 2025).

A practical boundary condition is verifier scale. The paper states that a 7B diffusion drafter is consistently effective for 70B/72B verifiers, but can be too large relative to a 13B or 14B verifier, in which case drafting latency begins to dominate. This makes SpecDiff-2 primarily a large-verifier method rather than a universal speculative decoding regime (Sandler et al., 1 Nov 2025).

6. Empirical results and scaling behavior

Across reasoning, code, and mathematical benchmarks, SpecDiff-2 reports state-of-the-art throughput among the compared speculative decoding methods. Its abstract summarizes the headline result as up to an average of $\gamma$5 tokens-per-second over previous baselines and up to $\gamma$6 average speed-up over standard decoding, without any loss of accuracy (Sandler et al., 1 Nov 2025).

The benchmark-level results are more specific. On Qwen2.5-72B with temperature $\gamma$7, mean speedup rises from $\gamma$8 for EAGLE-2 to $\gamma$9 for SpecDiff-2. On the same verifier with temperature $x_i$0, the mean improves from $x_i$1 to $x_i$2. For LLaMA-2-70B at temperature $x_i$3, mean speedup rises from $x_i$4 for EAGLE-2 to $x_i$5 for SpecDiff-2. The strongest individual result reported in the detailed tables is on HumanEval with Qwen2.5-72B at temperature $x_i$6, where SpecDiff-2 reaches $x_i$7 speedup with $x_i$8 (Sandler et al., 1 Nov 2025).

Setting	Baseline	SpecDiff-2
Qwen2.5-72B, Temp $x_i$9, mean	EAGLE-2: $i$0	$i$1
Qwen2.5-72B, Temp $i$2, mean	EAGLE-2: $i$3	$i$4
LLaMA-2-70B, Temp $i$5, mean	EAGLE-2: $i$6	$i$7
Qwen2.5-72B, HumanEval, Temp $i$8	EAGLE-2: $i$9	$$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$0

The comparison to the earlier, unaligned SpecDiff is equally important. On Math-500, Qwen2.5-72B improves from $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$1 with SpecDiff to $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$2 with SpecDiff-2, a $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$3 gain. Qwen2.5-14B improves from $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$4 to $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$5, a $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$6 gain. The paper attributes this improvement to the combination of streak-distillation and self-selection rather than to diffusion drafting alone (Sandler et al., 1 Nov 2025).

Because the decoding procedure is lossless, reported task accuracy is identical to that of the verifier. The paper therefore treats speed and accepted-streak statistics as the meaningful empirical variables. It further argues that accelerated decoding creates a secondary gain in budgeted reasoning: on Math-500 with a $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$7-second wall-time budget, SpecDiff-2 yields $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$8 accuracy over vanilla decoding and $$\Pr(\text{accept } x_i) = \min \left\{ 1, \frac{p(x_i \mid \mathbf{s})}{q(x_i \mid \mathbf{s})} \right\}.$$9 over unaligned SpecDiff by allowing substantially more reasoning tokens in the same latency budget (Sandler et al., 1 Nov 2025).

7. Relation to prior methods, limitations, and open questions

SpecDiff-2 is positioned against three families of prior methods. Relative to classic speculative sampling and DistillSpec-style alignment, it removes the autoregressive drafter bottleneck. Relative to EAGLE and EAGLE-2, it replaces autoregressive tree expansion with a non-autoregressive diffusion block proposal. Relative to the 2024 SpecDiff framework, it adds explicit alignment mechanisms tailored to the verifier–drafter mismatch (Christopher et al., 2024).

The method’s limitations are stated in practical rather than theoretical terms. It requires a discrete diffusion drafter and additional finetuning, which makes deployment more complex than plugging in a small autoregressive drafter. Its performance depends materially on $$\alpha(\mathbf{s}) \triangleq 1 - \tfrac{1}{2}\sum_{x \in \mathcal V} \left| P(x \mid \mathbf{s}) - Q(x \mid \mathbf{s}) \right|,$$0, $$\alpha(\mathbf{s}) \triangleq 1 - \tfrac{1}{2}\sum_{x \in \mathcal V} \left| P(x \mid \mathbf{s}) - Q(x \mid \mathbf{s}) \right|,$$1, drafter temperature, and verifier–drafter scale ratio. The paper also identifies several open directions: better scaling laws for drafter size relative to verifier size, analysis of alternative acceptance rules, cross-family and cross-tokenizer drafting, multiple specialized drafters for domains such as mathematics and code, and hardware-level kernel and cache optimizations (Sandler et al., 1 Nov 2025).

In that sense, SpecDiff-2 is best understood not merely as a faster drafter, but as a particular answer to a systems-and-alignment problem: how to make discrete diffusion viable as a speculative front-end for large autoregressive verifiers. Its contribution is to show that non-autoregressive drafting becomes competitive only when the objective is shifted from local token imitation to accepted-streak optimization, and when test-time diversity is converted into throughput by verifier-guided self-selection (Sandler et al., 1 Nov 2025).