Accelerating Speculative Diffusions via Block Verification

Abstract: Speculative decoding speeds up LLM inference by using a draft model to generate tokens, with an acceptance-rejection scheme that ensures that the output matches the target distribution. Adapting this to continuous diffusions is difficult because speculative sampling requires drawing from a residual distribution. While straightforward in discrete spaces, efficiently sampling this residual in continuous space is non-trivial. Consequently, existing diffusion adaptations either use computationally inefficient sampling techniques or rely on an alternative scheme. In this work, we introduce a novel scheme that efficiently implements the original speculative sampling mechanism for diffusion models. Our approach offers a critical advantage over current methods: it enables us to adapt block verification from LLMs to diffusions -- which provably improves the acceptance rate of drafts. Furthermore, we formalize and analyze the Free Drafter, a heuristic self-speculative drafter for diffusions that requires no training. By enabling block verification, our Free Drafter yields up to a 6.3% speedup over existing speculative methods with no additional training and negligible overhead beyond the existing parallel verification pass.

• The paper introduces an efficient block verification method for diffusion models that reduces residual sampling complexity via one-step orthogonal decomposition.
• It adapts speculative decoding from discrete LLMs to continuous data by employing stochastic residuals and block-wise acceptance probability maximization, yielding theoretical and empirical speedups.
• The Free Drafter mechanism recycles target outputs to achieve up to 6.3% wall-clock speedup on high-fidelity image benchmarks without compromising sample quality.

Accelerating Speculative Diffusions via Block Verification

Introduction and Motivation

Diffusion models have established themselves as premier generative models for high-dimensional continuous data, but their widespread adoption is hindered by the high computational cost associated with sampling. Each generated sample requires evaluating the (typically large) denoising network hundreds of times. While speculative decoding—originally introduced for accelerating inference in LLMs—offers a promising paradigm to exploit smaller “drafter” models for fast approximate sampling, its adaptation to the continuous, high-dimensional nature of diffusion models has faced significant algorithmic and computational challenges. In contrast to discrete settings, efficiently sampling from the residual distribution specified by the verification step is highly nontrivial in continuous space.

This paper, "Accelerating Speculative Diffusions via Block Verification" (2606.13426), addresses these fundamental challenges by introducing a practical, efficient implementation of speculative sampling for diffusion models, enabling block verification (previously restricted to discrete autoregressive models like LLMs). The work further provides a rigorous analysis of the method's optimality, computational complexity, and practical speedups, grounded both theoretically and empirically.

Technical Contributions

Efficient $\Gamma$-Maximal Coupling in Diffusions

A cornerstone of speculative decoding is maximal coupling: given a drafter $p$ and a target $q$, one constructs a coupling of samples with the highest possible chance of identity, using the residual distribution $r_\Gamma$. While trivial for LLMs, diffusion models’ continuous space complicates direct residual sampling, which prior works resolved only with expensive rejection sampling or suboptimal deterministic corrections (e.g., reflection coupling).

The authors introduce a low-complexity, one-step algorithm to efficiently sample from the residual $r_\Gamma$ between two multivariate Gaussian predictions (from drafter and target), reducing the sampling to a tractable 1D problem. This is achieved via an orthogonal decomposition: noise is split into a projection along the mean difference direction plus an orthogonal Gaussian variable. Sampling the univariate marginal is further streamlined using an analytic CDF, supporting efficient inverse sampling.

Figure 1: Plot of the \pdf $$f(u) \propto \max\{ 0, c \mathcal{N}(u; 0, 1) - \mathcal{N}(u - v; 0, 1) \}$$ across varying mean shifts, illustrating the structure of the univariate residual.

Figure 2: Plot of the PDF $$f(u) \propto \max\{ 0, c \mathcal{N}(u; 0, 1) - \mathcal{N}(u - v; 0, 1) \}$$ across varying scale factors, showing rapid adaptation of the residual's support.

Block Verification for Continuous Speculative Diffusion

Block verification, as realized in LLMs, improves computational throughput by validating entire drafter-proposed blocks in parallel—jointly optimizing acceptance rates and minimizing redundant computation. By making the residual sampling tractable for diffusions, the authors adapt block verification, shifting from sample-wise to block-wise acceptance probability maximization. This involves a recursive scheme for computing acceptance probabilities and a generalization of the residual distribution parameterized by block-level statistics. The analysis proves that deterministic residuals (e.g., reflection) are fundamentally incompatible with block verification in the continuous setting; stochastic correction, as provided by their decomposition method, is necessary.

The Free Drafter

To further reduce overhead, the work formalizes the "Free Drafter," a self-speculative strategy where the drafter’s predictions are derived by recycling target model outputs (or their extrapolation) from previous rounds, obviating extra passes through the target network. While the Frozen Drafter—for which a single target evaluation is reused for an entire draft—has superior theoretical block efficiency, in practice its cost is comparable to the full model call. The Free Drafter sustains high speedup with negligible additional computation.

Empirical Results

Experiments across multiple high-fidelity image generation benchmarks (e.g., ImageNet, CelebA, CIFAR10, LSUN) show up to 6.3% wall-clock speedup over prior speculative diffusion approaches at identical sample quality (as measured by FID). The gains from block verification are consistent and pronounced at higher numbers of denoising steps or on larger images/models, due to favorable scaling of acceptance rates and block efficiency.

Crucially, the Free Drafter—despite reduced theoretical block efficiency—always yields greater net speedups than Frozen Drafter due to its minimal compute and memory footprint.

Figure 3: Empirical verification acceptance probability across denoising steps on ImageNet LDM, highlighting acceptance decay near the clean image and the effect of block verification.

Implications and Theoretical Significance

From a theoretical lens, this work closes a longstanding gap between speculative decoding’s rigorous guarantees and the intractability of continuous maximal coupling. It establishes that for neural SDE-based generative models, block-level speculative sampling with exact distributional fidelity is feasible and practical. The formal impossibility of deterministic residuals in block verification precludes shortcut heuristics, emphasizing the necessity of the developed stochastic decompositions.

Practically, the method integrates seamlessly with parallel hardware, as verification can be vectorized, and removes a primary bottleneck for scaling diffusion-based generation to large domains and real-time inference. The ability to compose these speculative schemes with other acceleration methods—such as improved ODE solvers, score distillation, or model compression—suggests broad applicability across generative AI.

Moreover, self-speculative drafting (Free Drafter) sets the stage for a new class of autoregressive or Markovian acceleration protocols, blurring the line between teacher-student distillation and online sample proposal.

Limitations and Future Directions

The framework is constrained to stochastic samplers and requires that drafter and target share the underlying diffusion schedule. Memory overhead may increase due to parallel verification. Application to deterministic samplers is not addressed here.

Potential future extensions include:

• Application and optimized implementation for second-order SDE and Langevin integrators in molecular/physical simulation, per recent evidence in molecular dynamics [see references in the paper].
• Adaptive or learned drafter strategies that trade off sample fidelity and raw throughput.
• Integration with hardware-aware parallelization, dynamic batching, and mixed-precision inference.

Conclusion

This work delivers a rigorous, computationally practical realization of block verification for speculative sampling in diffusion models, yielding provable speedups without any degradation in output distribution. The method is efficient, hardware-friendly, and theoretically optimal in the sense of maximal acceptance. By formalizing and empirically endorsing the Free Drafter, the authors enable broad deployment of speculative sampling in continuous generative models, pointing toward scaling diffusion inference to unprecedented speeds with guaranteed sample quality.