MuLo-SD: Multi-Scale Local Speculative Decoding

• The paper introduces a hierarchical speculative decoding framework that leverages multi-scale drafting and localized verification to drastically reduce wall-clock latency in autoregressive models.
• MuLo-SD is a parallelized inference system that combines coarse-to-fine drafting with local token verification, ensuring both efficiency and high-fidelity outputs in chain-of-thought reasoning and image synthesis.
• Empirical benchmarks show that MuLo-SD achieves speedups of 1.5–2.1× while maintaining near-baseline performance metrics, demonstrating its potential for scalable generative inference.

Multi-Scale Local Speculative Decoding (MuLo-SD) is a parallelized inference framework designed to accelerate the generation of sequences in autoregressive (AR) models. By combining multi-scale drafting—coarse-to-fine for images or stepwise for text—with localized verification and resampling mechanisms, MuLo-SD offers substantially lower wall-clock latency than conventional token-level or single-scale speculative decoding methods. The framework has demonstrated state-of-the-art performance for both chain-of-thought reasoning in LLMs and high-fidelity image synthesis, with rigorous support across theoretical analysis and empirical benchmarks (Fu et al., 24 Jun 2025, Peruzzo et al., 8 Jan 2026).

1. Conceptual Foundations and Motivation

Traditional AR models, whether for text (sequences of tokens) or images (spatial grids of discrete tokens), generate outputs one unit at a time, resulting in $\mathcal{O}(T)$ or $\mathcal{O}(S^2)$ sequential operations for a sequence of length $T$ or image of size $S \times S$. Standard speculative decoding (SD) accelerates generation via a two-model setup: a lightweight drafter proposes several candidate tokens, and a full (target) model accepts or rejects these proposals in parallel via rejection sampling. However, token-level SD faces a fundamental acceptance rate bottleneck: the probability of an entire draft being accepted decays exponentially with the draft length, which sharply limits attainable speedup, especially as the sequence grows.

MuLo-SD advances the SD paradigm by incorporating both multi-scale drafting and local (stepwise/spatial) verification. For text, this exploits the hierarchical structure of chain-of-thought reasoning, while for images it leverages spatial redundancy and locality, addressing inefficiencies that arise from rasterscan rejection or failure to utilize multi-resolution structure (Fu et al., 24 Jun 2025, Peruzzo et al., 8 Jan 2026).

2. Multi-Scale Drafting and Local Verification

Text (Chain-of-Thought Reasoning)

• Step-level Drafting: Instead of speculating $\gamma$ tokens, MuLo-SD proposes $k$ full reasoning steps. Each step is generated autoregressively by a drafter $q$, then expanded in parallel by the target $p$.
• Inner Token-level SD: Within each reasoning step, conventional token-level SD operates, allowing additional fine-grained parallelism.
• Semantic Verification: Acceptance at the step level requires only that the draft step is semantically equivalent to the target step, not exactly token-matched. A verifier $V$ (e.g., LLM-as-Judge, similarity in embedding space) determines step acceptance.

Images (AR Synthesis)

• Low-to-High Resolution Drafting: MuLo-SD first generates image drafts at a reduced resolution using a low-res AR drafter $M_q$. Learned up-samplers $\mathcal{O}(S^2)$0 expand these drafts to high-res candidates.
• Local Verification: Rather than requiring global match, tokens are accepted if, within their codebook neighborhoods $\mathcal{O}(S^2)$1, the sum of target probabilities exceeds a threshold $\mathcal{O}(S^2)$2. Only small 2D spatial neighborhoods around rejected tokens are subject to resampling, drastically reducing unnecessary recomputation.

3. Formal Algorithmic Structure

MuLo-SD is best characterized as a two-stage hierarchical speculative inference method, summarized (for language) as:

• For each prefix, Drafter $\mathcal{O}(S^2)$3 generates $\mathcal{O}(S^2)$4 candidate steps, each internally utilizing token-level drafts of length $\mathcal{O}(S^2)$5.
• Target $\mathcal{O}(S^2)$6 expands all steps in batched parallel fashion.
• Verifier $\mathcal{O}(S^2)$7 checks each $\mathcal{O}(S^2)$8 draft–target pair for semantic equivalence; all accepted drafts are output, falling back to $\mathcal{O}(S^2)$9 for the failed step.

For images, the process may be summarized by:

• Down-sample current prefix, draft block of tokens at low-res.
• Up-sample candidates to high-res.
• Allow acceptance for tokens where the pooled target model probability within $T$0-nearest codebook neighbors exceeds $T$1; for rejected positions, expand to a local 2D spatial neighborhood and resample from $T$2.

Pseudocode (image domain, draft–verify–reject–resample loop):

$q$1 (Peruzzo et al., 8 Jan 2026)

4. Theoretical Speedup and Empirical Performance

Analytical Speedup

Let $T$3 be the per-token acceptance rate, $T$4 the drafter/target cost ratio, $T$5 the number of speculative steps (text), $T$6 the scale factor (images), and $T$7 the acceptance rate after local verification.

• Text: The overall theoretical speedup is $T$8, with $T$9 and $S \times S$0 denoting the step and token-level speedups. Under infinite concurrency,

$S \times S$1

and under hardware constraints $S \times S$2, optimal settings involve both $S \times S$3 and $S \times S$4 (Fu et al., 24 Jun 2025).

• Images: If $S \times S$5 is the full-sequence length and $S \times S$6 (for a scale ratio $S \times S$7), the ideal speedup is

$S \times S$8

Empirical Benchmarks

Text Domain:

• On GSM8K, token-only SD reaches $S \times S$91.4$\gamma$0 speedup, step-level lookahead $\gamma$1, and combined MuLo-SD achieves $\gamma$2, preserving answer accuracy within $\gamma$3.
• Across AIME, AMC, QA, and code generation, observed speedup of 1.5–2.0$\gamma$4 (Fu et al., 24 Jun 2025).

Image Domain:

• On MS-COCO 2017 (Tar-1.5B) at $\gamma$5p, MuLo-SD achieves 1.68$\gamma$6 speedup, with FID increase $\gamma$71.0 point, GenEval and DPG-Bench semantic alignment within 1% of baseline.
• Outperforms EAGLE-2 and LANTERN baselines, especially at high resolutions, maintaining negligible perceptual and semantic degradation (Peruzzo et al., 8 Jan 2026).

Method	Speedup	GenEval (%)	DPG-Bench (%)	FID	HPSv2
AR Sequential	1.00×	77.1	82.3	32.4	29.5
LANTERN	1.42×	75.4	82.3	31.1	28.5
MuLo-SD (4× up)	1.68×	76.3	82.0	32.8	28.4

5. Model Architecture and Verification Components

• Drafter/Target Models: For text, distillations such as DeepSeek-R1-Distill (1.5B $\gamma$8 32B) and Qwen3 were used. In images, Tar-1.5B serves as the AR backbone.
• Learnt Up-samplers: Composed of stacked masked convolutional ResNet blocks with pixel-shuffle, trained with mixed MSE, LPIPS, commitment and PatchGAN losses. Removal of adversarial or perceptual losses degrades output detail (Peruzzo et al., 8 Jan 2026).
• Verifier: LLM-as-Judge (Qwen2.5-7B-Instruct) and embedding-based methods (all-mpnet-base-v2, threshold $\gamma$9), the former providing highest alignment accuracy for text (Fu et al., 24 Jun 2025).

6. Ablation Studies and Design Choices

• Probabilistic Pooling: Pooling target probabilities over $k$0-nearest codebook neighbors raises acceptance rates at high speedup, with most gains realized via the threshold $k$1.
• Local Neighborhood Radius ($k$2): For images, $k$3 yields the most robust correction rate for $k$4, with larger radii leading to excessive resampling, smaller radii insufficient for correcting context.
• Multi-Branch Drafting: Exploring width $k$5 in text increases per-step acceptance but introduces overhead growing as $k$6. Optimal tradeoff in practice is at $k$7 (Fu et al., 24 Jun 2025).

7. Limitations and Prospects for Extension

• Step Segmentation: In text, step-level boundaries are detected via fixed delimiters (e.g., “\n\n”); adaptively learned segmentation could potentially improve draft–verifier alignment.
• Verifier Trade-offs: Embedding-based and target-scoring verifiers are faster, but risk higher misalignment rates; the design of lightweight, high-precision judges is an open question.
• Extension to Parallel Decoders: Application of MuLo-SD in combination with architectures such as ZipAR, or adaptation to 3D video or multi-modal decoding, remains an unexplored direction (Peruzzo et al., 8 Jan 2026).
• Adaptive Parameterization: The possibility of learning draft length $k$8, lookahead $k$9, or local radius $q$0 in an online fashion as a function of accept rates is an active area for further research.

MuLo-SD constitutes a unified, multi-scale, and locality-aware paradigm for AR model acceleration, establishing new state-of-the-art results in both language and vision, and providing a principled path forward for scalable, high-fidelity generative inference (Fu et al., 24 Jun 2025, Peruzzo et al., 8 Jan 2026).