---
title: 'MuLo-SD: Multi-Scale Local Speculative Decoding'
url: https://www.emergentmind.com/topics/multi-scale-local-speculative-decoding-mulo-sd
type: topic
---

# MuLo-SD: Multi-Scale Local Speculative Decoding

Multi-Scale Local Speculative Decoding (MuLo-SD) is a parallelized inference framework designed to accelerate the generation of sequences in autoregressive (AR) models. By combining multi-scale drafting—coarse-to-fine for images or stepwise for text—with localized verification and resampling mechanisms, MuLo-SD offers substantially lower wall-clock latency than conventional token-level or single-scale speculative decoding methods. The framework has demonstrated state-of-the-art performance for both chain-of-thought reasoning in language models and high-fidelity image synthesis, with rigorous support across theoretical analysis and empirical benchmarks [2506.19830], [2601.05149].

## 1. Conceptual Foundations and Motivation

Traditional AR models, whether for text (sequences of tokens) or images (spatial grids of discrete tokens), generate outputs one unit at a time, resulting in $\mathcal{O}(T)$ or $\mathcal{O}(S^2)$ sequential operations for a sequence of length $T$ or image of size $S \times S$. Standard speculative decoding (SD) accelerates generation via a two-model setup: a lightweight drafter proposes several candidate tokens, and a full (target) model accepts or rejects these proposals in parallel via rejection sampling. However, token-level SD faces a fundamental acceptance rate bottleneck: the probability of an entire draft being accepted decays exponentially with the draft length, which sharply limits attainable speedup, especially as the sequence grows.

MuLo-SD advances the SD paradigm by incorporating both multi-scale drafting and local (stepwise/spatial) verification. For text, this exploits the hierarchical structure of chain-of-thought reasoning, while for images it leverages spatial redundancy and locality, addressing inefficiencies that arise from rasterscan rejection or failure to utilize multi-resolution structure [2506.19830], [2601.05149].

## 2. Multi-Scale Drafting and Local Verification

### Text (Chain-of-Thought Reasoning)

- **Step-level Drafting:** Instead of speculating $\gamma$ tokens, MuLo-SD proposes $k$ full reasoning steps. Each step is generated autoregressively by a drafter $q$, then expanded in parallel by the target $p$.
- **Inner Token-level SD:** Within each reasoning step, conventional token-level SD operates, allowing additional fine-grained parallelism.
- **Semantic Verification:** Acceptance at the step level requires only that the draft step is semantically equivalent to the target step, not exactly token-matched. A verifier $V$ (e.g., LLM-as-Judge, similarity in embedding space) determines step acceptance.

### Images (AR Synthesis)

- **Low-to-High Resolution Drafting:** MuLo-SD first generates image drafts at a reduced resolution using a low-res AR drafter $M_q$. Learned up-samplers $U$ expand these drafts to high-res candidates.
- **Local Verification:** Rather than requiring global match, tokens are accepted if, within their codebook neighborhoods $B_k(\cdot)$, the sum of target probabilities exceeds a threshold $\tau$. Only small 2D spatial neighborhoods around rejected tokens are subject to resampling, drastically reducing unnecessary recomputation.

## 3. Formal Algorithmic Structure

MuLo-SD is best characterized as a two-stage hierarchical speculative inference method, summarized (for language) as:

- For each prefix, Drafter $q$ generates $k$ candidate steps, each internally utilizing token-level drafts of length $\gamma$.
- Target $p$ expands all steps in batched parallel fashion.
- Verifier $V$ checks each $(s_j, \hat{s}_j)$ draft–target pair for semantic equivalence; all accepted drafts are output, falling back to $p$ for the failed step.

For images, the process may be summarized by:

- Down-sample current prefix, draft block of tokens at low-res.
- Up-sample candidates to high-res.
- Allow acceptance for tokens where the pooled target model probability within $k$-nearest codebook neighbors exceeds $\tau$; for rejected positions, expand to a local 2D spatial neighborhood and resample from $M_p$.

##### Pseudocode (image domain, draft–verify–reject–resample loop):

```pseudo
Given draft block \tilde{x}_{n+1…n+L}, compute target p_i(·) in parallel
Initialize rejection set R_T ← ∅
For t = 1…L:
    If    sum_{v in B_k(\tilde{x}_{n+t})} p_{n+t}(v) ≥ τ
        accept: x_{n+t} ← \tilde{x}_{n+t}
    else
        reject: R_T ← R_T ∪ {n+t}
Let i₀ = min R_T
Define R_X ← union over all N((u_i,v_i); ℓ) for i ∈ R_T
For each j in sorted R_X:
    sequentially resample x_j ∼ M_p(·| x_{<j})
Append accepted & resampled tokens to prefix
```
[2601.05149]

## 4. Theoretical Speedup and Empirical Performance

### Analytical Speedup

Let $\alpha$ be the per-token acceptance rate, $c$ the drafter/target cost ratio, $k$ the number of speculative steps (text), $r$ the scale factor (images), and $a$ the acceptance rate after local verification.

- **Text:** The overall theoretical speedup is $h(k, \gamma) = f(k) \cdot g(\gamma)$, with $f(k)$ and $g(\gamma)$ denoting the step and token-level speedups. Under infinite concurrency,
  $$ h(k,\gamma) = f(k) \times g(\gamma) $$
  and under hardware constraints $(k \cdot \gamma \leq M)$, optimal settings involve both $k \geq 2$ and $\gamma \geq 2$ [2506.19830].

- **Images:** If $T_p$ is the full-sequence length and $T_q = T_p/r^2$ (for a scale ratio $r$), the ideal speedup is
  $$
  S = \dfrac{T_p}{(1-a) \cdot T_p + T_q}
  $$

### Empirical Benchmarks

**Text Domain:**

- On GSM8K, token-only SD reaches $\sim$1.4$\times$ speedup, step-level lookahead $f(6)\approx1.7\times$, and combined MuLo-SD achieves $2.11\times$, preserving answer accuracy within $\pm2\%$.
- Across AIME, AMC, QA, and code generation, observed speedup of 1.5–2.0$\times$ [2506.19830].

**Image Domain:**

- On MS-COCO 2017 (Tar-1.5B) at $1024$p, MuLo-SD achieves 1.68$\times$ speedup, with FID increase $<$1.0 point, GenEval and DPG-Bench semantic alignment within 1% of baseline.
- Outperforms EAGLE-2 and LANTERN baselines, especially at high resolutions, maintaining negligible perceptual and semantic degradation [2601.05149].

| Method            | Speedup | GenEval (%) | DPG-Bench (%) | FID  | HPSv2 |
|-------------------|---------|-------------|---------------|------|-------|
| AR Sequential     | 1.00×   | 77.1        | 82.3          | 32.4 | 29.5  |
| LANTERN           | 1.42×   | 75.4        | 82.3          | 31.1 | 28.5  |
| MuLo-SD (4× up)   | 1.68×   | 76.3        | 82.0          | 32.8 | 28.4  |

## 5. Model Architecture and Verification Components

- **Drafter/Target Models:** For text, distillations such as DeepSeek-R1-Distill (1.5B $\rightarrow$ 32B) and Qwen3 were used. In images, Tar-1.5B serves as the AR backbone.
- **Learnt Up-samplers:** Composed of stacked masked convolutional ResNet blocks with pixel-shuffle, trained with mixed MSE, LPIPS, commitment and PatchGAN losses. Removal of adversarial or perceptual losses degrades output detail [2601.05149].
- **Verifier:** LLM-as-Judge (Qwen2.5-7B-Instruct) and embedding-based methods (all-mpnet-base-v2, threshold $\theta$), the former providing highest alignment accuracy for text [2506.19830].

## 6. Ablation Studies and Design Choices

- **Probabilistic Pooling:** Pooling target probabilities over $k$-nearest codebook neighbors raises acceptance rates at high speedup, with most gains realized via the threshold $\tau$.
- **Local Neighborhood Radius ($\ell$):** For images, $\ell=3$ yields the most robust correction rate for $512 \rightarrow 1024p$, with larger radii leading to excessive resampling, smaller radii insufficient for correcting context.
- **Multi-Branch Drafting:** Exploring width $W>1$ in text increases per-step acceptance but introduces overhead growing as $W^k$. Optimal tradeoff in practice is at $W\sim2$ [2506.19830].

## 7. Limitations and Prospects for Extension

- **Step Segmentation:** In text, step-level boundaries are detected via fixed delimiters (e.g., “\n\n”); adaptively learned segmentation could potentially improve draft–verifier alignment.
- **Verifier Trade-offs:** Embedding-based and target-scoring verifiers are faster, but risk higher misalignment rates; the design of lightweight, high-precision judges is an open question.
- **Extension to Parallel Decoders:** Application of MuLo-SD in combination with architectures such as ZipAR, or adaptation to 3D video or multi-modal decoding, remains an unexplored direction [2601.05149].
- **Adaptive Parameterization:** The possibility of learning draft length $\gamma$, lookahead $k$, or local radius $\ell$ in an online fashion as a function of accept rates is an active area for further research.

MuLo-SD constitutes a unified, multi-scale, and locality-aware paradigm for AR model acceleration, establishing new state-of-the-art results in both language and vision, and providing a principled path forward for scalable, high-fidelity generative inference [2506.19830], [2601.05149].

Source: https://www.emergentmind.com/topics/multi-scale-local-speculative-decoding-mulo-sd