Mirror Speculative Decoding

• Mirror Speculative Decoding is an inference algorithm for LLMs that breaks the serial latency-acceptance tradeoff using early-exit proxies and branch-complete rollouts.
• It employs GPU and NPU parallelism with heterogeneous accelerator mapping to overlap draft generation and target verification, significantly reducing overall latency.
• Empirical evaluations show up to 5.8× speedups and about 30% improvement over previous methods, validating its effectiveness for server-scale LLM inference.

Mirror Speculative Decoding (Mirror-SD) is an inference algorithm for LLMs that breaks the serial latency-acceptance tradeoff of prior speculative decoding (SD) approaches. By orchestrating cross-accelerator parallelism between GPUs and NPUs, leveraging early-exit token proxies, and utilizing multi-token speculative streaming, Mirror-SD delivers substantially higher throughput and wall-time speedups for server-scale LLMs without compromising generation correctness or acceptance rate semantics (Bhendawade et al., 15 Oct 2025).

1. High-Level Architecture

Mirror-SD generalizes the classic speculative decoding framework, which accelerates slow, autoregressive LLM inference using a smaller draft model. Traditional SD proceeds serially: the draft proposes a “speculative window” of future tokens, which the target verifies, creating a coupling between draft generation latency and accepted token rate. Mirror-SD departs from this serial regime through two principal innovations:

Early-Exit Proxies & Branch-Complete Rollouts: At a selected early-exit layer $\ell_e$ in the target model, the algorithm extracts a top-$\kappa$ proxy distribution over possible next tokens. Only these $\kappa$ tokens and their log-probabilities are transmitted to the draft model via a lightweight token channel. The draft then initiates full $\gamma$-token branch-complete rollouts from each proxy root in parallel, speculating continuations for the target to verify.

Heterogeneous Accelerator Mapping: The target model runs on GPUs with Megatron tensor-parallel (TP) sharding, optimized for high-fidelity, latency-sensitive verification. The draft model runs on co-located NPUs under SPD sharding, requiring only two global collectives per forward pass. This design ensures that draft generation is entirely overlapped with the target’s suffix computation, effectively hiding draft latency as long as the draft computation time $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$ remains below the overlap budget $\Delta$.

This combination enables higher speculative window sizes (and thus higher acceptance rates) without incurring proportional increases in generation latency.

2. Stepwise Algorithmic Procedure

One Mirror-SD decoding step at position $t$ to generate a window of length $\gamma$ proceeds as follows:

1. Target Prefix & Early Exit: Compute activation up to $\ell_e$, extract the top-$\kappa$ token proxy, and transmit to the draft.
2. Target Suffix & Verification (in parallel): Run forward from $\kappa$0 to $\kappa$1 using cached key/value memory, then compare the default drafted branch against target outputs to determine the accepted prefix length $\kappa$2 and any required correction.
3. Draft Branch-Complete Speculation (in parallel): For each of the $\kappa$3 proxy roots, roll out $\kappa$4 tokens (branch-complete speculation), fully parallelized under SPD sharding.
4. Reuse or Fallback Decision: If the required correction exists in the draft hypothesis tree, it is reused; otherwise, the draft generates fresh continuations from the accepted prefix plus correction.
5. Commit Accepted Prefix: Accept $\kappa$5 tokens from the draft default branch.

This schedule is further detailed in the pseudocode provided in (Bhendawade et al., 15 Oct 2025), ensuring maximum overlap and low token-channel overhead.

3. Acceptance, Latency, and Stepwise Speedup Formulas

Define the accepted prefix length at step $\kappa$6 as $\kappa$7 and the window-normalized acceptance rate as

$\kappa$8

For vanilla SD, per-step latency is strictly serial:

$\kappa$9

where $\kappa$0 is the draft cost and $\kappa$1 the verification.

Mirror-SD introduces a parallel-overlap regime:

$\kappa$2

where $\kappa$3 is the rendezvous (token exchange) cost, and $\kappa$4 is the suffix compute time after early exit. If $\kappa$5, draft computation becomes “free,” decoupling accepted token rate from draft latency.

The stepwise speedup is given by

$\kappa$6

typically reducing to $\kappa$7 when $\kappa$8.

4. Orchestration of GPU/NPU Parallelism

The deployment architecture consists of clusters of Apple M2 Ultra nodes, each with a GPU and NPU linked via UltraFusion. Distributed computation uses:

• Target on GPU with Megatron TP ($\kappa$9): Each Transformer block performs two AllReduce collectives; total target communication

$\gamma$0

where $\gamma$1.

• Draft on NPU with SPD TP ($\gamma$2): Draft depth $\gamma$3 is split into segments requiring only two global synchronizations per forward pass; per-step communication is

$\gamma$4

with $\gamma$5.

• Cross-accelerator Token Channel: The rendezvous cost for token transmission is $\gamma$6s, negligible compared to compute latency.

By mapping the draft to NPUs and target to GPUs, and exploiting the low synchronization overhead of SPD, the system enables deep draft speculation to be overlapped under the target’s critical path.

5. Multi-Token Speculative Streaming

To further minimize $\gamma$7, Mirror-SD employs Speculative Streaming (SS) for the draft model. In SS, the draft maintains $\gamma$8 parallel lookahead streams, each forward pass potentially emitting multiple tokens. Let $\gamma$9 be tokens emitted at draft step $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$0; define

$T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$1

with total draft cost

$T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$2

where $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$3 and $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$4 are local computation and synchronization respectively. Empirically, $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$5–$T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$6, yielding a $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$7–$T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$8 reduction in draft compute steps and overall draft latency. SS is applied only to the draft, so acceptance semantics and target correctness remain lossless.

6. Empirical Evaluation on SpecBench

Mirror-SD is evaluated on models (Qwen3-14B, Qwen3-32B, Mistral-24B, OPT-66B) using the SpecBench suite, encompassing translation, summarization, question answering, mathematical reasoning, retrieval-augmented generation, and multi-turn conversation. Experiments are run with batch size $T_{\mathrm{draft}^{\mathrm{gen}}(\gamma)$9, $\Delta$0, $\Delta$1, and early exit at half-depth, comparing against Vanilla-SD, Medusa, Hydra, EAGLE, EAGLE-2, EAGLE-3, Recycling, PLD, SpS, REST, and Lookahead under both greedy ($\Delta$2) and stochastic ($\Delta$3) decoding.

Speedup results are summarized as follows:

Model/Task	EAGLE3	Medusa	Vanilla-SD	Mirror-SD
Qwen3-14B (Translation, $\Delta$4)	2.53×	1.65×	2.34×	4.13×
Qwen3-14B (Math Reasoning, $\Delta$5)	3.36×	2.12×	2.80×	5.32×
Qwen3-32B (Translation, $\Delta$6)	2.52×	1.56×	2.74×	3.72×

Mirror-SD consistently outperforms all baselines, delivering $\Delta$7–$\Delta$8 wall-time speedups and ~30% average improvement over the previous best (EAGLE-3).

7. Limitations and Prospective Research

Mirror-SD’s effectiveness is subject to several critical dependencies and operational nuances:

• Overlap Budget Sensitivity: Gains are realized only if $\Delta$9; overly large $t$0, $t$1, or shallow $t$2 choices reduce overlap, re-exposing draft latency.
• Hardware Heterogeneity: High-speed GPU/NPU interconnects are assumed; on platforms with lower inter-device bandwidth, rendezvous and collective communication may become bottlenecks.
• Draft Parity Requirement: Correctness requires that the draft branch used for verification matches vanilla SD’s AR draft distribution; deviation could bias acceptance.
• Batch Size Scaling: As batch size $t$3 increases, draft overhead grows ($t$4), reducing the overlap fraction; ablations show moderate batch sizes remain effective, but extreme batching diminishes benefit.

Potential avenues for further work include:

• Adaptive early-exit layer selection ($t$5) to maximize hardware overlap without accuracy loss.
• Dynamic scheduling of $t$6 according to uncertainty or fallback statistics.
• Porting to on-device SoCs and exploring real-time/energy tradeoffs.
• Integration with dynamic inference methods such as token merging or adaptive span selection.
• Architectural co-design of speculation engines within future accelerator hardware.

Mirror-SD demonstrates that systems–algorithm co-design enables breaking the serial speculative barrier in LLM inference, with multi-token streaming and branch-parallel rollouts on heterogeneous clusters enabling substantial decoding speedup while preserving correctness guarantees (Bhendawade et al., 15 Oct 2025).