MoE-Spec: Efficient Verification for MoE Decoding

• MoE-Spec is a verification-time method that enforces a fixed expert budget at each MoE layer, decoupling speculation depth from memory cost.
• It uses router-based ranking to select the most influential experts, thereby reducing the overhead from rarely activated experts.
• Empirical results show a 10–30% throughput improvement in speculative decoding without retraining or altering the draft model.

MoE-Spec is a verification-time method for speculative decoding in mixture-of-experts (MoE) LLMs that addresses a specific bottleneck of MoE verification: large draft trees activate many unique experts, causing expert-loading cost to grow enough to erode the speedups normally associated with speculative decoding (McDanel et al., 17 Feb 2026). It is explicitly training-free, leaves the draft model and the outer speculative decoding loop unchanged, and instead enforces a fixed expert capacity limit at each MoE layer during verification, loading only the experts that contribute most to the current draft tree while dropping the long tail of rarely used experts that drives memory pressure and bandwidth overhead. In the broader landscape of speculative MoE inference, it is distinct from methods centered on expert prefetching, offloading, or hardware–software co-design (Chen et al., 11 Oct 2025, Wang et al., 18 Nov 2025, Hoang et al., 3 Feb 2026, Li et al., 12 Feb 2026, Choi et al., 16 Apr 2026).

1. Verification bottleneck in speculative MoE decoding

Speculative decoding accelerates generation by having a draft model propose multiple future tokens and a target model verify them in parallel. For dense transformers, verifying more drafted tokens is often attractive because parameters are loaded once anyway, so larger draft structures can improve accepted length without proportionally increasing memory traffic. MoE verification behaves differently. If a draft tree is written as $$\mathcal{D} = \{\mathbf{h}_1, \ldots, \mathbf{h}_M\}$$ and each token activates its top-$k$ experts $\mathcal{T}_k(\mathbf{h}_t)$, then verification must execute over the expert union

$$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$

As $M$ grows, $|\mathcal{E}(\mathcal{D})|$ can approach the full expert count $N$, so speculative verification loses much of the sparse-activation advantage that makes MoE attractive in autoregressive decoding (McDanel et al., 17 Feb 2026).

The paper’s motivating example on OLMoE-1B-7B states that a 127-token tree activates 54 of 64 experts per layer on average, and its analysis also shows that at tree size 63 the top 32 of 64 experts capture 93% of the routing weight. This combination—a near-dense expert union but a heavy-tailed distribution of routing mass—motivates the central design of MoE-Spec: verification should not load every expert that appears once in the tree, because the straggler experts incur nearly the same loading cost as frequently used experts while contributing much less to the verifier’s output (McDanel et al., 17 Feb 2026).

2. Expert budgeting as the core mechanism

MoE-Spec keeps the standard MoE routing formulation. Given token representation $\mathbf{h} \in \mathbb{R}^d$, the router outputs

$$g_i(\mathbf{h}) = \frac{\exp(\mathbf{w}_i^\top \mathbf{h})}{\sum_{j=1}^{N} \exp(\mathbf{w}_j^\top \mathbf{h})},$$

with top-$k$ selected experts $k$0, and the full MoE layer output

$k$1

The method then introduces a per-layer expert budget $k$2 and constructs a shortlist $k$3 of size $k$4 for the current verification tree. Its main practical selector is router-based ranking:

$k$5

followed by choosing the top-$k$6 experts by $k$7. The result is a verifier that loads at most $k$8 experts per MoE layer regardless of how large the draft tree becomes (McDanel et al., 17 Feb 2026).

This is the sense in which MoE-Spec “decouples speculation depth from memory cost.” Under standard speculative verification, deeper trees activate more unique experts, so expert-loading cost rises with tree size. Under MoE-Spec, speculation depth can still increase accepted length, but verifier-side expert loading is bounded by a fixed capacity limit at each layer. The method is therefore a verifier-only modification: it does not retrain the model, does not alter draft-tree construction, and does not change the speculative acceptance logic inherited from EAGLE-style decoding (McDanel et al., 17 Feb 2026).

3. Ranking strategies and coverage policies

The paper studies three ways to construct the shortlist. Static ranking precomputes a fixed expert order from calibration data using

$k$9

Router-based ranking uses the runtime tree-specific score $\mathcal{T}_k(\mathbf{h}_t)$0 and is the main deployed method. Oracle ranking is an upper bound that greedily chooses experts to minimize reconstruction error relative to the full MoE output

$\mathcal{T}_k(\mathbf{h}_t)$1

but it requires executing all experts and is therefore not itself an acceleration mechanism (McDanel et al., 17 Feb 2026).

Once $\mathcal{T}_k(\mathbf{h}_t)$2 is chosen, MoE-Spec must define what happens when a token’s natural top-$\mathcal{T}_k(\mathbf{h}_t)$3 experts are not fully available. The paper studies truncation and substitution. Under truncation,

$\mathcal{T}_k(\mathbf{h}_t)$4

so missing experts are simply skipped. In the extreme case $\mathcal{T}_k(\mathbf{h}_t)$5, the MoE output becomes zero and only the residual path remains. Under substitution, routing is restricted to the shortlist:

$\mathcal{T}_k(\mathbf{h}_t)$6

so each token still receives exactly $\mathcal{T}_k(\mathbf{h}_t)$7 experts, but they are the top-$\mathcal{T}_k(\mathbf{h}_t)$8 available experts inside $\mathcal{T}_k(\mathbf{h}_t)$9. The paper reports that truncation and substitution perform similarly and uses substitution as the default because it preserves a constant active-expert count per token (McDanel et al., 17 Feb 2026).

4. Empirical behavior and reported performance

The evaluation covers OLMoE-1B-7B, Qwen3-30B-A3B, and Mixtral-8x7B; uses EAGLE-3 for OLMoE and Qwen3 and the original EAGLE for Mixtral; and reports results on GSM8K, MATH500, HumanEval, MBPP, and CNN/DailyMail under temperatures $$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$0 and $$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$1 on A100 80GB hardware. The main throughput claim is that MoE-Spec yields 10–30% higher throughput than state-of-the-art speculative decoding baselines at comparable quality, and the paper states that it improves speedup over EAGLE in 27 of 30 model-benchmark configurations while changing mean acceptance length by only 1.4% on average (McDanel et al., 17 Feb 2026).

The modelwise gains are uneven but consistent with the verifier-cost hypothesis: Mixtral-8x7B shows an average 27% relative speedup gain over EAGLE, Qwen3-30B-A3B 16%, and OLMoE-1B-7B 6%. A representative result is Mixtral-8x7B on MBPP at $$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$2, where EAGLE reports 1.7$$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$3 speedup and pass@1 = 45, while MoE-Spec reports 2.3$$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$4 speedup with pass@1 = 45. The paper also emphasizes that larger draft trees cease to help baseline EAGLE once expert loading becomes too expensive, whereas MoE-Spec can continue benefiting from larger trees because expert cost saturates at the budget. This produces an explicit latency–quality frontier: tighter budgets increase throughput further, but eventually degrade quality. Qwen3-30B-A3B exhibits the best trade-off, Mixtral degrades moderately at very tight budgets, and OLMoE-1B-7B is the most sensitive under aggressive budgeting (McDanel et al., 17 Feb 2026).

5. Position within speculative MoE inference

MoE-Spec belongs to a broader research line that asks how speculative decoding should be adapted to sparse expert architectures rather than dense transformers. One adjacent direction studies when speculative decoding is intrinsically favorable for sparse MoEs. In that line, MoESD argues that under medium batch sizes MoE can benefit more from speculative decoding than dense models, and introduces the metric

$$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$5

to separate acceptance-related effects from verifier-side system bottlenecks (Huang et al., 26 May 2025).

A different cluster of work treats speculative decoding primarily as a signal for memory management. SP-MoE uses the draft stage to prefetch likely experts before verification, together with a cutoff-layer policy and asynchronous batched I/O (Chen et al., 11 Oct 2025). MoE-SpeQ couples a quantized on-device draft model with proactive expert prefetching and offloading, framing MoE speculative decoding as a compute–I/O orchestration problem (Wang et al., 18 Nov 2025). SpecMD is broader than a single algorithm: it is a framework for benchmarking routing, prefetch, eviction, and miss-handling policies, and its main systems result is that speculative expert loading depends jointly on prediction, bandwidth, and eviction discipline rather than prediction quality alone (Hoang et al., 3 Feb 2026). MoE-SpAc repurposes speculative decoding as a short-horizon demand sensor for utility-guided prefetch, eviction, and CPU/GPU partitioning in heterogeneous edge settings (Li et al., 12 Feb 2026). ELMoE-3D pushes the same general agenda into hardware–software co-design by using Elastic Self-Speculative Decoding on a hybrid-bonded 3D-stacked architecture, where the draft structure also functions as an expert cache (Choi et al., 16 Apr 2026).

Against that background, MoE-Spec is unusually narrow and local: it does not prefetch, does not offload, does not introduce a new draft model, and does not require hardware-specific support. Its intervention point is the verifier’s MoE layer execution itself. A concise way to distinguish it is that the other systems mostly try to make the required experts arrive earlier or cheaper, whereas MoE-Spec asks which experts must be loaded at all for a given verification tree (McDanel et al., 17 Feb 2026).

6. Limitations, sensitivities, and open directions

The paper is explicit that MoE-Spec is not parameter-free. The expert budget $$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$6 must be selected for the deployment regime, and the selector is still heuristic. Router-based ranking is effective, but Oracle analysis shows that expert co-activation structure matters: Oracle can match Router’s reconstruction quality with about 25% fewer experts, which suggests that better shortlist construction could exploit redundancy among frequently co-activated experts rather than scoring each expert independently (McDanel et al., 17 Feb 2026).

Other limitations are structural. The reported selector adds 2–3% overhead to total generation time, which is small relative to the measured gains but nonzero. The experiments focus on single-request inference on A100 hardware. The method is derived for top-$$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$7 softmax-routed MoEs, so applying it to sigmoid-routed or otherwise different MoE architectures would require adapting the aggregation rule. The paper also does not study adaptive per-layer budgets; it uses a fixed scalar budget value per model and enforces that budget independently at each MoE layer. A plausible implication is that future verifier-side budgeting methods could become more effective by combining per-layer adaptation with co-activation-aware ranking, rather than using only the aggregate router score $$\mathcal{E}(\mathcal{D}) = \bigcup_{t=1}^{M} \mathcal{T}_k(\mathbf{h}_t).$$8 (McDanel et al., 17 Feb 2026).