MoE-SpAc: Speculative Inference for Heterogeneous MoE

• MoE-SpAc is an MoE inference framework that leverages speculative decoding to create informative utility signals for expert memory management.
• It integrates a Speculative Utility Estimator, Heterogeneous Workload Balancer, and Asynchronous Execution Engine to dynamically partition computation between CPU and GPU.
• Extensive experiments report a 42% TPS improvement and a 4.04x speedup over standard baselines, highlighting its efficiency in memory-constrained edge scenarios.

MoE-SpAc is an MoE inference framework for heterogeneous edge scenarios that repurposes speculative decoding not merely as a compute accelerator, but as an informative lookahead sensor for memory management. It integrates a Speculative Utility Estimator to track expert demand, a Heterogeneous Workload Balancer to dynamically partition computation via online integer optimization, and an Asynchronous Execution Engine to unify the prefetching and eviction in the same utility space. In extensive experiments on seven benchmarks, it reports a 42% improvement in TPS over the SOTA SD-based baseline, and an average 4.04x speedup over all standard baselines (Li et al., 12 Feb 2026).

1. Problem setting and motivation

MoE-SpAc is formulated for Mixture-of-Experts LLMs deployed on memory-constrained or heterogeneous devices, where VRAM is insufficient for holding all experts and most expert weights must reside in CPU DRAM. In an MoE layer with experts $\mathcal{E} = \{E_1,\dots,E_N\}$ and Top-$k$ routing, the layer computes

$$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$

Only $k \ll N$ experts are used per token, but all expert parameters must exist somewhere in memory (Li et al., 12 Feb 2026).

The framework is motivated by the observation that standard autoregressive decoding provides a weak signal for expert-demand prediction. In standard AR decoding, an inference step generates one token, so for each expert $E_i$ the activation frequency satisfies $f_{i,t} \in \{0,1\}$. The paper characterizes this as a binary, low-information signal that makes it difficult to distinguish marginally hot experts from truly cold ones. Existing offloading systems therefore suffer either from heavy PCIe traffic caused by on-demand loads or from static or greedy partitioning that cannot adapt well to dynamic per-step expert usage (Li et al., 12 Feb 2026).

MoE-SpAc treats speculative decoding differently. In conventional SD, a draft model generates $\gamma$ candidate tokens and the target model verifies them in parallel. MoE-SpAc adopts the same basic computation window, but interprets that window as an information source for memory management. Under SD, a single verification step processes $\gamma+1$ tokens in parallel, so expert activation frequency becomes $f_{i,t} \in [0,\gamma+1]$ rather than binary. This shift is central to the framework’s design (Li et al., 12 Feb 2026).

2. Core design: speculative activation utility

The key idea is to convert speculative-decoding activation frequencies into a unified utility signal for scheduling. The paper formalizes ground-truth utility for expert $E_i$ at step $k$0 as

$k$1

where $k$2 is monotone non-decreasing. Because $k$3 is unknown at decision time, MoE-SpAc estimates utility online from historical frequencies and prior utility states (Li et al., 12 Feb 2026).

The full system consists of three coupled components. The Speculative Utility Estimator consumes SD activation frequencies and maintains per-expert discrete utility $k$4. The Heterogeneous Workload Balancer solves a small online integer optimization problem at each MoE layer and SD step, selecting a threshold $k$5 that partitions experts into a hot GPU set and a cold CPU set. The Asynchronous Execution Engine implements concurrent prefetching and eviction using the same utility space, with priority queues for prefetch requests and a utility-ordered resident set for eviction (Li et al., 12 Feb 2026).

This architecture makes speculative decoding part of the runtime control loop rather than only part of the token-generation algorithm. A common misconception is that SD should be evaluated only through acceptance statistics or verifier-side compute amortization. MoE-SpAc instead uses SD as a lookahead sensor for expert-demand estimation and heterogeneous scheduling. A related line of work, MoESD, likewise argues that SD should not be treated as efficient only for dense models and introduces “target efficiency” to capture system-side effects beyond acceptance rate (Huang et al., 26 May 2025).

3. Speculative Utility Estimator

MoE-SpAc uses an inertial utility transition to update expert utilities from speculative activation frequencies. With fluctuation

$k$6

and expert-specific upward and downward thresholds $k$7, the update rule is

$k$8

All experts start with $k$9, and the initial thresholds are $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$0 (Li et al., 12 Feb 2026).

Thresholds are then adapted by AdaBC, a moving-average calibration with forgetting factor $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$1. If $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$2,

$$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$3

while if $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$4,

$$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$5

In the implementation described by the paper, $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$6, $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$7, and $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$8 (Li et al., 12 Feb 2026).

The paper supplies three theoretical arguments for this estimator. First, under SD the activation signal becomes more informative: if AR activation is modeled as $$\begin{aligned} \mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\ \mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\ g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\ \mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}). \end{aligned}$$9 and SD activation as $k \ll N$0, then $k \ll N$1 and the signal-to-noise ratio improves by a factor of $k \ll N$2. Second, a routing smoothness theorem bounds consecutive-step score variation:

$k \ll N$3

supporting inertial updates. Third, the paper argues that SD gives more fault tolerance around decision thresholds because prediction error need not be zero to preserve a correct hot/cold classification (Li et al., 12 Feb 2026).

Empirically, the paper reports that online hot/cold prediction accuracy on a MoE layer reaches about $k \ll N$4 for MoE-SpAc, while the AR-based comparator is lower and more fluctuating. This suggests that the additional information contained in speculative activation frequencies is operationally important rather than merely theoretical (Li et al., 12 Feb 2026).

4. Heterogeneous Workload Balancer and asynchronous execution

For each layer and SD step, the Heterogeneous Workload Balancer chooses a utility threshold $k \ll N$5 so that experts with $k \ll N$6 are treated as hot and targeted to GPU, while experts with $k \ll N$7 are treated as cold and assigned to CPU or eviction. The optimization objective is

$k \ll N$8

where

$k \ll N$9

This is constrained by I/O and VRAM budgets:

$E_i$0

and

$E_i$1

with $E_i$2 and $E_i$3 (Li et al., 12 Feb 2026).

Because $E_i$4 is monotonic in $E_i$5, the paper argues that the absolute imbalance objective is convex over the feasible integer interval and can be solved in $E_i$6 per layer and step by evaluating a small candidate set containing the boundary points and the crossing point. If the feasible set is empty, the method defaults to $E_i$7 (Li et al., 12 Feb 2026).

The execution engine then uses the same utility scores for both prefetch and eviction. Prefetching is organized as a multi-level priority queue, one queue per utility level. When the balancer outputs $E_i$8, the prefetcher ignores utility levels below $E_i$9 and serves queues from high utility to low. GPU-resident experts are managed by a state-ordered pool implemented as a Red–Black tree keyed by current utility; experts with $f_{i,t} \in \{0,1\}$0 become eviction candidates, while experts currently in use are marked with utility $f_{i,t} \in \{0,1\}$1 so they cannot be evicted mid-use. The implementation is built atop llama.cpp, with modifications to expert-id handling and a custom CUDA kernel that merges CPU and GPU expert outputs (Li et al., 12 Feb 2026).

A related SD-aware offloading system, SP-MoE, also exploits drafting-time information, but it does so through speculative expert prefetching, a cutoff-layer policy, and pipelined batched I/O for verification-time availability (Chen et al., 11 Oct 2025). MoE-SpAc differs in its explicit utility-space formulation and in its dynamic CPU/GPU partitioning via online threshold optimization (Li et al., 12 Feb 2026).

5. Evaluation and empirical results

The main evaluation uses Qwen3-30B-A3B as the target MoE model and Qwen3-4B-FP8 as the draft model. Compatibility experiments additionally use DeepSeek-V2-Lite with an INT4 draft. The reported hardware setting is a single NVIDIA RTX 4090 with 24 GB VRAM connected over PCIe 4.0, and the experiments use batch size 1 to emulate edge or single-user scenarios. Benchmarks include MMLU-Pro, MT-Bench, HumanEval, GSM8K, Alpaca, CNN/DailyMail, and Natural Questions (Li et al., 12 Feb 2026).

The headline result is an average 4.04x TPS speedup over all standard baselines, with a 41.9% average TPS improvement and a 24.6% average latency reduction relative to llama.cpp with SD. On DeepSeek-V2-Lite, the framework improves TPS by about 53% over llama.cpp+SD and reduces latency by about 45–49% on MT-Bench, HumanEval, and CNN/DailyMail. The paper states that MoE-SpAc is consistently the best TPS/latency configuration in its benchmark table (Li et al., 12 Feb 2026).

The ablation study attributes these gains to the interaction of all three major components. Removing SpecUE by collapsing to binary utility ($f_{i,t} \in \{0,1\}$2) causes a large drop in TPS and higher latency. Removing AdaBC with fixed thresholds $f_{i,t} \in \{0,1\}$3 yields small but consistent degradation. Replacing the adaptive balancer with a static threshold $f_{i,t} \in \{0,1\}$4 also degrades performance. Removing the utility-guided prefetcher or evictor drops TPS to near-baseline levels. Running without SD reduces performance to the level of llama.cpp or HybriMoE, indicating that SD and utility-guided heterogeneous scheduling are jointly necessary in the reported setting (Li et al., 12 Feb 2026).

Sensitivity analyses further show that when GPU cache ratio decreases, AdaBC and HetWB increase $f_{i,t} \in \{0,1\}$5 and shift more experts to CPU while maintaining a high TPS relative to baselines. The draft model consumes about 8% of VRAM and pushes the OOM threshold to about 21% effective cache, yet the paper reports that MoE-SpAc at 17% cache still outperforms baselines at higher ratios such as 25%. For the utility quantization parameter, $f_{i,t} \in \{0,1\}$6 hurts performance, while $f_{i,t} \in \{0,1\}$7 yields similar performance and motivates the choice $f_{i,t} \in \{0,1\}$8 (Li et al., 12 Feb 2026).

6. Positioning, related work, and limitations

MoE-SpAc belongs to a broader family of work on resource-constrained MoE deployment, but its emphasis is distinct. CoSMoEs targets on-device MoE viability through compact architectures, weight-decomposed experts, and Block-wise Expert Selection to reduce expert replacements during offloaded inference (Huber et al., 28 Feb 2025). Sub-MoE addresses memory and storage through post-training expert merging in a shared subspace (Li et al., 29 Jun 2025). Those efforts primarily alter model structure or training-time routing behavior, whereas MoE-SpAc operates as an inference-time framework for heterogeneous scheduling (Li et al., 12 Feb 2026).

Its closest conceptual neighbors are SD-aware inference systems. MoESD shows that, under medium batch sizes, MoE can benefit more from SD than dense models and argues that “target efficiency” captures system bottlenecks more comprehensively than acceptance rate alone (Huang et al., 26 May 2025). MoE-SpAc is compatible with that perspective but moves the analysis into an execution framework: speculative windows are used to derive utilities, drive prefetch and eviction, and partition work between CPU and GPU (Li et al., 12 Feb 2026). SP-MoE likewise combines speculative decoding with offloading, but its mechanisms center on cross-model expert prediction, cutoff-layer control, and pipelined verification-time availability (Chen et al., 11 Oct 2025).

The paper states several limitations. Gains diminish when SD acceptance rate is very low. The implementation requires substantial changes to the inference stack, including utility tracking, asynchronous execution, and CPU/GPU expert splitting. The evaluation scope is single-GPU, batch-1, edge-oriented inference rather than multi-GPU or large-batch data-center serving. This suggests that MoE-SpAc is best understood as a framework specialized for heterogeneous edge scenarios rather than a universal replacement for all MoE serving systems (Li et al., 12 Feb 2026).

A common misunderstanding is to reduce MoE-SpAc to “speculative decoding for MoE.” The paper’s own formulation is narrower and more specific: speculative decoding is repurposed as a lookahead sensor for expert activation and memory decisions, and the resulting speculative activation utility becomes the common currency for prediction, prefetch, eviction, and CPU/GPU assignment (Li et al., 12 Feb 2026).