---
title: 'MoE-SpAc: Speculative Inference for Heterogeneous MoE'
url: https://www.emergentmind.com/topics/moe-spac
type: topic
---

# MoE-SpAc: Speculative Inference for Heterogeneous MoE

MoE-SpAc is an MoE inference framework for heterogeneous edge scenarios that repurposes speculative decoding not merely as a compute accelerator, but as an informative lookahead sensor for memory management. It integrates a Speculative Utility Estimator to track expert demand, a Heterogeneous Workload Balancer to dynamically partition computation via online integer optimization, and an Asynchronous Execution Engine to unify the prefetching and eviction in the same utility space. In extensive experiments on seven benchmarks, it reports a 42% improvement in TPS over the SOTA SD-based baseline, and an average 4.04x speedup over all standard baselines [2603.09983].

## 1. Problem setting and motivation

MoE-SpAc is formulated for Mixture-of-Experts large language models deployed on memory-constrained or heterogeneous devices, where VRAM is insufficient for holding all experts and most expert weights must reside in CPU DRAM. In an MoE layer with experts $\mathcal{E} = \{E_1,\dots,E_N\}$ and Top-$k$ routing, the layer computes
$$
\begin{aligned}
\mathbf{s}&=\mathbf{W}_g \mathbf{h} \in \mathbb{R}^{N}\\
\mathcal{T}(\mathbf{h}) &= \text{Top-}k(\mathbf{s}) \subset\{1,\dots,N\}\\
g_i(\mathbf{h})&=\frac{\exp(\mathbf{s}_i)}{\sum_{j\in \mathcal{T}(\mathbf{h})}\exp(\mathbf{s}_j)}, \;\; i\in \mathcal{T}(\mathbf{h})\\
\mathbf{y} &= \sum\nolimits_{i \in \mathcal{T}(\mathbf{h})} g_i(\mathbf{h}) \cdot E_i(\mathbf{h}).
\end{aligned}
$$
Only $k \ll N$ experts are used per token, but all expert parameters must exist somewhere in memory [2603.09983].

The framework is motivated by the observation that standard autoregressive decoding provides a weak signal for expert-demand prediction. In standard AR decoding, an inference step generates one token, so for each expert $E_i$ the activation frequency satisfies $f_{i,t} \in \{0,1\}$. The paper characterizes this as a binary, low-information signal that makes it difficult to distinguish marginally hot experts from truly cold ones. Existing offloading systems therefore suffer either from heavy PCIe traffic caused by on-demand loads or from static or greedy partitioning that cannot adapt well to dynamic per-step expert usage [2603.09983].

MoE-SpAc treats speculative decoding differently. In conventional SD, a draft model generates $\gamma$ candidate tokens and the target model verifies them in parallel. MoE-SpAc adopts the same basic computation window, but interprets that window as an information source for memory management. Under SD, a single verification step processes $\gamma+1$ tokens in parallel, so expert activation frequency becomes $f_{i,t} \in [0,\gamma+1]$ rather than binary. This shift is central to the framework’s design [2603.09983].

## 2. Core design: speculative activation utility

The key idea is to convert speculative-decoding activation frequencies into a unified utility signal for scheduling. The paper formalizes ground-truth utility for expert $E_i$ at step $t+1$ as
$$
s_{i,t+1} = G(f_{i,t+1}),
$$
where $G:\mathbb{Z}_{\ge 0} \to \{0,\dots,K\}$ is monotone non-decreasing. Because $f_{i,t+1}$ is unknown at decision time, MoE-SpAc estimates utility online from historical frequencies and prior utility states [2603.09983].

The full system consists of three coupled components. The Speculative Utility Estimator consumes SD activation frequencies and maintains per-expert discrete utility $s_{i,t}\in\{0,\dots,K\}$. The Heterogeneous Workload Balancer solves a small online integer optimization problem at each MoE layer and SD step, selecting a threshold $\tau_t$ that partitions experts into a hot GPU set and a cold CPU set. The Asynchronous Execution Engine implements concurrent prefetching and eviction using the same utility space, with priority queues for prefetch requests and a utility-ordered resident set for eviction [2603.09983].

This architecture makes speculative decoding part of the runtime control loop rather than only part of the token-generation algorithm. A common misconception is that SD should be evaluated only through acceptance statistics or verifier-side compute amortization. MoE-SpAc instead uses SD as a lookahead sensor for expert-demand estimation and heterogeneous scheduling. A related line of work, MoESD, likewise argues that SD should not be treated as efficient only for dense models and introduces “target efficiency” to capture system-side effects beyond acceptance rate [2505.19645].

## 3. Speculative Utility Estimator

MoE-SpAc uses an inertial utility transition to update expert utilities from speculative activation frequencies. With fluctuation
$$
\Delta_{i,t} = f_{i,t} - f_{i,t-1},
$$
and expert-specific upward and downward thresholds $\theta^{\uparrow}_{i,t},\theta^{\downarrow}_{i,t}$, the update rule is
$$
s_{i, t+1} \leftarrow
\begin{cases}
\min(K, s_{i, t} + 1) & \text{if } \Delta_{i,t} \geq \theta^{\uparrow}_{i,t},\\
\max(0, s_{i, t} - 1) & \text{if } -\Delta_{i,t} \geq \theta^{\downarrow}_{i,t},\\
s_{i, t} & \text{otherwise}.
\end{cases}
$$
All experts start with $s_{i,1}=0$, and the initial thresholds are $\theta^{\uparrow}_{i,1}=\theta^{\downarrow}_{i,1}=\lfloor \gamma/2\rfloor$ [2603.09983].

Thresholds are then adapted by AdaBC, a moving-average calibration with forgetting factor $\lambda$. If $\Delta_{i,t}>0$,
$$
\theta^{\uparrow}_{i,t+1} = \lfloor (1 - \lambda)\theta^{\uparrow}_{i,t} + \lambda \cdot \Delta_{i,t} \rfloor,
$$
while if $\Delta_{i,t}<0$,
$$
\theta^{\downarrow}_{i,t+1} = \lfloor (1 - \lambda)\theta^{\downarrow}_{i,t} + \lambda \cdot |\Delta_{i,t}| \rfloor.
$$
In the implementation described by the paper, $\gamma=8$, $\lambda=0.1$, and $K=4$ [2603.09983].

The paper supplies three theoretical arguments for this estimator. First, under SD the activation signal becomes more informative: if AR activation is modeled as $X_i^{(1)} \sim \text{Bern}(p_i)$ and SD activation as $X_i^{(\gamma+1)} \sim \text{Bin}(\gamma+1,p_i)$, then $H(X_i^{(\gamma+1)}) \ge H(X_i^{(1)})$ and the signal-to-noise ratio improves by a factor of $\sqrt{\gamma+1}$. Second, a routing smoothness theorem bounds consecutive-step score variation:
$$
\|\mathbf{s}_t^l - \mathbf{s}_{t+1}^l\| \le C_R^l \cdot \delta_{in} \prod_{j=0}^{l-1} (1 + \sigma^j),
$$
supporting inertial updates. Third, the paper argues that SD gives more fault tolerance around decision thresholds because prediction error need not be zero to preserve a correct hot/cold classification [2603.09983].

Empirically, the paper reports that online hot/cold prediction accuracy on a MoE layer reaches about $0.85$ for MoE-SpAc, while the AR-based comparator is lower and more fluctuating. This suggests that the additional information contained in speculative activation frequencies is operationally important rather than merely theoretical [2603.09983].

## 4. Heterogeneous Workload Balancer and asynchronous execution

For each layer and SD step, the Heterogeneous Workload Balancer chooses a utility threshold $\tau$ so that experts with $s_{i,t}\ge\tau$ are treated as hot and targeted to GPU, while experts with $s_{i,t}<\tau$ are treated as cold and assigned to CPU or eviction. The optimization objective is
$$
\min_{\tau} |T_{cpu} - T_{gpu}|,
$$
where
$$
\begin{aligned}
T_{cpu} &= r_c(\tau) \cdot \gamma \cdot k \cdot T^{u}_{cpu}, \\
T_{gpu} &= r_g(\tau) \cdot b \cdot T^{u}_{gpu}.
\end{aligned}
$$
This is constrained by I/O and VRAM budgets:
$$
T^{u}_{io} \cdot n(\tau) \leq T_{total} + \frac{\gamma \cdot T^{u}_{draft}}{L_{target}},
$$
and
$$
W^{u}_e \cdot n(\tau) \leq \text{VRAM}_{left},
$$
with $0 < \tau \leq K$ and $\tau\in\mathbb{Z}$ [2603.09983].

Because $T_{cpu}(\tau)-T_{gpu}(\tau)$ is monotonic in $\tau$, the paper argues that the absolute imbalance objective is convex over the feasible integer interval and can be solved in $O(1)$ per layer and step by evaluating a small candidate set containing the boundary points and the crossing point. If the feasible set is empty, the method defaults to $\tau=K$ [2603.09983].

The execution engine then uses the same utility scores for both prefetch and eviction. Prefetching is organized as a multi-level priority queue, one queue per utility level. When the balancer outputs $\tau_t$, the prefetcher ignores utility levels below $\tau_t$ and serves queues from high utility to low. GPU-resident experts are managed by a state-ordered pool implemented as a Red–Black tree keyed by current utility; experts with $s_{i,t}<\tau_t$ become eviction candidates, while experts currently in use are marked with utility $K+1$ so they cannot be evicted mid-use. The implementation is built atop llama.cpp, with modifications to expert-id handling and a custom CUDA kernel that merges CPU and GPU expert outputs [2603.09983].

A related SD-aware offloading system, SP-MoE, also exploits drafting-time information, but it does so through speculative expert prefetching, a cutoff-layer policy, and pipelined batched I/O for verification-time availability [2510.10302]. MoE-SpAc differs in its explicit utility-space formulation and in its dynamic CPU/GPU partitioning via online threshold optimization [2603.09983].

## 5. Evaluation and empirical results

The main evaluation uses Qwen3-30B-A3B as the target MoE model and Qwen3-4B-FP8 as the draft model. Compatibility experiments additionally use DeepSeek-V2-Lite with an INT4 draft. The reported hardware setting is a single NVIDIA RTX 4090 with 24 GB VRAM connected over PCIe 4.0, and the experiments use batch size 1 to emulate edge or single-user scenarios. Benchmarks include MMLU-Pro, MT-Bench, HumanEval, GSM8K, Alpaca, CNN/DailyMail, and Natural Questions [2603.09983].

The headline result is an average 4.04x TPS speedup over all standard baselines, with a 41.9% average TPS improvement and a 24.6% average latency reduction relative to llama.cpp with SD. On DeepSeek-V2-Lite, the framework improves TPS by about 53% over llama.cpp+SD and reduces latency by about 45–49% on MT-Bench, HumanEval, and CNN/DailyMail. The paper states that MoE-SpAc is consistently the best TPS/latency configuration in its benchmark table [2603.09983].

The ablation study attributes these gains to the interaction of all three major components. Removing SpecUE by collapsing to binary utility ($K=1$) causes a large drop in TPS and higher latency. Removing AdaBC with fixed thresholds $\theta^{\uparrow}=3,\theta^{\downarrow}=1$ yields small but consistent degradation. Replacing the adaptive balancer with a static threshold $\tau=2$ also degrades performance. Removing the utility-guided prefetcher or evictor drops TPS to near-baseline levels. Running without SD reduces performance to the level of llama.cpp or HybriMoE, indicating that SD and utility-guided heterogeneous scheduling are jointly necessary in the reported setting [2603.09983].

Sensitivity analyses further show that when GPU cache ratio decreases, AdaBC and HetWB increase $\tau$ and shift more experts to CPU while maintaining a high TPS relative to baselines. The draft model consumes about 8% of VRAM and pushes the OOM threshold to about 21% effective cache, yet the paper reports that MoE-SpAc at 17% cache still outperforms baselines at higher ratios such as 25%. For the utility quantization parameter, $K<4$ hurts performance, while $K\ge 4$ yields similar performance and motivates the choice $K=4$ [2603.09983].

## 6. Positioning, related work, and limitations

MoE-SpAc belongs to a broader family of work on resource-constrained MoE deployment, but its emphasis is distinct. CoSMoEs targets on-device MoE viability through compact architectures, weight-decomposed experts, and Block-wise Expert Selection to reduce expert replacements during offloaded inference [2503.00245]. Sub-MoE addresses memory and storage through post-training expert merging in a shared subspace [2506.23266]. Those efforts primarily alter model structure or training-time routing behavior, whereas MoE-SpAc operates as an inference-time framework for heterogeneous scheduling [2603.09983].

Its closest conceptual neighbors are SD-aware inference systems. MoESD shows that, under medium batch sizes, MoE can benefit more from SD than dense models and argues that “target efficiency” captures system bottlenecks more comprehensively than acceptance rate alone [2505.19645]. MoE-SpAc is compatible with that perspective but moves the analysis into an execution framework: speculative windows are used to derive utilities, drive prefetch and eviction, and partition work between CPU and GPU [2603.09983]. SP-MoE likewise combines speculative decoding with offloading, but its mechanisms center on cross-model expert prediction, cutoff-layer control, and pipelined verification-time availability [2510.10302].

The paper states several limitations. Gains diminish when SD acceptance rate is very low. The implementation requires substantial changes to the inference stack, including utility tracking, asynchronous execution, and CPU/GPU expert splitting. The evaluation scope is single-GPU, batch-1, edge-oriented inference rather than multi-GPU or large-batch data-center serving. This suggests that MoE-SpAc is best understood as a framework specialized for heterogeneous edge scenarios rather than a universal replacement for all MoE serving systems [2603.09983].

A common misunderstanding is to reduce MoE-SpAc to “speculative decoding for MoE.” The paper’s own formulation is narrower and more specific: speculative decoding is repurposed as a lookahead sensor for expert activation and memory decisions, and the resulting speculative activation utility becomes the common currency for prediction, prefetch, eviction, and CPU/GPU assignment [2603.09983].

Source: https://www.emergentmind.com/topics/moe-spac