SPICE: Speculative Prefetching with Low-Rank Expert Surrogates and Heterogeneous Orchestration for MoE Inference Acceleration

Abstract: Mixture-of-Experts (MoE) models are increasingly used in LLMs because sparse activation decouples model capacity from compute cost. However, the large expert parameter footprint often exceeds GPU memory capacity, making inference latency dominated by the host-to-device PCIe transfers for expert loading. To address these challenges, this paper presents SPICE, a speculative prefetching framework for MoE offloading that combines lightweight expert prediction with confidence-aware CPU-GPU orchestration. On one hand, SPICE builds a lightweight draft model aligned with the target MoE architecture, using a confidence-aware adaptive lookahead algorithm to prefetch high-confidence experts. On the other hand, when speculative predictions miss, SPICE switches to a cost-aware CPU-GPU heterogeneous orchestration: low-confidence misses are approximated by the resident shared expert with low rank expert (LoRE) surrogates, while exact residual work is offloaded to the CPU and executed asynchronously in parallel with ongoing GPU computation. Evaluated on DeepSeek-V2-Lite and Qwen2-57B-A14B across diverse GPU platforms, SPICE achieves up to 3.12 speedup in Time Per Output Token (TPOT) with minimal quality loss, showing that effective MoE offloading requires not only predicting future experts, but also deciding which misses deserve approximation, which require exact recovery, and where exact residual work should execute.

• The paper proposes a system with three mechanisms for accelerating Mixture-of-Experts (MoE) inference: adaptive speculative prefetching, low-confidence miss substitution using LoRE surrogates, and heterogeneous CPU-GPU orchestration.
• SPICE achieves up to 3.12× speedup in Time Per Output Token (TPOT) over prior systems, with 3.0–3.9 percentage-point accuracy degradation on GSM8K and HumanEval.
• Evaluated frameworks include DeepSeek-V2-Lite and Qwen2-57B-A14B; SPICE successfully balances energy efficiency and latency, drawing more power but reducing the total energy needed, leading to a reduction of 51% in fallback rates from CG-MoE.

SPICE addresses a central deployment problem for Mixture-of-Experts (MoE) LLMs: when aggregate expert weights exceed GPU high-bandwidth memory (HBM), inference latency is dominated by host-to-device PCIe transfers rather than computation. The paper reports that expert loading accounts for 73–88% of per-layer latency across three representative GPUs, transforming MoE inference from a compute-bound into an I/O-bound workload. SPICE combines three mechanisms: structurally aligned speculative prefetching of experts, low-confidence miss substitution using resident shared experts augmented with Low-Rank Expert (LoRE) surrogates, and cost-aware CPU–GPU heterogeneous orchestration for misses requiring exact recovery. Evaluated on DeepSeek-V2-Lite and Qwen2-57B-A14B, it achieves up to 3.12× speedup in Time Per Output Token (TPOT) over prior systems, with 3.0–3.9 percentage-point accuracy degradation on GSM8K and HumanEval.

Problem setting and motivation

In an MoE layer, a router selects the top-$K$ routed experts per token while shared experts remain permanently active. Non-expert parameters and the KV cache reside on GPU; expert FFN weights must be offloaded to host DRAM. A cache miss places routing, weight transfer, expert computation, and aggregation serially on the critical path. The authors identify three coupled conditions under which conventional prefetching fails: (1) fixed lookahead depths mismatch layer- and token-dependent routing dynamics; (2) prefetch misses force synchronous reloads that erase prefetching's benefit; and (3) aggressive lookahead saturates PCIe bandwidth and evicts useful cached experts, delaying even correctly predicted transfers. These three scenarios—on-demand loading stalls, miss-induced blocking, and PCIe contention under speculative traffic—motivate a system that decides not only which experts to prefetch but also how to serve misses and where to execute residual work.

Speculative prefetching with adaptive lookahead

SPICE uses a lightweight draft model architecturally aligned with the target MoE to forecast expert usage over future layers without altering the decoding trajectory. Anchored at the current hidden state, the draft recursively projects states through layer-specific low-rank transitions, producing predicted Top-$K$ sets for each lookahead depth $\delta \in [1, H]$. Predicted sets are enqueued as asynchronous prefetch intents on a bounded-depth, low-priority copy queue so speculative traffic cannot monopolize the PCIe copy engine.

The horizon is chosen adaptively from two signals. First, a hardware-derived lower bound $H_{\min}$ ensures the transfer time $P \cdot M_e / B_{\mathrm{PCIe}}$ fits within the per-layer compute window $T_{\mathrm{attn}} + K \cdot T_{\mathrm{expert}}$. Second, prediction confidence—the probability mass assigned to the predicted expert set—is used to halt speculation early once depth exceeds $H_{\min}$ and confidence falls below threshold $\tau$. The paper shows confidence varies substantially across layers and workloads (HumanEval vs. GSM8K), which empirically justifies depth-adaptive halting over a fixed horizon.

Low-rank surrogate substitution

For low-confidence misses, SPICE avoids both synchronous reloads and CPU execution by approximating the missing routed expert as a correction atop the always-resident shared expert:

$$\tilde{y}_e^{(l)} = S^{(l)}(h^{(l)}) + B_e^{(l)} A_e^{(l)} h^{(l)},$$

where $A_e^{(l)} \in \mathbb{R}^{r \times d}$ and $K$0 with $K$1. The LoRE modules are fit offline by least-squares regression on calibration traces, minimizing the residual between the exact routed expert output and the shared-expert baseline per (layer, expert) pair. This design confines approximation to a controlled, lossy path: substitution applies only when routing confidence is low, while all other misses receive exact recovery. Notably, this mechanism presupposes an architecture with native shared experts (as in DeepSeek-V2 and Qwen-MoE); its applicability to MoE designs lacking shared experts is not addressed.

CPU–GPU heterogeneous orchestration

Misses that cannot be safely approximated are partitioned into a fetch set (transferred H2D and executed on GPU) and a CPU set (executed exactly on host-resident weights). The scheduler consults current PCIe pressure and a per-layer cost table, selecting a GPU-fetch split only when the estimated end-to-end recovery cost improves over all-CPU service by a safety margin. Execution proceeds by staging the hidden state in pinned memory, launching CPU residual computation concurrently with asynchronous H2D copies of fetched experts, then merging CPU and GPU outputs. Because the CPU path uses unmodified target weights, non-substituted misses preserve exact model semantics; only the LoRE path introduces approximation error.

Evaluation results

Across RTX 5090, RTX 4060, and A800 platforms, SPICE achieves 2.04–2.70× TPOT speedup over AdapMoE and consistently outperforms CG-MoE, with speedups of 2.44–3.12× sustained as prompt length grows from 64 to 1,024 tokens. Quality costs are modest: GSM8K accuracy drops 3.3 points for DeepSeek-V2-Lite (70.8 → 67.5) and 3.5 points for Qwen2-57B-A14B BF16 (87.6 → 84.1), with HumanEval Pass@1 drops of 2.4–3.9 points.

The energy and traffic analysis yields the paper's most instructive trade-off findings. Naive on-demand loading draws the least power yet consumes the most energy (5.51 J/token) because synchronous loading prolongs execution; SPICE consumes 3.48 J/token despite drawing more power. On transfer behavior, LRU and CG-MoE minimize H2D volume (30.2 GB) but leave 15.7% of accesses to fallback, whereas AdapMoE cuts fallback to 6.1% at 75.7 GB of traffic. SPICE attains the lowest fallback rate (3.0%) at intermediate traffic (54.7 GB)—a 28% traffic reduction and 51% fallback reduction relative to AdapMoE—indicating improved expert availability without proportional transfer overhead. PCIe utilization reaches 82–91%, versus 38–45% for AdapMoE, supporting the claim that multi-layer prediction exposes transfers early enough to overlap communication with computation rather than merely moving more data.

An ablation isolating four partial variants (deep vs. shallow prediction crossed with fetch-all/CPU-fallback vs. transfer scheduling) shows the full system reduces TPOT by 23.9% relative to the best ablation, confirming that cross-layer prediction and confidence-aware scheduling are complementary rather than redundant.

Limitations and open questions

Several constraints bound the reported results. First, the quality gains rest on a lossy substitution path whose error is characterized only through end-to-end benchmark deltas; no per-layer analysis of LoRE approximation error or its interaction with long-generation drift is provided. Second, the surrogate mechanism depends on architectures with resident shared experts, leaving coverage of shared-expert-free MoE designs unresolved. Third, the evaluation covers two models and context lengths up to 1,024 tokens; behavior at much longer contexts, where KV-cache growth shrinks the expert cache budget, remains unmeasured. Fourth, the energy accounting includes draft-model and CPU-recovery overheads, but the paper does not report how sensitive the fetch-versus-CPU split decision is to miscalibrated cost tables under varying batch sizes or concurrent serving loads. Finally, the ablation is conducted only under the exact-residual setting, so the marginal contribution of LoRE substitution alone versus combined with orchestration is not separately quantified.

Conclusion

SPICE reframes MoE offloading as a joint prediction-and-scheduling problem: predicting future expert demand with confidence-aware adaptive lookahead, serving low-confidence misses through resident shared-expert-plus-LoRE surrogates, and dispatching exact residual work between GPU fetch and asynchronous CPU execution based on PCIe pressure. The combination delivers up to 3.12× TPOT improvement with single-digit accuracy loss and the best observed fallback/traffic trade-off among compared policies. The main open questions concern generalization beyond shared-expert architectures, longer-context regimes, and isolated attribution of each recovery mechanism's contribution.