MoEpic: Efficient MoE Inference System

• MoEpic is an inference system for Mixture-of-Experts models that uses vertical expert splitting to mitigate VRAM constraints.
• It employs segmented caching and speculative prefetching to substantially lower GPU costs and inference latency by up to 65.73%.
• A dynamic, fixed-point iteration-based VRAM allocation algorithm optimizes expert split ratios, balancing transfer latency with computational overlap.

Searching arXiv for MoEpic and closely related MoE offloading/serving papers to ground the article in current literature. MoEpic is an inference system for Mixture-of-Experts (MoE) LLMs that targets the VRAM bottleneck created by expert-heavy architectures under CPU offloading. It introduces an expert split mechanism in which each expert is vertically divided into top and bottom segments, caches the top segments of hot experts, speculatively prefetches the experts predicted to be activated in the next layer, and uses a divide-and-conquer algorithm based on fixed-point iteration to determine each layer’s VRAM budget and expert split ratio. In the reported experiments, MoEpic can save about half of the GPU cost while lowering inference latency by about $37.51\%$–$65.73\%$ compared to the baselines (Yan et al., 10 Sep 2025).

1. Problem setting and architectural motivation

MoE LLMs scale by activating only a small subset of experts per token, but this sparsity does not remove the storage burden of the full expert pool. In the motivating examples for MoEpic, experts dominate parameter count: for DeepSeek-R1, $97.45\%$ of parameters are in experts, but only $3.13\%$ are activated per token. A fully resident deployment is correspondingly expensive: a 671B-parameter MoE model like DeepSeek-R1 requires $16\times$ A100-80GB GPUs, with a cost of about 300k USD (Yan et al., 10 Sep 2025).

MoEpic is designed for the common deployment regime in which non-expert parameters remain in GPU memory while most experts are offloaded to CPU RAM and fetched over PCIe on demand. The paper identifies two bottlenecks in prior offloading systems. First, cache-based schemes can retain only a small number of full experts per layer, which limits cache hit rate. On Qwen1.5-MoE with 60 experts per layer and 4 activated experts per token, caching 20 of 60 experts per layer yields cache hit rates of $33.02\%$ for RND, $38.88\%$ for LRU, $43.74\%$ for LFU, and $45.36\%$ for LCP; under an approximately $40\%$ hit rate, TPOT is about $65.73\%$0 the GPU-rich baseline where all experts fit in VRAM (Yan et al., 10 Sep 2025).

Second, prefetch-based schemes suffer from insufficient transfer-computation overlap. For Qwen1.5-MoE, per-layer prediction accuracy can be high, with on average $65.73\%$1–$65.73\%$2 of 4 active experts correctly predicted, but loading latency and compute remain imbalanced: the reported loading latency is about 165 ms versus 57 ms compute per token on some GPUs, so only $65.73\%$3–$65.73\%$4 of loading time can be hidden (Yan et al., 10 Sep 2025). MoEpic addresses both issues by changing the caching unit from full experts to expert segments and by re-optimizing prefetch and cache allocation around that choice.

2. Core design: expert splitting and MoE execution model

MoEpic assumes a standard MoE Transformer with $65.73\%$5 layers, where layer $65.73\%$6 contains an attention module $65.73\%$7, a router $65.73\%$8, and experts $65.73\%$9. Its layer computation is written as

$97.45\%$0

Router scores are

$97.45\%$1

If $97.45\%$2 denotes the activated top-$97.45\%$3 experts, the MoE output is

$97.45\%$4

The defining mechanism is “vertical” expert splitting. If an expert weight matrix is written as $97.45\%$5, MoEpic partitions it row-wise into $97.45\%$6 and $97.45\%$7. At the expert level, the paper abstracts this as a top segment occupying a fraction $97.45\%$8 of the expert parameters in layer $97.45\%$9, and a bottom segment occupying the remaining $3.13\%$0. If $3.13\%$1 is the total expert size and $3.13\%$2 experts are cached in layer $3.13\%$3, then the layer’s VRAM budget satisfies

$3.13\%$4

which the paper normalizes to

$3.13\%$5

This split has two effects. First, for a fixed VRAM budget $3.13\%$6, MoEpic can store $3.13\%$7 times more experts than a full-expert cache. Second, when an activated expert’s top segment is already resident, only the bottom segment must be fetched, which reduces transfer time and allows computation on the top segment to begin immediately. The paper treats the extreme cases as limiting baselines: $3.13\%$8 gives Prefetch Only, and $3.13\%$9 gives Cache Only. On Qwen1.5-MoE, Figure 3b reports an optimum around $16\times$0, where MoEpic outperforms both Cache Only and Prefetch Only by $16\times$1–$16\times$2 (Yan et al., 10 Sep 2025).

3. Cache management, speculative prefetching, and overlap model

MoEpic couples segmented experts with a cache policy intended to capture both long-term popularity and short-term reuse. For expert $16\times$3 in layer $16\times$4, it maintains an activation frequency $16\times$5 and an activation interval $16\times$6, then defines cache priority as

$16\times$7

where $16\times$8 is the observation window and $16\times$9 is a decay parameter. The default values are $33.02\%$0 tokens and $33.02\%$1. On a cache miss, the expert with minimum $33.02\%$2 is evicted; this Least Cache Priority policy is intended to interpolate between LFU and LRU. In the Qwen1.5-MoE/MMLU experiment with $33.02\%$3 full experts cached per layer, the reported hit rates are $33.02\%$4 for RND, $33.02\%$5 for LRU, $33.02\%$6 for LFU, and $33.02\%$7 for LCP (Yan et al., 10 Sep 2025).

Speculative prefetching is based on the observation that residual connections make adjacent-layer activations similar. During computation of layer $33.02\%$8, MoEpic feeds $33.02\%$9 into the next router $38.88\%$0 and uses the predicted scores

$38.88\%$1

to rank experts for layer $38.88\%$2. It then prefetches bottom segments for experts whose tops are cached, and full experts otherwise. For layer 1, it follows AdapMoE by using the last layer’s activation from the previous token as a proxy (Yan et al., 10 Sep 2025).

The paper formalizes overlap with three counts per layer: $38.88\%$3, the number of activated experts fully loaded in VRAM; $38.88\%$4, the number whose top segments are cached; and $38.88\%$5, the number completely offloaded, with

$38.88\%$6

The time window available to hide loading is

$38.88\%$7

while the loading time for missing segments is

$38.88\%$8

The exposed loading latency is therefore

$38.88\%$9

MoEpic also defines per-layer prefetch windows. For layer 1,

$43.74\%$0

and for $43.74\%$1,

$43.74\%$2

Within this model, MoEpic’s advantage over pure prefetching is that top-segment residency both shrinks the amount transferred and extends the interval during which transfer can be hidden by useful computation (Yan et al., 10 Sep 2025).

4. Adaptive cache configuration and fixed-point iteration

MoEpic does not treat cache allocation as a static hyperparameter. It optimizes per-layer VRAM budgets $43.74\%$3 and split ratios $43.74\%$4 to minimize total exposed loading latency under the global VRAM constraint

$43.74\%$5

where $43.74\%$6 is the size of non-expert parameters, $43.74\%$7 is a fixed buffer budget, and $43.74\%$8 is total GPU VRAM. The outer objective is

$43.74\%$9

Because $45.36\%$0, $45.36\%$1, and $45.36\%$2 depend on runtime behavior, MoEpic estimates them from online statistics gathered after $45.36\%$3 tokens. For each layer $45.36\%$4, it tracks the cache hit rate $45.36\%$5, the prediction accuracy $45.36\%$6 of the $45.36\%$7-th highest-scoring predicted expert, and the prediction cache hit rate $45.36\%$8 for that same predicted rank (Yan et al., 10 Sep 2025).

Given a fixed $45.36\%$9 and prefetch window $40\%$0, the inner “ExpertSplit” subproblem chooses a cache size $40\%$1, hence $40\%$2, to maximize the expected number $40\%$3 of active experts whose required data will already be in VRAM. The cache contribution is modeled as

$40\%$4

and for the $40\%$5-th predicted expert the expected prefetch time and expected contribution are

$40\%$6

$40\%$7

If the top $40\%$8 predicted experts can be prefetched within the available window, then

$40\%$9

subject to

$65.73\%$00

The resulting discrete optimization is

$65.73\%$01

The outer “VramAllocation” procedure starts from a uniform allocation $65.73\%$02 with $65.73\%$03, where $65.73\%$04. It then iteratively transfers a fraction $65.73\%$05 of VRAM from a layer with the smallest penalty for losing memory to a layer with the greatest benefit from gaining memory, using $65.73\%$06 by default. Reconfiguration is performed every $65.73\%$07 tokens, with $65.73\%$08, and only when the device is idle (Yan et al., 10 Sep 2025). The paper describes this as a divide-and-conquer algorithm based on fixed-point iteration.

5. Implementation and experimental results

MoEpic is implemented in Python with PyTorch and HuggingFace Transformers, with custom modifications for expert cache management, cache table lookup, and dynamic loading of expert segments. The evaluation hardware is an Intel Xeon Platinum 8358P CPU, $65.73\%$09 NVIDIA RTX A6000 GPUs with 48 GB memory each, 512 GB RAM, and PCIe 4.0 $65.73\%$10 with a theoretical 32 GB/s bandwidth. The system profiles $65.73\%$11, $65.73\%$12, $65.73\%$13, and $65.73\%$14 on the target hardware, and uses asynchronous CPU-to-GPU transfers overlapped with GPU kernels (Yan et al., 10 Sep 2025).

Model	Structural specification	Parameters
Qwen1.5-MoE	24 layers, 60 experts per layer, top-4 activated experts	14.3B total, 2.7B activated per token
Mixtral-8×7B	32 layers, 8 experts per layer, top-2 activated experts	46.7B total, 12.9B activated per token

The baselines are Pre-gated MoE, Mixtral-offloading, AdapMoE, and MoE-Infinity. The common expert-cache VRAM budget $65.73\%$15 is set equivalent to 240 full experts for Qwen1.5-MoE and 96 full experts for Mixtral-8×7B; the buffer $65.73\%$16 is set to $65.73\%$17 full experts (Yan et al., 10 Sep 2025).

For prefill latency, MoEpic is reported as the fastest method across all budgets. On Qwen1.5-MoE with a VRAM budget equivalent to 480 experts, TTFT is 2.58 s for Pre-gated MoE, 1.77 s for Mixtral-offloading, 1.78 s for AdapMoE, 1.73 s for MoE-Infinity, and 1.43 s for MoEpic. The reported prefill acceleration relative to baselines is about $65.73\%$18–$65.73\%$19 (Yan et al., 10 Sep 2025).

For decoding, the largest gains appear in TPOT. On Mixtral-8×7B with a VRAM budget of 160 full experts, TPOT is 248 ms for Pre-gated MoE, 162 ms for Mixtral-offloading, 144 ms for AdapMoE, 136 ms for MoE-Infinity, and 85 ms for MoEpic. Across the main experiments, MoEpic reduces TPOT by $65.73\%$20–$65.73\%$21 versus the state-of-the-art offloading baselines (Yan et al., 10 Sep 2025).

The VRAM-efficiency results emphasize the intended deployment use case. For Qwen1.5-MoE at a target TPOT of 140 ms, the required VRAM footprints are 11.97 GB for Mixtral-offloading, 10.42 GB for AdapMoE, 9.56 GB for MoE-Infinity, and 5.98 GB for MoEpic. In another comparison, a fully resident Qwen1.5-MoE deployment attains 57 ms TPOT with 26.67 GB VRAM, while MoEpic attains 60 ms TPOT with 12.38 GB VRAM, within $65.73\%$22 of the GPU-rich latency at about half the GPU memory cost (Yan et al., 10 Sep 2025).

The ablation study attributes gains to all three main components: speculative prefetching, LCP, and the Cache Configuration Algorithm. On Mixtral-8×7B, full MoEpic yields TTFT 720 ms and TPOT 132 ms; removing speculative prefetching gives 753 ms and 174 ms, removing LCP gives 782 ms and 157 ms, and replacing adaptive configuration with uniform VRAM allocation and fixed $65.73\%$23 gives 864 ms and 193 ms. The paper identifies the Cache Configuration Algorithm as the largest single contributor, with TPOT increasing by $65.73\%$24 when it is removed (Yan et al., 10 Sep 2025).

6. Scope, limitations, and relation to the broader MoE systems literature

MoEpic operates at the serving layer. It controls what fraction of each expert is kept in GPU VRAM, what remains in CPU RAM, and how those pieces are prefetched and scheduled, but it does not change model weights or training dynamics. This distinguishes it from systems work on large-scale MoE training such as DeepSpeed MoE, which combines expert parallelism, ZeRO, and ZeRO-Offload to support up to 3.5T parameters on 512 A100 GPUs (Kim et al., 2021). A plausible implication is that MoEpic is best understood as a runtime memory-management system rather than an MoE architecture or routing algorithm.

Its stated limitations are correspondingly specific. The paper assumes predictability of routing via adjacent-layer activation similarity, and notes that early and late layers have lower similarity and prediction accuracy. It maintains online statistics and runs periodic optimization only when the device is idle, which may limit reconfiguration opportunities under continuous high-load serving. It uses a single split ratio $65.73\%$25 per layer rather than per-expert split ratios, focuses on single-node CPU-GPU offloading rather than multi-node expert sharding, and requires integration of custom caching, segment-level prefetching, and dynamic split configuration into serving stacks (Yan et al., 10 Sep 2025).

In the broader inference literature, MoEpic addresses a different bottleneck from throughput-oriented expert-parallel schedulers such as EPS-MoE, which accelerates MoE FFN inference by dynamically choosing GroupGemm or DenseGemm and overlapping communication with computation, reporting at most $65.73\%$26 improvement in prefill throughput (Qian et al., 2024). It also targets a different operating regime from METRO, which focuses on expert-parallel serving in the memory-bound decode phase and argues that balancing activated experts rather than token counts reduces latency by 11–22% and can improve decode throughput by up to $65.73\%$27 at a fixed decode SLO (Yu et al., 10 Dec 2025). MoEpic’s contribution is narrower but orthogonal: it addresses the case where the dominant constraint is not expert-parallel routing across many GPUs, but limited VRAM and the cost of repeatedly moving expert parameters from CPU RAM into GPU memory (Yan et al., 10 Sep 2025).