ExpertFlow: Efficient Sparse MoE Inference

• ExpertFlow is a system for efficient inference in sparse Mixture-of-Experts transformers that utilizes predictive expert placement and token reorganization to significantly reduce GPU–CPU transfer latency.
• It integrates distinct formulations—one with a T5-based Routing Path Predictor and Expert Cache Engine, and another with a RandomForest-based predictor and adaptive prefetching—to dynamically manage expert offloading.
• Experimental results demonstrate that ExpertFlow achieves up to 10× speedups and 93% GPU memory savings by optimizing expert scheduling, cache locality, and communication-computation overlap.

Searching arXiv for the cited ExpertFlow papers to ground the article. ExpertFlow is a system for efficient inference in sparse Mixture-of-Experts (MoE) transformers under constrained GPU memory. In the 2024 formulation, it is presented as an inference-specific system that sits atop any sparse MoE transformer and inserts a Routing Path Predictor, an Expert Cache Engine, and a Token Scheduler between the standard embedding and expert layers, with the stated goal of accommodating flexible routing and enabling efficient expert scheduling between CPU and GPU (He et al., 2024). A later 2025 work using the same name describes a runtime system centered on adaptive expert prefetching, an Expert Predictor, a two-tiered LRU Memory Manager, and a Cache-Aware Router, coordinated by a lightweight feedback loop (Shen et al., 30 Oct 2025). Across these formulations, ExpertFlow addresses a common deployment bottleneck of MoE models: the latency and memory overhead induced by frequent expert parameter movement between host memory and GPU memory during autoregressive inference.

1. Definition and problem setting

Sparse MoE models activate only a subset of experts for each token or batch, which improves parameter efficiency relative to dense LLMs, but creates an inference-time systems problem when all experts cannot reside in GPU memory simultaneously. The 2024 ExpertFlow paper characterizes the core difficulty as the inadequacy of existing offloading techniques that swap activated and idle experts between GPU and CPU using rigid caching mechanisms that either fail to adapt to dynamic routing or incur prohibitive costs for prediction training (He et al., 2024).

Within that framing, ExpertFlow is defined by two linked objectives. The first is predictive expert placement: identifying which experts will be needed before computation begins, so that the system can prefetch those experts and reduce GPU↔CPU transfer frequency. The second is token-level reorganization: rearranging tokens across batches so that each batch activates fewer distinct experts while giving each activated expert more tokens to process, thereby improving cache locality and expert utilization (He et al., 2024).

The 2025 formulation retains the same high-level concern but recasts it as a runtime coordination problem. It emphasizes that conventional MoE inference often selects active experts independently at each layer, leading to latency because of frequent parameter transfers, and further argues that fixed-step cross-layer prediction lacks adaptability across hardware platforms and workloads. Its solution is a runtime system that continuously adjusts prediction horizon using runtime statistics such as transfer bandwidth, parameter dimensionality, and model feedback signals (Shen et al., 30 Oct 2025).

A plausible implication is that “ExpertFlow” denotes a systems family rather than a single invariant implementation: one version prioritizes predictive routing plus token regrouping, while another prioritizes adaptive prefetching plus cache-aware routing. Both are designed for resource-constrained MoE inference.

2. Core architecture

The 2024 architecture is explicitly layered onto an existing sparse MoE transformer. It inserts three modules: a Routing Path Predictor (RPP), an Expert Cache Engine (ECE), and a Token Scheduler. The RPP predicts which experts will be activated in every MoE layer before any expert computation begins. The ECE uses these predictions to prefetch experts into GPU memory, correct mispredictions at runtime, and orchestrate CPU↔GPU offloads. The Token Scheduler uses predicted routing paths to regroup input tokens into new batches so that each batch activates fewer experts and each active expert receives more tokens (He et al., 2024).

Two pipeline optimizations are also part of the 2024 design. A Multi-Stream Overlapping Pipeline overlaps expert prefetch and offload via PCIe streams with ongoing expert computations. A Dual-Batch Inference Pipeline treats pairs of batches as a unit and interleaves routing prediction and scheduling with decoding to hide their latency (He et al., 2024).

The 2025 architecture comprises four tightly integrated components: the Prefetch Manager, the Expert Predictor, the Memory Manager with a two-tiered LRU cache, and the Cache-Aware Router. The Prefetch Manager receives pre-gate output from the current MoE layer, estimates the number of experts likely to be needed over the next several layers, and issues asynchronous DMA fetch requests. The Expert Predictor is a host-side RandomForestRegressor that consumes a feature vector $x = [e, S, l, prev\_act]$ and outputs a correction term that refines naive top-$K$ pre-gate expert selection. The Memory Manager maintains LRU_high and LRU_low, estimates effective bandwidth and eviction cost, and updates the measured transfer bandwidth after each transfer. The Cache-Aware Router prioritizes cache-miss resolution and schedules tokens whose experts are already resident before those that would induce further I/O (Shen et al., 30 Oct 2025).

These two descriptions share a common systems pattern: prediction drives prefetch; memory management mediates residency and eviction; routing or scheduling exploits predicted structure to reduce stalls; and latency overhead is hidden by overlap between communication and computation.

3. Predictive expert activation and offloading

In the 2024 paper, predictive offloading is centered on the Routing Path Predictor. Its architecture is an off-the-shelf T5 encoder–decoder whose final classifier is replaced by $L$ separate MLP heads, one per MoE layer, each of output dimension $E$. For an input sequence of length $S$, the predictor outputs logits $p = \{p_{l,e}\}$ for every layer $l=1\dots L$ and expert $e=1\dots E$ (He et al., 2024).

Training data are constructed by running the MoE once for each input token $x_i$ to record its true routing path $r_i \in \{0,1\}^{L \times E}$, where $K$0 if expert $K$1 was used at layer $K$2. The per-token loss is given as

$K$3

The prediction objective is to estimate $K$4 and then threshold or top-$K$5 select to form a predicted binary batch-level routing matrix $K$6 (He et al., 2024).

The Expert Cache Engine implements Predictive Locality-aware Expert Caching (PLEC). At each MoE layer, ECE uses $K$7 to prefetch exactly the predicted experts into a layer-wise GPU cache of size $K$8. The paper contrasts this with LRU, arguing that focusing on spatial locality within each layer’s predicted active set avoids cache thrashing when many experts compete for eviction. At runtime, ECE corrects two error types: mis-prefetched experts, which were predicted active but are actually idle, and missed experts, which were predicted idle but are actually active. Missed experts are placed in a high-priority queue and swapped in by evicting either a mis-prefetched expert or an expert whose work is already done. This correction proceeds asynchronously during current expert computation via multi-stream overlap (He et al., 2024).

The 2025 paper presents a different predictive mechanism. Its central control variable is the lookahead horizon $K$9, the number of layers ahead for which experts should be prefetched:

$L$0

Here $L$1 is the expected number of experts to activate, $L$2 is the size in bytes of a single expert’s parameters, $L$3 is measured transfer bandwidth between host DRAM and GPU HBM, and $L$4 is average compute time per layer. The system then adapts $L$5 online using a stall-driven increment rule and an overfetch-driven decrement rule, with thresholds $L$6 and $L$7 (Shen et al., 30 Oct 2025).

That work also defines a hybrid cross-layer scoring function

$L$8

with default hyperparameters $L$9, $E$0, and $E$1, where pregate_score derives from normalized router logits, context_score from a softmax over a linear transform of mean-pooled token embeddings, and history_score from the previous layer’s activity mask (Shen et al., 30 Oct 2025).

A common misconception is that MoE efficiency follows automatically from sparse activation. These systems make clear that sparsity alone does not eliminate host–device transfer cost; predictive offloading and routing-aware memory coordination are necessary to convert parameter sparsity into practical latency and memory gains.

4. Token scheduling, routing order, and cache locality

A distinctive feature of the 2024 ExpertFlow design is dynamic token scheduling. It considers two batches of $E$2 tokens each, indexed globally as $E$3, with each token $E$4 having routing path matrix $E$5. The objective is to construct two new batches $E$6 and $E$7, each of size $E$8, that minimize the total number of distinct experts activated:

$E$9

To approximate this combinatorial optimization, the paper uses a balanced K-means heuristic with $S$0. It first computes Hamming-based similarity for all token pairs,

$S$1

then iteratively assigns tokens to the nearest centroid subject to cluster-size balancing, recomputes centroids as tokens maximizing average in-cluster similarity, and repeats until convergence or a maximum number of iterations. The reported complexity is $S$2 to build $S$3 and $S$4 per iteration in practice, with $S$5–$S$6 (He et al., 2024).

Because autoregressive decoding depends on attention state, ExpertFlow also specifies KV-cache adaptation after re-batching. It performs a Merge step to unify KV caches across old batches and a Reindex step to remap cache entries to new token positions, thereby maintaining consistency with MoE semantics (He et al., 2024).

The 2025 system does not re-batch tokens, but it introduces cache-aware routing order. During layer execution, tokens whose experts are already resident are routed first, while tokens that would induce further I/O are deferred. Swap-ins are overlapped with continuing FP16/GEMM computation to hide I/O latency (Shen et al., 30 Oct 2025).

These mechanisms instantiate two different locality strategies. The 2024 method reshapes the batch to reduce the number of activated experts per batch. The 2025 method preserves the batch but reorders execution based on expert residency. This suggests a broader design space in which MoE inference efficiency depends not only on which experts are selected, but also on how token work is grouped or sequenced relative to cache state.

5. Experimental results

The 2024 evaluation reports batch-level routing predictor accuracy on Switch-32, Switch-64, and Switch-128 with batch sizes 8, 16, and 32 on WMT16 and XSUM. The transformer predictor achieves 75–88%, compared with TLP/SLP’s 5–36% (He et al., 2024).

It further reports inference speedups against the SE-MoE baseline on an NVIDIA A40 with 48 GB. For the Switch series, speedups are 2.53× on WMT16 to 2.60× on XSUM for Switch-32, 5.12× to 6.34× for Switch-64, and 7.94× to 9.99× for Switch-128. For Mixtral-8×7B, the gains are 1.96× on WMT16 and 1.99× on XSUM over Cache-MoE (He et al., 2024).

GPU memory savings are reported relative to All-in-GPU. Average memory reduction is approximately 75%, with a peak of 93.72% on Switch-128, XSUM, cache size 4, and batch size 8. The paper also states that ExpertFlow without predictor, identical to ExpertFlow with predictor for memory cost, uses only about 1–3 GB for expert cache versus 15–16 GB for full expert storage (He et al., 2024).

Cache hit ratio improvements are a central systems metric. On Switch-32 for WMT16, PLEC with cache sizes $S$7 and batch sizes up to 16 maintains hit ratios of 80–92%, versus LRU’s 50–75%, corresponding to gains of 15–35 percentage points. In large-model tests, the paper describes hit ratio increases of 15–36 percentage points over LRU, for example from approximately 56% to approximately 92%, with offloading-stage I/O overhead reduced by up to 3–8× (He et al., 2024).

The end-to-end speed and memory excerpt includes two representative rows:

Task	Model / setting	Reported outcome
WMT16	Switch-128, cache 4, BS 8	AIG-GPU 15.2 GB; ExpertFlow Mem 0.96 GB; Saving 93.7%; Speedup 9.99×
XSUM	Mixtral-8×7B, cache 8, BS 16	AIG-GPU 96 GB (oom); ExpertFlow Mem 2.7 GB; Saving 82.5%; Speedup 1.99×

The 2025 paper reports results on NVIDIA A6000, H20, and Ascend 910B for DeepSeek, Qwen1.5, and Qwen2.0 under 20 GB GPU caps. End-to-end latency is reduced by 96–99.9% across models; for Qwen1.5 on A6000, latency falls from 1.05 s in the baseline to 0.038 s with Predictor and 0.022 s with Cache-Aware Routing. Combined stall and cache-miss latency drops to under 0.1% of baseline, and prediction accuracy improves by 21.8% on average versus pre-gate, up to +30.4%, raising asymptotic accuracy $S$8 from approximately 30% to approximately 60% (Shen et al., 30 Oct 2025).

6. Implementation characteristics and operational trade-offs

The 2024 paper emphasizes that ExpertFlow is designed for inference and does not require model retraining or invasive architectural changes to the MoE backbone. It also states that, in resource-constrained settings such as a single 48 GB GPU, the system makes large MoEs practicable while delivering high throughput and low latency (He et al., 2024).

Its reported gains depend on the synergy of four mechanisms. The predictor provides foresight on expert usage, enabling ECE’s PLEC to prefetch exactly those experts and correct mistakes with minimal I/O. The Token Scheduler reduces the number of experts activated per batch and packs more tokens per expert. Multi-Stream overlap and Dual-Batch pipelining hide the overhead of prediction, scheduling, and correction. The paper attributes its net 2×–10× speedups and up to 93.72% GPU memory savings on A40/A100-class hardware to this combined design (He et al., 2024).

The 2025 paper gives more explicit deployment guidance. It specifies a dedicated high-bandwidth PCIe or NVLink channel and notes that the system exploits up to 128 GB/s. It recommends host DRAM pre-partitioned into page-aligned blocks, a separate CPU thread for occupancy tracking and status polling, and a second CUDA stream dedicated to expert DMA. For tuning, it suggests a pregate cumulative probability threshold $S$9, thresholds $p = \{p_{l,e}\}$0 and $p = \{p_{l,e}\}$1, allocation of LRU_high to 30–40% of cache capacity, and warm-up profiling of $p = \{p_{l,e}\}$2 and initial $p = \{p_{l,e}\}$3 to initialize $p = \{p_{l,e}\}$4 (Shen et al., 30 Oct 2025).

These details highlight an important trade-off. ExpertFlow reduces GPU memory demand by moving experts to host memory and prefetching them on demand, but this shifts the optimization burden toward transport bandwidth, stream orchestration, eviction policy, and predictive accuracy. In other words, memory savings are not free; they are made viable by careful runtime coordination.

7. Relation between the two ExpertFlow formulations

The 2024 and 2025 papers present distinct but related system designs under the same name. The earlier work is centered on a T5-based routing path predictor, Predictive Locality-aware Expert Caching, a real-time correction mechanism, and dynamic token scheduling with KV-cache merge and reindex operations (He et al., 2024). The later work centers on adaptive expert pre-gating, a host-side RandomForestRegressor, a two-level LRU cache, cache-aware routing order, and online adaptation of the prediction horizon through stall and overfetch counters (Shen et al., 30 Oct 2025).

Several continuities are evident. Both formulations treat expert activation prediction as actionable systems information rather than merely an auxiliary diagnostic. Both coordinate CPU–GPU transfers asynchronously with ongoing expert computation. Both make cache residency a first-class concern, and both report that prediction quality directly affects stall behavior and overall latency.

At the same time, they differ in where they place the main optimization leverage. The 2024 system primarily reduces the active expert footprint through predictive routing and re-batching of tokens. The 2025 system primarily adapts prefetch depth and execution order using runtime statistics and feedback control. This suggests two complementary interpretations of MoE inference optimization: one based on restructuring the workload to better match cacheable expert sets, and another based on dynamically matching prefetch policy to hardware behavior and workload variability.

For researchers, these works frame ExpertFlow as part of a broader shift in MoE systems research from static offloading policies toward predictive, adaptive, and routing-aware runtimes. Their central claim is not that sparse activation alone solves inference efficiency, but that sparse activation must be coupled with expert scheduling and memory coordination mechanisms that are informed by routing structure and responsive to runtime conditions (He et al., 2024).