DuoServe-MoE: Dual-Phase Inference

• DuoServe-MoE is an MoE-LLM inference serving system that separates prefill and decode phases to efficiently manage expert weight scheduling and memory usage.
• It employs a two-stream CUDA pipeline for prefill and a lightweight offline-trained predictor for decode, ensuring optimal overlap of compute and transfer operations.
• Empirical results demonstrate up to 7.54× latency improvement and GPU memory usage reduction to 15% of full model size, highlighting its practical impact.

DuoServe-MoE is an MoE-LLM inference serving system built around the observation that prefill and decode are fundamentally different workloads and therefore should not be scheduled the same way. In autoregressive inference, the prefill phase processes the entire input prompt in parallel to produce the first token, whereas the decode phase generates subsequent tokens one by one; for MoE layers, this asymmetry makes prefill effectively dense and decode sparse. DuoServe-MoE explicitly separates these phases and combines CPU offloading, GPU expert caching, a two-stream CUDA pipeline for prefill, and a lightweight offline-trained predictor for decode. Experiments on 4-bit Mixtral-8×7B and 8×22B show end-to-end latency improvement of 1.42× to 7.54× while keeping peak memory usage at only 15 percent of the full model size (Zhang et al., 9 Sep 2025).

1. Workload asymmetry in MoE inference

The central premise of DuoServe-MoE is that MoE inference contains two qualitatively different stages. During prefill, many tokens are processed simultaneously; although each token activates only a few experts, the aggregate activation across the batch becomes dense. In Mixtral, prefill can effectively activate all experts in a layer. During decode, only one token is processed per iteration, so activation returns to being sparse, typically only the top-k experts for that token (Zhang et al., 9 Sep 2025).

This asymmetry makes a uniform expert-fetching policy suboptimal. A “load everything” strategy is fast in prefill but causes severe memory pressure. A “load only what is needed” strategy is reasonable in decode but can stall badly if expert prediction is wrong or too late. DuoServe-MoE therefore adopts a dense, pipelined strategy in prefill and a predicted prefetch strategy in decode.

A common misconception is that sparsity at the per-token level implies sparsity at the stage level. The prefill behavior described above shows why this is not generally true: sparse token-level routing can aggregate into dense layer-level expert usage when many prompt tokens are processed in parallel. This is the immediate rationale for phase-specific scheduling.

2. Dual-phase system organization

DuoServe-MoE keeps the non-MoE weights on GPU because they are only about 10% of the full model, while the large MoE expert weights are mostly offloaded to CPU. It creates a GPU expert cache sized to the number of experts actually active per layer, namely k for top-k gating. For Mixtral-8×7B, where top-2 is used, the GPU expert cache size is 2 (Zhang et al., 9 Sep 2025).

Phase	Activation pattern	Scheduling policy
Prefill	Dense in aggregate; can effectively activate all experts in a layer	Deterministic pipelined fetch/compute over all experts
Decode	Sparse; typically only top-k experts for one token	Prediction-driven next-layer expert prefetch

In prefill, DuoServe-MoE does not use prediction. All experts needed for the layer are eventually processed; tokens are grouped by expert after gating; each expert weight is fetched once from CPU to GPU and then reused for its token batch. In decode, after a token is routed by the gate in layer l, the system uses the observed activations plus learned statistics to predict the experts for layer l+1, then prefetches those predicted experts to GPU before the next layer executes.

This organization makes the memory policy phase-aware. Prefill assumes density and limits residency time; decode assumes sparsity and speculates on likely activations. A plausible implication is that the design treats expert weights less as static parameters than as transiently staged state whose lifetime depends on the inference phase.

3. Prefill pipeline and stream-level overlap

The prefill path is implemented as a two-stream CUDA pipeline with one computation stream and one communication stream. The purpose is to overlap PCIe transfer of expert weights with GPU execution of unrelated work, especially the computation of non-MoE layers (Zhang et al., 9 Sep 2025).

At the start of prefill, the communication stream begins prefetching the first expert weight into the GPU expert cache while the computation stream simultaneously runs non-MoE computation. Before entering the MoE layer, the two streams are synchronized so the needed expert is resident. After gating and token grouping, the current expert is computed in the computation stream, and as soon as one expert finishes, the communication stream fetches the next expert from CPU. The resulting pattern is a 2-stage pipeline in which one expert is being computed while the next is being transferred.

Several implementation details are structurally important. GPU expert cache size is only k, so the cache holds just the currently active experts. Synchronization points ensure that a weight is not used before transfer is complete. Because PCIe transfer is slower than expert computation, overlapping is essential. The paper’s scheduling pattern is therefore: fetch expert i while computing non-MoE layers or the previous expert, then fetch expert i+1 after expert i finishes, while keeping the GPU cache small so only the currently needed experts reside on GPU.

This pipeline is deterministic rather than predictive. Its effectiveness relies not on forecasting the expert set, but on the fact that dense prefill makes eventual processing of all experts likely enough that one-time fetch-and-reuse dominates speculative loading.

4. Decode-side prediction and expert-state construction

For decode, DuoServe-MoE introduces a lightweight layer-level predictor trained offline from activation traces and named ExpertMLP. The predictor does not require model modification, LLM fine-tuning, or architecture changes. It is formulated as a multi-label classification problem that predicts the set of experts likely to be activated in the next layer (Zhang et al., 9 Sep 2025).

An expert activation path is defined as

$$E = \{E_l \mid l \in [1, L], E_l \subseteq \{e_1, e_2, \dots, e_M\}\},$$

where E_l is the set of experts selected at layer l, L is the number of layers, and M is the number of experts. From a small profiling dataset, DuoServe-MoE records these paths and constructs two statistics.

For each layer l, expert popularity is

$P_l(i) = \frac{\sum_{n=1}^{N} \mathbbm{1}(e_i \in E_{l,n})}{\sum_{m=1}^{M} \sum_{n=1}^{N} \mathbbm{1}(e_m \in E_{l,n})}.$

For consecutive layers, expert affinity is

$A_{l,l+1}(i,j) = \frac{\sum_{n=1}^{N} \mathbbm{1}(e_i \in E_{l,n} \wedge e_j \in E_{l+1,n})}{\sum_{m=1}^{M} \sum_{n=1}^{N} \mathbbm{1}(e_i \in E_{l,n} \wedge e_m \in E_{l+1,n})}.$

The predictor input is constructed from history, popularity, and affinity:

$h_l = \{E_0, E_1, \dots, E_{l-1}\},$

$s_l = [h_l, p_l, a_{l-1,l}].$

These components are flattened and zero-padded into a fixed-length vector. The predictor itself is a 7-layer MLP with hidden sizes decreasing from 2048 to 64, together with BatchNorm, ReLU, and Dropout with default 0.1. It outputs a probability for each expert in the target layer and is trained with binary cross-entropy,

$$L(\theta) = - \sum_{n=1}^{N} \left[ y_n \log(\sigma(f_\theta(s_n))) + (1 - y_n) \log(1 - \sigma(f_\theta(s_n))) \right].$$

The predictor’s output is stored and cleared layer by layer by a State Constructor, which accumulates activation traces and feeds them into the MLP. Runtime includes two synchronization points: one to ensure prefetch is done before expert computation starts, and another so that, once the first expert’s computation and the next-layer prediction are ready, the system can start prefetching the new layer. The design is intentionally lightweight: it can run on the same GPU as inference, uses only a small amount of profiling data, and data collection can use only 2.5% of the dataset while preprocessing takes under 9 hours.

5. Scheduling policies, heuristics, and empirical behavior

DuoServe-MoE’s scheduling policies are simple and phase-specific. GPU cache sizing follows a top-k policy: for a top-k MoE layer, the GPU expert cache is sized to k. Prefill uses a deterministic pipelined heuristic: overlap computation and transfer, fetch the next expert as soon as the current one finishes, and synchronize only when necessary. Decode uses a prediction-driven prefetch policy: predict the next layer’s activated experts, prefetch them before the layer runs, compare predicted versus gated experts, and reload the correct experts from CPU if there is a mismatch (Zhang et al., 9 Sep 2025).

The empirical evaluation uses 4-bit AWQ-quantized Mixtral-8×7B and Mixtral-8×22B on single-GPU edge-server settings with A5000 and A6000 GPUs. Reported improvements relative to baselines are: TTFT improvement of 1.78× to 3.8×, end-to-end latency improvement of 1.42× to 7.54×, and throughput improvement of 1.78× to 7.04×. The paper attributes these gains to prefill overlap of compute and transfer, decode prefetching of likely experts, and reduced waiting on CPU-to-GPU transfers.

Memory reduction is equally central. For Mixtral-8×7B, GPU-only deployment requires 25.14 GB, whereas DuoServe-MoE uses 3.91 GB. For Mixtral-8×22B, GPU-only deployment requires 138 GB, whereas DuoServe-MoE uses 8.44 GB. Peak memory under DuoServe-MoE is reported as only about 15% of the full model size. Compared with baselines, DuoServe-MoE uses slightly more memory than pure activation-based offloading systems like Accelerate because it also stores the predictor, but less memory than MoESys, which preloads more expert weights. The additional predictor memory is around 300 MB.

Predictor quality is reported using hit-rate statistics. Top-2 hit rate is 66.85% on Mixtral-8×7B / Orca, 60.23% on Mixtral-8×7B / SQuAD, 56.21% on Mixtral-8×22B / Orca, and 54.16% on Mixtral-8×22B / SQuAD. The “at least one expert hit” rate is above 90% in all reported cases. The paper’s interpretation is that even partial correctness usually reduces transfer overhead enough to preserve good performance.

6. Scope, assumptions, and relation to adjacent MoE serving systems

DuoServe-MoE is evaluated on single-request workloads. The paper explicitly states that batching is not the intended scenario because batching would densify expert activation, reduce the benefit of sparse decode prediction, and increase communication overhead. CPU offloading is central: expert weights live primarily in CPU pinned memory, while the GPU holds only the non-MoE weights and a small expert cache. Prediction errors are tolerated rather than eliminated; if prediction is wrong, the correct experts are re-fetched after the gate result is known. The implementation stack comprises vLLM for inference, CUDA pinned memory for offloading, CUDA streams for overlap, and PyTorch for the offline predictor (Zhang et al., 9 Sep 2025).

These assumptions locate DuoServe-MoE within a broader design space of MoE serving systems. D$^{2}$MoE addresses resource-constrained edge devices through dual routing over expert ID and bit-width, matryoshka weight quantization, and the Hottest-Expert-Bit-First scheduling principle; its emphasis is dynamic bit allocation and I/O-computation overlap under constrained memory budgets rather than prefill/decode separation (Wang et al., 17 Apr 2025). METRO, by contrast, studies expert-parallel MoE serving in the memory-bound regime and argues that decode performance is governed by the number of activated expert replicas rather than token counts; it is applied only to the decode phase, while prefill still uses EPLB-style routing and placement (Yu et al., 10 Dec 2025).

This comparison suggests that DuoServe-MoE addresses a distinct regime: single-GPU or edge-style MoE serving in which CPU offloading and expert residency dominate system design, and where the chief systems insight is not expert-parallel routing but workload bifurcation between dense prefill and sparse decode. In that sense, the system belongs to a family of MoE-serving methods that derive performance not from changing the model, but from aligning memory movement, caching, and scheduling policy with the actual sparsity structure seen at inference time.