FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Abstract: Mixture-of-Experts (MoE) models have become a dominant paradigm for scaling LLMs, but their rapidly growing parameter sizes introduce a fundamental inefficiency during inference: most expert weights remain idle in GPU memory while competing with performance-critical runtime state such as the key-value (KV) cache. Since KV cache capacity directly determines serving throughput, this mismatch leads to underutilized memory and degraded performance. In this paper, we present FluxMoE, a new MoE inference system that decouples expert parameters from persistent GPU residency. FluxMoE introduces an expert paging abstraction that treats expert weights as streamed, transient resources, materializing them on demand and evicting them immediately after use, allowing GPU memory to be preferentially allocated to throughput-critical runtime state. We implement FluxMoE atop vLLM to enable efficient MoE inference under severe memory constraints. Experimental results demonstrate that FluxMoE achieves up to 3.0$\times$ throughput gains over vLLM in memory-intensive regimes, without compromising model fidelity.

• The paper introduces expert paging to decouple expert parameter residency from GPU memory, enabling efficient streaming of weights and reducing idle memory usage.
• It employs a hybrid storage system with a budget-aware residency planner to balance GPU memory allocation between the KV cache and expert streaming, achieving up to 3.0× throughput improvement.
• The design enables scalable MoE serving by allowing more concurrent contexts and longer input sequences under memory constraints while incurring minimal overhead.

FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving

Introduction and Motivation

Mixture-of-Experts (MoE) architectures have enabled the scaling of LLMs by replacing dense FFNs with sparse arrangements of many expert subnetworks. However, these designs come with a critical memory inefficiency at inference: the majority of the massive expert parameter pool remains idle in GPU memory while only a handful of experts are active per decoding step. This pressure is particularly impactful in production settings, where concurrent contexts and large batch sizes force the Key-Value (KV) cache to share limited GPU memory with resident weights.

Figure 1: Memory composition of Qwen3-Next-80B-A3B-Instruct during inference; model weights and KV cache occupy similar fractions of on-device memory, both scaling with workload size.

Prior work and system architectures have universally assumed static, GPU-resident weight parameters. For MoE models, whose expert pools can reach hundreds of gigabytes to terabytes [DeepSeekAI2025DeepSeekR1IR, Bai2025KimiKO], “idle” expert parameters consume the majority of GPU resources, constraining the KV cache and thus throughput.

FluxMoE Architecture and Core Abstractions

FluxMoE introduces an inference system predicated on decoupling expert parameters from persistent GPU residency. Its central abstraction is expert paging, where expert weights are streamed into GPU memory on demand and evicted immediately after use. This allows prioritization of GPU capacity for runtime state (KV cache and activations), directly improving throughput and model concurrency.

Key mechanisms include:

1. PagedTensor: A tensor virtualization layer that provides stable virtual addresses for all expert tensors, dynamically (re)binding on-device buffers to logical tensors prior to kernel launch, without modifying inference kernels.
2. Bandwidth-Balanced Storage Hierarchy: A multi-tier storage subsystem spanning compressed GPU memory and host DRAM, partitioning expert parameters in proportion to backend bandwidths, maximizing synchronous loading rates during expert materialization.
3. Budget-Aware Residency Planner: A runtime controller that dynamically determines the expert residency fraction in GPU memory by trading off KV cache pressure versus expert streaming bottlenecks based on observed system utilization.

   Figure 2: FluxMoE architecture—depicting PagedTensor, the hybrid storage hierarchy, and the residency planning feedback loop.

Expert Paging Formalism

Expert paging is realized as a sliding residency window accommodating the expert parameters of two consecutive layers at any time, minimizing resident expert footprint to $2/N$ of the full model size. For each MoE layer, preceding experts are evicted and subsequent ones asynchronously prefetched, pipelining parameter I/O with computation and ensuring high device utilization.

Figure 3: Overview of expert paging, showing paged-in, paged-out, and prefetch stages.

The PagedTensor apparatus provides programmable, stable virtual addresses for each logical weight tensor, maintaining a tight mapping between logical handles and recycled physical memory blocks. Asynchronous expert lifecycles and CUDA event barriers hide I/O latency behind layer computation, enforcing safe remapping (read-after-write, write-after-read) invariants for correctness and efficiency.

Storage Hierarchy and Compression

FluxMoE’s storage hierarchy comprises a compressed on-GPU backend (with selective lossless Huffman coding spanning expert exponent fields) and a host DRAM backend accessed via PCIe. Partitioning is bandwidth-tracking, with placement fractions $x_k$ for each backend $S_k$ allocated proportionally to their I/O throughput, thus eliminating backend-induced pipeline stalls.

Figure 4: Exponent (left) and mantissa (right) value distributions for MoE expert parameters, validating high compressibility in exponent fields.

This scheme allows the system to compress major share (>90%) of parameter size, saving at least ~20% of expert footprint for BFloat16 models with negligible fidelity risk; non-expert weights (e.g., attention projections) remain uncompressed and GPU resident for critical path latency.

Runtime Planning and Adaptation

The budget-aware residency planner controls the residency fraction $\alpha$, adjusting as follows:

• Decrease $\alpha$ when computation time dominates and less expert memory is needed, reclaiming memory for the KV cache.
• Increase $\alpha$ when parameter loading becomes a bottleneck, promoting experts to GPU residency.

It always respects a hard constraint giving absolute priority to maintaining a maximal KV cache. Adaptation is driven by a compute-to-load ratio metric $\rho = \tau_\text{comp(theory)} / \tau_\text{load}$, targeting $\rho \approx 1$ for optimal pipeline overlap.

Figure 5: Trends of aggregate expert loading time and computation time vs. expert residency fraction, indicating the compute-to-I/O-bound operating regime transition.

Evaluation and Results

The experimental evaluation spans both performance-bound (adequate memory) and capacity-bound (severe constraints) scenarios using Mixtral-8x7B-Instruct [jiang24] and Qwen3-Next-80B-A3B-Instruct [qwen3technicalreport] on NVIDIA L40 GPUs.

In performance-bound regimes, FluxMoE with all experts compressed in GPU memory achieves up to 3.0× throughput improvement over vLLM at high batch sizes and long contexts, primarily due to maximized on-device KV cache residency and deferred onset of swapping.

Figure 6: Throughput scaling vs. batch size and context length in the performance-bound regime.

Under capacity-bound settings—where dense parameter residency is infeasible and even expert offloading baselines become PCIe-bound—FluxMoE’s hybrid storage design significantly outperforms host-offload-only or single-layer granularity approaches, achieving 28.5% and 22.9% higher throughput (for short and long contexts, respectively) relative to hybrid ablations.

Figure 7: Throughput scaling under capacity-bound regime; FluxMoE achieves consistently higher throughput under severe GPU memory constraint.

Runtime adaptation experiments show the planner maintaining throughput comparable to the fixed-residency baseline, with dynamic expert offloading events as KV cache grows having negligible negative effect on throughput due to effective I/O-compute overlap.

Figure 8: Throughput stability during runtime adaptation—FluxMoE sustains stable throughput despite dynamic expert offloading and memory reclamation.

PagedTensor's management overhead is empirically negligible (<3%) across all workload settings when all experts are GPU-resident, indicating that virtualization does not sacrifice inference speed.

Figure 9: PagedTensor overhead analysis—FluxMoE throughput closely tracks vLLM in an uncompressed, all-resident expert configuration.

Discussion and Implications

FluxMoE redefines memory management in MoE inference by discarding the constraint that parameters must always reside in device memory. The expert paging abstraction enables new hybrid memory-compute execution models with reduced static memory commitment, which has several important implications:

• For large-scale deployments, more concurrent contexts and longer input sequences become tractable on fixed hardware budgets, maximizing infrastructure utilization.
• For multi-tenant or on-premise clusters, the recovered GPU memory may be directly redeployed for co-located tasks, batch co-scheduling, or dynamic expansion of KV cache size, further increasing aggregate system throughput.
• For model scaling, as MoE parameter counts approach multi-terabyte scale, expert paging provides the only practical path toward efficient inference without resorting to lossy compression or coarse, hardware-intensive expert parallelism.
• Compatibility with model fidelity is ensured, as FluxMoE only employs lossless compression for expert parameters—avoiding precision or reasoning degradation [li25quantization, liu25].

Potential future directions include generalization to multi-node distributed settings, prioritization policies for expert caching under adversarial workloads, and closer integration with dynamic KV cache resizing and adaptive sequence dropout strategies.

Conclusion

FluxMoE systematically decouples expert parameter residency from GPU memory for high-performance MoE inference, achieving strong throughput gains and robust resource adaptation under real hardware constraints. By formalizing expert paging, layering PagedTensor virtualization, bandwidth-optimized expert storage, and a closed-loop residency planner, FluxMoE presents a practical and theoretically sound framework for scalable MoE serving. Its architectural insights and quantitative results set a new standard for memory-aware model serving and will underpin subsequent advances in large-scale LLM system design.