MoEless: Efficient MoE LLM Serving via Serverless Computing

Abstract: LLMs have become a cornerstone of AI, driving progress across diverse domains such as content creation, search and recommendation systems, and AI-assisted workflows. To alleviate extreme training costs and advancing model scales, Mixture-of-Experts (MoE) has become a popular backbone for modern LLMs, which are commonly served in distributed deployment using expert parallelism (EP). However, MoE's sparse activation mechanism leads to severe expert load imbalance, where a few experts become overloaded while others remain idle, resulting in expert stragglers that inflate inference latency and serving cost. Existing expert load balancing solutions assume static resource configurations on serverful infrastructures, limiting expert scalability and elasticity, and resulting in either costly real-time expert swapping or degraded generation quality. We present MoEless, the first serverless MoE serving framework that mitigates expert load imbalance and accelerates inference via serverless experts. MoEless employs lightweight, layer-aware predictors to accurately estimate incoming expert load distributions and proactively identify stragglers. We design optimized expert scaling and placement strategies to maximize function locality, improve GPU utilization, and balance loads across experts and GPUs. MoEless is prototyped on top of Megatron-LM and deployed on an eight-GPU testbed. Experiments with open-source MoE models and real-world workloads show that MoEless reduces inference latency by 43% and inference cost by 84% compared to state-of-the-art solutions.

• The paper introduces MoEless, a serverless MoE serving framework that decouples experts from non-expert modules and combines layer-aware load prediction with elastic replica scaling and placement.
• MoEless reduces average MoE-layer forward latency by 43.19% versus Megatron-LM and 21.89% versus EPLB while preserving the model’s original gate routing and generation quality.
• The system cuts total inference cost by 92.68% versus Megatron-LM, 84.06% versus a perfect-balance Oracle, and 95.11% versus EPLB, though large-scale deployment and automatic parameter adaptation remain open challenges.

Motivation and problem

MoE-based LLMs reduce training and inference FLOPs by activating only a sparse subset of experts per token, but this sparsity produces severely skewed expert loads in distributed serving under expert parallelism (EP). Overloaded experts become stragglers that stall the all-to-all scatter/gather pipeline and inflate both latency and cost. The authors characterize this imbalance empirically for Mixtral-8×7B and Phi-3.5-MoE on ShareGPT and LMSYS-Chat-1M, showing layer-wise and time-varying expert demand driven by fluctuating request arrivals (replayed from Azure LLM traces) and intrinsic expert popularity skew.

Existing remedies—EPLB-style periodic expert replication and expert swapping in systems such as FasterMoE and Lina—assume fixed resource configurations on serverful infrastructure. They either pay costly real-time expert swapping with limited effectiveness, or, in lossy re-routing schemes such as Capacity-Aware Inference, sacrifice generation quality by overriding gate routing. The paper argues that fine-grained, elastic, and accurate expert scaling is required to balance loads without quality loss, and proposes serverless computing as the mechanism for that elasticity.

Design of MoEless

MoEless is presented as the first serverless MoE serving framework. Rather than encapsulating an entire LLM in serverless functions (as ServerlessLLM and successors do), it decouples experts from the MoE model: experts are packaged as independent functions executed in GPU-enabled Docker containers, while non-expert modules (attention, gating, embeddings) remain in data parallelism within Megatron-LM. The authors justify this split by noting that expert computation dominates latency and cost, and that experts' stateless, all-to-all communication pattern maps naturally onto serverless execution. The serving workflow comprises four stages: load prediction, expert scaling, expert placement, and serving with load evenly split across replicas.

Expert Load Predictor. Exploiting the high cosine similarity between the inputs of gate networks in nearby layers—a consequence of residual connections—the predictor feeds Layer $l$'s hidden states into a replica of Layer $(l{+}d)$'s gate network to speculate its load distribution, where $d$ is the prediction distance. The key design choice is layer-aware fine-tuning: profiling shows that early layers have lower gate-input similarity and less stable load distributions than later layers, so only layers below a target accuracy threshold (80%) are fine-tuned. Predictors share the gate networks' architecture (at most 80K parameters, under 4.16 MB per model), run asynchronously on dedicated CUDA streams, and fine-tune within five minutes on one GPU.

Expert Scaler. Given predicted loads $W_l$ for each layer, a greedy heuristic iteratively adds a replica to the most overloaded expert (selected via a max heap) and splits its load evenly, until the coefficient of variation (CV) of per-expert loads falls below a threshold $V$ (0.2) or a per-layer memory cap is reached.

Expert Placer. Replicas are assigned to GPUs via Join-the-Shortest-Queue, preferring reuse of previously placed (warm) replicas to avoid cold starts, and otherwise assigning the most-loaded replica to the least-loaded GPU. Cold starts are mitigated with standard pre-warming and keep-alive techniques; scaling and placement are asynchronous.

Problem formulation

The paper formulates joint replica-count and placement optimization as a multi-objective ILP minimizing total inference latency $T$ and cost $C$ (cost defined as memory-footprint × latency aggregated over iterations), subject to per-GPU memory constraints. Since the ILP is NP-hard and workloads are dynamic, the system uses the greedy heuristics above rather than an exact solver.

Evaluation

The prototype runs on Megatron-LM atop an eight-GPU NVIDIA A6000 testbed (pairwise NVLink, 48 GB per GPU), serving Mixtral-8×7B, Phi-3.5-MoE, and Llama-4-Scout with request arrivals replayed from Azure LLM traces over LMSYS-Chat-1M and ShareGPT. Baselines are vanilla Megatron-LM EP, EPLB (DeepSeek's periodic redundancy scheme), and a lossy Oracle upper bound that achieves perfect balance by ignoring gate routing.

Headline results:

• Latency: MoEless reduces average MoE layer forward latency by 43.19% versus Megatron-LM and 21.89% versus EPLB, and stays consistently closest to the Oracle CDF across all model–dataset combinations.
• Cost: total inference cost falls by 92.68% versus Megatron-LM, 84.06% versus Oracle, and 95.11% versus EPLB. The comparison against Oracle is notable: MoEless achieves lower cost than a baseline that attains ideal load balance, because Oracle still pays serverful resource costs while MoEless exploits pay-per-use elasticity.
• Prediction accuracy: the layer-aware fine-tuned gate predictors improve accuracy by up to 18% over Mixtral-offloading (which reuses raw gate networks) and 15% over ProMoE (which trains large external predictors), while using less than 2% of ProMoE's predictor memory footprint. Predicted versus actual load distributions show strong positive Pearson correlation.
• Sensitivity: increasing the prediction distance from 1 to 5 trades prediction accuracy for coarser estimates, raising forward time while reducing replica counts; the system uses $d{=}1$. Raising the CV threshold from 0.2 to 1.0 reduces replicas but increases straggler-induced latency; $V{=}0.2$ is chosen. Ablations confirm that removing any of the predictor, scaler, or placer degrades latency toward serverful baselines.
• Overheads: prediction delay is below 0.2 ms per layer; nearly all expert scaling and placement operations are warm-started.

Limitations and open questions

The paper is explicit about several constraints. Cold-start mitigation relies on standard pre-warming and fixed-duration keep-alive periods rather than more advanced runtime techniques. System parameters—the prediction distance and the CV threshold—are determined through offline profiling and fixed at deployment, rather than adapted automatically across models, datasets, or traffic regimes; the sensitivity results indicate these choices materially affect the latency–cost trade-off, so their transferability to unseen workloads remains an open question. The evaluation is confined to an eight-GPU testbed with pairwise NVLink, homogeneous A6000 GPUs, and Megatron-LM without native continuous batching (batching is emulated by second-granularity request aggregation), leaving behavior at datacenter scale, on heterogeneous interconnects, and under finer-grained batching untested. The linear processing-time and communication-time models underlying the ILP formulation are approximations whose fidelity under large batches is not separately validated. Finally, whether layer-aware fine-tuning generalizes to MoE architectures with different routing structures (e.g., shared experts or finer-grained experts) is not examined.

Conclusion

MoEless demonstrates that decoupling experts as serverless functions under expert parallelism, combined with lightweight layer-aware load predictors and greedy scaling/placement heuristics, can eliminate expert stragglers without degrading generation quality. On a small-scale testbed it reduces MoE layer forward latency by up to 43% and inference cost by up to 84–95% relative to serverful baselines, including a lossy perfect-balance Oracle on cost. The main open issues are automatic parameter adaptation, cold-start management beyond fixed keep-alive policies, and validation at production scale.