MobileMoE: Scaling On-Device Mixture of Experts

Abstract: Mixture-of-Experts (MoE) has become the de facto architecture for hundred-billion-parameter LLMs, yet its advantages at sub-billion scales for on-device deployment remain largely unexplored. To close this gap, we present MobileMoE, a family of on-device MoE LLMs with sub-billion active parameters (0.3-0.9B active and 1.3-5.3B total) that establish a new Pareto frontier for on-device LLMs. We first formulate an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints, identifying an on-device sweet spot - moderate sparsity with fine-grained and shared experts - that is simultaneously memory and compute-optimal. Building on the derived architectures, we train MobileMoE with a four-stage recipe covering pre-training, mid-training, instruction fine-tuning, and quantization-aware training, all on open-source datasets. Across 14 benchmarks, MobileMoE matches or exceeds leading on-device dense LLMs with 2-4$\times$ fewer inference FLOPs, and matches or surpasses the state-of-the-art MoE OLMoE-1B-7B with up to 60% fewer parameters. To bridge the last mile to mobile deployment, we provide the first efficient MoE inference on commodity smartphones with comprehensive on-device profiling. At comparable INT4 weight memory, MobileMoE-S delivers $1.8$-$3.8\times$ faster prefill and $2.2$-$3.4\times$ faster decode than the dense baseline MobileLLM-Pro.

• The paper introduces a novel on-device MoE scaling law that jointly optimizes expert count, granularity, and shared experts under compute and memory constraints.
• It employs a four-stage training pipeline—pre-training, mid-training, supervised fine-tuning, and quantization-aware training—to enhance expert specialization and efficiency.
• Empirical results demonstrate that MobileMoE outperforms dense models and prior MoE baselines on benchmark tasks while reducing inference FLOPs and memory usage.

MobileMoE: Scaling Mixture-of-Experts for Efficient On-Device Language Modeling

Motivation and Problem Formulation

The paper introduces MobileMoE, a class of Mixture-of-Experts (MoE) LLMs explicitly designed for deployment on resource-constrained edge devices. While MoE architectures have become standard in high-capacity server LLMs, their adoption in the sub-billion parameter regime—necessary for typical on-device LLMs—has not been systematically investigated. This gap is critical given the increasing demand for edge intelligence on smartphones and wearables, which imposes stringent memory and compute budgets. The authors propose an on-device MoE scaling law that jointly optimizes model architecture under real-world device constraints, enabling parameter-efficient, inference-efficient, and memory-optimal design.

On-Device Scaling Law and Architectural Optimization

A central contribution is the formulation of a generalized scaling law for the on-device regime. Unlike prior scaling laws focused on server-side models [kaplan2020scaling, hoffmann2022chinchilla], this law incorporates both compute and memory constraints relevant for mobile hardware. The law guides the joint optimization of:

• Expert Count ($E$): Controls model sparsity and total parameter count.
• Expert Granularity ($g$): Dictates sub-expert configuration for routing flexibility.
• Shared Expert ($s$): Adds an always-on pathway for generalist computation.

Through systematic scaling studies and parametric sweeps, MobileMoE identifies a sweet spot: moderate sparsity (e.g., $E=8$), fine-grained granularity ($g=8$), and a shared expert pathway, achieving optimal trade-offs between inference compute and memory footprint.

Figure 1: Left: MoE architectural factors; Right: MobileMoE configurations at Small, Medium, Large scales with optimized active parameter counts and base settings.

Scaling studies show that moderate sparsity outperforms dense and highly sparse alternatives both in compute and memory efficiency, and that fine-grained experts yield substantial improvements at fixed compute, saturating at $g=8$. The addition of a shared expert further reduces loss at fixed compute, as predicted by the fitted scaling law.

Figure 2: Model loss as a function of the number of experts ($E$) for different memory and compute budgets, validating the sweet spot for $E=8$ in the on-device regime.

Figure 3: Compute-optimal loss curves for variations in $E$, $g$, and shared expert configurations, highlighting the optimal trade-offs and diminishing returns for increased complexity.

Training Pipeline and Data Composition

To realize the architectural benefits, MobileMoE employs a four-stage training pipeline:

1. Pre-Training: Diverse web and domain-specific data to encourage expert specialization.
2. Mid-Training: Upweighting high-quality, domain-centric sources, and extending context length.
3. Supervised Fine-Tuning (SFT): Instruction-following datasets for downstream adaptability.
4. Quantization-Aware Training (QAT): INT4 precision for deployment, preserving nearly all accuracy with $g$0 reduced weight memory.

Auxiliary-loss-free load balancing and router regularization stabilize MoE-specific training. Grouped MLP and expert-parallel GPU strategies ensure efficient throughput.

Figure 4: Four-stage training pipeline: PT → MT → SFT → QAT, each tailored for MoE-specific efficiency and stability.

Figure 5: Domain composition per training stage, shifting from web-heavy to domain-specialized mixtures across PT, MT, SFT.

Empirical Results: Benchmark and On-Device Performance

Benchmarking and Pareto Frontier

MobileMoE establishes new Pareto frontiers on 14 foundational benchmarks spanning commonsense, knowledge, science, reading, and reasoning.

• MobileMoE-S/M: Matches or exceeds dense baselines with 2–4$g$1 fewer inference FLOPs.
• MobileMoE-L: Surpasses state-of-the-art MoE OLMoE-1B-7B with 30% fewer active parameters and 23% smaller memory.

Numerical results show MobileMoE-L averages 60.1% benchmark accuracy post-SFT, compared to OLMoE-1B-7B's 55.6% at higher parameter scale.

Figure 6: MobileMoE achieves a new Pareto frontier: average accuracy plotted against inference compute ($g$2) and total parameters/memory.

Training Efficiency and Capability Progression

Training loss vs. wall-clock performance confirms MobileMoE's architectural choices are also efficient in real training throughput. Capability progression across PT, MT, and SFT stages demonstrates that mid-training primarily improves knowledge and reading tasks, while SFT unlocks multi-step reasoning capabilities.

Figure 7: Training efficiency curves for varying $g$3, $g$4, $g$5, validating architectural sweet spots for memory and throughput.

Figure 8: Benchmark accuracy progression for MobileMoE-S/M/L across PT, MT, SFT stages; dashed line denotes reference MoE baseline.

On-Device Deployment: Runtime and Memory Profiling

MobileMoE models are deployed on flagship smartphones with custom fused MoE kernels. At comparable INT4 memory:

• MobileMoE-S: $g$6–$g$7 faster prefill and $g$8–$g$9 faster decode than MobileLLM-Pro.
• Peak RSS memory: At 8k context, MobileMoE-S uses 22% less runtime memory than MobileLLM-Pro.

Empirical profiling on both Qualcomm and Apple silicons across CPU/GPU backends verifies consistency and portability.

(Figure 6, revisited)

Figure 6: MobileMoE achieves superior accuracy at lower compute/memory cost, shifting the Pareto frontier for on-device LLMs.

Theoretical and Practical Implications

The results strongly validate the applicability of MoE for edge deployment in the sub-billion parameter regime. The architectural and training pipeline innovations enable local intelligence with lower power, reduced cloud dependency, and enhanced privacy. The scaling law provides a principled methodology for further optimization without brute-force architecture sweeps.

On-device inference demonstrates that MoE's theoretical advantages in compute and memory translate into real gains, enabled by bespoke kernel implementations. Fine-grained expert specialization opens further research directions in runtime compression, dynamic routing, and adaptive expert loading. The work sets the stage for multimodal, multimodel, and distillation-enhanced edge LLMs.

Conclusion

MobileMoE provides a principled framework for scaling MoE architectures under mobile constraints, achieving state-of-the-art on-device LLM performance in both accuracy and efficiency. The combination of architectural optimization, targeted training pipeline, and customized deployment establishes MoE as a superior alternative to dense approaches in the edge regime. Future work may extend MoE runtime and training through distillation, expert pruning, NPUs, and multimodal adaptation, further unlocking efficient local AI.