MobileMoE: On-Device Mixture-of-Experts

• MobileMoE is a design of on-device sparse language models that decouples total capacity from active compute by activating a subset of experts per token.
• It employs fine-grained routing, cache-aware techniques, and INT4 quantization to optimize performance under stringent mobile memory and compute constraints.
• Scalable model variants demonstrate significant speed and memory efficiency improvements on devices like the Samsung Galaxy S25 and iPhone 16 Pro.

MobileMoE denotes a line of research on making Mixture-of-Experts LLMs practical on mobile and edge hardware. In its primary usage, it refers to a family of on-device MoE LLMs with sub-billion active parameters, specifically 0.3–0.9B active and 1.3–5.3B total parameters, trained and deployed under explicit mobile memory and compute constraints (Chen et al., 26 May 2026). In a broader systems usage, the term also encompasses mobile- and edge-oriented MoE inference techniques that modify routing, caching, offloading, and scheduling so that sparse expert models remain efficient when DRAM, VRAM, bandwidth, and batch size are tightly constrained (Skliar et al., 2024).

1. Concept and problem setting

MobileMoE starts from a simple systems observation: dense on-device LLMs pay the full parameter cost on every token, whereas MoE decouples total capacity from active compute by activating only a subset of experts per token (Chen et al., 26 May 2026). The on-device motivation is explicitly phase-dependent. In the reported formulation, prefill is mostly compute-bound, so reducing active parameters reduces latency; decode is often memory-bandwidth-bound, so reducing the active weights per token reduces memory traffic. The same work argues that modern phones now have enough DRAM to host a few-billion-parameter model in low precision, making sparse models viable in a way that earlier mobile deployments were not.

The central design tension is that mobile deployment is constrained simultaneously by inference FLOPs, weight memory, and KV-cache growth. MobileMoE therefore does not treat “smallest possible model” as the objective. Instead, it targets the best architecture under both compute and memory limits, with active parameters governing per-token compute and total parameters governing weight memory (Chen et al., 26 May 2026). This is a different optimization target from server-side MoE systems, where expert parallelism and aggregate memory are often assumed.

A second problem setting appears in mobile runtime work that applies MoE to existing backbones rather than training a new family from scratch. In "Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference" (Skliar et al., 2024), the bottleneck is not training or architecture selection but sequential batch-1 autoregressive generation on devices where only a subset of expert weights fit in DRAM. That work formalizes the cache hit rate as

$$\operatorname{hit-rate} = \frac{\operatorname{count}(\mathbf{r}[:K] \cap C)}{K},$$

where $C$ is the expert cache set, and argues that naive LRU suffers because expert selection varies significantly across adjacent tokens. Taken together, these papers establish MobileMoE as both an architectural program for on-device sparse models and a runtime program for cache- and transfer-aware expert execution.

2. Scaling-law formulation and architectural sweet spot

The defining theoretical contribution of the 2026 MobileMoE paper is an on-device MoE scaling law that jointly optimizes MoE architecture under mobile memory and compute constraints (Chen et al., 26 May 2026). The reported conclusion is that the best on-device regime is not extreme sparsity but moderate sparsity with fine-grained and shared experts. In the architecture sweep, the key findings are that the best expert count is around $E=8$, that finer sub-expert granularity improves loss with diminishing returns beyond $g=8$, and that adding a shared always-on expert improves performance at fixed compute.

The resulting “sweet spot” is described as

$\mathrm{MoE}(E=8, g=8, s=\checkmark),$

which corresponds in the final design to 60 routed fine-grained experts plus one shared expert, with top-4 routing (Chen et al., 26 May 2026). The backbone remains aligned with contemporary on-device design norms: a decoder-only transformer with GQA with 4 KV heads, a width-depth aspect ratio around 40, and SwiGLU FFNs.

Three model scales instantiate this design:

Model	Active / total parameters	INT4 weight memory
MobileMoE-S	272M / 1.3B	0.68 GB
MobileMoE-M	528M / 2.8B	1.48 GB
MobileMoE-L	922M / 5.3B	2.75 GB

The corresponding proxy memory including KV cache at 8k context is reported as 0.76 / 1.58 / 2.88 GB for S, M, and L, respectively (Chen et al., 26 May 2026). A plausible implication is that MobileMoE treats expert count as a hardware-constrained variable rather than a purely modeling variable: too few experts lose the benefits of sparsity, whereas too many experts inflate total memory even if active compute remains low.

3. Training pipeline and quantization

MobileMoE uses a four-stage recipe consisting of pre-training, mid-training, instruction fine-tuning, and quantization-aware training (Chen et al., 26 May 2026). The pre-training stage uses context length 2048 and ~6T tokens from an open-licensed web-heavy mixture plus code, math, science, and knowledge sources. Mid-training raises context length to 8192 and adds ~500B tokens from more curated data emphasizing knowledge, code, math, science, and long-document sources. Instruction tuning is also run at 8192 context over >80M samples spanning math, code, general chat/instructions, science/knowledge, reasoning, tool use, and safety.

The stabilization details are characteristic of modern sparse training. The reported recipe includes auxiliary-loss-free load balancing, router z-loss, sigmoid gating with top-k normalization, router computations in FP32, grouped MLP / fused grouped matmul, drop-and-pad token dispatching, and expert parallelism (Chen et al., 26 May 2026). These are not incidental engineering choices; they define the regime in which the scaling law is made operational.

The deployable models are produced by quantization-aware training with INT4 weights and INT8 activations, while the router remains FP32. Weight quantization is described as symmetric group-wise INT4 with group size 32, using

$$\tilde{\mathbf{W}}_g = s_g \cdot \text{clamp}\!\left(\left\lfloor \frac{\mathbf{W}_g}{s_g} \right\rceil, q_{\min}, q_{\max}\right), \qquad s_g = \frac{2\max(|\mathbf{W}_g|)}{2^b - 1},$$

with $b=4$, $q_{\min}=-2^{b-1}$, and $q_{\max}=2^{b-1}-1$ (Chen et al., 26 May 2026). The reported BF16-to-INT4 drops are modest: 46.7 → 44.0 for S, 55.3 → 52.5 for M, and 60.1 → 57.8 for L. This suggests that the MobileMoE design is quantization-aware not only at the kernel level but already at the model-family level.

4. Smartphone runtime and systems implementation

A distinctive feature of MobileMoE is that it includes direct deployment on Samsung Galaxy S25 with Snapdragon 8 Elite and iPhone 16 Pro with Apple A18 Pro (Chen et al., 26 May 2026). The implementation uses a custom fused MoE kernel in ExecuTorch, motivated by the observation that existing mobile inference stacks generally do not support efficient MoE natively.

The kernel design is explicitly sparse-runtime-aware. It reorders tokens by expert IDs, uses dense grouped GEMMs per expert, and fuses top-k selection, dispatch, projections, activation, and unscatter (Chen et al., 26 May 2026). The runtime methodology uses batch size 1, INT4 weights, INT8 dynamic activations, real prompts from code, knowledge, and math domains, and context lengths from 256 to 8k in prefill.

At comparable INT4 memory, MobileMoE-S is reported to be faster than MobileLLM-Pro on both phones. On Galaxy S25, prefill is about 1.8–2.2× faster and decode about 2.2–2.6× faster; on iPhone 16 Pro, prefill is about 2.7–3.1× faster and decode about 2.8–3.4× faster. The abstract summarizes this as up to 1.8–3.8× faster prefill and 2.2–3.4× faster decode (Chen et al., 26 May 2026).

The memory profile is equally important. Real prompts are reported to consume more memory than dummy prompts because expert routing is input-dependent and activates diverse experts. Relative to MobileLLM-Pro, MobileMoE-S uses about 9% lower peak RSS at 4k and about 22% lower at 8k, despite slightly larger static INT4 weights. Even the large variant remains within mobile budgets: MobileMoE-L stays under 5 GB peak RSS even at 8k context (Chen et al., 26 May 2026). A common misconception is that MoE necessarily worsens mobile memory pressure because total parameters increase; these measurements show that active-parameter efficiency can dominate the end-to-end memory profile.

5. Benchmark performance and Pareto trade-offs

MobileMoE is evaluated on 14 foundational tasks spanning commonsense, knowledge, science, reading, and reasoning, and on 8 advanced tasks after instruction tuning (Chen et al., 26 May 2026). The reported pattern is monotonic with scale, summarized as S < M < L, and the scaling law is said to predict this monotonic improvement.

Against dense on-device baselines, MobileMoE matches or beats models including Gemma 3 270M / 1B, SmolLM2 360M / 1.7B, MobileLLM-Pro, Llama 3.2 1B, OLMo 2 1B, and Qwen3.5 0.8B / 2B (Chen et al., 26 May 2026). The abstract states that across 14 benchmarks MobileMoE matches or exceeds leading on-device dense LLMs with 2–4× fewer inference FLOPs. The strongest gains are reported on knowledge, reading comprehension, and, after SFT, math and code.

The comparison with the prior MoE baseline OLMoE-1B-7B is especially consequential. MobileMoE-M is reported to match OLMoE-1B-7B with about 60% fewer active and total parameters. MobileMoE-L exceeds OLMoE-1B-7B with 30% fewer active parameters and 23% smaller total model memory. In the foundational table, MobileMoE-L reaches about 59.8 average after mid-training, versus 52.4 for OLMoE-1B-7B base (Chen et al., 26 May 2026).

The paper also emphasizes token efficiency. MobileMoE reaches these results with ~6T pre-training tokens, compared with 9T for Llama 3.2 1B and 11T for SmolLM2 1.7B (Chen et al., 26 May 2026). This suggests that MobileMoE is not merely a deployment optimization but also a data-efficient modeling strategy at the sub-billion active-parameter scale.

6. Relation to adjacent mobile and edge MoE systems

The broader literature shows that MobileMoE is not a single technique but a design space. One branch redesigns the model family for phones, as in MobileMoE itself (Chen et al., 26 May 2026). Another retains an existing MoE backbone and changes runtime behavior. "Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference" modifies routing at inference time by adding a cache prior to router logits,

$$\mathbf{z}' = \mathbf{z} + \lambda \cdot \Delta_{\text{avg}} \cdot \tilde{\mathbf{m}}_t,$$

using $C$0 only for reranking while preserving the original router probabilities for the actual MoE output; on two Qualcomm Snapdragon-based Android 14 devices running Qwen1.5-MoE-A2.7B, it reports over 2× speedup on-device while preserving output quality reasonably well (Skliar et al., 2024). This establishes a training-free MobileMoE interpretation in which routing is optimized subject to cache locality rather than learned relevance alone.

A more aggressive runtime branch treats offloading as the main problem. "Enabling MoE on the Edge via Importance-Driven Expert Scheduling" uses router scores to distinguish top-score experts, low-score active experts, and alternative experts around the $C$1-th score boundary, replacing low-score off-GPU experts with similar GPU-cached alternatives. The reported result is 48% lower decoding latency with over 60% expert cache hit rate and nearly lossless accuracy (Zhu et al., 26 Aug 2025). "MoE-Beyond" replaces heuristic caching with a learned transformer predictor trained on 66 million activation traces and reports ~72% cache hit rate when only 10% of experts fit in GPU cache, versus 17% for MoE-Infinity (Gavhane et al., 23 Aug 2025). "D$C$2MoE" adds dynamic bit-width routing, Matryoshka Weight Quantization, and HEBF scheduling, yielding up to 1.39× throughput improvement and up to 53% lower peak memory on real edge devices (Wang et al., 17 Apr 2025).

The literature also includes approaches that depart from caching entirely. "OD-MoE" argues that edge-friendly MoE inference should not rely on a long-lived expert cache, using distributed worker nodes and a quantized shadow model for multi-layer-ahead prediction. It reports 99.94% expert activation prediction accuracy, about 75.51% of the decoding throughput of a fully GPU-cached baseline, and only 1/3 of the GPU memory, with less than 1 GB GPU memory required per worker node (Wang et al., 3 Dec 2025). By contrast, "MoBiLE" keeps the offloading model but changes the activation budget, running a little model with $C$3 experts for unimportant tokens and falling back to the full big model when confidence is below $C$4; it reports 1.60× to 1.72× speedups with negligible degradation in accuracy on a consumer GPU (Zhao et al., 14 Oct 2025).

A further extension distributes experts across the edge network. "WDMoE" places the gating network and preceding neural layer at the base station while distributing expert networks across mobile devices, jointly optimizing expert selection and bandwidth allocation through a Weight-to-Latency Ratio formulation and validating the design on a Jetson-based hardware testbed (Xue et al., 2024). "CoMoE" pushes the same idea toward heterogeneous mobile edge systems by jointly optimizing expert aggregation and expert offloading; for Switch-Base-128, it reduces memory from 15.6 GB to 4.7 GB and reports 10.5% lower inference latency than existing expert offloading techniques while maintaining performance stability (Li et al., 10 Aug 2025).

Taken together, these results suggest that “MobileMoE” now spans two complementary meanings. One is a specific family of on-device sparse LLMs whose architecture, training, quantization, and runtime are all co-designed for smartphones (Chen et al., 26 May 2026). The other is a broader methodological category in which MoE deployment on mobile and edge hardware is made feasible through cache-aware routing, importance-driven substitution, learned activation prediction, dynamic precision, just-in-time loading, or wireless distribution. The shared premise across both meanings is that mobile viability depends less on nominal parameter count than on controlling active parameters, memory movement, and expert locality under real device constraints.