---
title: 'MobileMoE: On-Device Mixture-of-Experts'
url: https://www.emergentmind.com/topics/mobilemoe
type: topic
---

# MobileMoE: On-Device Mixture-of-Experts

MobileMoE denotes a line of research on making Mixture-of-Experts language models practical on mobile and edge hardware. In its primary usage, it refers to a family of on-device MoE language models with sub-billion active parameters, specifically **0.3–0.9B active** and **1.3–5.3B total** parameters, trained and deployed under explicit mobile memory and compute constraints [2605.27358]. In a broader systems usage, the term also encompasses mobile- and edge-oriented MoE inference techniques that modify routing, caching, offloading, and scheduling so that sparse expert models remain efficient when DRAM, VRAM, bandwidth, and batch size are tightly constrained [2412.00099].

## 1. Concept and problem setting

MobileMoE starts from a simple systems observation: dense on-device LLMs pay the full parameter cost on every token, whereas MoE decouples **total capacity** from **active compute** by activating only a subset of experts per token [2605.27358]. The on-device motivation is explicitly phase-dependent. In the reported formulation, **prefill** is mostly compute-bound, so reducing active parameters reduces latency; **decode** is often memory-bandwidth-bound, so reducing the active weights per token reduces memory traffic. The same work argues that modern phones now have enough DRAM to host a few-billion-parameter model in low precision, making sparse models viable in a way that earlier mobile deployments were not.

The central design tension is that mobile deployment is constrained simultaneously by inference FLOPs, weight memory, and KV-cache growth. MobileMoE therefore does not treat “smallest possible model” as the objective. Instead, it targets the best architecture under both compute and memory limits, with active parameters governing per-token compute and total parameters governing weight memory [2605.27358]. This is a different optimization target from server-side MoE systems, where expert parallelism and aggregate memory are often assumed.

A second problem setting appears in mobile runtime work that applies MoE to existing backbones rather than training a new family from scratch. In "Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference" [2412.00099], the bottleneck is not training or architecture selection but **sequential batch-1 autoregressive generation** on devices where only a subset of expert weights fit in DRAM. That work formalizes the cache hit rate as
$$
\operatorname{hit-rate} = \frac{\operatorname{count}(\mathbf{r}[:K] \cap C)}{K},
$$
where \(C\) is the expert cache set, and argues that naive LRU suffers because expert selection varies significantly across adjacent tokens. Taken together, these papers establish MobileMoE as both an architectural program for on-device sparse models and a runtime program for cache- and transfer-aware expert execution.

## 2. Scaling-law formulation and architectural sweet spot

The defining theoretical contribution of the 2026 MobileMoE paper is an **on-device MoE scaling law** that jointly optimizes MoE architecture under mobile memory and compute constraints [2605.27358]. The reported conclusion is that the best on-device regime is not extreme sparsity but **moderate sparsity with fine-grained and shared experts**. In the architecture sweep, the key findings are that the best expert count is around **\(E=8\)**, that finer sub-expert granularity improves loss with diminishing returns beyond **\(g=8\)**, and that adding a shared always-on expert improves performance at fixed compute.

The resulting “sweet spot” is described as
$$
\mathrm{MoE}(E=8, g=8, s=\checkmark),
$$
which corresponds in the final design to **60 routed fine-grained experts plus one shared expert**, with **top-4 routing** [2605.27358]. The backbone remains aligned with contemporary on-device design norms: a decoder-only transformer with **GQA with 4 KV heads**, a **width-depth aspect ratio around 40**, and **SwiGLU FFNs**.

Three model scales instantiate this design:

| Model | Active / total parameters | INT4 weight memory |
|---|---:|---:|
| MobileMoE-S | 272M / 1.3B | 0.68 GB |
| MobileMoE-M | 528M / 2.8B | 1.48 GB |
| MobileMoE-L | 922M / 5.3B | 2.75 GB |

The corresponding proxy memory including KV cache at 8k context is reported as **0.76 / 1.58 / 2.88 GB** for S, M, and L, respectively [2605.27358]. A plausible implication is that MobileMoE treats expert count as a hardware-constrained variable rather than a purely modeling variable: too few experts lose the benefits of sparsity, whereas too many experts inflate total memory even if active compute remains low.

## 3. Training pipeline and quantization

MobileMoE uses a four-stage recipe consisting of **pre-training**, **mid-training**, **instruction fine-tuning**, and **quantization-aware training** [2605.27358]. The pre-training stage uses **context length 2048** and **~6T tokens** from an open-licensed web-heavy mixture plus code, math, science, and knowledge sources. Mid-training raises context length to **8192** and adds **~500B** tokens from more curated data emphasizing knowledge, code, math, science, and long-document sources. Instruction tuning is also run at **8192** context over **>80M samples** spanning math, code, general chat/instructions, science/knowledge, reasoning, tool use, and safety.

The stabilization details are characteristic of modern sparse training. The reported recipe includes **auxiliary-loss-free load balancing**, **router z-loss**, **sigmoid gating with top-k normalization**, **router computations in FP32**, **grouped MLP / fused grouped matmul**, **drop-and-pad token dispatching**, and **expert parallelism** [2605.27358]. These are not incidental engineering choices; they define the regime in which the scaling law is made operational.

The deployable models are produced by **quantization-aware training** with **INT4 weights** and **INT8 activations**, while the router remains **FP32**. Weight quantization is described as **symmetric group-wise INT4** with **group size 32**, using
$$
\tilde{\mathbf{W}}_g = s_g \cdot \text{clamp}\!\left(\left\lfloor \frac{\mathbf{W}_g}{s_g} \right\rceil, q_{\min}, q_{\max}\right),
\qquad
s_g = \frac{2\max(|\mathbf{W}_g|)}{2^b - 1},
$$
with \(b=4\), \(q_{\min}=-2^{b-1}\), and \(q_{\max}=2^{b-1}-1\) [2605.27358]. The reported BF16-to-INT4 drops are modest: **46.7 → 44.0** for S, **55.3 → 52.5** for M, and **60.1 → 57.8** for L. This suggests that the MobileMoE design is quantization-aware not only at the kernel level but already at the model-family level.

## 4. Smartphone runtime and systems implementation

A distinctive feature of MobileMoE is that it includes direct deployment on **Samsung Galaxy S25 with Snapdragon 8 Elite** and **iPhone 16 Pro with Apple A18 Pro** [2605.27358]. The implementation uses a **custom fused MoE kernel in ExecuTorch**, motivated by the observation that existing mobile inference stacks generally do not support efficient MoE natively.

The kernel design is explicitly sparse-runtime-aware. It **reorders tokens by expert IDs**, uses **dense grouped GEMMs per expert**, and **fuses top-k selection, dispatch, projections, activation, and unscatter** [2605.27358]. The runtime methodology uses **batch size 1**, **INT4 weights**, **INT8 dynamic activations**, real prompts from code, knowledge, and math domains, and context lengths from **256** to **8k** in prefill.

At comparable INT4 memory, **MobileMoE-S** is reported to be faster than **MobileLLM-Pro** on both phones. On Galaxy S25, prefill is about **1.8–2.2× faster** and decode about **2.2–2.6× faster**; on iPhone 16 Pro, prefill is about **2.7–3.1× faster** and decode about **2.8–3.4× faster**. The abstract summarizes this as **up to 1.8–3.8× faster prefill** and **2.2–3.4× faster decode** [2605.27358].

The memory profile is equally important. Real prompts are reported to consume more memory than dummy prompts because expert routing is input-dependent and activates diverse experts. Relative to MobileLLM-Pro, MobileMoE-S uses about **9% lower** peak RSS at **4k** and about **22% lower** at **8k**, despite slightly larger static INT4 weights. Even the large variant remains within mobile budgets: **MobileMoE-L stays under 5 GB peak RSS even at 8k context** [2605.27358]. A common misconception is that MoE necessarily worsens mobile memory pressure because total parameters increase; these measurements show that active-parameter efficiency can dominate the end-to-end memory profile.

## 5. Benchmark performance and Pareto trade-offs

MobileMoE is evaluated on **14 foundational tasks** spanning commonsense, knowledge, science, reading, and reasoning, and on **8 advanced tasks** after instruction tuning [2605.27358]. The reported pattern is monotonic with scale, summarized as **S < M < L**, and the scaling law is said to predict this monotonic improvement.

Against dense on-device baselines, MobileMoE **matches or beats** models including **Gemma 3 270M / 1B**, **SmolLM2 360M / 1.7B**, **MobileLLM-Pro**, **Llama 3.2 1B**, **OLMo 2 1B**, and **Qwen3.5 0.8B / 2B** [2605.27358]. The abstract states that across 14 benchmarks MobileMoE **matches or exceeds leading on-device dense LLMs with 2–4× fewer inference FLOPs**. The strongest gains are reported on **knowledge**, **reading comprehension**, and, after SFT, **math and code**.

The comparison with the prior MoE baseline **OLMoE-1B-7B** is especially consequential. **MobileMoE-M** is reported to match OLMoE-1B-7B with about **60% fewer active and total parameters**. **MobileMoE-L** exceeds OLMoE-1B-7B with **30% fewer active parameters** and **23% smaller total model memory**. In the foundational table, **MobileMoE-L reaches about 59.8 average after mid-training**, versus **52.4** for **OLMoE-1B-7B base** [2605.27358].

The paper also emphasizes token efficiency. MobileMoE reaches these results with **~6T pre-training tokens**, compared with **9T** for **Llama 3.2 1B** and **11T** for **SmolLM2 1.7B** [2605.27358]. This suggests that MobileMoE is not merely a deployment optimization but also a data-efficient modeling strategy at the sub-billion active-parameter scale.

## 6. Relation to adjacent mobile and edge MoE systems

The broader literature shows that MobileMoE is not a single technique but a design space. One branch redesigns the model family for phones, as in MobileMoE itself [2605.27358]. Another retains an existing MoE backbone and changes runtime behavior. "Mixture of Cache-Conditional Experts for Efficient Mobile Device Inference" modifies routing at inference time by adding a cache prior to router logits,
$$
\mathbf{z}' = \mathbf{z} + \lambda \cdot \Delta_{\text{avg}} \cdot \tilde{\mathbf{m}}_t,
$$
using \(\mathbf{z}'\) only for reranking while preserving the original router probabilities for the actual MoE output; on two Qualcomm Snapdragon-based Android 14 devices running **Qwen1.5-MoE-A2.7B**, it reports **over 2× speedup** on-device while preserving output quality reasonably well [2412.00099]. This establishes a training-free MobileMoE interpretation in which routing is optimized subject to cache locality rather than learned relevance alone.

A more aggressive runtime branch treats offloading as the main problem. "Enabling MoE on the Edge via Importance-Driven Expert Scheduling" uses router scores to distinguish **top-score experts**, **low-score active experts**, and **alternative experts** around the \((k+1)\)-th score boundary, replacing low-score off-GPU experts with similar GPU-cached alternatives. The reported result is **48% lower decoding latency** with **over 60% expert cache hit rate** and **nearly lossless accuracy** [2508.18983]. "MoE-Beyond" replaces heuristic caching with a learned transformer predictor trained on **66 million** activation traces and reports **~72% cache hit rate** when only **10% of experts fit in GPU cache**, versus **17%** for **MoE-Infinity** [2508.17137]. "D\(^{2}\)MoE" adds dynamic bit-width routing, Matryoshka Weight Quantization, and HEBF scheduling, yielding **up to 1.39× throughput improvement** and **up to 53%** lower peak memory on real edge devices [2504.15299].

The literature also includes approaches that depart from caching entirely. "OD-MoE" argues that edge-friendly MoE inference should not rely on a long-lived expert cache, using distributed worker nodes and a quantized shadow model for multi-layer-ahead prediction. It reports **99.94%** expert activation prediction accuracy, about **75.51%** of the decoding throughput of a fully GPU-cached baseline, and only **1/3** of the GPU memory, with **less than 1 GB** GPU memory required per worker node [2512.03927]. By contrast, "MoBiLE" keeps the offloading model but changes the activation budget, running a **little model** with \(\frac{K}{2}\) experts for unimportant tokens and falling back to the full **big model** when confidence is below \(\gamma\); it reports **1.60× to 1.72×** speedups with negligible degradation in accuracy on a consumer GPU [2510.12357].

A further extension distributes experts across the edge network. "WDMoE" places the gating network and preceding neural layer at the base station while distributing expert networks across mobile devices, jointly optimizing expert selection and bandwidth allocation through a **Weight-to-Latency Ratio** formulation and validating the design on a Jetson-based hardware testbed [2411.06681]. "CoMoE" pushes the same idea toward heterogeneous mobile edge systems by jointly optimizing expert aggregation and expert offloading; for **Switch-Base-128**, it reduces memory from **15.6 GB** to **4.7 GB** and reports **10.5% lower inference latency** than existing expert offloading techniques while maintaining performance stability [2508.09208].

Taken together, these results suggest that “MobileMoE” now spans two complementary meanings. One is a specific family of on-device sparse language models whose architecture, training, quantization, and runtime are all co-designed for smartphones [2605.27358]. The other is a broader methodological category in which MoE deployment on mobile and edge hardware is made feasible through cache-aware routing, importance-driven substitution, learned activation prediction, dynamic precision, just-in-time loading, or wireless distribution. The shared premise across both meanings is that mobile viability depends less on nominal parameter count than on controlling **active parameters**, **memory movement**, and **expert locality** under real device constraints.

Source: https://www.emergentmind.com/topics/mobilemoe