X-MoE: Adaptive Sparse Expert Models

• X-MoE is a specialized Mixture-of-Experts architecture that uses fine-grained and adaptive routing to select small, task-specific experts.
• Threshold-based routing dynamically adjusts the number of active experts per token, reducing computational load by over 50% in language modeling benchmarks.
• System-level innovations such as padding-free buffers and hybrid parallelism enhance scalability on HPC systems and heterogeneous hardware.

X-MoE is a term used in recent Mixture-of-Experts research for several closely related developments concerned with making sparse expert models more adaptive, efficient, and scalable. In one specific usage, “XMoE” denotes a sparse model with fine-grained and adaptive expert selection, built from small experts and a threshold-based router (Yang et al., 2024). In another, “X-MoE” denotes a cross-platform training system for emerging expert-specialized architectures such as DeepSeek-MoE, emphasizing padding-free training, redundancy-bypassing dispatch, and sequence-sharded MoE blocks on Frontier-class HPC systems (Yuan et al., 18 Aug 2025). This suggests a broader MoE design trajectory in which routing, expert granularity, and system execution are co-optimized rather than treated as separable problems.

1. Sparse expert computation and the baseline MoE formulation

The technical point of departure is the standard sparse MoE layer. One common formulation writes the output as

$y = \sum_{i=1}^{N} G(x)_i E_i(x),$

where $E_i$ are experts and $G(x)$ is the router or gating function (Vavre et al., 2024). In the standard trainable-router setting, expert probabilities are computed as

$p=\mathrm{Softmax}(Wh),$

and top-$k$ routing produces

$$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$

so every token uses exactly $k$ experts (Yang et al., 2024).

This baseline is sparse only at expert granularity. The original XMoE analysis argues that this remains computationally inefficient because each selected expert is still a full FFN, so many parameters inside the activated expert are used even when they contribute little or nothing to the token (Yang et al., 2024). A second inefficiency appears in later work on expert-specialized architectures: fixed top-$k$ is usually applied uniformly across layers or uniformly between training and inference, even though different layers, workloads, and hardware regimes respond differently to expert activation patterns (Chitty-Venkata et al., 2 Sep 2025).

The resulting research program keeps the sparse-expert premise but changes what “expert selection” means. Some papers make the unit of selection finer than a conventional FFN expert, some make the number of active experts adaptive rather than fixed, some let layers reuse experts beyond their own local pool, and some focus on how routing interacts with memory hierarchy, interconnects, or heterogeneous devices.

2. Fine-grained and adaptive selection in the original XMoE model

The 2024 XMoE architecture replaces FFN layers with many small FFN experts and uses a threshold-based router instead of fixed top-$k$ selection (Yang et al., 2024). After computing router probabilities $p$ and sorting them in descending order, XMoE selects the minimum number of top experts such that cumulative probability mass exceeds a threshold $E_i$0: $E_i$1 This makes the number of active experts adaptive per token. The paper explicitly notes that $E_i$2 sends every token to all experts, whereas $E_i$3 behaves like top-1 routing.

This design couples two claims. First, smaller experts make expert selection more granular, so the model can activate only the necessary parameters. Second, threshold routing allocates variable compute: easy tokens can stop early, while harder tokens can use more experts. XMoE models the expert capacity as

$E_i$4

and approximates MoE-layer complexity as

$E_i$5

with the explicit strategy of reducing $E_i$6 via small experts while allowing a larger capacity factor $E_i$7 (Yang et al., 2024).

Empirically, XMoE reports that it can enhance model performance while decreasing the computation load at MoE layers by over $E_i$8 without sacrificing performance, and it also applies the same decomposition idea to dense models so that training uses $E_i$9 and $G(x)$0, while inference can later become sparse (Yang et al., 2024). On language modeling, the best reported configuration around $G(x)$1M parameters beats top-$G(x)$2 routing by $G(x)$3 perplexity points and Switch Transformer by $G(x)$4 perplexity points on OpenWebText; on WikiText-103, an XMoE configuration with expert size $G(x)$5 outperforms Switch Transformer while using only $G(x)$6 of the FLOPs of the baseline (Yang et al., 2024). The same paper also records a practical caveat: decreasing FLOPs does not map monotonically to wall time, because sparse computation is not fully efficient on current GPU/TPU hardware and very small experts can raise routing and sorting overhead.

3. X-MoE as an HPC training system for expert-specialized architectures

The 2025 X-MoE system addresses a different, but related, problem: scalable training of emerging expert-specialized MoE architectures on non-NVIDIA HPC platforms (Yuan et al., 18 Aug 2025). Its target regime is exemplified by DeepSeek-style models with many more, much smaller experts and larger top-$G(x)$7 routing. The paper formalizes a size-equivalent comparison between a conventional MoE,

$G(x)$8

and an expert-specialized MoE,

$G(x)$9

Under this transformation, total parameters and activated parameters stay roughly constant, but the system bottlenecks change.

The key systems observation is that activation memory shifts from expert MLP internals to the routing pipeline itself. In expert-specialized MoEs, dispatched inputs and combine tensors scale as

$p=\mathrm{Softmax}(Wh),$0

so large top-$p=\mathrm{Softmax}(Wh),$1 turns routing buffers into the dominant memory term (Yuan et al., 18 Aug 2025). X-MoE addresses this with three integrated techniques: padding-free token buffers rather than fixed-capacity expert buffers; hierarchical redundancy-bypassing dispatch, which avoids sending duplicate token copies across slow inter-node links; and hybrid parallelism with sequence-sharded MoE blocks, which shards the sequence dimension inside MoE blocks to reduce activation footprint.

The padding-free representation removes the padded $p=\mathrm{Softmax}(Wh),$2 expert-buffer layout and stores only valid routed tokens plus routing metadata. In the paper’s analysis, conventional GShard-style dispatch has memory overhead roughly $p=\mathrm{Softmax}(Wh),$3, whereas the padding-free representation needs only $p=\mathrm{Softmax}(Wh),$4 (Yuan et al., 18 Aug 2025). The implementation uses Triton-based gather, scatter, and sequential GeMM kernels so that the design is backend-agnostic rather than CUDA-specific.

The evaluation on Frontier uses AMD MI250X GPUs and scales to $p=\mathrm{Softmax}(Wh),$5 effective GPUs. On the Medium model, X-MoE achieves a $p=\mathrm{Softmax}(Wh),$6 speedup over DeepSpeed-TED and a $p=\mathrm{Softmax}(Wh),$7 speedup over Tutel; on the Super model, a $p=\mathrm{Softmax}(Wh),$8B-parameter DeepSeek-style MoE trains on $p=\mathrm{Softmax}(Wh),$9 GPUs with aggregated throughput of $k$0 PFLOPs, while prior systems fail with out-of-memory errors (Yuan et al., 18 Aug 2025). The paper states that this is $k$1 larger than the largest trainable model with existing methods under the same hardware budget. A related training study also argues that step time, rather than FLOPs or activated parameters, is the more accurate complexity measure for MoE-vs-dense comparisons, and that with efficient 3D sharding the dense-to-MoE step-time increase can remain within a healthy range while MoE still outperforms dense models on the speed-accuracy trade-off curve (Du et al., 2024).

4. Cross-layer reuse, layer-adaptive budgets, and inference-time elasticity

A major extension of the X-MoE design space is the relaxation of layer-local, fixed-budget routing. ReXMoE begins from the claim that standard MoE LLMs are layer-local: layer $k$2 routes only into its own expert pool $k$3, which forces a trade-off between expert dimensionality and routing diversity (Tan et al., 20 Oct 2025). ReXMoE introduces expert reuse across adjacent layers with reuse frequency $k$4, so that a layer routes over an expanded pool

$k$5

with up to $k$6 candidate experts. To stabilize training over the enlarged pool, it uses Progressive Scaling Routing (PSR), which begins with local experts only and gradually increases the visible candidate set. On the $k$7B series, the paper reports that the baseline average of $k$8 rises to $k$9 with reuse factor $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$0, while perplexity drops from $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$1 to $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$2 once PSR is added (Tan et al., 20 Oct 2025). The paper also records a systems asymmetry: prefill becomes slower because more experts are involved in MoE block I/O, while decode throughput remains about comparable to the baseline.

LExI modifies a different assumption: the optimal number of active experts does not have to be the same in every layer (Chitty-Venkata et al., 2 Sep 2025). It is a data-free post-training method that profiles each MoE layer with synthetic Gaussian inputs, measures sensitivity to changing top-$$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$3 by the Frobenius norm of output perturbation, and then solves for a layer-wise allocation vector

$$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$4

under a total budget $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$5. The central claim is that weights alone are enough to estimate relative layer importance. Empirically, LExI reports that on Qwen1.5-MoE it reaches the same throughput on NVIDIA H100 GPU as traditional expert pruning but with $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$6 better accuracy, and that across language and vision MoE benchmarks it dominates inter- and intra-pruning in the accuracy-throughput plane (Chitty-Venkata et al., 2 Sep 2025).

Elastic MoE (EMoE) relaxes the fixed relation between training-time and inference-time expert count (Gu et al., 26 Sep 2025). It shows that standard MoE performance often peaks near the training-time $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$7 and then degrades rapidly as inference activates more experts, because the model has not learned collaboration among those extra expert combinations. EMoE introduces stochastic co-activation sampling from a larger candidate pool and a hierarchical router loss

$$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$8

with $$y=\sum_{i\in\mathcal{E}} p_i E_i(h), \qquad |\mathcal{E}|=k,$$9 encouraging a sharper expert hierarchy. The reported result is that the effective scaling range extends to about $k$0–$k$1 the training-time $k$2, while peak performance also increases (Gu et al., 26 Sep 2025). This suggests that recent X-MoE-like work increasingly treats routing as a budget-allocation problem across depth, time, and inference budgets rather than as a fixed top-$k$3 rule.

5. Compression, quantization, and cache-aware routing

Another cluster of X-MoE-adjacent work concerns deployment under memory and bandwidth constraints. MoE-I$k$4 decomposes MoE redundancy into two levels: redundancy across experts and redundancy inside each expert (Yang et al., 2024). Its inter-expert pruning stage uses layer importance, non-uniform pruning ratios, layer-wise genetic search, and block-wise KT-Reception Field; its intra-expert stage uses expert-specific low-rank decomposition ranks. After pruning, decomposition, and LoRA fine-tuning, reported expert-parameter reduction exceeds $k$5, with memory dropping from $k$6 GB to $k$7 GB on Mixtral-8$k$87B and speedups reaching $k$9 on Mixtral, $k$0 on Qwen1.5-MoE-A2.7B, and $k$1 on DeepSeek-V2-Lite (Yang et al., 2024).

MxMoE treats MoE quantization as an accuracy-and-performance co-design problem rather than uniform bitwidth assignment (Duanmu et al., 9 May 2025). Its two governing observations are that linear blocks such as gate_proj, up_proj, and down_proj have different quantization sensitivity, and that expert activation frequencies differ by over $k$2, leading to heterogeneous runtime behavior. MxMoE formulates a mixed-precision optimization with an objective of the form

$k$3

subject to one quantization scheme per block and a total memory budget. It also generates mixed-precision GroupGEMM kernels for parallel execution across experts with different precisions. The paper reports $k$4 lower WikiText-2 perplexity than GPTQ at $k$5-bit, up to $k$6 speedup over full precision, and up to $k$7 speedup over uniform quantization at equivalent accuracy with $k$8-bit weight-activation quantization (Duanmu et al., 9 May 2025).

For batch-size-one autoregressive inference on mobile devices, “Mixture of Cache-Conditional Experts” modifies routing to favor experts already present in DRAM cache (Skliar et al., 2024). Its main method, Cache Prior Reranking, adds a cache-dependent bias to router logits,

$k$9

but uses the modified logits only for ranking, not for the final expert weights. The method always preserves Top-$k$0 experts, because the paper’s sensitivity analysis shows that removing or replacing lower-ranked experts hurts less than changing Top-1 or Top-2 choices. On-device results on Android 14 report up to $k$1 throughput speedup over standard routing with LRU caching, with expert lifetime on Qwen1.5-MoE increasing from $k$2 to $k$3 tokens and miss rate dropping from $k$4 to $k$5 (Skliar et al., 2024).

6. High-throughput serving and heterogeneous MoE execution

At the systems level, several papers extend the X-MoE agenda from routing design to end-to-end serving architecture. MoE-Lens studies throughput-oriented offline inference when GPU memory cannot hold full MoE weights and CPU-GPU hybrid execution is mandatory (Yuan et al., 12 Apr 2025). Its central modeling claim is that CPU memory capacity for the KV cache is a first-class resource, not merely a storage detail. The paper derives a theoretical throughput bound

$k$6

where PME is Parallelism-Memory Efficiency, $k$7 is KV capacity, and $k$8 is weight-transfer time. A second-stage end-to-end model adds finite batch size, paged KV cache, and overlap effects. The reported system outperforms MoE-Lightning by $k$9 on average and up to $p$0, while the model predicts throughput with $p$1 average accuracy (Yuan et al., 12 Apr 2025).

CoX-MoE addresses the single-GPU, tight-VRAM regime by combining AMX-enabled CPU execution with coalesced expert execution and static expert-aware stratification (Son et al., 18 May 2026). Its key change is that expert computation is not micro-batched: experts are executed on the full batch, while other layer operations may still use micro-batches if needed for memory fit. This restores arithmetic intensity in the expert path and makes CPU offloading worthwhile with Intel AMX. Compared to FlexGen and MoE-Lightning, CoX-MoE reports up to $p$2 and $p$3 higher throughput, respectively (Son et al., 18 May 2026).

TriMoE generalizes binary offloading into a three-way GPU-CPU-NDP partition (Pan et al., 1 Mar 2026). It argues that non-hot experts are not homogeneous: warm experts, often about $p$4–$p$5 of experts but up to $p$6 of tokens, are poorly matched to DIMM-NDP and also do not justify GPU residency, while cold experts, typically over $p$7 of experts but only about $p$8 of tokens, are well suited to NDP. TriMoE therefore maps hot experts to GPU, warm experts to an AMX-enabled CPU, and cold experts to DIMM-NDP, then uses makespan-aware scheduling and prediction-driven relayout. The paper reports $p$9–$E_i$00 speedup over the best baseline (Pan et al., 1 Mar 2026).

FlashDMoE pursues the opposite extreme: fully GPU-resident distributed execution (Aimuyo et al., 5 Jun 2025). It replaces CPU-orchestrated scheduling and host-managed collectives with one persistent GPU kernel that fuses gate, dispatch, expert compute, combine, and device-initiated one-sided communication via NVSHMEM. Evaluated on a single $E_i$01-H100 node, it reports up to $E_i$02 higher GPU utilization, $E_i$03 lower latency, $E_i$04 higher throughput, and $E_i$05 better overlap efficiency than state-of-the-art baselines, despite using FP32 while baselines use FP16 (Aimuyo et al., 5 Jun 2025). Across these systems, a recurring conclusion is that sparse routing efficiency depends as much on payload shape, overlap, and memory hierarchy as on nominal activated-parameter counts.

7. Adjacent directions: upcycling, recurrent experts, and routing theory

A distinct but related line of work studies how MoE capacity can be obtained or interpreted without conventional from-scratch training. “Llama 3 Meets MoE: Efficient Upcycling” converts selected feed-forward layers of a pretrained Llama 3-8B checkpoint into an $E_i$06-Expert Top-$E_i$07 MoE by copying the dense FFN weights $E_i$08 times, once per expert, and initializing only the router randomly (Vavre et al., 2024). The resulting Llama 3-E8T2 model has $E_i$09B total parameters and $E_i$10B active parameters. Trained on $E_i$11B tokens, it is reported to use less than $E_i$12 of typical pre-training compute, consuming about $E_i$13K GPU hours versus an estimated $E_i$14 million GPU hours for from-scratch MoE pretraining, while improving MMLU $E_i$15-shot by $E_i$16 and reaching $E_i$17 MFU in the best configuration (Vavre et al., 2024). The paper also introduces online upcycling in NeMo so that dense checkpoints can be sharded directly under the target distributed configuration.

MoxE replaces transformer FFN experts with heterogeneous xLSTM experts and uses token difficulty to bias routing between mLSTM and sLSTM experts (Thiombiano et al., 1 May 2025). With

$E_i$18

the router adds a positive bias $E_i$19 to mLSTM experts and a negative bias to sLSTM experts, then aligns the learned difficulty score with normalized router entropy through an auxiliary loss. The paper trains a model of about $E_i$20M parameters and reports that MoxE matches the xLSTM baseline on Fineweb-Edu validation while outperforming both Transformer and xLSTM baselines on LAMBADA (Thiombiano et al., 1 May 2025). Its ablations show that removing entropy bias, removing the group-wise balancing loss, or replacing xLSTM experts with FFNs all sharply worsens perplexity.

A third adjacent direction reframes routing information-theoretically. “Expert Routing for Communication-Efficient MoE via Finite Expert Banks” treats the gate as a stochastic channel and defines routing rate as

$E_i$21

where $E_i$22 is the selected expert index (Salehi et al., 6 May 2026). In a finite-bank MNIST construction with pretrained CNN experts, the paper estimates both routing information $E_i$23 and algorithmic dependence $E_i$24, finding that $E_i$25 monotonically tracks the generalization gap as the gate becomes more sample-dependent. It also gives a Blahut-Arimoto procedure for tracing an empirical accuracy-rate curve over the expert bank (Salehi et al., 6 May 2026). This suggests a useful conceptual synthesis: in the broader X-MoE landscape, the gate is increasingly treated not only as a learning component but also as a communication interface that governs computation, bandwidth, and generalization simultaneously.