---
title: 'ReXMoE: Efficient Cross-Layer Expert Reuse'
url: https://www.emergentmind.com/topics/rexmoe
type: topic
---

# ReXMoE: Efficient Cross-Layer Expert Reuse

ReXMoE (Reusing Experts with Minimal Overhead in Mixture-of-Experts) is a Mixture-of-Experts (MoE) architecture for Transformer-based large language models (LLMs) that enables the reuse of experts across adjacent layers. By decoupling expert dimensionality from per-layer routing and introducing a progressive scaling routing (PSR) strategy, ReXMoE achieves increased routing diversity and expressiveness under fixed parameter and computational budgets, leading to improved language modeling and downstream task performance with minimal overhead [2510.17483].

## 1. Motivation and Background

Traditional MoE frameworks in Transformers assign to each layer $l$ an independent pool of $N$ experts $\mathcal{E}^{(l)} = \{E_{l,1}, \dots, E_{l,N}\}$. Token-level routing (e.g., Top-$K$) selects a subset of these local experts per token, providing computational efficiency. Under a fixed total parameter budget $P_{\rm tot}$, there is an inherent trade-off: increasing $N$ enhances routing diversity but reduces the representational capacity of each expert ($d_f$), unless overall model size is increased. As model sizes grow, state-of-the-art MoE LLMs (e.g., Mixtral, Qwen3, DeepSeek-V3) push $N$ to 128–256, fragmenting the FFN into fine-grained experts. However, each layer’s router remains confined to its own local pool, exacerbating the diversity/capacity trade-off and often reducing per-expert width in large $N$ regimes [2510.17483].

The standard Top-$K$ MoE routing operates via:
$$
\mathbf{s} = \mathrm{Softmax}(W\,\mathbf{h}) \in \mathbb{R}^N,\qquad
g_i = \begin{cases}
s_i & \text{if } s_i \in \mathrm{TopK}(\{s_j\}), \\
0 & \text{otherwise},
\end{cases}
$$
with each token output
$$
\mathbf{h}' = \sum_{i=1}^N g_i\, E_{l,i}(\mathbf{h}).
$$
Here, $W \in \mathbb{R}^{N \times d}$ are routing weights, and each $E_{l,i}$ is an independent FFN with hidden size $d_f$.

## 2. ReXMoE Architecture: Cross-Layer Expert Reuse

ReXMoE mitigates the layer-local routing bottleneck by introducing cross-layer expert reuse. Every $r$ consecutive layers are grouped into a reuse block, allowing the router at a given layer $l$ to access a pooled set of $rN$ experts, $\mathcal{U}^{(l)} = \bigcup_{i \in G(l)} \mathcal{E}^{(i)}$, where $G(l)$ covers $r$ adjacent layers. Routing thus operates as:
$$
\mathbf{h}' = \sum_{i=1}^{rN} g_i^{(l)} U^{(l)}_i(\mathbf{h}),\quad
g^{(l)} = \mathrm{TopK}\left( \mathrm{Softmax}(W^{(l)} \mathbf{h}) \right),
$$
where $U^{(l)}_i$ indexes into the expert pool pooled from $r$ layers, parameterized via $W^{(l)} \in \mathbb{R}^{rN \times d}$.

This cross-layer pool allows routers to "look sideways" in the depth dimension, leveraging a much larger candidate set without increasing the total number of FFN parameters. Expert selection therefore becomes combinatorially richer, decoupling routing diversity from per-expert capacity under constant $P_{\rm tot}$.

| Symbol    | Meaning                                          |
|:----------|:-------------------------------------------------|
| $L$       | Total Transformer layers                         |
| $N$       | Experts per layer                                |
| $r$       | Reuse group size (reuse frequency)               |
| $\mathcal{E}^{(l)}$ | Experts local to layer $l$             |
| $\mathcal{U}^{(l)}$ | Expanded pool across $r$ layers at $l$ |

## 3. Progressive Scaling Routing (PSR) Strategy

Directly routing over $rN$ experts from training onset introduces severe load imbalance among experts, with many never activated. To address this, PSR gradually increases the candidate set. For training iterations $t$, let $t_s$ and $t_e$ denote the PSR start and end steps:
$$
N_t =
\begin{cases}
N, & t \leq t_s \\
\left\lfloor N + \frac{(r-1)(t-t_s)}{t_e-t_s}N \right\rfloor, & t_s < t \leq t_e \\
rN, & t > t_e
\end{cases}
$$
At each step, $(rN - N_t)$ experts in $\mathcal{U}$ are randomly masked before routing. This curriculum facilitates early specialization (local candidates only), with gradual exploration of cross-layer combinations as $N_t$ rises. PSR thus smooths the path from exploitation to exploration without requiring explicit diversity regularization.

## 4. Capacity, Complexity, and Scalability

In baseline MoE, total FFN parameters are $P_{\rm experts}^{\rm base} = L N \times 2 d d_f$, and routers add $P_{\rm router}^{\rm base} = L N d$, giving
$P_{\rm tot}^{\rm base} \approx P_{\rm experts}^{\rm base} + P_{\rm router}^{\rm base}$.
ReXMoE introduces no new FFN parameters but expands routers to $L (rN) d$, resulting in $P_{\rm router}^{\rm ReXMoE} / P_{\rm router}^{\rm base} = r$. Since routers are parameter-light compared to FFNs, overhead is modest: $P_{\rm tot}^{\rm ReX} \approx P_{\rm tot}^{\rm base} + \mathcal{O}(L d r N)$.

Per-token compute increases by $\Delta{\rm FLOPs} \approx rN d$ from the wider router, but expert computation cost is unchanged. This enables expert width $d_f$ to remain large even as routing diversity (size of expert pool) scales with $rN$. The architecture thus decouples expert width from routing diversity, a key advantage for parameter-efficient scaling.

## 5. Experimental Protocols

ReXMoE was evaluated across 0.5B, 2.3B, and 7B parameter LLM variants using the following setup:
- Architectures: MoE-0.5B-A0.07B (16L, $d=768,N=4$), MoE-2.3B-A0.3B (32L, $d=512,N=8$), MoE-7B-A3B-SE (32L, $d=2048,N=6$ routed + 2 shared).
- 100B-token pretraining dataset (fineweb-edu), batch: 2M tokens, sequence length: 4096.
- Optimization: AdamW ($\beta_1=0.9$, $\beta_2=0.95$, weight decay 0.1), gradient clipping 1.0, LR: $3 \times 10^{-4}$ warmup to $3 \times 10^{-5}$ cosine decay, warmup: 100 steps.
- PSR schedule: $t_s=10$k steps, $t_e=30$k.
- Expert parallelism used for routing >8 experts. Hardware: 4 nodes $\times$ 32 Hopper GPUs.

Eval metrics:
- Language modeling perplexity (WikiText validation split)
- Average and per-task zero-shot accuracy (ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, LAMBADA, LogiQA, OpenBookQA, PIQA, SciQ, SIQA, WinoGrande via lm-eval-harness)
- Inference throughput (prefill, decoding via vLLM)

## 6. Core Results and Ablations

Across all tested model sizes, ReXMoE demonstrates improved downstream performance and perplexity compared to baseline MoE:

| Model                | Avg. Accuracy | WikiText PPL |
|----------------------|---------------|--------------|
| MoE-2.3B-A0.3B       | 49.15%        | 21.19        |
| ReX-R2               | 49.65%        | —            |
| ReX-R4               | 50.23%        | 20.73        |

Key findings:
- ReXMoE-R4 (reuse group $r=4$) yields the highest average accuracy across all scales.
- Simple cross-layer reuse ($r=4$) yields marginal improvement (+0.13% accuracy, small PPL drop).
- The PSR curriculum adds a further +0.95% accuracy and significant reduction in perplexity.
- On open-source benchmarks, ReX-7B-A3B-SE-R3 trained with 1T tokens matches or surpasses LLaMA-7B on LogiQA, SciQ, and other tasks.
- Prefill throughput for short sequences drops up to 15% (increased router overhead); negligible impact on decoding throughput ($<5$\% change).
- Optimal reuse group size is $r=4$. Larger $r$ (16, 32) initially match but later underperform due to expert under-utilization and load imbalance, as measured by Load Balance Violation (LBV).
- Activation ratio heatmaps show that ReXMoE enables specialization of certain experts for specific tasks, while vanilla MoE experts remain uniformly utilized.

## 7. Analysis, Limitations, and Future Prospects

ReXMoE breaks the core limitation of layer-local routing, allowing parameter-efficient decoupling of expert width and routing diversity. Cross-layer reuse of experts, in combination with progressive scaling routing, enables richer expert combinations without increasing FFN parameters and encourages both exploration and expert specialization during training.

Salient points:
- The choice of $r=4$ emerges as a strong sweet spot, balancing diversity and overhead.
- Excessively large $r$ ($\geq16$) causes expert collapse and imbalance unless load-balancing regularization is introduced.
- Some prefill I/O overhead marginally reduces inference speed for short context windows.
- Future work could enhance large-$r$ performance using explicit load-balancing objectives or learned/adaptive groupings (non-uniform $r$). Extending cross-layer reuse to multi-task or continual settings is a promising direction for knowledge sharing.

In summary, ReXMoE establishes a new design dimension in MoE-based LLMs: cross-layer expert reuse, operationalized via a lightweight router parameter extension and a curriculum-based progressive scaling routing schedule. This architecture achieves consistent improvements in language modeling and downstream tasks with minimal incremental overhead, advancing the pursuit of scalable, parameter-efficient MoE LLMs [2510.17483].

Source: https://www.emergentmind.com/topics/rexmoe