Layerwise Recurrent Router (RMoE)

• RMoE is a routing mechanism that incorporates recurrent state propagation using GRU updates to leverage cross-layer dependencies in Mixture-of-Experts models.
• It employs layer-wise projections and either stateful recurrent or weight-sharing strategies to achieve robust expert specialization and improved token routing.
• Empirical results in language and speech tasks demonstrate enhanced performance, increased expert diversity, and minimal computational overhead compared to standard SMoE.

The Layerwise Recurrent Router (RMoE) is a routing mechanism for Mixture-of-Experts (MoE) architectures that introduces explicit cross-layer dependencies via lightweight recurrent computation within the routing function. It addresses the common limitation of MoE models wherein routing decisions at each layer are made independently, potentially leading to inefficient token–expert assignments and reduced expert diversity. By incorporating mechanisms for recurrent state propagation (typically through a compact Gated Recurrent Unit or via weight sharing), RMoE architectures leverage historical routing information and inter-layer correlations, resulting in improved performance and more robust specialization among experts for both language and speech applications (Qiu et al., 2024, Gu et al., 8 Jul 2025).

1. Background: Standard MoE Routing Paradigms

In the canonical sparse Mixture-of-Experts (SMoE) architecture, each Transformer layer replaces its feed-forward sub-network with an MoE block. For a given layer $\ell$, the router computes gating logits over the layer’s incoming hidden state $\mathbf{x}_\ell\in\mathbb{R}^h$ via a layer-specific weight matrix $\mathbf{G}_\ell$:

$$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$

The top-$k$ entries in $\mathbf{g}_\ell$ are retained (yielding sparse, softmax-normalized gating), and the corresponding experts—independent two-layer FFNs—are activated for each token. This mechanism is highly layer-local; each router acts in isolation, and there is no flow of expert assignment information between layers. This design leads to suboptimal expert utilization, poor expert specialization, and token–expert assignment collapse, especially as model scale increases (Qiu et al., 2024).

2. Recurrent Routing: Architectural Design

The RMoE introduces recurrence across the routing paths, establishing explicit statistical or parameter dependencies between routing decisions at consecutive layers. The two major classes are:

• Stateful Recurrent Routers: Routing decisions at layer $\ell$ are conditioned on the routing trajectory up to layer $\ell-1$, succinctly summarized as a compact hidden state. Typical instantiation employs a shared GRU cell for recurrent state updates:

a) Project $\mathbf{x}_\ell$ into a lower-dimensional $$\mathbf{x}'_\ell = \mathrm{Proj}_\ell(\mathbf{x}_\ell)\in\mathbb{R}^p$$ ($\mathbf{x}_\ell\in\mathbb{R}^h$0).

b) Update GRU hidden state for each token:

$\mathbf{x}_\ell\in\mathbb{R}^h$1

c) Route using:

$\mathbf{x}_\ell\in\mathbb{R}^h$2

The current hidden state $\mathbf{x}_\ell\in\mathbb{R}^h$3 explicitly encodes expert choices and summary statistics from all previous layers, informing token routing at subsequent depths (Qiu et al., 2024).

• Parameter-Tied (Weight-Sharing) Routers: An alternative instantiation, as in Omni-Router, ties a single router weight matrix $\mathbf{x}_\ell\in\mathbb{R}^h$4 across all MoE layers, so that every layer shares the same gating decision boundary. This enforces a form of recurrence and encourages consistent routing patterns throughout depth. Importantly, only router parameters are shared; expert FFNs are still independent per-layer (Gu et al., 8 Jul 2025).

3. Mathematically Formalized Routing Dynamics

For stateful RMoE models (Qiu et al., 2024), token-wise routing proceeds as:

• Projection:

$\mathbf{x}_\ell\in\mathbb{R}^h$5

• GRU Update:

$\mathbf{x}_\ell\in\mathbb{R}^h$6

• Routing (TopK-sparse gating):

$\mathbf{x}_\ell\in\mathbb{R}^h$7

• Expert Output Aggregation:

$\mathbf{x}_\ell\in\mathbb{R}^h$8

Here, all GRU parameters $\mathbf{x}_\ell\in\mathbb{R}^h$9 are shared across layers. Each layer maintains its own projection and router weight.

In Omni-Router-style models (Gu et al., 8 Jul 2025), for all layers $\mathbf{G}_\ell$0, routing scores for token representations $\mathbf{G}_\ell$1 are computed as $\mathbf{G}_\ell$2, normalized via softmax, and expert assignment proceeds with $\mathbf{G}_\ell$3 or top-$\mathbf{G}_\ell$4 gating.

4. Computational Overhead and Integration

The RMoE routing stages add marginal computational costs relative to standard SMoE routers:

• Projection step: $\mathbf{G}_\ell$5 per layer ($\mathbf{G}_\ell$6 = batch size).
• GRU step: $\mathbf{G}_\ell$7 per layer; $\mathbf{G}_\ell$8 ensures negligible increase.
• Router step: $\mathbf{G}_\ell$9 versus $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$0 in SMoE.

Typical values ($$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$1, $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$2, $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$3 for large LMs) yield router overheads of only a few percent. Memory usage increases by $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$4 (typically $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$51 GB for large batches). Since the recurrent update is along depth, there is no change to token-wise parallelism within batches or degradation of overall model pipeline parallelism. Only the current $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$6 needs to be propagated layerwise (Qiu et al., 2024). In weight-tied variants, parameter sharing means no increase in parameter count; the sole change is the replacement of per-layer routers with a single global router matrix (Gu et al., 8 Jul 2025).

5. Empirical Findings and Comparative Analysis

Metric	SMoE	RMoE	RMoE (Large LM)
Enwiki8 BPC	1.128	1.116	-
WikiText-103 PPL	33.06	32.87	-
Avg downstream acc (%)	47.63	49.20	51.20 (SFT)

On small-scale language modeling, RMoE improves bit-per-character (BPC) and perplexity (PPL) over standard SMoE at nearly constant parameter count ($$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$71.3% slowdown, $$\mathbf{g}_\ell = \mathsf{softmax}(\mathbf{x}_\ell \mathbf{G}_\ell)$$8 GB memory). For large-scale pretraining (24L-1280d-16E, top-4), RMoE achieves pre-SFT scores of 49.20 vs. SMoE's 47.63 (+1.57), and after SFT, 51.20 vs. 48.97 (+2.23). With 40B tokens, SMoE 54.26 vs. RMoE 55.31 (Qiu et al., 2024).

Ablation studies reveal that removing recurrence collapses RMoE's advantage (matching SMoE), and detaching hidden-state gradients further degrades performance. Adding a residual path or stateless recurrence does not replicate the gains. RMoE demonstrates elevated cross-layer mutual information in gating decisions (near zero in SMoE, XMoE, HyperMoE), improved expert diversity (lower cosine similarity among experts), and robustly higher entropy in gating—avoiding both collapse and randomness.

In ASR, the shared-router approach yields an average WER reduction of 8.2% (vs. SwitchMoE), 11.2% (vs. dense) and strong routings aligned with data structure. Cramér’s V of routing between adjacent layers is near one for Omni-Router (RMoE), near zero for untied SwitchMoE (Gu et al., 8 Jul 2025).

6. Integration with Broader MoE Architectures

The RMoE mechanism is orthogonal to most other MoE enhancements and can be overlaid on architectures such as XMoE, HyperMoE, and ReXMoE (Tan et al., 20 Oct 2025). Implementation requires only insertion of the GRU update and per-layer projections before the router, with minimal code changes. In frameworks such as Megatron-MoE, Fairseq, or DeepSpeed-MoE, the recurrent router can be realized as a few additional lines around router invocation. A plausible implication is that RMoE can naturally combine with cross-layer expert pools, advanced gating/objective functions, and various expert aggregation schemes for further gains.

7. Significance and Limitations

RMoE establishes explicit depthwise information flow at the token–expert assignment level. This enables steadier routing, improved expert utilization, and greater resistance to route collapse as networks scale. Its empirical superiority is robust across both language modeling and ASR. However, the method introduces mild recurrent dependency along depth, which could, in extremely deep models, have implications for parallel throughput if not efficiently handled (although no practical slowdowns are observed). Additionally, RMoE depends on the assumption that inter-layer token representations are sufficiently aligned to benefit from recurrent propagation; in domains or architectures departing from this assumption, gains may be diminished. Empirical results indicate the method is most beneficial when contrasted with independent per-layer routing and that the recurrence itself, rather than increased parameterization, is key to the effect (Qiu et al., 2024, Gu et al., 8 Jul 2025).