---
title: Linear-MoE Architecture
url: https://www.emergentmind.com/topics/linear-moe-architecture
type: topic
---

# Linear-MoE Architecture

A Linear-MoE (Mixture-of-Experts with linear experts) architecture couples classical mixture-of-experts modeling with sparse, high-capacity neural architectures via a gating network assigning inputs to expert linear mappings. This approach yields models that combine strong theoretical guarantees, architectural modularity, and scalable, efficient implementation for both parametric and nonparametric function approximation, sequence modeling, and large-scale transformer-based systems. The following sections cover principal structure, theoretical properties, sparse routing, integration into sequence and transformer models, scaling laws, and recent architectural innovations.

## 1. Core Architecture and Mathematical Specification

A Linear-MoE consists of two primary components: a gating network and a collection of $K$ expert networks, each parameterizing a (vector-valued) linear map. For input $x \in \mathbb{R}^p$, the model operates as follows:

- **Gating Network**: Computes mixture weights
  $$
  g_i(x) = \frac{\exp(w_i^\top x + c_i)}{\sum_{j=1}^K \exp(w_j^\top x + c_j)}, \quad i = 1, \ldots, K
  $$
  where $w_i \in \mathbb{R}^p$ and $c_i \in \mathbb{R}$ are gating parameters, enforcing $g_i(x) > 0$ and $\sum_i g_i(x) = 1$.

- **Expert Networks**: In the scalar-output case, each expert computes a linear response
  $$
  f_i(x) = a_i^\top x + b_i, \quad a_i \in \mathbb{R}^p, b_i \in \mathbb{R}
  $$
  For multi-output $\mathbb{R}^q$, experts generalize to
  $$
  f_i(x) = A_i x + b_i, \quad A_i \in \mathbb{R}^{q \times p}, b_i \in \mathbb{R}^q
  $$
- **Model Output**: The final prediction is the mixture
  $$
  \mu(x) = \sum_{i=1}^K g_i(x) f_i(x)
  $$
  or, if modeling conditional densities,
  $$
  p(y \mid x) = \sum_{i=1}^K g_i(x) \mathcal{N}(y; A_i x + b_i, C_i)
  $$
  where $C_i \in \mathbb{R}^{q \times q}$ is a positive-definite covariance [1704.00946].

The parameter counts scale as follows:

| Component          | Parameters per Expert       | Total Parameters                         |
|--------------------|-------------------------|-----------------------------------------|
| Gating (softmax)   | $w_i \in \mathbb{R}^p$, $c_i \in \mathbb{R}$ | $K \cdot (p+1)$               |
| Expert             | $A_i \in \mathbb{R}^{q \times p}$, $b_i \in \mathbb{R}^q$ | $K \cdot q \cdot (p+1)$      |
| Covariance         | (optional) $C_i$: $q(q+1)/2$ | $K \cdot q(q+1)/2$               |

## 2. Theoretical Properties and Approximation Results

Linear-MoE architectures possess strong universal approximation properties for both mean functions and conditional densities:

- **Density Approximation**: For any collection of true conditional marginal densities $g_{Y_j|X}(y_j|x)$ on compact $X \subset \mathbb{R}^p$, there exists a Linear-MoE (with sufficiently large $K$) that approximates each $p(y_j|x)$ to arbitrary precision in conditional KL-divergence given suitable smoothness and positivity conditions [1704.00946].

- **Mean-Function Denseness**: The model class forms a dense subset in the space of continuous vector-valued functions. For any continuous $\mu: X \to \mathbb{R}^q$ and $\epsilon > 0$, one can choose $K$ and parameters so that
  $$
  \|\hat{\mu} - \mu\|_{q,\infty} = \max_{x \in X} \sum_{j=1}^q |\hat{\mu}_j(x) - \mu_j(x)| < \epsilon
  $$
  Thus, Linear-MoE can model arbitrary continuous multivariate regressions [1704.00946].

These properties are enabled by closure under summation (for mean functions) and permutation (for conditional densities) of the MoE class, allowing one to build multivariate approximators by composition of independent univariate MoEs and mixture-of-Gaussians [1704.00946].

## 3. Sparse Routing and Load Balancing

The canonical sparsity procedure is *top-$k$ gating* in which, for each token or input, the $k$ experts with highest gating probabilities $g_i(x)$ are selected. The normalized weights for the chosen subset are:
$$
w_e(x) = \frac{\max\{g_e(x), 0\}}{\sum_{e' \in \mathcal{S}_x} \max\{g_{e'}(x), 0\}}
$$
Strict capacity constraints are enforced by this per-token top-$k$ selection [2506.16419, 2503.05447].

To promote balanced expert usage and avoid "stragglers," an auxiliary load-balancing loss term is often introduced:
$$
\mathcal{L}_{\mathrm{aux}} = \alpha_{\mathrm{aux}} E \sum_{e=1}^E f_e g_e \;\;\text{or}\;\; = \alpha_{\mathrm{aux}} E \sum_{e=1}^E (g_e)^2 f_e
$$
where $f_e$ is the fraction of tokens routed to expert $e$, $g_e$ is the average routing probability, and $\alpha_{\mathrm{aux}}$ is a hyperparameter [2506.16419]. This regularizer encourages uniform utilization but can degrade specialization if set too strongly [2511.10971].

## 4. Integration with Sequence and Transformer Architectures

The modern instantiations of Linear-MoE deploy MoE layers within various high-throughput sequence modeling and Transformer frameworks:

- **Linear Sequence Modeling (LSM) + MoE**: Linear-MoE integrates linear-complexity sequence modules including linear attention ($O(N)$), structured state-space models (SSMs), and linear RNNs, chaining each block as:
  $$
  \mathrm{LN} \rightarrow \mathrm{LSM} \rightarrow + \rightarrow \mathrm{LN} \rightarrow \mathrm{MoE} \rightarrow +
  $$
  All MoE layers share the gating/routing machinery above, while the LSM component provides the token-mixing operator, facilitating $O(N)$ runtime and efficient parallelism (Data, Tensor, Pipeline, Expert, Sequence Parallelism) [2503.05447].

- **Shared Experts in Attention and FFN**: The UMoE model reformulates multi-head attention as a MoE sublayer, enabling shared expert modules across both attention and FFN and efficient parameter reuse. Each expert block is a two-layer FFN, and routers can be separated for attention and FFN sub-modules or tied for additional savings [2505.07260].

- **Hybrid Linear-MoE/Transformer-MoE**: For tasks demanding inductive biases from both softmax attention and LSM, hybrid models alternate Linear-MoE and standard Transformer-MoE blocks, using the same routing/gating layers [2503.05447].

## 5. Scaling Laws, Efficiency, and Architectural Tradeoffs

Scaling theory for Linear-MoE predicts compute advantage and capacity via the *Efficiency Leverage (EL)* metric:
$$
EL(X_{\mathrm{MoE}} \mid X_{\mathrm{Dense}}; C_{\mathrm{target}})
= \frac{C_{\mathrm{dense}}}{C_{\mathrm{moe}}}
$$
where $C$ is the training compute required to reach the same loss (up to a small $\epsilon$) [2507.17702]. Key empirical findings:

- **Activation Ratio ($A$):** The fraction of experts activated per token ($A = K / E$) primarily determines EL, with $EL \propto A^{-a}$ for $a \in [1, 1.5]$; lower $A$ (greater sparsity) yields higher leverage.

- **Expert Granularity ($G$):** Defined as $G = 2 d_{\mathrm{model}} / d_{\mathrm{expert}}$; optimal EL emerges in the range $G \approx 8$–$12$ due to a U-shaped dependence in $\log EL(\log G)$.

- **Compute Budget ($C$):** Scaling EL as a power law in $C$, with modest increases for large budgets.

- **Unified Law:** 
  $$
  EL(A, G, C) = \bar{A}^{a + d \log C + \gamma (\log G)^2 + \beta \log G}
  $$
  Coefficients are empirically fitted (see Table 3, [2507.17702]). This law accurately predicts that, for example, a model with $A=3.4\%$, $G=12$, and $C=1{\times}10^{22}$ attains $EL \approx 7$, consistent with Ling-mini-beta ($0.85$B active params) matching dense-6.1B with $1/7$ the training FLOPs.

## 6. Comparative Routers and Recent Innovations

Several routing mechanisms have been studied for their trade-offs in expressivity, efficiency, and stability:

- **Linear Router**: Minimal overhead, a single affine projection ($\sim6$ k params for typical settings), extremely fast routing (0.07 ms/token), moderate entropy, and smooth, uniform expert usage [2506.16419]. However, limited semantic awareness and nonlinear selectivity compared to multilayer alternatives.

- **Attention and MLP Routers**: Offer greater expressivity, higher routing entropy ($\sim2.08$ bits vs. $1.95$ for Linear), and improved feature-space partitioning, but incur 4–16× parameter and latency costs [2506.16419].

- **Eigenbasis/ERMoE Routing**: ERMoE replaces gating logits with a cosine similarity (“Eigenbasis Score”) between the input and a learned orthonormal basis for each expert. This content-aware, geometry-based routing achieves highly stable expert utilization, obviates balancing losses, and produces anatomically or semantically interpretable specializations [2511.10971]. ERMoE achieves state-of-the-art accuracy on ImageNet, COCO, and clinical imaging benchmarks, and in 3D MRI brain-age prediction achieves 2.31 MAE vs. 2.83 for the best dense model.

- **UMoE Shared Experts**: Demonstrates that simultaneous expert-sharing across attention and FFN unlocks additional parameter efficiency and outperforms dense or prior MoE architectures in both perplexity and downstream accuracy [2505.07260].

## 7. Practical Implementation and Empirical Results

Linear-MoE systems support state-of-the-art empirical performance in production-scale settings. Key workflow steps include:

1. **Selection of $K$ (number of experts):** Chosen via information criterion or cross-validation to balance approximation power, overfitting, and computational footprint [1704.00946].
2. **Parametrization and Training:** Gating network (softmax or geometry-based), linear/MLP experts, and, if needed, auxiliary regularizers (for classic Linear routing); all parameters trained jointly with gradient or EM-based optimization [2503.05447].
3. **Parallel Execution:** Leveraging advanced parallelism (Data/Tensor/Pipeline/Expert/Sequence) for strong-scaling efficiency at long sequence lengths and large parameter counts [2503.05447].
4. **Empirical Benchmarks:** Across multiple model sizes (e.g., A0.3B–2B, A1B–7B), Linear-MoE attains efficiency gains (up to $2\times$ inference speed and $O(1)$ memory with up to 16K context) with accuracy competitive to dense and baseline MoE architectures [2503.05447, 2507.17702]. In vision and language tasks, modern Linear-MoE architectures consistently match or outperform their dense counterparts using $7\times$ less compute [2507.17702].

## Summary Table: Router Variants in Linear-MoE

| Router Type       | Param Count (E=8, d=768) | Entropy (H(P)) | Latency (ms/token) | Notable Property                    |
|-------------------|-------------------------|----------------|--------------------|--------------------------------------|
| Linear            | 6,144                   | 1.95           | 0.07               | Minimal, classical, stable           |
| Attention         | 49,664                  | 2.08           | 0.29               | Embedding-based, higher awareness    |
| MLP               | 101,000                 | 2.08           | 0.23               | Nonlinear, expressive                |
| ERMoE (basis)     | $O(d^2)$ (basis, small) | Flattest       | 0.3–0.4            | Content-aware, interpretable         |
| MLP-Hadamard      | 101,000                 | 1.10           | 0.88               | Structured, sharp two-expert splits  |
| Hash              | 0                       | 0              | 85.0               | Deterministic, not used in practice  |

This comparative summary underscores both the efficiency–expressivity frontier and recent advances in geometry- or content-aware routing.

## References

- [1704.00946] Approximation results regarding the multiple-output mixture of linear experts model
- [2503.05447] Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
- [2505.07260] UMoE: Unifying Attention and FFN with Shared Experts
- [2506.16419] Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
- [2507.17702] Towards Greater Leverage: Scaling Laws for Efficient Mixture-of-Experts Language Models
- [2511.10971] ERMoE: Eigen-Reparameterized Mixture-of-Experts for Stable Routing and Interpretable Specialization

Source: https://www.emergentmind.com/topics/linear-moe-architecture