---
title: Developmental Mixture of Experts
url: https://www.emergentmind.com/topics/developmental-mixture-of-experts-moe
type: topic
---

# Developmental Mixture of Experts

Searching arXiv for the cited MoE papers and closely related work to ground the article.
arxiv_search query: "2112.14397 EvoMoE evolutional mixture-of-experts training framework dense-to-sparse gate"
Developmental Mixture of Experts (MoE) denotes a class of mixture-of-experts architectures and statistical models in which expert capacity, routing behavior, or coefficient structure changes along training time or along an explicit index such as developmental stage, time, or space. In this usage, “developmental” refers to staged growth, progressive specialization, or smoothly varying subpopulation structure rather than a fixed, randomly initialized sparse MoE trained end-to-end from the outset. The clearest deep-learning exemplar is EvoMoE, which begins with a single shared expert and gradually evolves into a sparse multi-expert architecture through an expert-diversify phase and a gate-sparsify phase [2112.14397]. In a complementary statistical sense, developmental MoE includes models such as the Varying-Coefficient Mixture of Experts (VCMoE), where both gating and expert coefficients vary along an indexing variable \(U\), making the mixture itself explicitly developmental or longitudinal [2601.01699]. Across these formulations, the central premise is that specialization and routing should emerge or vary over time rather than be fixed a priori.

## 1. Conceptual definition and scope

Mixture-of-experts models combine multiple specialized submodels through input-dependent gating functions. In the standard formulation, given an input token \(x_s \in \mathbb{R}^D\), experts \(\{e_1,\dots,e_N\}\), and gating weights \(g(x_s)_i\), the MoE layer output is
\[
y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).
\]
For Transformer-based MoEs, each expert typically replaces an FFN:
\[
e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.
\]
A conventional sparse gate is often written as
\[
g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),
\]
with \(W_g \in \mathbb{R}^{D \times N}\) [2112.14397].

Developmental MoE differs from this static picture by altering either the architecture or the parameter functions during learning. In EvoMoE, the model evolves from a small dense network with one expert and no gate into a large sparse MoE with many experts and Top‑1 routing [2112.14397]. In VCMoE, the mixture components remain present, but the gating coefficients \(\boldsymbol{\beta}_c(u)\), expert coefficients \(\boldsymbol{\alpha}_c(u)\), and dispersion terms \(\delta_c(u)\) vary smoothly with an index \(U\), so the effective mixture changes across developmental stage or time [2601.01699]. A plausible unifying interpretation is that developmental MoE treats specialization as a trajectory rather than as a fixed design choice.

The broader theoretical backdrop is that MoE mean functions are dense in the class of all continuous functions over arbitrary compact domains of estimation, giving a universal approximation theorem for MoE models [1602.03683]. This suggests that developmental schemes concern how MoE capacity is organized and learned, not whether the family is expressive enough in principle.

## 2. Static MoE as the baseline from which developmental formulations depart

Static MoE training typically initializes multiple experts randomly, adopts a fixed routing rule such as Top‑1 or Top‑2, and trains experts and gate jointly from scratch. EvoMoE identifies two difficulties with this regime: immature experts and unstable sparse gate [2112.14397]. In the described GPT-MoE with 16 experts, one expert monopolizes the load for hundreds of steps while other experts are underutilized and undertrained, and routing for a token such as “the” is highly unstable early in training [2112.14397]. These observations motivate a developmental alternative in which shared knowledge is learned before specialization.

The probabilistic MoE literature provides the more general conditional form
\[
p(y \mid x; \Theta) = \sum_{k=1}^K \pi_k(x; \alpha)\, p_k(y \mid x; \theta_k),
\]
with covariate-dependent gating probabilities and expert-specific conditional models [1707.03538]. In this perspective, developmental behavior can arise by increasing \(K\), by refining the gating partition, or by allowing coefficients to vary across an index \(U\) [2601.01699]. The universal approximation result that the full MoE class is dense in \(C(\Omega)\) over arbitrary compact \(\Omega\subset\mathbb{R}^d\) supports this divide-and-conquer view of capacity allocation [1602.03683].

A common misconception is that “developmental MoE” necessarily means only expert growth. The literature in the supplied sources suggests a broader usage. EvoMoE is developmental because both experts and gate mature over training iterations [2112.14397]. VCMoE is developmental because latent subpopulation structure and covariate effects evolve along \(U\) [2601.01699]. A plausible implication is that developmental MoE is best understood as a family of staged or indexed MoE formalisms rather than a single architecture class.

## 3. EvoMoE: staged growth from one expert to sparse routing

EvoMoE is an end-to-end MoE training framework that starts from one single expert and gradually evolves into a large and sparse MoE structure [2112.14397]. Its developmental process has two phases.

In the expert-diversify phase, the model first trains a single shared expert with no gate:
\[
y_s = e(x_s).
\]
All tokens flow through the same expert, effectively using a dense Transformer FFN. Training proceeds for \(T_S\) iterations using only the task loss \(\mathcal{L}_{\text{task}}\) [2112.14397]. After this warm-up, the single expert is diversified into \(N\) experts by copying its parameters and applying expert-specific random masks:
\[
W_i = M_i \odot W.
\]
This yields multiple diverse experts, all initialized from a trained base expert rather than random weights [2112.14397].

In the gate-sparsify phase, the model introduces a gate and continues training the MoE with the Dense-to-Sparse gate, written abstractly as
\[
g(x_s) = \text{DTS\_Gate}(x_s; \tau, T_D).
\]
The design decouples early expert learning from sparse routing and focuses on learning basic shared knowledge with a single expert at the early stage [2112.14397]. This staged procedure is the primary reason EvoMoE is naturally described as a developmental or evolving MoE.

The empirical setting spans RoBERTa for masked language modeling, GPT for language modeling, and Transformer for machine translation [2112.14397]. For RoBERTa, the model uses 24 layers, hidden size 1024, 16 attention heads, replaces every other FFN with MoE-FFN with 16 experts per layer, and reports shared params 259M, expert params 1536M, total 1759M. EvoMoE uses Stage 1 = 5k steps and Stage 2 = 5k steps in that configuration [2112.14397]. For GPT, the model has 24 layers, hidden 1024, 16 heads, and 12 MoE layers \(\times\) 16 experts \(=192\) experts, with Stage 1 = 10k steps and Stage 2 annealing from \(\tau=2.0\) to \(0.3\) over 5k steps [2112.14397]. These setup details instantiate developmental MoE as a concrete training curriculum rather than only a conceptual analogy.

## 4. Dense-to-Sparse gate and the maturation of routing

The Dense-to-Sparse Gate (DTS-Gate) is the core mechanism by which EvoMoE makes routing developmental [2112.14397]. The gate begins as a dense Gumbel-Softmax router. Given logits
\[
z(x_s) = x_s W_g \in \mathbb{R}^N,
\]
the gate first computes
\[
g'(x_s)_i =
\frac{\exp\big((z(x_s)_i + \zeta_i)/\tau\big)}
{\sum_{j=1}^{N}\exp\big((z(x_s)_j + \zeta_j)/\tau\big)},
\]
where \(\zeta\) is sampled from \(\text{Gumbel}(0,1)\) [2112.14397]. Large \(\tau\) yields a more uniform distribution and dense routing, while small \(\tau\) yields a sharper distribution approaching one-hot behavior.

Instead of fixing \(k\) as in Top‑\(k\), EvoMoE uses content-based thresholding:
\[
g(x_s)_i =
\begin{cases}
g'(x_s)_i, & \text{if } g'(x_s)_i > c,\\
0, & \text{otherwise}.
\end{cases}
\]
The number of active experts per token is therefore adaptive, and the selected experts’ weights are not renormalized after thresholding [2112.14397]. The temperature is annealed from a large value such as \(\tau_{\max}\approx 2.0\) to a smaller value such as \(\tau_{\min}\approx 0.3\) over \(T_D\) iterations, after which EvoMoE switches to a strict Top‑1 gate [2112.14397]. The resulting training trajectory is dense \(\rightarrow\) adaptive \(\rightarrow\) Top‑1.

This gate maturation is paired with a balance loss:
\[
\mathcal{L} = \mathcal{L}_{\text{task}} + \mathcal{L}_{\text{balance}},
\]
where
\[
\mathcal{L}_{\text{balance}} = \alpha N \sum_{i=1}^N
\left(
\frac{\sum_{x_s \in \mathcal{B}} \mathbb{I}\{g(x_s)_i > 0\}}{|\mathcal{B}|}
\cdot
\frac{\sum_{x_s \in \mathcal{B}} g'(x_s)_i}{|\mathcal{B}|}
\right).
\]
This penalizes uneven loads and skewed gate probabilities, encouraging balanced specialization [2112.14397]. In developmental terms, the gate is not merely sparsified; it is regularized while it matures.

A common misunderstanding is that developmental routing is equivalent to selecting a fixed Top‑\(k\) later in training. EvoMoE’s mechanism is more specific: the gate starts dense, uses threshold-based adaptive sparsity during transition, and only later becomes Top‑1 [2112.14397]. This suggests that developmental MoE is concerned with the path by which selectivity emerges, not only the terminal sparsity level.

## 5. Quantitative behavior, optimization, and systems implications

EvoMoE reports that it outperforms Switch, BASE Layer, Hash Layer, and StableMoE [2112.14397]. On GLUE with RoBERTa, EvoMoE achieves average score 90.875 versus Switch 90.313, BASE 90.650, Hash 90.513, and StableMoE 90.413, with gains up to \(+0.562\) over standard Transformer and \(+0.403\) over the best MoE baseline [2112.14397]. On GPT language modeling, perplexity is 12.24 for EvoMoE versus 12.45 for BASE, 12.87 for Hash, 12.91 for StableMoE, 13.12 for Switch, and 15.14 for dense TRM [2112.14397]. On machine translation, the reported BLEU values are En–De: Transformer 28.1, Switch 28.4, EvoMoE 29.6; De–En: Transformer 34.8, Switch 34.6, EvoMoE 36.7; En–Fr: Transformer 39.2, Switch 39.1, EvoMoE 40.3; Fr–En: Transformer 38.1, Switch 38.2, EvoMoE 39.2 [2112.14397].

The framework also reports compute-related improvements. For GPT-MoE with OpenWebText, the DTS gate achieves approximately \(2\times\) fewer iterations to reach the same validation perplexity as Switch and approximately \(1.42\times\) FLOPs-efficiency improvement [2112.14397]. On machine translation, EvoMoE needs only approximately \(75\%\)–\(82\%\) of Switch’s compute to reach similar BLEU, with larger gains when more experts are used, and achieves up to \(1.33\times\) speed-up [2112.14397]. These values support the view that developmental schedules can improve convergence and effective utilization, not merely final quality.

Optimization details reported for EvoMoE include Adam with \(\beta_1=0.9\), \(\beta_2=0.98\), polynomial learning rate scheduler, dropout \(=0.1\), weight decay \(=0.1\), and warm-up steps such as 5k–10k [2112.14397]. The threshold may use a fixed small value such as \(c=0.001\), and the paper states that temperature schedules have relatively low sensitivity except at extreme values such as \(\tau_{\max}=1.0\), \(\tau_{\min}=0.1\), which can harm gradient variance [2112.14397].

The implementation notes also identify systems-level developmental support. EvoMoE uses expert parallelism with All-to-All communication, adds a Topology-Aware Hierarchical AllToAll to improve communication efficiency, and introduces MoE-aware recomputation that stores inputs to All-to-All rather than recomputing entire layers [2112.14397]. Although these are engineering mechanisms, they matter because developmental MoE in practice often entails changing expert count, routing density, and communication pattern during training.

## 6. Statistical developmental MoE: varying coefficients, identifiability, and inference

The VCMoE model provides a statistical formulation of developmental MoE in which the mixture varies along a known index \(U\) [2601.01699]. For observations \(i=1,\dots,n\), with response \(Y_i\), gating covariates \(\boldsymbol{x}_i\), expert covariates \(\boldsymbol{z}_i\), and index \(U_i\in\mathcal U\subset\mathbb R\), the model is
\[
f(y_i\mid U_i=u,\boldsymbol{x}_i,\boldsymbol{z}_i)
=
\sum_{c=1}^C
g\big(\boldsymbol{x}_i^\top \boldsymbol{\beta}_c(u)\big)\;
\phi\!\left\{
y_i \mid \eta_c(\boldsymbol{z}_i;\boldsymbol{\alpha}_c(u)),\,\delta_c(u)
\right\},
\]
where \(\eta_c(\boldsymbol{z}_i;\boldsymbol{\alpha}_c(u)) = w(\boldsymbol{z}_i^\top \boldsymbol{\alpha}_c(u))\) [2601.01699]. All coefficient effects in both the gating functions and expert models vary along \(U\).

Estimation is based on a label-consistent EM algorithm and local weighted likelihood
\[
\ell_n(u)=
\frac{1}{n}\sum_{i=1}^n
\log\left(
\sum_{c=1}^C
\pi_c(\boldsymbol{x}_i;\boldsymbol{\beta}_c(u))
\phi\!\left\{Y_i \mid \eta_c(\boldsymbol{z}_i;\boldsymbol{\alpha}_c(u)),\delta_c(u)\right\}
\right)
K_h(U_i-u),
\]
with local linear approximation
\[
\beta_p(U_i) \approx \beta_p(u) + \beta_p'(u)(U_i-u) = a_p(u)+b_p(u)(U_i-u)
\]
and analogous expansions for expert coefficients and dispersions [2601.01699]. The global E-step computes responsibilities using the same labels for all \(u\), while the local M-step updates class-specific coefficient functions, thereby avoiding label switching across the developmental axis [2601.01699].

VCMoE establishes identifiability and consistency under conditions including continuously differentiable functional coefficients, open domains, distinctness of components so that no two component trajectories are identical or tangent, identifiability of the static finite mixture for fixed \(u\), and known number of components \(C\) [2601.01699]. The local estimator satisfies
\[
\hat{\boldsymbol{\theta}}(u)-\boldsymbol{\theta}(u)=O_p\big((nh)^{-1/2}+h^2\big),
\]
and
\[
\sqrt{nh}\Big\{
\hat{\boldsymbol{\theta}}(u)-\boldsymbol{\theta}(u)-
\frac{h^2}{2}\boldsymbol{\theta}''(u)v_2+o_p(h^2)
\Big\}
\xrightarrow{D}
\mathcal{N}\big(\mathbf{0},\,\tau f^{-1}(u)\mathcal{I}^{-1}(u)\big)
\]
under the stated regularity conditions [2601.01699]. The model also constructs simultaneous confidence bands and develops a generalized likelihood ratio test for whether a coefficient function is genuinely varying across the index variable [2601.01699]. This gives developmental MoE a full inferential apparatus rather than only predictive semantics.

The embryonic mouse application makes the developmental interpretation concrete. Using single nucleus gene expression data, the model characterizes temporal dynamics in the associations between Satb2 and Bcl11b across two latent neuron subpopulations, yielding results consistent with prior findings [2601.01699]. The reported ROC AUC for predicting upper-layer versus deep-layer labels on a test set is 0.885 [2601.01699]. Here, developmental MoE means that both mixture proportions and within-subpopulation covariate effects evolve over embryonic time.

## 7. Theoretical foundations, model selection, and emerging developmental theory

The universal approximation theorem for MoE models establishes that the class of MoE mean functions is dense in \(C(\Omega)\) for arbitrary compact domains of estimation [1602.03683]. This result generalizes earlier work that assumed sufficiently differentiable target functions from Sobolev space and compact unit hypercube domains [1602.03683]. For developmental MoE, the implication is not about training dynamics, but about representational sufficiency: adding experts and refining gating can, in principle, approximate arbitrary continuous structure on arbitrary compact domains.

Model selection becomes especially significant once development is interpreted as growth and consolidation of experts. For Gaussian-gated Gaussian MoE, dendrograms of mixing measures provide a mechanism to estimate the true number of experts from a single overfitted fit, avoiding the need to train and compare a range of models with varying numbers of components [2505.13052]. The paper defines a dissimilarity between atoms
\[
d\big(\pi_i\delta_{\theta_i},\pi_j\delta_{\theta_j}\big)
=
\frac{\pi_i\pi_j}{\pi_i+\pi_j}
\Big(
\|\mu_i-\mu_j\|^2+\|\Sigma_i-\Sigma_j\|+\|a_i-a_j\|+\|b_i-b_j\|^2+\|\sigma_i-\sigma_j\|
\Big)
\]
and builds a dendrogram by repeatedly merging the closest pair [2505.13052]. The Dendrogram Selection Criterion is
\[
\mathrm{DSC}_N^{(\kappa)} := -\Big(h_N^{(\kappa)} + \omega_N \bar{l}_N^{(\kappa)}\Big),
\]
with recommended \(\omega_N=\log N\), and the selected number of experts is
\[
\widehat{K}_N := \arg\min_{\kappa\in[2,K]} \mathrm{DSC}_N^{(\kappa)}.
\]
The paper proves \(\widehat{K}_N \to K_0\) in probability and reports that on synthetic data this method outperforms Akaike information criterion, Bayesian information criterion, and integrated completed likelihood while recovering the number of experts accurately [2505.13052]. A plausible implication is that developmental MoE need not only grow experts; it may also merge or prune them in a statistically principled way.

A more recent theoretical perspective studies developmental specialization under gradient-based learning. In a regression setting with latent clustered single-index structure, one paper shows that a vanilla neural network fails to detect the latent organization, whereas a MoE succeeds by dividing the problem into easier subproblems [2506.01656]. The student MoE has multiple experts, each a two-layer network,
\[
f_m(x;W_m) = \frac{1}{J}\sum_{j=1}^J a_{m,j}\,\sigma_m(w_{m,j}^\top x + b_{m,j}),
\]
and a softmax router
\[
\pi_m(x;\Theta) = \frac{\exp(h_m(x))}{\sum_{m'}\exp(h_{m'}(x))},\qquad h(x;\Theta)=\Theta^\top x
\]
[2506.01656]. Training is split into four phases: exploration, router learning, expert learning under router, and second-layer optimization [2506.01656]. The paper proves that with
\(T_1=\tilde{\Theta}(d^{k^*-1})\), \(T_2=\Theta(d)\), \(T_3=\tilde{\Theta}(d^{k^*-1}\vee d\epsilon^{-2}\vee\epsilon^{-3})\), and \(T_4=\tilde{\Theta}(\epsilon^{-2})\), the MoE achieves expected error at most \(\epsilon\) with probability at least \(0.99\) [2506.01656]. In contrast, the vanilla network’s hidden neurons remain at random-initialization scale relative to the global direction \(w_g^*\) [2506.01656]. This formalizes a developmental account in which weak specialization emerges first, router differentiation follows, and only then does strong expert specialization occur.

Taken together, these results suggest three distinct but compatible interpretations of developmental MoE. The first is staged architectural and routing maturation during optimization, exemplified by EvoMoE [2112.14397]. The second is smooth variation of mixture structure across an explicit developmental axis, exemplified by VCMoE [2601.01699]. The third is adaptive growth, clustering, and consolidation of experts, supported by dendrogram-based model selection and SGD-based latent-cluster discovery [2505.13052; 2506.01656]. The term therefore names not a single canonical model, but a research program centered on MoE architectures whose specialization is learned, scheduled, or indexed over developmental time.

Source: https://www.emergentmind.com/topics/developmental-mixture-of-experts-moe