Developmental Mixture of Experts

• Developmental Mixture of Experts is a class of architectures where expert specialization and routing evolve over time or stages during training.
• It introduces phased processes—such as expert-diversify and gate-sparsify—to overcome early training instability and promote efficient expert growth.
• Empirical results, including improved language modeling and translation metrics, validate the effectiveness of developmental MoE in optimizing performance and compute efficiency.

Searching arXiv for the cited MoE papers and closely related work to ground the article. arxiv_search query: "(Nie et al., 2021) EvoMoE evolutional mixture-of-experts training framework dense-to-sparse gate" Developmental Mixture of Experts (MoE) denotes a class of mixture-of-experts architectures and statistical models in which expert capacity, routing behavior, or coefficient structure changes along training time or along an explicit index such as developmental stage, time, or space. In this usage, “developmental” refers to staged growth, progressive specialization, or smoothly varying subpopulation structure rather than a fixed, randomly initialized sparse MoE trained end-to-end from the outset. The clearest deep-learning exemplar is EvoMoE, which begins with a single shared expert and gradually evolves into a sparse multi-expert architecture through an expert-diversify phase and a gate-sparsify phase (Nie et al., 2021). In a complementary statistical sense, developmental MoE includes models such as the Varying-Coefficient Mixture of Experts (VCMoE), where both gating and expert coefficients vary along an indexing variable $U$, making the mixture itself explicitly developmental or longitudinal (Zhao et al., 5 Jan 2026). Across these formulations, the central premise is that specialization and routing should emerge or vary over time rather than be fixed a priori.

1. Conceptual definition and scope

Mixture-of-experts models combine multiple specialized submodels through input-dependent gating functions. In the standard formulation, given an input token $x_s \in \mathbb{R}^D$, experts $\{e_1,\dots,e_N\}$, and gating weights $g(x_s)_i$, the MoE layer output is

$y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$

For Transformer-based MoEs, each expert typically replaces an FFN: $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$ A conventional sparse gate is often written as

$$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$

with $W_g \in \mathbb{R}^{D \times N}$ (Nie et al., 2021).

Developmental MoE differs from this static picture by altering either the architecture or the parameter functions during learning. In EvoMoE, the model evolves from a small dense network with one expert and no gate into a large sparse MoE with many experts and Top‑1 routing (Nie et al., 2021). In VCMoE, the mixture components remain present, but the gating coefficients $\boldsymbol{\beta}_c(u)$, expert coefficients $\boldsymbol{\alpha}_c(u)$, and dispersion terms $x_s \in \mathbb{R}^D$0 vary smoothly with an index $x_s \in \mathbb{R}^D$1, so the effective mixture changes across developmental stage or time (Zhao et al., 5 Jan 2026). A plausible unifying interpretation is that developmental MoE treats specialization as a trajectory rather than as a fixed design choice.

The broader theoretical backdrop is that MoE mean functions are dense in the class of all continuous functions over arbitrary compact domains of estimation, giving a universal approximation theorem for MoE models (Nguyen et al., 2016). This suggests that developmental schemes concern how MoE capacity is organized and learned, not whether the family is expressive enough in principle.

2. Static MoE as the baseline from which developmental formulations depart

Static MoE training typically initializes multiple experts randomly, adopts a fixed routing rule such as Top‑1 or Top‑2, and trains experts and gate jointly from scratch. EvoMoE identifies two difficulties with this regime: immature experts and unstable sparse gate (Nie et al., 2021). In the described GPT-MoE with 16 experts, one expert monopolizes the load for hundreds of steps while other experts are underutilized and undertrained, and routing for a token such as “the” is highly unstable early in training (Nie et al., 2021). These observations motivate a developmental alternative in which shared knowledge is learned before specialization.

The probabilistic MoE literature provides the more general conditional form

$x_s \in \mathbb{R}^D$2

with covariate-dependent gating probabilities and expert-specific conditional models (Nguyen et al., 2017). In this perspective, developmental behavior can arise by increasing $x_s \in \mathbb{R}^D$3, by refining the gating partition, or by allowing coefficients to vary across an index $x_s \in \mathbb{R}^D$4 (Zhao et al., 5 Jan 2026). The universal approximation result that the full MoE class is dense in $x_s \in \mathbb{R}^D$5 over arbitrary compact $x_s \in \mathbb{R}^D$6 supports this divide-and-conquer view of capacity allocation (Nguyen et al., 2016).

A common misconception is that “developmental MoE” necessarily means only expert growth. The literature in the supplied sources suggests a broader usage. EvoMoE is developmental because both experts and gate mature over training iterations (Nie et al., 2021). VCMoE is developmental because latent subpopulation structure and covariate effects evolve along $x_s \in \mathbb{R}^D$7 (Zhao et al., 5 Jan 2026). A plausible implication is that developmental MoE is best understood as a family of staged or indexed MoE formalisms rather than a single architecture class.

3. EvoMoE: staged growth from one expert to sparse routing

EvoMoE is an end-to-end MoE training framework that starts from one single expert and gradually evolves into a large and sparse MoE structure (Nie et al., 2021). Its developmental process has two phases.

In the expert-diversify phase, the model first trains a single shared expert with no gate: $x_s \in \mathbb{R}^D$8 All tokens flow through the same expert, effectively using a dense Transformer FFN. Training proceeds for $x_s \in \mathbb{R}^D$9 iterations using only the task loss $\{e_1,\dots,e_N\}$0 (Nie et al., 2021). After this warm-up, the single expert is diversified into $\{e_1,\dots,e_N\}$1 experts by copying its parameters and applying expert-specific random masks: $\{e_1,\dots,e_N\}$2 This yields multiple diverse experts, all initialized from a trained base expert rather than random weights (Nie et al., 2021).

In the gate-sparsify phase, the model introduces a gate and continues training the MoE with the Dense-to-Sparse gate, written abstractly as

$\{e_1,\dots,e_N\}$3

The design decouples early expert learning from sparse routing and focuses on learning basic shared knowledge with a single expert at the early stage (Nie et al., 2021). This staged procedure is the primary reason EvoMoE is naturally described as a developmental or evolving MoE.

The empirical setting spans RoBERTa for masked language modeling, GPT for language modeling, and Transformer for machine translation (Nie et al., 2021). For RoBERTa, the model uses 24 layers, hidden size 1024, 16 attention heads, replaces every other FFN with MoE-FFN with 16 experts per layer, and reports shared params 259M, expert params 1536M, total 1759M. EvoMoE uses Stage 1 = 5k steps and Stage 2 = 5k steps in that configuration (Nie et al., 2021). For GPT, the model has 24 layers, hidden 1024, 16 heads, and 12 MoE layers $\{e_1,\dots,e_N\}$4 16 experts $\{e_1,\dots,e_N\}$5 experts, with Stage 1 = 10k steps and Stage 2 annealing from $\{e_1,\dots,e_N\}$6 to $\{e_1,\dots,e_N\}$7 over 5k steps (Nie et al., 2021). These setup details instantiate developmental MoE as a concrete training curriculum rather than only a conceptual analogy.

4. Dense-to-Sparse gate and the maturation of routing

The Dense-to-Sparse Gate (DTS-Gate) is the core mechanism by which EvoMoE makes routing developmental (Nie et al., 2021). The gate begins as a dense Gumbel-Softmax router. Given logits

$\{e_1,\dots,e_N\}$8

the gate first computes

$\{e_1,\dots,e_N\}$9

where $g(x_s)_i$0 is sampled from $g(x_s)_i$1 (Nie et al., 2021). Large $g(x_s)_i$2 yields a more uniform distribution and dense routing, while small $g(x_s)_i$3 yields a sharper distribution approaching one-hot behavior.

Instead of fixing $g(x_s)_i$4 as in Top‑$g(x_s)_i$5, EvoMoE uses content-based thresholding: $g(x_s)_i$6 The number of active experts per token is therefore adaptive, and the selected experts’ weights are not renormalized after thresholding (Nie et al., 2021). The temperature is annealed from a large value such as $g(x_s)_i$7 to a smaller value such as $g(x_s)_i$8 over $g(x_s)_i$9 iterations, after which EvoMoE switches to a strict Top‑1 gate (Nie et al., 2021). The resulting training trajectory is dense $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$0 adaptive $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$1 Top‑1.

This gate maturation is paired with a balance loss: $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$2 where

$y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$3

This penalizes uneven loads and skewed gate probabilities, encouraging balanced specialization (Nie et al., 2021). In developmental terms, the gate is not merely sparsified; it is regularized while it matures.

A common misunderstanding is that developmental routing is equivalent to selecting a fixed Top‑$y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$4 later in training. EvoMoE’s mechanism is more specific: the gate starts dense, uses threshold-based adaptive sparsity during transition, and only later becomes Top‑1 (Nie et al., 2021). This suggests that developmental MoE is concerned with the path by which selectivity emerges, not only the terminal sparsity level.

5. Quantitative behavior, optimization, and systems implications

EvoMoE reports that it outperforms Switch, BASE Layer, Hash Layer, and StableMoE (Nie et al., 2021). On GLUE with RoBERTa, EvoMoE achieves average score 90.875 versus Switch 90.313, BASE 90.650, Hash 90.513, and StableMoE 90.413, with gains up to $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$5 over standard Transformer and $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$6 over the best MoE baseline (Nie et al., 2021). On GPT language modeling, perplexity is 12.24 for EvoMoE versus 12.45 for BASE, 12.87 for Hash, 12.91 for StableMoE, 13.12 for Switch, and 15.14 for dense TRM (Nie et al., 2021). On machine translation, the reported BLEU values are En–De: Transformer 28.1, Switch 28.4, EvoMoE 29.6; De–En: Transformer 34.8, Switch 34.6, EvoMoE 36.7; En–Fr: Transformer 39.2, Switch 39.1, EvoMoE 40.3; Fr–En: Transformer 38.1, Switch 38.2, EvoMoE 39.2 (Nie et al., 2021).

The framework also reports compute-related improvements. For GPT-MoE with OpenWebText, the DTS gate achieves approximately $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$7 fewer iterations to reach the same validation perplexity as Switch and approximately $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$8 FLOPs-efficiency improvement (Nie et al., 2021). On machine translation, EvoMoE needs only approximately $y_s = \sum_{i=1}^{N} g(x_s)_i \cdot e_i(x_s).$9–$$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$0 of Switch’s compute to reach similar BLEU, with larger gains when more experts are used, and achieves up to $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$1 speed-up (Nie et al., 2021). These values support the view that developmental schedules can improve convergence and effective utilization, not merely final quality.

Optimization details reported for EvoMoE include Adam with $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$2, $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$3, polynomial learning rate scheduler, dropout $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$4, weight decay $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$5, and warm-up steps such as 5k–10k (Nie et al., 2021). The threshold may use a fixed small value such as $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$6, and the paper states that temperature schedules have relatively low sensitivity except at extreme values such as $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$7, $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$8, which can harm gradient variance (Nie et al., 2021).

The implementation notes also identify systems-level developmental support. EvoMoE uses expert parallelism with All-to-All communication, adds a Topology-Aware Hierarchical AllToAll to improve communication efficiency, and introduces MoE-aware recomputation that stores inputs to All-to-All rather than recomputing entire layers (Nie et al., 2021). Although these are engineering mechanisms, they matter because developmental MoE in practice often entails changing expert count, routing density, and communication pattern during training.

6. Statistical developmental MoE: varying coefficients, identifiability, and inference

The VCMoE model provides a statistical formulation of developmental MoE in which the mixture varies along a known index $$e_i(x_s) = \text{FFN}_i(x_s) = W_{2,i} \cdot \text{ReLU}(W_{1,i} x_s + b_{1,i}) + b_{2,i}.$$9 (Zhao et al., 5 Jan 2026). For observations $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$0, with response $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$1, gating covariates $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$2, expert covariates $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$3, and index $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$4, the model is

$$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$5

where $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$6 (Zhao et al., 5 Jan 2026). All coefficient effects in both the gating functions and expert models vary along $$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$7.

Estimation is based on a label-consistent EM algorithm and local weighted likelihood

$$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$8

with local linear approximation

$$g(x_s) = \text{softmax}\big(\text{TopK}(x_s W_g)\big),$$9

and analogous expansions for expert coefficients and dispersions (Zhao et al., 5 Jan 2026). The global E-step computes responsibilities using the same labels for all $W_g \in \mathbb{R}^{D \times N}$0, while the local M-step updates class-specific coefficient functions, thereby avoiding label switching across the developmental axis (Zhao et al., 5 Jan 2026).

VCMoE establishes identifiability and consistency under conditions including continuously differentiable functional coefficients, open domains, distinctness of components so that no two component trajectories are identical or tangent, identifiability of the static finite mixture for fixed $W_g \in \mathbb{R}^{D \times N}$1, and known number of components $W_g \in \mathbb{R}^{D \times N}$2 (Zhao et al., 5 Jan 2026). The local estimator satisfies

$W_g \in \mathbb{R}^{D \times N}$3

and

$W_g \in \mathbb{R}^{D \times N}$4

under the stated regularity conditions (Zhao et al., 5 Jan 2026). The model also constructs simultaneous confidence bands and develops a generalized likelihood ratio test for whether a coefficient function is genuinely varying across the index variable (Zhao et al., 5 Jan 2026). This gives developmental MoE a full inferential apparatus rather than only predictive semantics.

The embryonic mouse application makes the developmental interpretation concrete. Using single nucleus gene expression data, the model characterizes temporal dynamics in the associations between Satb2 and Bcl11b across two latent neuron subpopulations, yielding results consistent with prior findings (Zhao et al., 5 Jan 2026). The reported ROC AUC for predicting upper-layer versus deep-layer labels on a test set is 0.885 (Zhao et al., 5 Jan 2026). Here, developmental MoE means that both mixture proportions and within-subpopulation covariate effects evolve over embryonic time.

7. Theoretical foundations, model selection, and emerging developmental theory

The universal approximation theorem for MoE models establishes that the class of MoE mean functions is dense in $W_g \in \mathbb{R}^{D \times N}$5 for arbitrary compact domains of estimation (Nguyen et al., 2016). This result generalizes earlier work that assumed sufficiently differentiable target functions from Sobolev space and compact unit hypercube domains (Nguyen et al., 2016). For developmental MoE, the implication is not about training dynamics, but about representational sufficiency: adding experts and refining gating can, in principle, approximate arbitrary continuous structure on arbitrary compact domains.

Model selection becomes especially significant once development is interpreted as growth and consolidation of experts. For Gaussian-gated Gaussian MoE, dendrograms of mixing measures provide a mechanism to estimate the true number of experts from a single overfitted fit, avoiding the need to train and compare a range of models with varying numbers of components (Thai et al., 19 May 2025). The paper defines a dissimilarity between atoms

$W_g \in \mathbb{R}^{D \times N}$6

and builds a dendrogram by repeatedly merging the closest pair (Thai et al., 19 May 2025). The Dendrogram Selection Criterion is

$W_g \in \mathbb{R}^{D \times N}$7

with recommended $W_g \in \mathbb{R}^{D \times N}$8, and the selected number of experts is

$W_g \in \mathbb{R}^{D \times N}$9

The paper proves $\boldsymbol{\beta}_c(u)$0 in probability and reports that on synthetic data this method outperforms Akaike information criterion, Bayesian information criterion, and integrated completed likelihood while recovering the number of experts accurately (Thai et al., 19 May 2025). A plausible implication is that developmental MoE need not only grow experts; it may also merge or prune them in a statistically principled way.

A more recent theoretical perspective studies developmental specialization under gradient-based learning. In a regression setting with latent clustered single-index structure, one paper shows that a vanilla neural network fails to detect the latent organization, whereas a MoE succeeds by dividing the problem into easier subproblems (Kawata et al., 2 Jun 2025). The student MoE has multiple experts, each a two-layer network,

$\boldsymbol{\beta}_c(u)$1

and a softmax router

$\boldsymbol{\beta}_c(u)$2

(Kawata et al., 2 Jun 2025). Training is split into four phases: exploration, router learning, expert learning under router, and second-layer optimization (Kawata et al., 2 Jun 2025). The paper proves that with $\boldsymbol{\beta}_c(u)$3, $\boldsymbol{\beta}_c(u)$4, $\boldsymbol{\beta}_c(u)$5, and $\boldsymbol{\beta}_c(u)$6, the MoE achieves expected error at most $\boldsymbol{\beta}_c(u)$7 with probability at least $\boldsymbol{\beta}_c(u)$8 (Kawata et al., 2 Jun 2025). In contrast, the vanilla network’s hidden neurons remain at random-initialization scale relative to the global direction $\boldsymbol{\beta}_c(u)$9 (Kawata et al., 2 Jun 2025). This formalizes a developmental account in which weak specialization emerges first, router differentiation follows, and only then does strong expert specialization occur.

Taken together, these results suggest three distinct but compatible interpretations of developmental MoE. The first is staged architectural and routing maturation during optimization, exemplified by EvoMoE (Nie et al., 2021). The second is smooth variation of mixture structure across an explicit developmental axis, exemplified by VCMoE (Zhao et al., 5 Jan 2026). The third is adaptive growth, clustering, and consolidation of experts, supported by dendrogram-based model selection and SGD-based latent-cluster discovery (Thai et al., 19 May 2025, Kawata et al., 2 Jun 2025). The term therefore names not a single canonical model, but a research program centered on MoE architectures whose specialization is learned, scheduled, or indexed over developmental time.