Dynamic-Capacity MoE Design

• Dynamic-capacity MoEs are adaptive neural architectures that select variable expert subnetworks based on input complexity to optimize computational efficiency.
• They utilize soft or hard routing with cumulative confidence criteria and load-balancing losses to ensure effective utilization of experts.
• Empirical results show these models outperform fixed-capacity variants, achieving higher accuracy with fewer computational resources on benchmarks like BBH and ARC.

A Dynamic-Capacity Mixture-of-Experts (MoE) design in deep neural architectures provides an adaptive mechanism for activating variable expert subnetworks conditioned on input complexity, task demands, or token-level routing confidence. Unlike fixed-capacity MoEs, which dispatch a predetermined number of experts (often via Top-K routing), dynamic-capacity MoEs determine the number and composition of activated experts per input, optimizing the trade-off between computational efficiency and representational power. Key approaches leverage soft or hard routing in Transformers, cumulative confidence criteria, stratified gating, and various forms of grouped, hierarchical, or reusable expert pools. The goal is to align model capacity with per-instance difficulty, minimize redundant computation on simple data, and ensure scalability under hardware or resource constraints.

1. Adaptive Expert Selection: Core Algorithmic Principles

Dynamic-capacity MoE designs replace dense feed-forward blocks with an expert pool and a learnable routing (gating) network. For each input token $x$, the router computes a probability distribution $P(x)$ across $N$ experts. Unlike traditional Top-K routing, which activates a static number of experts, dynamic-capacity mechanisms select a variable subset $S(x)$ based on cumulative confidence.

The canonical dynamic routing (as in "Harder Tasks Need More Experts: Dynamic Routing in MoE Models" (Huang et al., 12 Mar 2024)) proceeds as follows:

1. Compute $P(x) = \text{Softmax}(W_r x^T) \in \mathbb{R}^N$.
2. Sort $P_i(x)$ descending; form cumulative sum until a threshold $p$ is reached:

$$t(x) = \min \{ k \mid \sum_{j=1}^k P_{I_j}(x) \ge p \}$$

1. Activate experts $S(x) = \{e_{I_1}, \dots, e_{I_{t(x)}}\}$.
2. MoE output: $\text{MoE}(x) = \sum_{i=1}^N g_i(x) \cdot e_i(x)$, where $g_i(x) = P_i(x)$ if $i \in S(x)$, zero otherwise.

This variable-t mechanism flexibly scales computation, dispatching more experts only for data requiring complex reasoning, confirmed by results on tasks such as BBH and ARC.

2. Load Balancing, Regularization, and Training Objectives

Dynamic-capacity MoEs incorporate specialized losses to prevent gating collapse and maintain expert diversity:

• Language modeling loss: Standard next-token cross-entropy.
• Dynamic loss (entropy regularizer):

$\text{Loss}_d = -\sum_{i=1}^N P_i(x) \log P_i(x)$

Penalizes uniform distributions, promoting confident routing.

• Load-balance auxiliary loss: Forces uniform utilization across the batch:

$\text{Loss}_b = N \sum_{i=1}^N f_i Q_i$

Where $f_i$ is the fraction of tokens routed to expert $i$, and $Q_i$ is the mean $P_i(x)$ across the batch.

• Composite loss:

$$\text{Loss} = \text{Loss}_{lm} + \alpha \text{Loss}_b + \beta \text{Loss}_d$$

Hyperparameters such as $\alpha$ and $\beta$ (typically $\alpha=10^{-2}$, $\beta=10^{-4}$) tune the regularization balance; threshold $p$ is recommended in the range $[0.3, 0.5]$ to prevent under-allocation or excessive sparsity (Huang et al., 12 Mar 2024).

3. Heterogeneous, Layer-wise, and Grouped Dynamic Designs

Empirical analysis reveals significant variation in average experts activated per layer—bottom Transformer layers tend to route to up to 4 experts per token, while top layers stabilize to 1 (monotonic t-decrease). This supports heterogeneous MoE designs: high-capacity (large $N$, high $t$) lower layers for rich representations; low-capacity upper layers to avoid overprocessing (Huang et al., 12 Mar 2024).

Advanced variants (e.g., AT-MoE (Li et al., 12 Oct 2024)) extend dynamic-capacity to grouped and task-specific architectures:

• Experts are organized into groups (e.g., domain, function, style).
• Routing proceeds via staged Softmax normalizations: group-level allocation, then within-group expert selection.
• Layerwise routing adapts fusion and capacity shift to data complexity, supporting multi-dimensional balance and interpretability.

4. Resource and Quality Trade-offs under System Constraints

Deployment of dynamic-capacity MoEs in real-world systems often involves additional adaptation:

• Partial quantization (mixture of precisions) enables Pareto-efficient throughput/perplexity trade-offs under memory constraints. Precision and compute locations (CPU/GPU) are dynamically assigned by solving optimizations:

$$\min_{s \in \mathcal{S}} P(s) \text{ s.t. } M_{GPU}(s) \le M_{\max}$$

Enumerating allowed quantizations and allocations, with observed throughput adjustable from $0.63$ to $13.00$ tokens/sec and perplexity increase marginal under maximal quantization (Imani et al., 19 Jul 2024).

5. Empirical Performance, Efficiency, and Practical Guidelines

Dynamic-capacity MoE models consistently outperform fixed Top-K approaches on diverse benchmarks:

Model	Activated Params	PIQA	Hellaswag	ARC-e	CSQA	BBH	Avg
MoE-Top2	581M	68.1	43.9	40.4	32.1	23.3	41.6
MoE-Dynamic	<2 experts/token	68.1	44.3	39.9	33.6	25.6	42.3

MoE-Dynamic yields $+0.7\%$ average gain over Top-2, $+2.3\%$ on BBH, with $12\%$ fewer parameters and FLOPs per token (Huang et al., 12 Mar 2024). The average experts per token during inference $\bar{t} \approx 1.76$ (vs. Top-2's $2$); efficient load balancing is sustained through the described losses.

Design recommendations include:

• Careful tuning of threshold $p$ and auxiliary loss weights.
• Monitoring of $\bar{t}$; aim for fewer active experts than fixed-K to preserve efficiency.
• For heterogeneous designs, layer-wise allocation of $N$ and $t$ (e.g., $N=$ 16/8/4 descending).

6. Extensions, Interpretability, and Future Directions

Dynamic-capacity MoEs are extensible along multiple axes:

• Multi-stage routing (e.g., AT-MoE (Li et al., 12 Oct 2024)), cross-layer sharing (ReXMoE (Tan et al., 20 Oct 2025)), stratified tiering (SMoE (Xu et al., 2023)), evolutionary diversification (EvoMoE (Jing et al., 28 May 2025)).
• Integration with PEFT techniques (LoRA, AdaLoRA, IA^3, prefix-tuning).
• Supplementary HyperExperts (HyperMoE (Zhao et al., 20 Feb 2024)) transfer knowledge from unselected experts without inflating K.
• System-level adaptations for latency, memory, and throughput (e.g., mixture of precisions (Imani et al., 19 Jul 2024), capacity-aware token drop (He et al., 7 Mar 2025)).

Challenges include regularization against expert collapse, task grouping, and hardware scheduling. Research continues into dynamic expert addition/removal, budget-aware routing, and joint expert-router adaptation.

7. Comparative Summary and Conceptual Significance

Dynamic-capacity Mixture-of-Experts frameworks mark a significant advance over statically routed sparse architectures, demonstrating that model capacity can be efficiently aligned to input complexity, heterogeneity, and external constraints—yielding superior accuracy and hardware efficiency. They operationalize adaptive computation at scale in both language and vision domains, supported by rigorous algorithmic, empirical, and theoretical foundations (Huang et al., 12 Mar 2024, Li et al., 12 Oct 2024, He et al., 7 Mar 2025, Imani et al., 19 Jul 2024).