Mixture-of-Experts Configurations

• Mixture-of-Experts configurations are defined by their architecture, gating mechanisms, and hyperparameters that route data to specialized subnetworks.
• They leverage conditional computation to activate only a subset of experts, maximizing model capacity while controlling inference cost.
• Practical tuning involves balancing expert count, sparsity, and routing strategies to adhere to memory and throughput constraints.

A Mixture-of-Experts (MoE) configuration refers to the complete specification of the architecture, gating/routing mechanisms, hyperparameter values, and resource allocation strategies that govern how a model dynamically routes data to multiple expert subnetworks. MoE architectures exploit conditional computation to enlarge model capacity without proportionally increasing compute per sample. Optimal configuration of MoE systems is an active domain of both empirical innovation and theoretical analysis, as they are central to the efficient scaling of state-of-the-art LLMs, vision models, multitask systems, and generative models in the current deep learning landscape.

1. Formal Parameterization and Core Design Variables

A standard Transformer-style MoE layer is fully characterized by a set of structural and sparsity parameters:

• Depth ($l$), model (hidden) dimension ($d$)
• Number of experts per layer ($n_{\rm exp}$)
• Expert hidden dimension ($d_{\rm exp}$), with granularity $g \coloneqq d / d_{\rm exp}$
• Number of experts activated per token ($n_{\rm topk}$), controlling sparsity $s := n_{\rm exp}/n_{\rm topk}$

Parameter counts are given by: $$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$ where $N_{\rm total}$ is the total (memory) parameter count and $N_{\rm active}$ is the number of parameters used per token in a forward pass (Liew et al., 13 Jan 2026).

Key roles:

• $d$0 sets the number of distinct, parameter-disjoint expert MLPs.
• $d$1 defines how many experts contribute per token, trading off between model utilization and inference cost.
• $d$2 controls the granularity of expert partitioning, allowing precise modulation of intermediate FFN width and the overall gain from sparsity (Krajewski et al., 2024).

2. Memory, Inference, and Compute Constraints

MoE configurations are universally dictated by deployment-specific constraints: $d$3 where $d$4 is the available model memory budget (non-embedding parameters) and $d$5 is the maximum active parameter count per sample (related to throughput/latency) (Liew et al., 13 Jan 2026).

The practical configuration task is to maximize held-out performance (e.g., minimize loss $d$6) over quadruples $d$7 given these constraints.

Recent large-scale analyses establish that $d$8 dominates performance variance, but both the expert sparsity $d$9 and the absolute value of $n_{\rm exp}$0 exert secondary but significant effects due to their implicit impact on core model width/depth. This is captured in the empirical scaling law: $n_{\rm exp}$1 with a small penalty on large $n_{\rm exp}$2 at fixed $n_{\rm exp}$3 (Liew et al., 13 Jan 2026).

3. Routing, Gating, and Expert Selection Mechanisms

MoE configurations support several architectural and algorithmic alternatives for routing:

• Standard top-$n_{\rm exp}$4 gating: A router projects each token to a softmax over experts, then selects the $n_{\rm exp}$5 experts with highest scores:

$n_{\rm exp}$6

where $n_{\rm exp}$7 is the normalized gate for expert $n_{\rm exp}$8 (Zhang et al., 15 Jul 2025).

• Hierarchical routing: Groups experts into super-experts and applies staged top-$n_{\rm exp}$9 selection at each level, reducing gate computational load for large expert pools (Zhang et al., 15 Jul 2025).
• Maximum Score Routing (MaxScore): Formulates routing as a minimum-cost maximum-flow problem to enforce hard capacity constraints per expert and prevent token drop, yielding near-perfect load balance under hardware constraints (Dong et al., 18 Aug 2025).
• Multi-head MoE (MH-MoE): Splits input representations into multiple “heads” and applies independent per-head MoE gating and routing to each subspace, before concatenating outputs. This provides consistently improved perplexity and compatibility with 1-bit quantized LLMs (Huang et al., 2024).
• Mixture of Precisions: Selectively quantizes experts to different bit-widths (e.g., FP16 and INT4) and offloads to CPU/GPU based on memory/throughput constraints, enabling real-time Pareto trade-offs between speed and quality (Imani et al., 2024).
• Shared and task-adaptive experts: Supplement sparse task-specific experts with shared, always-active experts and normalize the gating over both; exemplified in LoRA-based MoEs for multi-task transfer (Yang et al., 1 Oct 2025).

The router mechanism is the locus of specialization, capacity balancing, and regularization. Auxiliary losses (e.g., load balancing, entropy, or mutual-distillation) are often required to prevent expert collapse or load imbalance (Zhang et al., 15 Jul 2025, Yang et al., 1 Oct 2025, Dong et al., 18 Aug 2025).

4. Empirical and Theoretical Configuration Principles

Joint scaling laws and empirical ablations provide actionable, quantitative recipes for expert count, width, granularity, and sparse activation patterns:

Design Variable	Principle	Practical Range / Recipe
$d_{\rm exp}$0	Minimize for fixed $d_{\rm exp}$1; higher incurs penalty.	Small powers of two (64, 128) (Liew et al., 13 Jan 2026).
$d_{\rm exp}$2	Max out under $d_{\rm exp}$3; increases utilization.	$d_{\rm exp}$4 (Liew et al., 13 Jan 2026).
Granularity $d_{\rm exp}$5	$d_{\rm exp}$6	Empirically optimal in $d_{\rm exp}$7 (Krajewski et al., 2024, Liew et al., 13 Jan 2026).
Width-to-depth $d_{\rm exp}$8	$d_{\rm exp}$9	$g \coloneqq d / d_{\rm exp}$0 (Liew et al., 13 Jan 2026).
Routing Mechanism	Gating/routing choice	Top-$g \coloneqq d / d_{\rm exp}$1, Maximum Score Routing, shared experts, adaptive quantization.
MoE FFN width	Reduce per expert for more experts at fixed activated cost	Expansion factors $g \coloneqq d / d_{\rm exp}$2–$g \coloneqq d / d_{\rm exp}$3 better than $g \coloneqq d / d_{\rm exp}$4 at scale (Liu et al., 1 Dec 2025).

Scaling laws show that (a) compute-optimal MoE configurations use finer-grained experts ($g \coloneqq d / d_{\rm exp}$5), and (b) the efficiency gap between MoE and dense Transformers grows with scale, often yielding $g \coloneqq d / d_{\rm exp}$6–$g \coloneqq d / d_{\rm exp}$7 FLOP savings at large $g \coloneqq d / d_{\rm exp}$8 and budget $g \coloneqq d / d_{\rm exp}$9 (Krajewski et al., 2024). The rule-of-thumb “expert width = FFN width” ($n_{\rm topk}$0) is systematically sub-optimal.

5. Practical Configuration and Tuning Procedures

A robust practitioner workflow (Liew et al., 13 Jan 2026, Liu et al., 1 Dec 2025, Krajewski et al., 2024) proceeds as follows:

1. Establish budgets: Set $n_{\rm topk}$1 (max parameters in memory) and $n_{\rm topk}$2 (max active parameters/inference latency).
2. Granularity & structural ratios: Choose $n_{\rm topk}$3 and width-to-depth $n_{\rm topk}$4 within validated ranges.
3. Expert sweep: For each candidate $n_{\rm topk}$5, maximize $n_{\rm topk}$6 under $n_{\rm topk}$7 and calculate $n_{\rm topk}$8 to saturate $n_{\rm topk}$9.
4. Score configurations: Evaluate held-out loss proxy

$s := n_{\rm exp}/n_{\rm topk}$0

and select the configuration that minimizes $s := n_{\rm exp}/n_{\rm topk}$1 (Liew et al., 13 Jan 2026).

1. Sanity checks: Ensure all hyperparameters respect their tested ranges; saturate active parameter budgets.

For applications requiring multi-task transfer, domain conflict resolution, or adaptation, additional configuration axes include shared (“global”) experts, LoRA-based low-rank expert construction, and sparse routing over lightweight adaptation modules (Yang et al., 1 Oct 2025, Chen et al., 2024, Liu et al., 4 Aug 2025).

6. Configuration Trade-offs, Empirical "Sweet Spots", and Limitations

Empirical results across modalities reveal critical trade-offs induced by expert count, activation sparsity, and layer placement:

• In image classification, the sweet spot is moderate $s := n_{\rm exp}/n_{\rm topk}$2 and $s := n_{\rm exp}/n_{\rm topk}$3 (or $s := n_{\rm exp}/n_{\rm topk}$4 in ViT), with late-stage (last two) layer insertion preferred. Beyond this, accuracy benefits vanish or decline due to data fragmentation, under-trained experts, and rising routing overhead. Sample-wise activated parameter limit is $s := n_{\rm exp}/n_{\rm topk}$5M (Videau et al., 2024).
• In multitask and multi-modal instruction tuning, $s := n_{\rm exp}/n_{\rm topk}$6 LoRA experts per FFN, top-1 routing, and aggressive load-balancing losses yield best reuse/conlict avoidance (Chen et al., 2024).
• For Diffusion MoE architectures, improved FID/IS is obtained with $s := n_{\rm exp}/n_{\rm topk}$7 sparse experts, aggressive reduction of MLP expansion factor, and inclusion of “shared” always-on experts for regularization, without increasing per-token compute (Liu et al., 1 Dec 2025).

Limitations persist: very high $s := n_{\rm exp}/n_{\rm topk}$8 and $s := n_{\rm exp}/n_{\rm topk}$9 fragment data unduly, defeating the benefit of increased parameter count. Routing overhead becomes dominant as expert/tok-distribution becomes imbalanced. The optimal values remain sensitive to dataset size, task diversity, and hardware constraints.

7. Theoretical and Bayesian Perspectives on MoE Configuration

Beyond practical recipes, rigorous statistical theory informs expert count and specialization:

• Nonparametric convergence rates balance approximation error (favoring many simple experts) versus estimation error (favoring few complex experts). Optimal choices minimize:

$$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$0

for $$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$1 experts of order-$$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$2 experts in $$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$3 dimensions (Mendes et al., 2011).

• Bayesian selection: Placing a prior on number of experts $$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$4 and employing ELBO-based or full Bayesian model selection reliably recovers $$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$5 and achieves nearly parametric posterior contraction rates in density estimation (Bariletto et al., 22 Apr 2026).

This theoretical foundation establishes (a) why $$N_{\rm total} \approx l\,d^2 \left[4 + \frac{3 n_{\rm exp}}{g} \right], \qquad N_{\rm active} \approx l\,d^2 \left[4 + \frac{3 n_{\rm topk}}{g} \right]$$6 and expert capacity must scale with both data size and intrinsic function complexity, and (b) the value of load-balancing, identifiability constraints, and regularization in practical MoE configuration.

---

In conclusion, Mixture-of-Experts configuration is governed by a joint regime of parameter count, sparsity, expert/activation matching, and deployment constraints, underpinned by both empirically validated design rules and nonparametric statistical theory. The optimal configuration maximizes total parameters, minimizes sparsity, judiciously selects expert count, tunes granularity, and rigorously aligns with memory/inference budgets—a synthesis systematically derived and operationalized in recent large-scale and multimodal MoE research (Liew et al., 13 Jan 2026, Krajewski et al., 2024, Videau et al., 2024, Liu et al., 1 Dec 2025, Yang et al., 1 Oct 2025, Bariletto et al., 22 Apr 2026).