Slicing and Dicing: Configuring Optimal Mixtures of Experts

Abstract: Mixture-of-Experts (MoE) architectures have become standard in LLMs, yet many of their core design choices - expert count, granularity, shared experts, load balancing, token dropping - have only been studied one or two at a time over narrow configuration ranges. It remains an open question whether these choices can be optimized independently, without considering interactions. We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters, in which we exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms. We find that at every active-parameter scale that we study, performance consistently improves with total MoE parameters even at extreme active expert parameter ratios like 128.Further, the optimal expert size is nearly invariant to total parameter count and depends only on active parameter count. Third, we see that other choices like shared experts, heterogeneous experts and load-balancing settings have small effects relative to expert count and granularity, although dropless routing yields a consistent gain. Overall, our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.

• The paper shows that performance gains in MoE LMs primarily stem from increasing total expert count at a fixed coarse granularity within compute and memory constraints.
• It rigorously decouples hyperparameters to demonstrate that expert heterogeneity and always-active modules offer minimal benefit or can even impair performance.
• Empirical analysis across scales indicates that proper routing and load balancing are secondary to efficient parameter allocation in enhancing language modeling accuracy.

Slicing and Dicing: A Systematic Analysis of Mixture-of-Experts Configuration Space

Overview and Motivation

This paper provides the most comprehensive empirical study to date of Mixture-of-Experts (MoE) architectural choices for LLMs, interrogating the interplay among expert count, expert granularity, sharing/heterogeneity, and routing/load-balancing mechanisms across over 2,000 pretraining runs. Spanning model scales from 10M to 6.6B parameters, the work rigorously decouples multiple often-tied hyperparameters and exposes how performance emerges as a function of both active and inactive parameter allocation. The principal finding is that, within compute- and memory-matched settings, performance gains trace almost entirely to maximizing total expert parameterization—primarily by increasing expert count—while auxiliary design choices yield only minimal additional benefit. This significantly simplifies the practical optimization landscape for MoE LMs.

Detailed Experimental Design

The experimental suite exhaustively explores a multi-dimensional MoE design space, including both homogeneous and heterogeneous expert sets, optionally supplemented by always-active "generalist" FFNs, as well as systematic sweeps over load-balancing hyperparameters and routing choices. Models are FLOP-matched on the basis of active FFN parameters, isolating performance effects attributable to varying total expert count $n$, granularity $g$ (fractional expert width versus dense baseline), or activation sparsity $s = n \cdot g$. Furthermore, expert size and expert count are manipulated independently to tease apart their contributions (Figure 1).

Figure 1: MoE layer schematic. Each token traverses shared self-attention, is routed to high-affinity experts (possibly of heterogeneous sizes), and the expert outputs are combined. Generalist (G) modules are always active (not routed); heterogeneity can arise from mixing expert granularities within a layer.

At all scales, language modeling performance is evaluated using macro-average cross-entropy on diverse held-out domains; ablation analyses are conducted on downstream tasks (e.g., BoolQ, HellaSwag, MMLU) as well. The models are pretrained on a multilingual, high-quality corpus following Chinchilla guidelines for data-to-parameter ratio, with all hyperparameters precisely documented for rigorous comparison.

Key Empirical Findings

1. Performance Is Predominantly Parameter-Driven

Across all activation scales $\geq 50$M, MoEs consistently and substantially outperform parameter-matched dense baselines so long as the total (inactive) parameter budget is maximized—primarily by increasing $n$ at nearly fixed $g$ (Figures 2, 8, 9, 11). This effect subsides only at very small compute, where dense models saturate data fit before sparsity provides leverage (Figures 14, 18, 19).

Figure 2: Validation performance scales monotonically with total expert parameter count, holding active parameters fixed.

Figure 3: Scaling trends for 50-110M active, up to 1.4B total parameters reinforce the dominance of total parameter count over other settings.

Notably, expert granularity $g$ (i.e., the size of each expert) should remain coarse (e.g., $1/4$ to $1/2$ of dense width), only slowly increasing with extreme activation sparsity, rather than becoming ultra-fine-grained (Figures 2, 8, 20, 21). Increasing $n$ at near-constant $g$0 provides the best utilization of memory for performance uplift.

2. Expert Heterogeneity and Generalists Are Ineffective or Detrimental

Introducing expert heterogeneity (mixed granularities within a layer) or always-active generalist layers does not confer any systematic improvement over the best homogeneous MoE at fixed memory/compute. Heterogeneous MoEs interpolate between the best matching homogeneous settings but provide negligible or no gains (Figures 6, 12, 13).

Figure 4: Heterogeneous expert pools underperform relative to the envelope traced out by homogeneous configurations at the same sparsity and parameter budget.

Adding a generalist FFN module consistently reduces validation accuracy, even when allocated considerable width (Figure 5):

Figure 5: The inclusion of a generalist module consistently hurts performance regardless of configuration.

These results suggest the architectural flexibility for mixing expert types is unnecessary or even counterproductive under optimal parameter allocation.

3. Routing and Load Balancing: Marginal Gains, Little Sensitivity

MoE performance is robust to a wide range of load balancing hyperparameters and auxiliary routing losses, requiring only coarse tuning to avoid degenerate expert loading. Dropless routing (preventing token loss due to routing overflow) provides a modest but consistent gain, particularly at lower expert counts, but as expert count increases, its impact wanes (Figure 6).

Figure 6: Dropless routing marginally outperforms default (token-dropping) routing, especially at moderate expert count.

Poorly chosen load balancing settings (especially excessively high loss-free bias) can impair performance (Figure 7), but within moderate ranges, the sensitivity is low. These findings demote the importance of elaborate auxiliary losses or routing innovations relative to macro architectural decisions.

Figure 7: Load balancing mechanisms must be adequately tuned; very strong load-free bias can harm results, but moderate settings are robust.

4. MoE Advantage Emerges Only Above a Compute Floor

At low activation scales (10-20M active parameters), even sweeping expert count and granularity, MoE LMs underperform dense baselines under Chinchilla-optimal token count (Figures 14-17). Only by increasing the data budget (by 5-20$g$1), thus matching the compute of larger models, do MoEs begin to surpass dense models. This aligns the empirical regime with scaling law predictions.

Figure 8: At 10M active, MoEs underperform at Chinchilla-optimal tokens, but win after 20$g$2 more data (compute-matched with 50M scale).

Figure 9: Similarly, at 20M active, accessing more data is required for MoEs to overtake dense models.

5. Fine-Grained Divisions Without Sparsity Harm Performance

Dividing FFN parameters into multiple always-active smaller components (without exploitation of conditional routing/sparsity) does not improve—and often impairs— performance, further underscoring the necessity of both sparsity and overall parameter scaling for MoE benefit (Figure 10).

Figure 10: The performance boost of MoEs is not due to granularity alone; splitting dense FFNs without sparsity severely underperforms.

Implications and Theoretical Reflection

The results paint a strikingly simple picture: Optimal MoE model design for a fixed compute/memory budget, and in the scaling-law regime, is almost entirely a matter of allocating as many total parameters as possible to conditional experts, maximizing expert count at a fixed, coarse granularity, and discarding most other architectural flexibility. Sophisticated tuning of auxiliary router losses, capacity factors, granularity mixing, and shared generalist modules confer minimal to negative benefit once global sparsity and parameter count are properly set.

This finding contradicts recent claims in the literature favoring intricate heterogeneity [wang2025hmoe], pervasive use of shared experts [zhao2025comprehensivescalinglawmixtureofexperts], or the necessity of highly fine-grained experts [tian2025greaterleveragescalinglaws]. Instead, the present results favor a scale-dependent, not fixed, recipe, advocating for aggressive increase in expert count as scale and budget permit. Consequently, theoretical analyses or practical recipes that tie optimal MoE configuration “constants” across very different scales may fail to generalize (cf. [misfitting]).

Practically, this enables the design of high-performing MoE LMs with minimal architecture search: select active parameter target, choose a coarse expert granularity ($g$3 is near-optimal across most scales), allocate maximal expert count within the memory envelope, and adjust only minimally for router and load-balancing settings.

Future Directions

This systematic analysis opens several additional avenues:

• Scaling Laws at Trillion-Parameter Regimes: How persistently do these trends extrapolate to ultra-large models? What are the practical hardware/throughput breakpoints for further increasing expert count?
• Domain-Specific Experts: While heterogeneous granularity offers no consistent gain in language modeling, could domain-conditional experts for specialized modalities produce different outcomes?
• Efficient Distributed Implementation: The hardware cost (especially memory) of extremely large $g$4 must be balanced against model throughput and training complexity; system engineering advances (e.g., MegaBlocks routing [megablocks]) may further optimize for requested architectures.
• Downstream Impact: While the paper’s downstream task ablations show some gain mirroring language modeling loss, task-specific trends and possible transfer improvements remain under-explored.

Conclusion

This paper provides strong evidence that most MoE design complexity—including expert heterogeneity, generalists, and fine-grained auxiliary routing—is unnecessary for optimal performance: maximizing total expert parameter (via count, not width) and guaranteeing effective routing/load balancing are sufficient and almost entirely determinant of achieved accuracy in language modeling regimes above the compute threshold. These findings should incentivize both researchers and practitioners to prioritize these global configuration choices, abandoning excessive micro-architectural tuning except as required by resource constraints or hardware idiosyncrasies (2605.11689).