The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

Abstract: Mixture of Experts (MoEs) are now ubiquitous in LLMs, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions ($\sim$60\%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

• The paper reveals that routing in sparse MoEs is dictated by the geometry of hidden states, not by domain-specific expertise.
• It employs rigorous empirical and theoretical analyses to show that token embedding similarity primarily drives expert assignments.
• The study illustrates how auxiliary loss and layer depth shape expert specialization, with broad implications for model design and privacy.

Mechanistic Deconstruction of Expert Specialization in Sparse MoEs

Introduction

This work systematically interrogates the canonical interpretation of expert specialization in sparse Mixture-of-Experts (MoE) LLMs. Contrary to the prevailing view that routing decisions induce or reflect domain-specific expertise, the authors empirically and theoretically demonstrate that expert activation is driven fundamentally by the geometry of the internal hidden states. Specialized expert patterns are mathematically inevitable consequences of the specific manner in which token embeddings are distributed in high-dimensional space, not emergent phenomena of the routing mechanism per se. This paradigm challenges the attribution of semantic or domain-driven explanations to observed expert utilization, reorienting the study of MoE specialization toward a geometric and representational lens.

Theoretical Foundation: Routing Reflects Geometry

The MoE router in modern LLMs operates as a linear projection from the normalized hidden state of each token to expert selection logits. The core mechanistic result (Prop. 1) establishes a tight, data-adaptive upper bound for logit distance between two tokens as a function of their projected hidden-state distance within the top principal subspace:

Figure 1: Empirical validation -- tokens with similar hidden states (x-axis) have similar router logits (y-axis), tightly supporting Prop. 1.

When the difference in hidden states projects predominantly into the leading singular vectors of the data matrix, the logit (and thus expert usage) distance is sharply limited. Empirically, this yields a triangular pattern: if hidden states are close in the geometric sense, their expert assignments are strictly similar; for more distant vectors, expert assignments may diverge but can also coincide by chance alignment.

Furthermore, the introduction of an auxiliary load-balancing loss influences this process by actively suppressing the impact of directions shared across tokens (i.e., the dominant vectors in the hidden state covariance). The analytic result (Prop. 2) demonstrates that, under such regularization, the router becomes invariant to these shared directions, promoting diversity in expert usage. This theoretical result aligns with known phenomena in training collapse for small-batch scenarios and criteria for specialization robustness, offering a principled explanation for empirical findings on specialization collapse and the design of effective MoE pretraining.

Figure 2: Hidden-state correlation and its suppression by the router across layers under AdamW and auxiliary load balancing loss.

Figure 3: Analysis of a representative large MoE model, ERNIE-4.5-21B, in the context of router confidence.

Empirical Consequences: Specialization Without Domain Semantics

Extending the analysis to the sequence and dataset level, the direct correspondence between sequence-level embedding similarity and expert usage similarity persists across domains and model architectures.

Figure 4: Sequence-pair analysis demonstrates strong alignment between pooled hidden-state similarity and expert-usage similarity.

Crucially, the following empirical properties emerge and contradict the assumption of semantically interpretable expert specialization:

• Low Inter-Model Consistency: Different models answering the same question exhibit expert overlap ($\sim$60%) indistinguishable from that seen for different questions. Thus, specialization is not a robust function of semantic content across model instantiations.

  Figure 5: Expert overlap (Jaccard index) shows limited agreement across models, even for identical queries.

• Prompt-Rollout Dissociation: Expert usage during the prompt (prefilling) phase is highly correlated across semantically disparate inputs or prompt rephrasings, but diverges sharply during autoregressive rollout and is not predictive of rollout-phase specialization.

  Figure 6: Prompt-phase expert usage is similar even for disparate queries, but rollout-phase similarity decays unpredictably.

• Layer-Dependent Collapse: In deep transformer layers, hidden states—and consequently expert usage—collapse for unrelated sequences, a phenomenon accentuated during prefilling but reversed during rollout. Consequential expert capacity is only utilized post-prefill.

  Figure 7: Pruning experts at deeper layers has negligible effect on prompt NLL, but greatly increases NLL during generation.

  

  Figure 8: During the prefill phase, unrelated inputs yield identical per-expert activation, but generation quickly differentiates their activation profiles.

• Geometric Amplification by Duplication: Artificially duplicating inputs amplifies shared principal directions and further enforces router collapse, confirming geometric, not semantic, control.

  Figure 9: Duplication increases token and sequence hidden-state similarity, which induces increased expert-usage similarity.

Contextual and Layerwise Forces on Specialization

The above findings are interpreted in the broader context of representational geometry across the transformer stack. Early layers reflect token identity (thus, show context-independence), while deeper layers encode context and anticipated outputs, yielding layer-dependent shifts that decouple surface content from expert selection.

Divergence in expert usage under alternative optimizers, training objectives, and auxiliary regularization mechanisms is articulated visually and quantitatively in additional figures.

Figure 10: Optimizer choice modulates hidden-state norm and pairwise similarity, affecting the geometry and thus specialist patterns.

Figure 11: In dense models, increasing depth correlates with increased token similarity (collapse), except when architectural modifications (e.g., sandwich norm in Gemma3) disrupt this trend.

Figure 12: Quantitative comparison of bound tightness confirms projection-based bounds capture empirical behavior more precisely.

Implications and Prospects for MoE Research

This reframing of expert specialization has both practical and theoretical ramifications. On the practical side, traditional specialization metrics are insufficient proxies for domain expertise, and router assignment patterns cannot be reliably interpreted in terms of semantic axis or content domains. For model design and evaluation, this implies that architectural or regularization changes affecting representation geometry yield substantial, sometimes non-intuitive, effects on routing diversity and specialization collapse.

Theoretically, the results foreground the necessity of understanding high-dimensional geometry of LLM hidden states to make progress on MoE interpretability, a deeply challenging and unresolved area. Designing explicit objectives for semantic or functional specialization (e.g., explicit divergence objectives) may be essential if domain-aligned routing is desired rather than simply geometric partitioning.

Additionally, since expert usage patterns are deterministic functions of input hidden states, the results intersect with privacy and side-channel concerns: routing can leak more about query structure than model designers may intend.

Conclusion

Expert specialization in MoE LLMs is rigorously shown to be a geometric epiphenomenon of representation space rather than evidence of semantic or domain-level expertise. The mapping from inputs to experts is dictated by hidden-state similarity, modulated by auxiliary loss mechanisms and shaped by representation collapse across depth, context, and sequence. As such, claims about interpretability, efficiency, and domain alignment in MoEs must be carefully recast in terms of representational geometry, and future progress depends on advances in the mechanistic understanding of high-dimensional model spaces. Explicit specialization objectives or interpretability tools are essential for meaningful domain-driven MoE analysis.