Geometric Regularization in Mixture-of-Experts: The Disconnect Between Weights and Activations

Abstract: Mixture-of-Experts (MoE) models achieve efficiency through sparse activation, but the role of geometric regularization in expert specialization remains unclear. We apply orthogonality loss to enforce expert diversity and find it fails on multiple fronts: it does not reduce weight-space overlap (MSO actually increases by up to 114%), activation-space overlap remains high (~0.6) regardless of regularization, and effects on performance are inconsistent -- marginal improvement on WikiText-103 (-0.9%), slight degradation on TinyStories (+0.9%), and highly variable results on PTB (std > 1.0). Our analysis across 7 regularization strengths reveals no significant correlation (r = -0.293, p = 0.523) between weight and activation orthogonality. These findings demonstrate that weight-space regularization neither achieves its geometric goal nor reliably improves performance, making it unsuitable for MoE diversity.

• The paper demonstrates that while weight orthogonality increases weight MSO, it does not affect activation MSO, undermining its assumed benefits in MoE specialization.
• It employs an orthogonality loss on expert weights and uses Mean Squared Overlap metrics on both weights and activations to highlight the ineffective influence of geometric regularization.
• The findings motivate a shift from parameter-based to activation-based regularization strategies, suggesting methods like gradient-space orthogonality and contrastive objectives.

Geometric Regularization in Mixture-of-Experts: Decoupling Weights and Activations

Introduction

This paper rigorously investigates the underlying assumption in Mixture-of-Experts (MoE) architectures that geometric regularization—explicitly encouraging weight orthogonality between experts—should enhance expert diversity and model performance. Standard rationales for this hypothesis are grounded in linear algebraic principles: orthogonal weight vectors should pass disjoint representations, thereby minimizing expert interference. Through theoretical and empirical analysis, the authors demonstrate a significant disconnect between weight-space orthogonality and functional diversity as measured in activation-space, challenging prevailing assumptions regarding MoE diversity regularization.

Methodology

The study applies orthogonality loss on the up-projection weights of each expert, formulated as the sum of squared pairwise inner products between normalized, flattened expert weight matrices. The primary quantitative measure of geometric diversity employed is the Mean Squared Overlap (MSO), computed for both weights and activations. The orthogonality loss is integrated into the language modeling objective with varied strengths ($\lambda$), and MSO is monitored for both parameter space (weights) and functional space (activations) across multiple datasets. For activation MSO, the authors examine pairwise cosine similarities between the outputs of the top-$k$ selected experts for a given input.

Experiments utilize NanoGPT-MoE (130M parameters, 8 experts, 6 layers, top-2 routing) trained on TinyStories for 10K iterations, with additional evaluations on WikiText-103 and Penn Treebank (PTB) for cross-dataset validation. Multiple seeds are employed to assess sample variance and statisical reliability.

Empirical Findings

A principal finding is that orthogonality regularization does not decrease weight MSO—contrary to its objective; instead, it increases it by up to 114%. Baseline training already produces near-orthogonal weights ($\sim$10^{-4}$$MSO), suggesting implicit geometric regularization via optimization dynamics. Furthermore, activation MSO remains approximately$$0.57$$regardless of the strength of geometric regularization. <img src="https://emergentmind-storage-cdn-c7atfsgud9cecchk.z01.azurefd.net/paper-images/2601-00457/weight_activation_gap.png" alt="Figure 1" title="" class="markdown-image" loading="lazy"> <p class="figure-caption">Figure 1: Weight-Activation Gap. Weight MSO responds to regularization; activation MSO does not. No significant correlation ($$r = -0.293$,$p = 0.523$$).</p></p> <p>Statistical analysis reveals no significant correlation between weight and activation MSO across a sweep of seven regularization strengths ($$r = -0.293$,$p = 0.523$$). The ratio of activation to weight MSO is dramatic—activation MSO can be$$10^3$$times higher than weight MSO, and this factor remains largely unaltered by increased regularization. Perplexity metrics do not show consistent improvements: effects are marginal and dataset-dependent (+0.9 <p>Notably, the variance in perplexity increases under strong regularization (standard deviation from 0.08 to 0.32 on TinyStories), indicating destabilization of training dynamics with little benefit.</p> <h2 class='paper-heading' id='mechanisms-underlying-the-weight-activation-gap'>Mechanisms Underlying the Weight-Activation Gap</h2> <p>The authors hypothesize that the gap between weight and activation geometric properties is driven by the presence of non-linearities (SiLU activations, LayerNorm) and the structure of input distributions. Even with orthogonal weight matrices, non-linear transformations and normalization can compress or distort angular separations between outputs, leading to persistent activation overlap.</p> <p>Analysis at the level of quadratic forms reveals Frobenius orthogonality on weights guarantees only that the trace of$$W_1^T W_2$ is zero, but does not enforce entrywise independence or output orthogonality in the space of typical inputs. Furthermore, correlated input projections across experts, induced by natural language input distribution, can yield strongly overlapping activations despite orthogonal parameterization.

Cross-Dataset Validation and the Role of Dataset Scale

Regularization effects are not consistent across datasets. On larger datasets such as WikiText-103, geometric regularization gives marginal, statistically insignificant improvement; on smaller datasets (e.g., PTB), results are seed-sensitive and show large variance, indicating a lack of reliability and reproducibility. Optimization trajectories may essentially dominate the realized diversity patterns, making weight orthogonality a poor universal target.

Implications and Future Directions

This investigation demonstrates that weight-based geometric regularization, commonly assumed to promote expert differentiation in MoE models, is an unreliable and ineffective strategy. The functional diversity required for MoE specialization is not captured by parameter-space metrics such as weight MSO. Activation MSO is invariant to weight-space regularization in the presence of modern network nonlinearities and input statistics, casting doubt on analogies to phenomena like Neural Collapse.

The paper provides several actionable prescriptions for future research:

• Weight-space regularization is unreliable for MoE expert diversity.
• Activation-space regularization, directly penalizing activation MSO, may provide a more grounded functional diversity metric.
• Methods such as gradient-space orthogonality, routing diversity losses, or contrastive objectives on activations should be considered.
• Dataset scale and architecture details interact nontrivially with regularization strategies.

Conclusion

The analysis persuasively refutes the utility of geometric regularization via weight orthogonality in MoE models, revealing a substantial disconnect between parameter diversity and functional specialization. The empirical and theoretical evidence highlights that activation overlap is mostly invariant to weight geometry under standard MoE settings, rendering weight-based regularization strategies ineffective for ensuring expert functional diversity or improving perplexity. This directs the field toward functional, rather than parametric, regularization strategies for advancing MoE model efficacy.