Mean Squared Overlap (MSO)

• Mean Squared Overlap (MSO) is a quantitative metric that measures the similarity between components using geometric and statistical methods in high-dimensional models.
• It employs rigorous definitions in both weight-space and activation-space to evaluate correlations and functional overlaps, crucial in MoE networks and spin glass systems.
• MSO provides actionable insights into model specialization and phase behavior, informing the design of neural architectures and the analysis of disordered systems.

The mean squared overlap (MSO) is a quantitative metric used to characterize the statistical or geometric similarity between components in high-dimensional models, serving as a key indicator of correlation structure, specialization, or phase behavior depending on context. MSO has been employed both in the analysis of mixture-of-experts (MoE) neural architectures—measuring diversity and functional overlap between experts—and in the mean field theory of spin glass models—acting as an order parameter reflecting phase organization and self-averaging. Its formalism, empirical behavior, and interpretive value are deeply tied to the class of system and methodological context in which it is deployed.

1. Formal Definitions of Mean Squared Overlap

Weight-Space MSO in MoE

For a MoE layer containing $N$ experts, each with weight matrix $W_i$, the weight-space MSO is computed by first flattening and $\ell_2$-normalizing each weight matrix:

$$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$

The mean squared overlap across all expert pairs is

$$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$

This metric measures geometric orthogonality in parameter space: lower values indicate more orthogonal (diverse) expert weights (Kim, 1 Jan 2026).

Activation-Space MSO in MoE

During inference, each input $x$ selects a subset $\mathcal S(x)$ of $k$ experts (typically $k=2$, under top-2 routing). For the normalized activations $h_i(x)$ from the selected experts, the functional overlap is quantified as

$W_i$0

This measures output alignment or specialization in activation space, evaluated on actual data (Kim, 1 Jan 2026).

MSO in Spin Glass Theory

In the Ghatak–Sherrington mean-field model for spin glasses, the overlap between two replicas $W_i$1 of size $W_i$2 is

$W_i$3

The mean squared overlap is then

$W_i$4

where the expectation is taken with respect to the quenched Gibbs measure and the limiting value is given by the square of the replica-symmetric overlap parameter (Sheng et al., 2023).

2. Methodologies for MSO Measurement

MoE Empirical Protocol

MSO was empirically measured in 130M-parameter NanoGPT-MoE models with 6 MoE layers, each layer containing 8 experts, using top-2 routing. For each regularization parameter $W_i$5 in $W_i$6, weight-space MSO was computed per layer (then averaged), while activation-space MSO was computed over 1K validation samples, considering only top-2 expert activations per input and unweighted by gating scores. Data splits included TinyStories, WikiText-103, and PTB across multiple random seeds. Orthogonality regularization was implemented as

$W_i$7

and added to the standard language modeling objective (Kim, 1 Jan 2026).

Spin Glass Analytical Approach

In the mean-field Ghatak–Sherrington model, the MSO was analyzed using the moment method and cavity approach to establish joint central limit theorems for overlap and self-overlap arrays in the high-temperature regime. Explicit expressions for finite-$W_i$8 corrections to $W_i$9 are given in terms of model-specific constants $\ell_2$0, $\ell_2$1, and $\ell_2$2. The regime of validity includes all $\ell_2$3, for fixed external and crystal fields (Sheng et al., 2023).

3. Key Empirical and Theoretical Results

MoE: Effect of Orthogonality Loss

Introduction of an explicit orthogonality loss in MoE layers produced a counterintuitive increase in weight-space MSO:

• Baseline ($\ell_2$4): $\ell_2$5
• $\ell_2$6: $\ell_2$7 (+39%)
• $\ell_2$8: $\ell_2$9 (+114%)
• Trend persists for larger $$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$0 (up to $$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$1 at $$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$2)

Activation-space MSO remained stable across the sweep ($$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$30.57–0.59), showing no substantial effect of regularization. The Pearson correlation between $$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$4 and $$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$5 was statistically insignificant ($$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$6, $$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$7) (Kim, 1 Jan 2026).

Summary Table: Weight vs. Activation MSO (TinyStories sweep)

$$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$8	Weight MSO	Activation MSO	Ratio (act/weight)
0	$$\tilde W_i = \frac{\mathrm{vec}(W_i)}{\|\mathrm{vec}(W_i)\|}.$$9	0.572	1053×
0.001	$$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$0	0.581	773×
0.01	$$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$1	0.577	496×
0.1	$$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$2	0.593	290×
0.2	$$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$3	0.564	203×

Spin Glass: High-Temperature Limit

In the Ghatak–Sherrington model, the high-temperature (replica-symmetric) mean squared overlap is $$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$4 with $$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$5 determined by fixed-point equations. The finite-size correction is

$$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$6

where the $$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$7 are closed-form functions of the model parameters (Sheng et al., 2023).

4. Interpretations and Theoretical Significance

MoE: Disconnection between Weight and Activation Diversity

Weight-space orthogonality constraints, as implemented via a Frobenius-trace penalty, act only on the leading trace $$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$8, not the full matrix product, creating a mismatch between the geometric regularization objective and actual weight configuration. This perturbation can cause weights to drift into higher-magnitude directions, increasing pairwise inner products—and thus MSO—paradoxically under explicit regularization. Nonlinearities (e.g., SiLU) and LayerNorm further disrupt any correspondence between weight and activation overlaps; even if $$\mathrm{MSO}_\text{weight} = \frac{2}{N(N-1)} \sum_{1 \le i < j \le N} |\langle \tilde W_i, \tilde W_j\rangle|^2.$$9, the post-activation outputs $x$0 can remain highly aligned on natural data. The input distribution in natural language tasks, living on a low-rank manifold, can also force different experts to exhibit similar activations (Kim, 1 Jan 2026).

Spin Glasses: MSO as an Order Parameter

In spin glass systems, MSO encapsulates the self-averaging and long-range order of the system, stabilizing to the square of the replica-symmetric overlap in the thermodynamic limit. The value of $x$1 tracks the system's phase and responds predictably to external and crystal fields. A higher crystal-field parameter $x$2 tends to reduce MSO, while a stronger external field $x$3 increases it. The finite-size corrections and covariance structure from the central limit theorem provide a rigorous quantitative framework for fluctuations and critical behavior near the high-temperature limit (Sheng et al., 2023).

5. Implications for Model Specialization and Performance

MoE: Limitations and Design Recommendations

Empirically, weight-space MSO is not a reliable proxy for expert specialization: orthogonality regularization neither achieves its geometric goal nor reliably improves downstream performance. For instance, on TinyStories, orthogonality loss marginally worsens perplexity (+0.9%, $x$4), while WikiText-103 shows a marginal improvement (–0.9%, $x$5), and PTB exhibits high variance with no consistent effect. The lack of correlation between weight and activation MSO suggests that MoE diversity must be promoted at the functional (activation) level or via routing and diversity objectives acting on actual expert outputs (Kim, 1 Jan 2026).

Spin Glass Theory: Control via Field Parameters

In the statistical mechanics of disordered systems, MSO provides a direct readout of the overlap structure and is modulated via field parameters $x$6 and $x$7. This supports precise analytical investigation into the relationship between microscopic correlations and macroscopic phases in high-dimensional, interacting systems (Sheng et al., 2023).

6. Directions for Future Investigation

A plausible implication emerges that, in neural MoE systems, activation-space diversity metrics (such as $x$8 or analogous losses) and routing-aware objectives may provide more direct and robust mechanisms for specialization and performance gains, in contrast to weight-space geometric regularization. In spin glass models, further refinement of MSO analysis beyond the high-temperature regime—especially in context of phase transitions or competing disorder—remains a substantive avenue for theoretical exploration (Kim, 1 Jan 2026, Sheng et al., 2023).