Spectral Superposition: A Theory of Feature Geometry

Abstract: Neural networks represent more features than they have dimensions via superposition, forcing features to share representational space. Current methods decompose activations into sparse linear features but discard geometric structure. We develop a theory for studying the geometric structre of features by analyzing the spectra (eigenvalues, eigenspaces, etc.) of weight derived matrices. In particular, we introduce the frame operator $F = WW^\top$, which gives us a spectral measure that describes how each feature allocates norm across eigenspaces. While previous tools could describe the pairwise interactions between features, spectral methods capture the global geometry (``how do all features interact?''). In toy models of superposition, we use this theory to prove that capacity saturation forces spectral localization: features collapse onto single eigenspaces, organize into tight frames, and admit discrete classification via association schemes, classifying all geometries from prior work (simplices, polygons, antiprisms). The spectral measure formalism applies to arbitrary weight matrices, enabling diagnosis of feature localization beyond toy settings. These results point toward a broader program: applying operator theory to interpretability.

• The paper presents a formal spectral theory that explains how superposed features interact through shared eigenspaces in neural networks.
• It leverages the frame operator and spectral measures to characterize feature localization, capacity saturation, and interference.
• Empirical validation via toy models shows that spectral localization leads to fixed tight-frame configurations and predictable superposition geometries.

Spectral Superposition: A Theory of Feature Geometry

Motivation and Theoretical Framework

Neural networks routinely represent more features than the dimension of their activation spaces, a phenomenon called superposition. This inherently induces interference between features, whereby individual representations cannot be uniquely assigned to independent axes—multiple, sometimes structured, features must share dimensions. Conventional interpretability methods, including sparse autoencoders (SAEs), aim to recover an orthogonal or sparse basis to render features interpretable, but these methods generally discard the geometric structure intrinsic to the learned superpositions. Empirical findings indicate that this geometric structure captures meaningful relationships among features, such as cyclicity or hierarchy, which reflect inductive biases or domain regularities encoded by the models.

This work introduces a formal, basis-invariant theory for feature geometry in superposition, leveraging spectral analysis of weight-derived operators. The central technical object is the frame operator $F = WW^\top$, derived from the weight matrix $W$. Spectral analysis of $F$—its eigenspaces and eigenvalues—yields a compact, invariant description of how features organize, interact, and allocate representational capacity through superposition. In this framework, interference between features only occurs within shared eigenspaces; features localized to orthogonal eigenspaces are non-interfering. This provides a principled distinction between incidental and structural superposition, capturing not only pairwise but also global patterns of feature geometry.

Figure 1: The games-of-chance example—rock, paper, scissors as an equilateral triangle in the $xy$-plane; heads/tails as an antipodal pair along $z$—illustrates symmetry and subspace sharing as encoded by feature clusters.

The spectral formalism is validated both theoretically and empirically in the canonical Toy Models of Superposition (TMS) framework, demonstrating that spectral localization and capacity saturation drive the emergence of highly structured, classifiable geometries.

Spectral Analysis and Feature Geometry

The paper establishes that geometric structure among features is best captured via spectral methods, rather than the arbitrary coordinate bases produced by most feature extraction algorithms. Key results include:

• Activation-space invariance: The Gram matrix $M = W^\top W$ is sensitive to column permutations (feature relabeling), whereas the frame operator $F = WW^\top$ is invariant under such transformations. Thus, spectral analysis of $F$ yields basis-independent structure.
• Group-theoretic encoding of symmetry: Feature clusters with intrinsic symmetry (e.g., cyclicity in rock-paper-scissors) manifest as association schemes—a class of algebraic structures whose spectral decomposition directly encodes invariant subspaces and their geometrical relationships.

This enables reconstruction of latent feature geometry from spectral signatures alone, even when features are indexed or rotated arbitrarily. For the illustrative toy example, the RPS cluster forms a two-dimensional subspace (triangle in the plane), while the HT pair is localized along an orthogonal axis.

Figure 2: The elements of the $D_3$ dihedral group, underlying the permutation symmetries of the triangle (RPS feature cluster).

Figure 3: Decomposition of association schemes into orthogonal strata, each corresponding to an eigenspace relevant for feature allocation and interference characterization.

Spectral Measures, Localization, and Capacity

The spectral measure $\mu_i$ associated to each feature captures the fraction of its norm that projects onto each eigenspace of $F$, providing a probability distribution over eigenmodes. This measure describes how “localized” or “delocalized” a feature is in activation space:

• Localized features: $\mu_i$ collapses to a Dirac delta at one eigenvalue; the feature occupies a single eigenspace.
• Delocalized features: $\mu_i$ has broad support; the feature participates in several interaction modes.

The fractional dimensionality $D_i = \frac{\|W_i\|^4}{(W_i^\top F W_i)}$ formalizes the degree of superposition. When all spectral measures are highly localized and the sum $\sum_i D_i$ equals the model rank (the number of hidden dimensions), the model is said to have saturated its capacity.

This is empirically seen in high-throughput TMS experiments, where the aggregate feature dimensionality approaches the dimension bound across thousands of runs, confirming that neural networks allocate their entire representational budget, driving features into the minimal number of eigenspaces necessary for the task.

Figure 4: Capacity saturation across 3,200 runs—total feature fractional dimensionality approaches the hidden dimension across sparsity and hidden size regimes.

Classification of Superposition Geometries

Key theoretical results include the Spectral Localization Theorem: At capacity saturation, every feature is an eigenvector of $F$, i.e., spectral measure is perfectly localized. Features cluster according to the eigenvalues of $F$, and each cluster forms a tight frame—a set of vectors spanning an eigenspace with maximal symmetry and minimal redundancy. This enables discrete classification of cluster geometry via association schemes, recovering known geometries (simplex, polygon, antiprism) and predicting new structured configurations.

Further, the Projective Linearity Corollary states that fractional dimensionality $D_i$ is linearly determined by the feature norm with a slope set by the reciprocal of the corresponding eigenvalue. This provides a direct observable diagnostic for spectral localization.

The paper also connects these static results to training dynamics: gradient descent can be interpreted in terms of mass transport between spectral modes and shows that tight-frame configurations are fixed points, explaining the empirical stability of observed superposition geometries.

Practical and Theoretical Implications

From a practical perspective, this framework moves interpretability beyond the identification of sparse features to a characterization of how features interact globally—diagnosing interference, redundancy, and robustness across all features, not just those that can be rendered monosemantic. This has direct consequences for mechanistic interpretability, model alignment, and intervention design: interventions that do not respect the underlying spectral geometry risk unintended cross-feature effects.

Theoretically, the spectral operator perspective suggests a unification of representation analysis in neural networks with methods from operator theory, random matrix theory, and group representation theory. The analogy to quantum mechanical spectral decompositions (states as eigenvectors of observables) is precise, not poetic: algebraic and spectral objects determine the structure and stability of learned representations.

This work also contextualizes the observed “heavy-tailed” spectral statistics in trained weight matrices—central to the heavy-tailed self-regularization literature—as manifestations of capacity allocation and localization phenomena.

Future Directions

Open directions include extending spectral measure diagnostics to deep and nonlinear architectures, establishing the generality of spectral localization beyond toy models, and developing new operator-theoretic tools for global interpretability. Further, this line of research could inform regularization and model design strategies that nudge learning dynamics toward preferred geometric configurations or reduce undesired interference between task-relevant features.

Conclusion

This paper advances a rigorous, spectral approach to the study of feature geometry in neural network superposition. By analyzing weight-derived operators rather than sparse bases, it provides basis-invariant diagnostics for feature allocation, interaction, and interference, offering a unified theoretical toolkit for classifying and understanding the geometric organization of representations. These results have both immediate experimental validation in canonical toy settings and broad potential impact on methodological advances for interpretability and model design.