Asymptotic Spectral Theory for Attention Matrices

• Asymptotic spectral theory for attention matrices is an analysis of eigenvalue distributions in self-attention mechanisms using high-dimensional random matrix theory and free probability.
• It reveals phenomena such as rank-one plus noise decomposition, soft-thresholding effects under regularization, and deviations from classic spectral laws.
• The insights guide the design and training of Transformer models by characterizing dynamical modes, transport properties, and the impact of inductive bias.

Asymptotic spectral theory for attention matrices studies the limiting behavior of the spectrum—eigenvalues and singular values—of matrices arising in self-attention mechanisms, particularly in regimes where data, model, and sequence dimensions scale to infinity in fixed proportions. This theory draws heavily on random matrix theory (RMT), spin-glass techniques, approximate message passing (AMP), free probability, and operator-theoretic frameworks. Its main objectives are to precisely characterize the spectral distributions of random and learned attention matrices, to understand the effects of inductive bias and regularization, and to analyze the dynamical or transport properties that govern information routing in neural architectures such as Transformers.

1. High-Dimensional Scaling Regimes and Attention Architectures

Spectral analysis in self-attention models focuses on regimes where the number of tokens $T$, embedding dimension $d$, attention width $m$, and dataset/sample size $n$ all diverge to infinity with fixed ratios such as $\kappa=m/d$ and $\alpha=n/d^2$ remaining $O(1)$. In this context, the archetypal object of study is the row-wise softmax matrix

$$A_W(x_{\text{in}}) = \mathrm{softmax}_\mathrm{row}\left( \frac{x_{\text{in}} W W^\top x_\mathrm{in}^\top - \mathrm{Tr}(WW^\top) I_T}{d\sqrt{m}} \right)$$

where $W\in\mathbb{R}^{d\times m}$ parameterizes both the query and key projections in a single-head, tied-attention model (Boncoraglio et al., 29 Sep 2025, Hayase et al., 8 Oct 2025).

The basic objects for spectral study are:

• The empirical spectral distribution (ESD) of $W$, $d$0, or their normalization.
• The limiting ESD of the attention matrix $d$1 at random initialization or after training.
• The degree-normalized, possibly causal-masked, transport operator $d$2 and associated symmetric/antisymmetric decompositions (Dahlem et al., 6 May 2026).

These settings support rigorous asymptotic analysis via free probability and Gaussian equivalence principles.

2. Asymptotic Spectral Distributions: Theory and Universality

Under random Gaussian initialization for weights and normalized inputs, the singular value spectrum of the attention matrix $d$3 is captured by an exact Gaussian equivalence principle: in the proportional limit $d$4 with $d$5, the squared singular values (after removing the Perron outlier at $d$6) converge to the empirical law of a linear Gaussian model,

$d$7

with $d$8 the normalized score matrix, $d$9 an independent Gaussian, and $m$0, $m$1 (Hayase et al., 8 Oct 2025). The limiting spectral density is characterized via its Stieltjes transform $m$2 satisfying the free additive convolution equation

$m$3

which deviates strictly from the Marchenko–Pastur law owing to the nontrivial structure induced by the softmax normalization even at random initialization.

A critical finding is the emergence of a macroscopic singular value $m$4 (the mean mode), while the remainder of the spectrum is of scale $m$5, forming a rank-one plus noise decomposition. For moderate softmax inverse temperature $m$6, Taylor expansion and concentration bounds validate the reduction to polynomial models, but for $m$7 at and beyond the critical threshold $m$8, higher-order corrections and distinctly non-Gaussian effects emerge (Hayase et al., 8 Oct 2025).

3. Spectral Effects of Regularization and Inductive Bias

When training the tied-attention model by empirical risk minimization with an explicit $m$9 (weight-decay) penalty on $n$0, the perceivable inductive bias at the spectral level is an implicit nuclear-norm regularization on the effective attention kernel $n$1. The asymptotic solution is exactly characterized by

$n$2

which, due to the variational form of the nuclear norm, enforces a soft-thresholding effect on the eigenvalues of $n$3:

• Any eigenvalue $n$4 below a threshold $n$5 is set to zero;
• Eigenvalues above threshold are shifted down by $n$6 (Boncoraglio et al., 29 Sep 2025).

The limiting spectral law of the learned weights is thus described as "soft-thresholded free convolution" of the ground-truth spectrum $n$7 with a semicircle law of variance $n$8: $n$9 Here, the presence of nonzero bulk, mass at zero, and possible outlier spikes closely mirrors empirical spectra observed in fully-trained Transformers.

4. Spectral Mode-Selection in Symmetric Attention Dynamics

For continuous-time idealized self-attention flows on the sphere under the symmetry condition $\kappa=m/d$0, the evolution admits a gradient-flow structure governed entirely by the spectrum of $\kappa=m/d$1 (Kuehn et al., 28 Apr 2026). The key findings are:

• Homogeneous consensus: If a single eigenvalue $\kappa=m/d$2 of $\kappa=m/d$3 strictly dominates in magnitude, the dynamics aligns all tokens along the corresponding eigendirection.
• Bipolar polarization: For $\kappa=m/d$4 negative-definite (all $\kappa=m/d$5), tokens polarize into groups aligning with the most negative eigenvector.
• Stability of pure-mode equilibria is determined by spectral gaps and, for sign-split states, by a $\kappa=m/d$6- and partition-ratio-dependent spectral window.

Spectral selection manifests globally—positive-dominant spectra yield global alignment, while negative-definite spectra drive nearly all initial configurations to maximal anti-alignment along the most negative mode. These insights define a mode-selection principle: long-time asymptotics of attention-induced interactions are controlled by extremal spectral properties.

5. Transport, Capacity, and Asymmetry in Degree-Normalized Attention

Beyond raw spectral content, operator-theoretic frameworks analyze the degree-normalized attention transport operator $\kappa=m/d$7 and its symmetric ($\kappa=m/d$8) and antisymmetric ($\kappa=m/d$9) decompositions: $\alpha=n/d^2$0 Spectral diagnostics based on $\alpha=n/d^2$1 (e.g., spectral gap, Cheeger conductance, Laplacian eigenvalues) rigorously cannot distinguish $\alpha=n/d^2$2 from its transpose—a phenomenon termed orientation-blindness (Dahlem et al., 6 May 2026). The unique control parameter for measuring directionality is the asymmetry coefficient

$\alpha=n/d^2$3

with $\alpha=n/d^2$4 only if $\alpha=n/d^2$5 is symmetric. Thus, spectral capacity (measured by Cheeger-type conductance or the second singular value) and directionality (measured by $\alpha=n/d^2$6) constitute two fundamentally orthogonal diagnostic axes.

The exact Cheeger landscape for canonical attention masks demonstrates that uniform causal attention imposes a universal lower bound $\alpha=n/d^2$7 on transport conductance, while windowed attention can approach bottleneck regime at $\alpha=n/d^2$8 rates. This dichotomy gives rise to two shape-distinct failure modes: bottleneck (low conductance, high localization) and diffuse (high conductance, weak localization).

6. Empirical Parallels and Interpretational Framework

Empirical transformer spectra show features precisely predicted by these asymptotic laws: a compact-supported spectral bulk governed by free convolution, outlier spikes correlated with true signal dimensions, and direct observability of mass at zero under strong regularization (Boncoraglio et al., 29 Sep 2025). In transport analysis, the orientation-blindness of symmetric spectral methods implies that identification of routing directionality and temporal causality depends entirely on antisymmetric operator content (Dahlem et al., 6 May 2026).

A falsifiable prediction is established for hallucination regimes: bottleneck-dominated evaluation benchmarks will yield low spectral conductance (low $\alpha=n/d^2$9), whereas diffuse-dominated benchmarks yield high conductance; directionality $O(1)$0 flags only temporal isolation or breakdown of standard causal routing.

7. Technical Instruments and Theoretical Underpinnings

The rigorous analysis of asymptotic attention spectra employs:

• Free additive convolution and the Pastur equation for limiting ESD computation.
• Gaussian min-max inequalities (Gordon's theorem), Gaussian equivalence, and polynomial linearization for nonlinearly transformed random matrices (Hayase et al., 8 Oct 2025, Boncoraglio et al., 29 Sep 2025).
• Approximate message passing (AMP) and spin-glass state evolution for precise solution characterization in ERM and nuclear-norm penalized objectives.
• Variational principles connecting $O(1)$1 regularization in factorizable parametrization to nuclear-norm objectives.

All results rely on core assumptions: rotational invariance or Gaussianity of inputs, replica symmetry for AMP convergence, and technical control of outlier eigenvalues and high-moment bounds for RMT methods.

---

Key references: "Inductive Bias and Spectral Properties of Single-Head Attention in High Dimensions" (Boncoraglio et al., 29 Sep 2025), "Gaussian Equivalence for Self-Attention: Asymptotic Spectral Analysis of Attention Matrix" (Hayase et al., 8 Oct 2025), "Spectral Selection in Symmetric Self-Attention Dynamics" (Kuehn et al., 28 Apr 2026), "Self-Attention as Transport: Limits of Symmetric Spectral Diagnostics" (Dahlem et al., 6 May 2026).