Spectrum-Adaptive Generalization Bounds for Trained Deep Transformers

Abstract: Understanding why trained Transformers generalize well is a fundamental problem in modern machine learning theory, and complexity-based generalization bounds provide a principled way to study this question. While existing norm-based bounds for Transformers remove the explicit polynomial dependence on the hidden dimension, they typically impose fixed norm constraints specified a priori and can exhibit unfavorable exponential dependence on depth. In this paper, we derive spectrum-adaptive post hoc generalization bounds for multi-layer Transformers. Under layerwise spectral norm control, the bounds are expressed in terms of layerwise Schatten quantities of the query-key, value, and feedforward weight matrices. Since the Schatten indices need not be fixed a priori and can instead be selected after training, separately for each matrix type and layer, the bounds adaptively trade off spectral complexity against the dimension- and depth-dependent factors according to the learned singular-value profiles. Empirical comparisons of BERT-adapted proxies for the leading complexity factors suggest that the proxies induced by our bounds grow more slowly with depth and hidden dimension than the corresponding norm-based proxies. Overall, our results provide a complexity-based perspective on how the spectral structure of trained Transformers is reflected in generalization analyses.

• The paper presents post hoc generalization bounds that adaptively interpolate between rank- and Frobenius-norm regimes by selecting optimal Schatten indices for each weight matrix.
• It introduces a parametric interpolation for covering numbers that yields tighter depth and width scaling compared to traditional norm-based generalization measures.
• Empirical results on Transformer checkpoints demonstrate that leveraging spectral structures leads to effective model compression and improved diagnostic assessments.

Spectrum-Adaptive Generalization for Deep Transformers

Motivation and Context

The generalization phenomenon in overparameterized deep Transformers remains an unresolved theoretical question, paralleling the empirical success of these architectures in NLP, vision, and multimodal domains. Standard complexity-based generalization bounds, particularly norm-based variants, provide insights but impose constraining assumptions—such as a priori fixed norm constraints and often carry exponential dependence on depth via layerwise propagation factors. Moreover, these approaches fail to exploit the heterogeneous, layer- and matrix-type-specific spectral structure observed in trained Transformers. Contemporary studies (e.g., weight compression analyses and structured pruning) underscore significant variation in singular-value decay profiles across subcomponents of the architecture.

Spectrum-Adaptive, Post Hoc Bounds: Conceptual Overview

This work presents post hoc generalization bounds for multi-layer Transformers that are spectrum-adaptive in the sense of using the layer- and matrix-specific spectral structure of trained weights as complexity controls. Central to this approach is reliance on Schatten quantities—$\|W\|_{s,p}$, $p \in [0,2]$—of the query-key, value, and feedforward matrices, with the nuance that the Schatten index $p$ is chosen after training and may be selected independently for each weight matrix. The bounds thus interpolate smoothly between rank-based ($p\to0$) and Frobenius-norm-based ($p=2$) regimes, adapting to the singular value decay profile of each transformation.

The main results establish that, for a broad class of Lipschitz losses and under layerwise spectral norm constraints, with high probability the uniform generalization gap satisfies: $$\sup_{f \in \mathcal{F}_{\text{out}},\, p \in [0,2]^{3L}} \text{GAP}(f) \lesssim \inf_{p} \left\{ \Bigg(\sum_{l=1}^L \sum_{* \in \{\text{QK}, \text{V},\text{M}\}} \frac{\|W_*^{(l)}\|_{s,p_*^{(l)}}^{p_*^{(l)}+2} C^{p_*^{(l)}+2} L^{p_*^{(l)}+2} N^{p_*^{(l)}+2}}{n}\Bigg)^{1/2} +\dots \right\}$$ The infimum over $p$ is executed post hoc, meaning for each model instance the index is tuned to minimize the complexity expression. This adaptivity aligns the theory with empirical reality—spectrally compressible or low-rank weights yield rank-like control (smaller propagation and polynomial factors in $L$ and $N$); “unstructured” weights revert to norm-type scaling.

Theoretical Innovations

Parametric Interpolation for Matrix Classes

The key technical foundation is a parametric interpolation for covering numbers of matrix-valued function classes under simultaneous spectral norm and Schatten-$p$ constraints. Using a singular value threshold decomposition, the covering entropy splits into low-rank and Frobenius-norm components, and optimizing the interpolation threshold yields the exponent structure of the generalization bound.

Multi-layer Composition

The analysis extends to deep compositional settings relevant for Transformers, handling the interplay of softmax attention, activation, and normalization steps. The covering argument is carefully propagated through $p \in [0,2]$0 layers, with complexity scaling controlled by both the spectral structure and the composition's Lipschitz constants.

Post Hoc Selection and Adaptive Bounds

The method allows the post hoc selection of the Schatten indices, employing a discretization and union bound argument to ensure that, after observing trained weights, one can select the $p \in [0,2]$1 for each matrix to minimize the bound. The result holds uniformly over all possible index assignments, up to a logarithmic penalty depending on $p \in [0,2]$2 and $p \in [0,2]$3.

Numerical and Empirical Results

Numerical proxies for the leading complexity factors, computed for BERT Miniatures checkpoints, demonstrate that the spectrum-adaptive quantities grow substantially more slowly with both depth and hidden dimension compared to norm-based proxies from prior work (Edelman et al. 2022). Empirically, the proxies’ minimization is achieved at $p \in [0,2]$4 (rank), confirming persistent low-rank structure in key matrices, while the slow growth aligns with improved theoretical depth and width scaling.

Contradictory to traditional assumptions, the optimal complexity control is not provided by a single, fixed norm constraint but by dynamically balancing spectral complexity, hidden dimension, and layerwise propagation, chosen after observing each trained layer’s singular value spectrum.

Implications and Directions

Theoretical Implications

• Tighter Depth and Width Dependence: The bounds reduce polynomial and propagation factor scaling relative to prior norm-based analyses, especially in regimes where weight matrices exhibit rapid spectral decay or effective low-rankness.
• Spectral Adaptivity: By not requiring fixed indices or universal norm radii, the theory carefully matches observed multi-layer spectral heterogeneity.
• Sharp Interpolation: The framework rigorously interpolates between norm-type and rank-type bounds, encompassing architectural and data-driven regimes.

Practical Implications

• Model Selection and Analysis: Post hoc complexity evaluation provides practitioners with a diagnostic tool to assess trained models’ generalization capacity based on empirical spectral structure.
• Compression and Pruning: The insensitivity of spectrum-adaptive bounds to hidden dimension and depth in compressible matrices provides a rationale for aggressive structural compression and pruning in Transformer applications.

Limitations and Future Directions

• The current analysis is agnostic to the optimization dynamic (e.g., SGD trajectory, implicit regularization). Incorporating optimization-dependent complexity control (e.g., path-norms, margin-based terms) may elucidate additional mechanisms.
• The treatment is in terms of spectral measures of weights; incorporating data-dependent structures in activations and representations could further sharpen the bounds.
• The methodology readily extends to combine with offset Rademacher complexity and alternative capacity measures, suggesting new avenues for analyzing generalization in large-scale LLMs and structured architectures.

Conclusion

This work provides the first spectrum-adaptive post hoc generalization bounds for deep Transformers, demonstrating how learned, layerwise spectral profiles yield substantially improved complexity scaling relative to fixed-norm approaches. The results not only advance the theoretical understanding of generalization in contemporary deep architectures but also have concrete implications for model analysis, selection, and compression. The adaptive, post hoc selection mechanism reorients the analytical focus from predetermined architectural assumptions to data-driven, learned properties, constituting a significant conceptual shift in generalization theory for deep learning.