Spectrum-Adaptive Generalization Bounds

• The paper demonstrates that post-training generalization bounds adapt via layerwise spectral measures (using Schatten norms) to capture effective model complexity.
• It leverages covering-number techniques and Rademacher complexity through matrix interpolation, balancing rank-like and Frobenius-like descriptions.
• It also addresses invariance issues in ReLU networks by proposing lifted representations to ensure robust, functionally-aligned complexity estimates.

Spectrum-adaptive post hoc generalization bounds are generalization guarantees evaluated after training in which the complexity term adapts to the learned spectral structure of the model—typically singular values through rank, Frobenius, or intermediate Schatten quantities—rather than to a single norm chosen a priori. In the cited literature, the most explicit realization of this idea is a covering-number and Rademacher-complexity theory for trained deep Transformers (Sakai et al., 8 May 2026). Surrounding work supplies broader post hoc frameworks—disintegrated PAC-Bayes, information-density bounds, online-to-PAC conversions, and algorithm-dependent Rademacher complexity—that are not themselves spectral but define technical templates into which spectral complexity can be inserted (Viallard et al., 2024, Hellström et al., 2020, Lugosi et al., 2023, Sachs et al., 2023). A recurrent structural issue is that spectrum-sensitive quantities computed directly in parameter space can fail to reflect functional complexity when neural-network symmetries are ignored, especially ReLU rescaling invariance (Rouchouse et al., 30 Sep 2025).

1. Conceptual scope and terminology

In the post-training generalization literature, “post hoc” refers to guarantees that are evaluated after training on the realized predictor, posterior, or algorithm output. The dependence is therefore on trained weights, realized posterior distributions, realized hypothesis classes, or realized information quantities, rather than on a fixed hypothesis class specified before training. Spectrum adaptivity adds a second layer: the complexity description itself is chosen after training so that it reflects the observed singular-value profile of the learned matrices.

The direct spectral formulation in the cited literature uses layerwise Schatten quantities. For a matrix $W$ and $p\in(0,2]$,

$$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$

so $p=0$ yields rank, $p=2$ yields Frobenius structure through $\|W\|_{S,2}^2=\|W\|_F^2$, and $0Sakai et al., 8 May 2026). The post hoc aspect is that the admissible indices $p$ are selected after training, separately by layer and matrix type, under a uniform high-probability event (Sakai et al., 8 May 2026).

A separate body of work uses “post hoc generalization” in the adaptive-data-analysis sense. There, an algorithm $M$ satisfies $(\varepsilon,\delta)$-post hoc generalization if, for every distribution $p\in(0,2]$0 and every analyst $p\in(0,2]$1 that chooses a bounded query $p\in(0,2]$2 after seeing $p\in(0,2]$3, one has

$p\in(0,2]$4

with $p\in(0,2]$5 (Nissim et al., 2018). This is a different notion from post-training complexity control for a fixed trained network. The distinction matters because lower bounds and non-composition phenomena in adaptive data analysis do not automatically transfer to post-training spectrum-sensitive bounds, although they do delimit what is possible under stronger worst-case quantifiers (Nissim et al., 2018).

2. Direct spectrum-adaptive theorems for trained Transformers

The most direct theorem of this type in the cited set studies a simplified multi-layer Transformer with input $p\in(0,2]$6, single-head attention

$p\in(0,2]$7

a normalized block

$p\in(0,2]$8

and scalar readout

$p\in(0,2]$9

under layerwise spectral norm control $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$0 and $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$1 (Sakai et al., 8 May 2026).

The simplified main theorem states that, if $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$2 and every nonzero layer matrix satisfies

$$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$3

then with probability at least $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$4,

$$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$5

$$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$6

simultaneously for all admissible trained Transformers (Sakai et al., 8 May 2026). The infimum is over $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$7, so each layer $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$8 and each matrix type $$\|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W),$$9 may choose its own Schatten index after training.

The fixed-index precursor theorem holds on classes $p=0$0 with prescribed Schatten radii $p=0$1, while a common-$p=0$2 theorem improves depth allocation when all matrices share a single index $p=0$3 (Sakai et al., 8 May 2026). The theory is therefore genuinely post hoc: the high-probability event is uniform over all admissible Schatten regimes, and only after observing the trained weights does one select the indices that minimize the bound.

The leading term interpolates between rank-like and Frobenius-like descriptions. When $p=0$4, the dependence is rank-based; when $p=0$5, it becomes Frobenius-based; intermediate $p=0$6 values trade spectral sparsity against the architecture-dependent factors $p=0$7 and $p=0$8 (Sakai et al., 8 May 2026). In matched comparisons reported in the paper, the resulting leading factor improves on prior norm-based Transformer bounds of Edelman et al. (2022) and Trauger and Tewari (2024), which are expressed through fixed $p=0$9- or $p=2$0-norm constraints and can scale like $p=2$1 (Sakai et al., 8 May 2026).

3. Technical mechanisms of spectrum adaptivity

The Transformer bounds are covering-number results converted to generalization bounds by a Dudley-type entropy integral and a Rademacher complexity bound for Lipschitz losses (Sakai et al., 8 May 2026). They are therefore neither PAC-Bayes nor margin bounds. The central mechanism is a matrix interpolation argument for linear maps $p=2$2 under simultaneous spectral norm and Schatten control.

The proof decomposes a matrix as

$p=2$3

using a singular-value threshold $p=2$4. If $p=2$5, then

$p=2$6

Thus $p=2$7 is treated as a low-rank, spectral-norm-bounded component, while $p=2$8 is treated as a Frobenius-bounded tail; optimizing $p=2$9 yields the characteristic exponent $\|W\|_{S,2}^2=\|W\|_F^2$0 that appears in the final complexity term (Sakai et al., 8 May 2026).

For the Transformer architecture, the covering argument is applied separately to the query-key, value, and feedforward matrices. The proof uses that rowwise normalization $\|W\|_{S,2}^2=\|W\|_F^2$1 is $\|W\|_{S,2}^2=\|W\|_F^2$2-Lipschitz in $\|W\|_{S,2}^2=\|W\|_F^2$3, and that softmax is $\|W\|_{S,2}^2=\|W\|_F^2$4-Lipschitz from $\|W\|_{S,2}^2=\|W\|_F^2$5 to $\|W\|_{S,2}^2=\|W\|_F^2$6 (Sakai et al., 8 May 2026). Layerwise errors then accumulate through a propagation factor

$\|W\|_{S,2}^2=\|W\|_F^2$7

so the contribution of layer $\|W\|_{S,2}^2=\|W\|_F^2$8 is multiplied by the Lipschitz growth of all subsequent layers (Sakai et al., 8 May 2026).

The post hoc uniformity over all $\|W\|_{S,2}^2=\|W\|_F^2$9 is obtained by discretizing $0

$0

performing dyadic peeling over realized Schatten radii, and rounding arbitrary continuous indices upward to the grid (Sakai et al., 8 May 2026). The appendix’s exact theorem packages the adaptive term into $0Sakai et al., 8 May 2026).

The empirical evidence in that work is proxy-based rather than an exact theorem evaluation. Using BERT Miniatures checkpoints with depths $0

4. Invariance, quotient geometry, and functional alignment

Spectrum-sensitive post hoc bounds for neural networks face a foundational obstacle absent from the Transformer results above: in ReLU networks, raw parameter-space quantities can be arbitrarily changed by function-preserving rescalings. For a ReLU network on a DAG $0

$p$0

and positive homogeneity gives

$p$1

for all $p$2 (Rouchouse et al., 30 Sep 2025).

The cited ReLU analysis shows on a one-hidden-neuron example,

$p$3

that a standard PAC-Bayes complexity can diverge under rescaling even though the predictor is unchanged. With

$p$4

the KL behaves as

$p$5

(Rouchouse et al., 30 Sep 2025). The paper explicitly notes that this pathology is relevant not only to PAC-Bayes but also to Euclidean norms, layerwise norms, path norms, spectral norms, products of spectral norms, margins normalized by norms, perturbation sensitivity, sharpness, local curvature, and Hessian traces or eigenvalues when computed in raw coordinates (Rouchouse et al., 30 Sep 2025).

The proposed remedy is a lifted representation $p$6 such that

$p$7

for some measurable $p$8, and PAC-Bayes is applied to the pushforwards $p$9 and $M$0. For ReLU networks, the paper highlights the rescaling-invariant path+sign lift

$M$1

The lifted PAC-Bayes inequality replaces $M$2 by

$M$3

and the paper proves the comparison chain

$M$4

(Rouchouse et al., 30 Sep 2025).

The direct theorem there is KL/PAC-Bayes-centric rather than spectrum-centric, but its relevance to spectrum-adaptive bounds is structural. A plausible implication is that any spectrum-adaptive post hoc bound for ReLU networks should either be formulated in a rescaling-invariant lifted or quotient representation, or explicitly optimized over the rescaling orbit, because layerwise singular values can change under hidden-unit rescaling without changing the realized function (Rouchouse et al., 30 Sep 2025). The same paper reports that deterministic rescaling optimization typically reduces the KL by about a factor of 4 and the final PAC-Bayes bound by about a factor of 2, and in some cases turns a vacuous bound into a non-vacuous one (Rouchouse et al., 30 Sep 2025).

5. General post hoc templates compatible with spectral complexity

Several cited frameworks are not spectrum-adaptive by themselves but are broad enough to host spectral complexity measures.

The most explicit PAC-Bayes template is the disintegrated Gibbs framework. For any measurable score $M$5, define a Gibbs posterior

$M$6

Then, with probability at least $M$7 over

$M$8

the paper proves

$M$9

(Viallard et al., 2024). The theorem is hypothesis-level rather than posterior-expectation-level. Because $(\varepsilon,\delta)$0 is user-chosen and need only be measurable with a well-defined Gibbs density, the framework is structurally broad enough to accommodate spectral penalties. The cited summary is explicit that the paper does not derive a spectrum-specific theorem, but that it provides a direct route to post hoc, hypothesis-dependent PAC-Bayes bounds with user-chosen complexity measures (Viallard et al., 2024).

A second template is algorithm-dependent Rademacher complexity. Given a deterministic algorithm $(\varepsilon,\delta)$1, two ghost halves $(\varepsilon,\delta)$2, and mixed samples $(\varepsilon,\delta)$3, define

$(\varepsilon,\delta)$4

The paper proves

$(\varepsilon,\delta)$5

and, under boundedness assumptions, a corresponding high-probability bound through the essential supremum of $(\varepsilon,\delta)$6 (Sachs et al., 2023). Since $(\varepsilon,\delta)$7 is controlled by covering numbers and finite Minkowski dimension, a plausible implication is that spectral control of the covering entropy of $(\varepsilon,\delta)$8 would immediately yield a spectrum-adaptive post hoc bound (Sachs et al., 2023).

Two further frameworks are post hoc and data-dependent but not spectral in the cited papers. Information-density bounds yield single-draw guarantees of the form

$(\varepsilon,\delta)$9

under sub-Gaussian loss (Hellström et al., 2020). Online-to-PAC conversions yield the exact identity

$p\in(0,2]$00

from which PAC-Bayes-style and generalized regularizer-based bounds follow by plugging in regret theorems (Lugosi et al., 2023). Both are highly relevant to post hoc theory; neither paper develops an explicit eigenvalue- or effective-rank-adaptive bound.

6. Limitations, controversies, and open directions

The direct Transformer theory is deliberately narrow. Its main results apply to a simplified architecture with single-head attention, rowwise normalization $p\in(0,2]$01, one feedforward matrix per block, and no explicit residual connections, LayerNorm, or positional encodings (Sakai et al., 8 May 2026). The bounds still depend on depth $p\in(0,2]$02, hidden dimension $p\in(0,2]$03, token length through $p\in(0,2]$04, and propagation factors built from spectral norms (Sakai et al., 8 May 2026). They are post hoc in the sense of depending only on final trained weights, not on optimization trajectory, initialization, or implicit regularization (Sakai et al., 8 May 2026). The experiments evaluate leading-factor proxies rather than the exact theorem on full BERT models (Sakai et al., 8 May 2026).

The invariance literature introduces a different limitation: exact function-invariant post hoc complexity terms are often intractable. In the lifted PAC-Bayes framework, the ideal invariant quantity $p\in(0,2]$05 is usually unavailable in closed form, and the practical method is a deterministic-rescaling proxy (Rouchouse et al., 30 Sep 2025). Likewise, information-density bounds are post hoc in form but generally require unknown reference laws such as $p\in(0,2]$06 or $p\in(0,2]$07, so they are not automatically computable from a single trained model and dataset alone (Hellström et al., 2020).

A distinct controversy concerns the phrase “post hoc generalization” itself. In adaptive data analysis, the strongest output-level notion has sharp limitations: any algorithm that is both accurate and post hoc generalizing for $p\in(0,2]$08 adaptive bounded statistical queries requires

$p\in(0,2]$09

and post hoc generalization is not closed under composition (Nissim et al., 2018). This is not a theorem about spectral bounds for trained networks, but it does indicate that no uniform spectrum-adaptive refinement should be expected under the full worst-case quantifiers of post-selection generalization. The cited discussion explicitly suggests that stronger results would require extra structure such as restricted query classes, low-dimensional or effective-support assumptions, average-case analysts, or stronger composable notions such as differential privacy (Nissim et al., 2018).

The main open direction stated in the direct spectrum-adaptive paper is to combine weight spectral structure with data-dependent activation structure (Sakai et al., 8 May 2026). The surrounding frameworks suggest complementary routes: spectral penalties inside disintegrated PAC-Bayes (Viallard et al., 2024), spectral entropy control of algorithm-dependent output classes (Sachs et al., 2023), and geometry-sensitive online regularizers in online-to-PAC conversions (Lugosi et al., 2023). Taken together, these works suggest that spectrum-adaptive post hoc generalization theory is not a single method but a junction of three requirements: post-training evaluability, spectral sensitivity, and invariance to function-preserving parameter symmetries.