--- title: Spectrum-Adaptive Generalization Bounds url: https://www.emergentmind.com/topics/spectrum-adaptive-post-hoc-generalization-bounds type: topic --- # Spectrum-Adaptive Generalization Bounds Spectrum-adaptive post hoc generalization bounds are generalization guarantees evaluated after training in which the complexity term adapts to the learned spectral structure of the model—typically singular values through rank, Frobenius, or intermediate Schatten quantities—rather than to a single norm chosen a priori. In the cited literature, the most explicit realization of this idea is a covering-number and Rademacher-complexity theory for trained deep Transformers [2605.07297]. Surrounding work supplies broader post hoc frameworks—disintegrated PAC-Bayes, information-density bounds, online-to-PAC conversions, and algorithm-dependent Rademacher complexity—that are not themselves spectral but define technical templates into which spectral complexity can be inserted [2402.13285][2005.08044][2305.19674][2307.02501]. A recurrent structural issue is that spectrum-sensitive quantities computed directly in parameter space can fail to reflect functional complexity when neural-network symmetries are ignored, especially ReLU rescaling invariance [2509.26149]. ## 1. Conceptual scope and terminology In the post-training generalization literature, “post hoc” refers to guarantees that are evaluated after training on the realized predictor, posterior, or algorithm output. The dependence is therefore on trained weights, realized posterior distributions, realized hypothesis classes, or realized information quantities, rather than on a fixed hypothesis class specified before training. Spectrum adaptivity adds a second layer: the complexity description itself is chosen after training so that it reflects the observed singular-value profile of the learned matrices. The direct spectral formulation in the cited literature uses layerwise Schatten quantities. For a matrix \(W\) and \(p\in(0,2]\), \[ \|W\|_{S,p}=\left(\sum_i \sigma_i(W)^p\right)^{1/p}, \qquad \|W\|_{S,0}:=\operatorname{rank}(W), \] so \(p=0\) yields rank, \(p=2\) yields Frobenius structure through \(\|W\|_{S,2}^2=\|W\|_F^2\), and \(0\varepsilon\big]\le \delta \] with \(X\sim \mathcal P^n\) [1806.06100]. This is a different notion from post-training complexity control for a fixed trained network. The distinction matters because lower bounds and non-composition phenomena in adaptive data analysis do not automatically transfer to post-training spectrum-sensitive bounds, although they do delimit what is possible under stronger worst-case quantifiers [1806.06100]. ## 2. Direct spectrum-adaptive theorems for trained Transformers The most direct theorem of this type in the cited set studies a simplified multi-layer Transformer with input \(X\in\mathbb R^{T\times N}\), single-head attention \[ f_{\mathrm{head}}(X;W^{QK},W^V)=\mathrm{SoftMax}(XW^{QK}X^\top)\,XW^V, \] a normalized block \[ f_{\mathrm{block}}(X;W^{QK},W^V,W^M) = \Pi_{\mathrm{norm}}\!\left( \sigma\!\bigl(\Pi_{\mathrm{norm}}(f_{\mathrm{head}}(X;W^{QK},W^V))\bigr)\,W^M \right), \] and scalar readout \[ f_{\mathrm{out}}(X;W^{(1:L)},w) = w^\top [f^{(L)}_{\mathrm{tf}}(X;W^{(1:L)})]_{\mathrm{CLS},\cdot} \] under layerwise spectral norm control \(\|W_{*,(\ell)}\|_2\le C^{(2)}_{*,(\ell)}\) and \(\|w\|_2\le C_{\mathrm{out}}\) [2605.07297]. The simplified main theorem states that, if \(n\ge 3\) and every nonzero layer matrix satisfies \[ \|W_{*,(\ell)}\|_2 \ge \exp[-c_0(L+\log N)], \] then with probability at least \(1-\delta\), \[ \mathrm{GAP}(f_{\mathrm{out}}) \;\lesssim\; \frac{\sqrt{\log(nT)}}{\sqrt n} \inf_{p\in[0,2]^{3L}} \sum_{\ell=1}^L\sum_{*\in\{QK,V,M\}} \left(\|W_{*,(\ell)}\|_{S,p_{*,(\ell)}}^{p_{*,(\ell)}}\right)^{\frac{1}{p_{*,(\ell)}+2}} C^{\frac{p_{*,(\ell)}}{p_{*,(\ell)}+2}} L^{\frac{p_{*,(\ell)}+1}{p_{*,(\ell)}+2}} N^{\frac{p_{*,(\ell)}+1}{p_{*,(\ell)}+2}} \] \[ \qquad +\frac{\log n}{\sqrt n} +\sqrt{\frac{\log(1/\delta)+L\log(L+\log N)}{n}} \] simultaneously for all admissible trained Transformers [2605.07297]. The infimum is over \(p\in[0,2]^{3L}\), so each layer \(\ell\) and each matrix type \(*\in\{QK,V,M\}\) may choose its own Schatten index after training. The fixed-index precursor theorem holds on classes \(\mathcal F^{\mathrm{out}}(p;C_S)\) with prescribed Schatten radii \(C_{*,(\ell)}\), while a common-\(p\) theorem improves depth allocation when all matrices share a single index \(p\) [2605.07297]. The theory is therefore genuinely post hoc: the high-probability event is uniform over all admissible Schatten regimes, and only after observing the trained weights does one select the indices that minimize the bound. The leading term interpolates between rank-like and Frobenius-like descriptions. When \(p=0\), the dependence is rank-based; when \(p=2\), it becomes Frobenius-based; intermediate \(p\) values trade spectral sparsity against the architecture-dependent factors \(L\) and \(N\) [2605.07297]. In matched comparisons reported in the paper, the resulting leading factor improves on prior norm-based Transformer bounds of Edelman et al. (2022) and Trauger and Tewari (2024), which are expressed through fixed \((2,1)\)- or \((1,1)\)-norm constraints and can scale like \(C^L L^2\) [2605.07297]. ## 3. Technical mechanisms of spectrum adaptivity The Transformer bounds are covering-number results converted to generalization bounds by a Dudley-type entropy integral and a Rademacher complexity bound for Lipschitz losses [2605.07297]. They are therefore neither PAC-Bayes nor margin bounds. The central mechanism is a matrix interpolation argument for linear maps \(X\mapsto XW\) under simultaneous spectral norm and Schatten control. The proof decomposes a matrix as \[ W=W_1+W_2 \] using a singular-value threshold \(\tau>0\). If \(\|W\|_{S,p}^p\le C_S\), then \[ \operatorname{rank}(W_1)\le \frac{C_S}{\tau^p}, \qquad \|W_2\|_F^2 \le \tau^{2-p}\min\{l,m\}. \] Thus \(W_1\) is treated as a low-rank, spectral-norm-bounded component, while \(W_2\) is treated as a Frobenius-bounded tail; optimizing \(\tau\) yields the characteristic exponent \(1/(p+2)\) that appears in the final complexity term [2605.07297]. For the Transformer architecture, the covering argument is applied separately to the query-key, value, and feedforward matrices. The proof uses that rowwise normalization \(\Pi_{\mathrm{norm}}\) is \(1\)-Lipschitz in \(\|\cdot\|_{2\to\infty}\), and that softmax is \(2\)-Lipschitz from \(\ell_\infty\) to \(\ell_1\) [2605.07297]. Layerwise errors then accumulate through a propagation factor \[ a^{(\ell)} = \prod_{k=\ell+1}^{L} L_\sigma C_{M,(k)} C_{V,(k)}\bigl(1+4C_{QK,(k)}\bigr), \] so the contribution of layer \(\ell\) is multiplied by the Lipschitz growth of all subsequent layers [2605.07297]. The post hoc uniformity over all \(p\in[0,2]^{3L}\) is obtained by discretizing \([0,2]\) via \[ P_m=\left\{0,\frac1m,\frac2m,\dots,\frac{2m-1}{m},2\right\}, \] performing dyadic peeling over realized Schatten radii, and rounding arbitrary continuous indices upward to the grid [2605.07297]. The appendix’s exact theorem packages the adaptive term into \(B_p(W^{(1:L)})\) and the post hoc logarithmic penalty into \(\mathcal H_p(W^{(1:L)})\) [2605.07297]. The empirical evidence in that work is proxy-based rather than an exact theorem evaluation. Using BERT Miniatures checkpoints with depths \(L\in\{2,4,6,8,10,12\}\) and hidden dimensions \(N\in\{128,256,512,768\}\), the authors compute BERT-adapted leading-factor proxies and report that the proxies induced by their bounds grow more slowly with depth and hidden dimension than the corresponding Edelman-type proxies [2605.07297]. They also report that the post hoc optimizer selects \(p=0\) for all matrices on those checkpoints; the paper attributes this to the tradeoff between spectral decay and the architectural penalties in \(L\) and \(N\), not to identical spectra across matrices [2605.07297]. ## 4. Invariance, quotient geometry, and functional alignment Spectrum-sensitive post hoc bounds for neural networks face a foundational obstacle absent from the Transformer results above: in ReLU networks, raw parameter-space quantities can be arbitrarily changed by function-preserving rescalings. For a ReLU network on a DAG \(G=(V,E)\), hidden-unit rescaling is \[ \bigl(\diamond^\lambda(w)\bigr)_{u\to v}=\frac{\lambda_v}{\lambda_u}w_{u\to v}, \qquad \bigl(\diamond^\lambda(w)\bigr)_v=\lambda_v w_v, \] and positive homogeneity gives \[ f_{\diamond^\lambda(w)}=f_w \] for all \(\lambda\in\mathbb R_{>0}^{H}\) [2509.26149]. The cited ReLU analysis shows on a one-hidden-neuron example, \[ f_w(x)=w_2\max(w_1x,0), \qquad \diamond^\lambda(w)=(\lambda w_1,w_2/\lambda), \] that a standard PAC-Bayes complexity can diverge under rescaling even though the predictor is unchanged. With \[ P\sim\mathcal N(0,\sigma^2 I_2), \qquad Q\sim\mathcal N(w,\operatorname{diag}(w^2)), \] the KL behaves as \[ \mathrm{KL}(\diamond^\lambda_\sharp Q\|P)\sim \lambda^2 w_1^2/\sigma^2 \quad\text{as }\lambda\to\infty \] [2509.26149]. The paper explicitly notes that this pathology is relevant not only to PAC-Bayes but also to Euclidean norms, layerwise norms, path norms, spectral norms, products of spectral norms, margins normalized by norms, perturbation sensitivity, sharpness, local curvature, and Hessian traces or eigenvalues when computed in raw coordinates [2509.26149]. The proposed remedy is a lifted representation \(\Psi:W\to Z\) such that \[ f_w=g(\Psi(w)) \] for some measurable \(g\), and PAC-Bayes is applied to the pushforwards \(\Psi_\sharp Q\) and \(\Psi_\sharp P\). For ReLU networks, the paper highlights the rescaling-invariant path+sign lift \[ \Psi(w)=(\Phi(w),\operatorname{sgn}(w)), \qquad \Psi\circ\diamond^\lambda=\Psi. \] The lifted PAC-Bayes inequality replaces \(\mathrm{KL}(Q\|P)\) by \[ \mathrm{KL}(\Psi_\sharp Q\|\Psi_\sharp P), \] and the paper proves the comparison chain \[ \mathrm{KL}(\Psi_\sharp Q\|\Psi_\sharp P) \le \inf_{\boldsymbol{\lambda},\boldsymbol{\lambda}'} \mathrm{KL}(\diamond^{\boldsymbol{\lambda}}_\sharp Q \,\|\, \diamond^{\boldsymbol{\lambda}'}_\sharp P) \le \inf_{\lambda,\lambda'} \mathrm{KL}(\diamond^\lambda_\sharp Q \,\|\, \diamond^{\lambda'}_\sharp P) \le \mathrm{KL}(Q\|P) \] [2509.26149]. The direct theorem there is KL/PAC-Bayes-centric rather than spectrum-centric, but its relevance to spectrum-adaptive bounds is structural. A plausible implication is that any spectrum-adaptive post hoc bound for ReLU networks should either be formulated in a rescaling-invariant lifted or quotient representation, or explicitly optimized over the rescaling orbit, because layerwise singular values can change under hidden-unit rescaling without changing the realized function [2509.26149]. The same paper reports that deterministic rescaling optimization typically reduces the KL by about a factor of 4 and the final PAC-Bayes bound by about a factor of 2, and in some cases turns a vacuous bound into a non-vacuous one [2509.26149]. ## 5. General post hoc templates compatible with spectral complexity Several cited frameworks are not spectrum-adaptive by themselves but are broad enough to host spectral complexity measures. The most explicit PAC-Bayes template is the disintegrated Gibbs framework. For any measurable score \(\mu:\mathcal H\times(\mathcal X\times\mathcal Y)^m\to\mathbb R\), define a Gibbs posterior \[ \rho(h)\propto e^{-\mu(h,S)}. \] Then, with probability at least \(1-\delta\) over \[ h'\sim \pi,\quad S\sim\mathcal D^m,\quad h\sim \rho, \] the paper proves \[ \phi(R_{\mathcal D}(h),R_S(h)) \le \mu(h',S)-\mu(h,S)+\ln\frac{\pi(h')}{\pi(h)} + \ln\!\left[ \frac{4}{\delta^2} \mathbb E_{S\sim\mathcal D^m}\mathbb E_{g\sim\pi} e^{\phi(R_{\mathcal D}(g),R_S(g))} \right] \] [2402.13285]. The theorem is hypothesis-level rather than posterior-expectation-level. Because \(\mu\) is user-chosen and need only be measurable with a well-defined Gibbs density, the framework is structurally broad enough to accommodate spectral penalties. The cited summary is explicit that the paper does not derive a spectrum-specific theorem, but that it provides a direct route to post hoc, hypothesis-dependent PAC-Bayes bounds with user-chosen complexity measures [2402.13285]. A second template is algorithm-dependent Rademacher complexity. Given a deterministic algorithm \(\mathcal A\), two ghost halves \(S_-^n,S_+^n\), and mixed samples \(S_\sigma^n\), define \[ \hat\Theta^n=\{\mathcal A(S_\sigma^n):\sigma\in\{-1,+1\}^n\}. \] The paper proves \[ \mathbb E_{S^n}\bigl[R(\hat\theta)-\hat R(\hat\theta,S^n)\bigr] \le 2\,\mathbb E_{S_-^n,S_+^n}\bigl[\Rad_\ell(\hat\Theta^n,S_+^n)\bigr], \] and, under boundedness assumptions, a corresponding high-probability bound through the essential supremum of \(\Rad_\ell(\hat\Theta^n,S_+^n)\) [2307.02501]. Since \(\Rad_\ell(\hat\Theta^n,S^n)\) is controlled by covering numbers and finite Minkowski dimension, a plausible implication is that spectral control of the covering entropy of \(\hat\Theta^n\) would immediately yield a spectrum-adaptive post hoc bound [2307.02501]. Two further frameworks are post hoc and data-dependent but not spectral in the cited papers. Information-density bounds yield single-draw guarantees of the form \[ |\mathrm{gen}(W,Z)| \le \sqrt{\frac{2\sigma^2}{n-1} \left( \imath(W,Z)+\log\frac{\sqrt n}{\delta} \right)} \] under sub-Gaussian loss [2005.08044]. Online-to-PAC conversions yield the exact identity \[ \overline{gen} = \frac{regret_{\Pi_n}(P_{W_n\mid S_n})}{n} - M_{\Pi_n}, \] from which PAC-Bayes-style and generalized regularizer-based bounds follow by plugging in regret theorems [2305.19674]. Both are highly relevant to post hoc theory; neither paper develops an explicit eigenvalue- or effective-rank-adaptive bound. ## 6. Limitations, controversies, and open directions The direct Transformer theory is deliberately narrow. Its main results apply to a simplified architecture with single-head attention, rowwise normalization \(\Pi_{\mathrm{norm}}\), one feedforward matrix per block, and no explicit residual connections, LayerNorm, or positional encodings [2605.07297]. The bounds still depend on depth \(L\), hidden dimension \(N\), token length through \(\log(nT)\), and propagation factors built from spectral norms [2605.07297]. They are post hoc in the sense of depending only on final trained weights, not on optimization trajectory, initialization, or implicit regularization [2605.07297]. The experiments evaluate leading-factor proxies rather than the exact theorem on full BERT models [2605.07297]. The invariance literature introduces a different limitation: exact function-invariant post hoc complexity terms are often intractable. In the lifted PAC-Bayes framework, the ideal invariant quantity \(\mathrm{KL}(\Psi_\sharp Q\|\Psi_\sharp P)\) is usually unavailable in closed form, and the practical method is a deterministic-rescaling proxy [2509.26149]. Likewise, information-density bounds are post hoc in form but generally require unknown reference laws such as \(P_W\) or \(P_{W\mid \tilde Z}\), so they are not automatically computable from a single trained model and dataset alone [2005.08044]. A distinct controversy concerns the phrase “post hoc generalization” itself. In adaptive data analysis, the strongest output-level notion has sharp limitations: any algorithm that is both accurate and post hoc generalizing for \(k\) adaptive bounded statistical queries requires \[ n=\Omega\!\left(\frac{\sqrt{k}}{\varepsilon^2}\right), \] and post hoc generalization is not closed under composition [1806.06100]. This is not a theorem about spectral bounds for trained networks, but it does indicate that no uniform spectrum-adaptive refinement should be expected under the full worst-case quantifiers of post-selection generalization. The cited discussion explicitly suggests that stronger results would require extra structure such as restricted query classes, low-dimensional or effective-support assumptions, average-case analysts, or stronger composable notions such as differential privacy [1806.06100]. The main open direction stated in the direct spectrum-adaptive paper is to combine weight spectral structure with data-dependent activation structure [2605.07297]. The surrounding frameworks suggest complementary routes: spectral penalties inside disintegrated PAC-Bayes [2402.13285], spectral entropy control of algorithm-dependent output classes [2307.02501], and geometry-sensitive online regularizers in online-to-PAC conversions [2305.19674]. Taken together, these works suggest that spectrum-adaptive post hoc generalization theory is not a single method but a junction of three requirements: post-training evaluability, spectral sensitivity, and invariance to function-preserving parameter symmetries. Source: https://www.emergentmind.com/topics/spectrum-adaptive-post-hoc-generalization-bounds