Free Decompression with Algebraic Spectral Curves

Abstract: Tools from random matrix theory have become central to deep learning theory, using spectral information to provide mechanisms for modeling generalization, robustness, scaling, and failure modes. While often capable of modeling empirical behavior, practical computations are limited by matrix size, often imposing a restriction to models that are too small to be realistic. This motivates the inference of properties of larger models from the behavior of smaller ones. Free decompression (FD) is a recently proposed method for extrapolating spectral information across matrix sizes, but its utility is currently limited by strong assumptions that preclude its implementation on more realistic ML models. We use algebraic spectral curve theory to provide a general FD methodology for spectral densities whose Stieltjes transform satisfies an algebraic relation, a modeling assumption that is more likely to hold in practice. This recasts FD as an evolution along spectral curves which can be readily integrated. Our framework enables the expansion of spectral densities that have multiple or multi-modal bulks, that exist at multiple scales, and that contain atoms, all characteristic of real-world data and popular ML models. We demonstrate the efficacy of our framework on models of interest in modern ML, including Hessian and activation matrices associated with neural networks and large-scale diffusion models.

• The paper introduces a robust algebraic spectral curve framework that generalizes free decompression to handle multi-bulk and atom-rich spectral densities.
• It develops numerical methods using geometric continuation and regularized least squares to accurately track the physical branch of the Stieltjes transform.
• Empirical results on neural network Hessians and diffusion model matrices demonstrate significant improvements in computational efficiency and spectral accuracy over traditional methods.

Free Decompression with Algebraic Spectral Curves: An Expert Analysis

Introduction and Motivation

The spectral analysis of large Hermitian matrices is fundamental in both high-dimensional statistics and the understanding of deep learning systems—particularly for diagnosing generalization, overfitting, and robustness via eigenspectrum properties. However, direct eigendecomposition or even implicit access is infeasible for matrices whose size precludes full storage or manipulation (“impalpable matrices”), a typical scenario in modern machine learning. Existing approaches to spectral estimation, notably Free Decompression (FD) (2605.03634), permit extrapolating the empirical spectral density (ESD) of a large matrix from the ESD of its principal submatrices. While foundational, previous FD frameworks suffered crucial limitations: they assumed unimodal, connected bulks and imposed restrictive algebraic conditions that proved inadequate for the complex, multi-bulk, or atom-containing spectra prevalent in realistic ML settings.

This paper advances the state of the art with a significant generalization: a practical, scalable FD methodology rooted in algebraic spectral curve theory. The approach assumes only that the Stieltjes transform of the spectral density satisfies a (possibly high-degree) bivariate polynomial relation—a property verified for all classical random matrix ensembles and a wide swath of relevant ML-induced matrices. The authors develop robust numerical methods for inferring spectral properties of large-scale matrices from small samples, elucidating the practical FD flow of key spectral features such as atoms, moments, and spectral edges, and demonstrate empirical efficacy on matrices from neural network Hessians and diffusion models.

Theoretical Foundations

FD and the Algebraic Ansatz

The central technical assumption is that the Stieltjes transform $m(z)$ associated with the matrix's ESD is algebraic, i.e., there exists a nonzero bivariate polynomial $P(z, m)$ so that $P(z, m(z)) = 0$. While prior FD was limited to quadratic or a few canonical cases, this polynomial method (PM) substantially enlarges the representational capacity and coherence of FD by exploiting the structure of algebraic curves—a perspective originally noted by Rao and Edelman [39], but not previously operationalized at this scale or generality.

The authors show that free decompression corresponds to an evolution along the algebraic spectral curve, and that the evolved Stieltjes transform after decompression (with decompression ratio $\tau \geq 1$) satisfies a derived polynomial:

$$P\left(z + \frac{\tau-1}{\tau m}, \tau m\right) = 0$$

This formulation circumvents the ill-conditioned PDE and analytic continuation issues that hampered earlier approaches.

Implementation Challenges and Solutions

Polynomial root selection and numerical curve fitting are nontrivial, especially in the multi-modal, high-degree case. The authors develop a geometric continuation procedure—anchored at points where the physical sheet is identified either via large $|z|$ asymptotics or empirical Stieltjes values, and transported along the complex plane using implicit differentiation and Newton-corrected steps—to consistently track the physical Stieltjes branch and avoid sheet switching near branch points or bulk splits. This increases robustness in cases involving cusp singularities, multi-bulk densities, and spectral atoms.

The methodology is enriched by moment-matching constraints and branch-point analysis, used both to fit the spectral curve and to guarantee physicality (i.e., that the solution corresponds to an appropriate probability measure). Numerical stability and overfitting are addressed via regularized least squares and careful selection of monomial supports in the polynomial fit.

Computation and Evolution of Key Spectral Features

A core advantage of the algebraic framework is the capacity to directly compute and trace quantitatively relevant spectral observables, including:

• Location and evolution of atoms—handled through residues of the spectral curve and evolved deterministically under FD.
• Spectral edges and bulk supports, which are linked to real branch points of the algebraic relation and can be continued even through mergers or splits of bulks (e.g., at cusp points, which are algebraically characterized and correspond to phase transitions in the ESD support).
• Spectral moments—both high-order and lower-order moments can be computed recursively from the coefficients of the polynomial, providing a principled mechanism for enforcing empirical fidelity and monitoring scaling laws or generalization metrics.

Finite-size corrections, including Tracy-Widom-type behavior for the extreme eigenvalues, are incorporated using a heuristic correction derived from local expansions around the spectral edge, improving quantitative accuracy for condition number estimation and support estimation near singularities.

Empirical Validation and Numerical Results

The methodology is validated on three principal testbeds:

Compound Free Poisson Laws

Multi-bulk, atom-containing synthetic matrices—such as those arising from compound free Poisson constructions—demonstrate that the method can recover both the splitting and merging of bulks as the matrix size is "decompressed," as well as the correct evolution of atomic weights.

Neural Network Hessian Models

The Hessians of two-layer neural networks (as modeled by the Pennington–Bahri law and realistic autoencoder architectures) are empirically shown to have spectra whose multi-bulk structure, index (fraction of negative eigenvalues), and moments are accurately predicted by FD from much smaller subsamples, outperforming previous PDE-based techniques and at a fraction of the computational cost.

Diffusion Model Activation Matrices

Spectra with extreme log-scale bulk structure are correctly tracked, with the narrow atom-like bulks, multiple bulk splitting, and large dynamical range all being handled robustly by the FD polynomial method. Moment and distributional discrepancies remain under 1% in Wasserstein and MMD metrics for full-sized matrices extrapolated from small submatrices.

A summary of quantitative performance is given below:

Matrix Size	Direct Eigendecomp (sec)	FD Compute Time (sec)	Wasserstein Error	MMD Error	Index Rel. Error
$2^{11}$	32.2	32.2 + 0.0	0.01%	0.01%	0.01%
$2^{15}$	192447.7	32.2 + 6.8	2.19%	0.53%	0.08%

(Adapted from Table 1 of the paper.)

Implications for Spectral Inference and Large-Scale Machine Learning

The algebraic FD framework is the first practical technique that enables accurate, high-fidelity extrapolation of spectral features from limited principal submatrices to full, intractably large matrices, in regimes where the spectral measure can be highly nontrivial—multi-scale, multi-modal, and with spectral atoms. This is essential for model tuning, monitoring of implicit regularization, diagnosing failure modes, and even hardware-aware computation in very large ML systems.

On a theoretical level, the method solidifies the link between free probability, algebraic geometry, and random matrix theory, and generalizes the class of models for which explicit spectral computations are tractable.

Potential future developments include automated model selection for the degree and structure of the polynomial ansatz, integration with neural scaling law diagnostics, and further refinement of finite-size corrections for extreme eigenvalue statistics. Incorporation of this framework in practical tools (as evidenced by the open-source freealg implementation) lays groundwork for broader adoption in ML pipelines.

Conclusion

This paper presents a comprehensive algebraic framework for free decompression, rigorously connecting the evolution of spectral densities—across both bulks and atoms—under matrix scaling to algebraic spectral curve theory. By delivering robust, numerically stable inference of spectral properties for matrices beyond the reach of direct eigendecomposition, and by empirically validating the method on relevant machine learning examples, the work marks a step forward in computational spectral theory and its application to high-dimensional learning. While some limitations remain concerning heavy-tailed distributions and extreme-edge corrections, the algebraic polynomial method establishes a solid foundation for further exploration of large-scale, algebraically-structured spectral inference in ML and statistics.