Potential Hessian Ascent: The Sherrington-Kirkpatrick Model (2408.02360v2)

Abstract: We present the first iterative spectral algorithm to find near-optimal solutions for a random quadratic objective over the discrete hypercube, resolving a conjecture of Subag [Subag, Communications on Pure and Applied Mathematics, 74(5), 2021]. The algorithm is a randomized Hessian ascent in the solid cube, with the objective modified by subtracting an instance-independent potential function [Chen et al., Communications on Pure and Applied Mathematics, 76(7), 2023]. Using tools from free probability theory, we construct an approximate projector into the top eigenspaces of the Hessian, which serves as the covariance matrix for the random increments. With high probability, the iterates' empirical distribution approximates the solution to the primal version of the Auffinger-Chen SDE [Auffinger et al., Communications in Mathematical Physics, 335, 2015]. The per-iterate change in the modified objective is bounded via a Taylor expansion, where the derivatives are controlled through Gaussian concentration bounds and smoothness properties of a semiconcave regularization of the Fenchel-Legendre dual to the Parisi PDE. These results lay the groundwork for (possibly) demonstrating low-degree sum-of-squares certificates over high-entropy step distributions for a relaxed version of the Parisi formula [Open Question 1.8, arXiv:2401.14383].

• The paper introduces the Potential Hessian Ascent algorithm that iteratively applies spectral methods to optimize a quadratic Hamiltonian over a binary hypercube.
• It extends traditional Hessian ascent techniques by incorporating an entropy-based potential correction and free probability analysis to ensure convergence.
• Empirical and theoretical analyses confirm that the iterative process closely follows the Auffinger-Chen SDE and advances high-dimensional optimization methods.

Potential Hessian Ascent: The Sherrington-Kirkpatrick Model

The paper presents the Potential Hessian Ascent (PHA) algorithm, an iterative spectral method for finding near-optimal solutions to a random quadratic objective over the discrete hypercube. The authors extend the Hessian ascent methodology originally proposed by Subag for spherical spin glasses to the more complex geometry of the hypercube.

Overview of the Approach

The core problem addressed by the PHA algorithm involves maximizing a Hamiltonian derived from the Sherrington-Kirkpatrick (SK) model over binary strings. The authors introduce an iterative process leveraging the eigenanalysis of the Hessian of a modified objective function, which includes instance-independent potential corrections to enforce regularity and continuity.

Detailed Analysis and Results

1. Potential Function Definition:
   • The potential function integrates an entropy term characterized by the Fenchel-Legendre conjugate of a regularized solution to the Parisi PDE, and an additional correction based on random matrix theory.
   • This functional form allows extensions to the interior of the hypercube while maintaining consistency with entropy-based regularization techniques used in similar high-dimensional problems.
2. Iterative Algorithm Design:
   • The algorithm initializes at the origin and executes updates by projecting along the top eigenvectors of the modified Hessian.
   • At each step, it generates increments through Gaussian vectors with covariance matrices designed to focus on the main part of the Hessian’s spectrum.
   • Additionally, truncation methods are employed during rounding to ensure the feasibility of solutions within the binary space.
3. Spectral Analysis:
   • The paper explores the spectral properties of the modified Hessian using free probability theory, which provides the groundwork for the iterative process by establishing controls over step distributions and spectral gaps.
   • The convergence arguments crucially depend on random matrix theory to estimate eigenvalues and high-dimensional concentration inequalities.
4. Convergence to SDE:
   • The authors demonstrate that empirical distributions of iterations approximate the solution to the Auffinger-Chen stochastic differential equation (SDE). This establishes a rigorous connection between the algorithm’s iterate distributions and the Parisi PDE.
   • This rigorous control over increments through Gaussian concentration allows the algorithm to follow the leading eigenvectors efficiently.
5. Energy Analysis:
   • A careful Taylor expansion analysis of the modified objective function across iterations shows the reduction in discrepancy from stepwise iterations.
   • Concentration inequalities for Gaussian processes and properties of the Parisi PDE contribute to bounding the difference between the actual objective and its expected change, ensuring convergence.

Numerical Results and Performance

The authors provide probabilistic guarantees that the PHA algorithm approximates near-optimal configurations of the SK model with high confidence. The critical insight lies in translating the advances in understanding the Parisi PDE, beyond the theoretical framework to practical algorithmic steps, while maintaining computational efficiency.

Theoretical Implications

The PHA algorithm pioneers bridging spectral analysis with high-dimensional Gaussian processes for combinatorial optimization. It adds a new dimension to Hessian ascent methods beyond spherical configurations and shows promising directions for potential extensions to non-binary domains or higher-order interactions.

Practical Applications and Future Work

The practical implications of the PHA algorithm traverse fields like optimization in machine learning, particularly in scenarios where optimization landscapes are complex and high-dimensional. Extensions could include mixed p-spin models and non-binary domains, using similar entropy-inspired corrections and spectral properties.

Conclusion

The PHA algorithm exemplifies a significant leap in combinatorial optimization via iterative spectral methods. It effectively utilizes mathematical frameworks from free probability, random matrix theory, and stochastic analysis to produce a highly reliable and theoretically grounded optimization path for the SK model, laying the foundation for future explorations in high-dimensional optimization spaces.