Explaining Machine Learning and Memorization with Statistical Mechanics

Abstract: Artificial neural networks (NNs) and ML algorithms are poorly understood from a theoretical perspective, which makes it difficult to fully realize their potential and overcome their weaknesses. For instance, ML algorithms train NN weights by moving them along a low-dimensional subspace of their allowed values, but this implicitly low-dimensional learning structure is not properly exploited to improve training because its nature is not well understood. Moreover, trained NNs are easily confused by pervasive adversarial attacks whose theoretical underpinnings are still unclear. This thesis aims to improve our theoretical understanding of NNs and ML, with a particular focus on adversarial attacks and implicitly low-dimensional learning. For this purpose, we use mathematical tools from statistical mechanics to study different types of NNs and ways in which they can fit the data. In particular, we study two classes of models that fit the data with various degrees of learning and memorization: dense associative memory (DAM) and restricted Boltzmann machines (RBM). In the process, we investigate connections between different versions of these models that are useful to make analytical investigations more efficient.

• The paper develops an exact statistical mechanics framework to characterize phase transitions, learning, and memorization in dense neural networks.
• It demonstrates that higher-order interactions lead to enhanced adversarial robustness and first-order transitions, validated by Monte Carlo experiments.
• The work extends the lottery ticket hypothesis by identifying robust subnetworks within overparameterized models using targeted pruning and replica theory.

Statistical Mechanics Perspectives on Machine Learning and Memorization

Introduction

"Explaining Machine Learning and Memorization with Statistical Mechanics" (2606.31110) delivers a comprehensive, mathematically rigorous analysis of learning and memorization phenomena in modern neural network architectures through the frameworks of statistical mechanics and spin glass theory. The focus is on the interplay between learning in low-dimensional manifolds, adversarial robustness, and phase transitions in energy-based models, specifically dense associative memory (DAM) systems such as dense Hopfield networks (DHNs) and Restricted Boltzmann Machines (RBMs). The work leverages the teacher-student protocol, replica theory, and analytical calculations to clarify the limits and mechanisms of neural information storage, learning, memorization, and vulnerability to adversarial attacks.

Theoretical Foundations: Energy-Based Models and Phase Diagrams

The paper systematically derives the statistical mechanical foundations of Hopfield networks, dense HNs, and RBMs. These networks are modeled as high-dimensional energy-based systems, where training and inference correspond to the minimization of suitable Hamiltonians and the exploration of their Gibbs distributions at "temperature" $T$. This analogy enables explicit calculation of equilibrium measures and phase diagrams, exposing transitions between paramagnetic (uninformative), spin glass (frozen disorder), retrieval (feature-based), and example (prototype-based) phases as a function of model parameters and data complexity.

Figure 1: In the left panel, a linear function $y_{\text{model}}$ fit to noisy data; the right shows residuals—illustrating how learning can be framed as probability density estimation.

The analysis formalizes how more "structured" (low-temperature, low-noise) data leads to the emergence of alignment between learned parameters and true generative structures, quantified via overlaps and order parameters. The statistical mechanics approach, prominent in spin glass models, provides exact and approximate predictions for critical thresholds in storage capacity and phase boundaries.

Figure 2: Phase diagram of Hopfield networks in the teacher-student setting with one pattern, quantitatively delineating regions of retrieval, spin glass behavior, and paramagnetic disorder.

Dense Hopfield Networks: Learning, Memorization, and Adversarial Phenomena

A key contribution is the exact solution of the teacher-student problem for $p$-body dense Hopfield networks, which generalize standard pairwise Hopfield models by allowing higher-order interactions. The work demonstrates that, on the Nishimori line (i.e., when student and teacher hyperparameters match), the boundary between retrieval of the generative structure and failure coincides with the paramagnetic–spin-glass transition derived for the associated "direct" problem with random patterns. This formalizes and extends empirically observed regimes, such as the prototype (example-retrieval) and feature (signal-retrieval) phases, conjectured to underlie robust and fragile operation modes in neural systems.

Figure 3: RS phase diagrams for direct models with $p=3$ and $p=10$, highlighting the emergence and stability of retrieval, spin glass, and paramagnetic phases.

A notable analytic result is that, for $p \geq 3$, the transition from uninformative to retrieval behavior is first-order, leading to coexistence ("hard phases") and computational barriers that align with observed difficulties in practical inference.

Further, the analysis uncovers how increasing the student model's interaction order $p$ (e.g., $p=4$) confers extensive noise tolerance and adversarial robustness, at the expense of requiring superlinear sample sizes. In these regimes, the student can recover a teacher's pattern information even if the teacher's data is highly noisy, up to $T^* \sim \mathcal{O}(N^{1-2/p})$.

Figure 4: RS phase diagrams for $p^* = 2$, $y_{\text{model}}$0 in the student, revealing enhanced noise tolerance regimes not present in $y_{\text{model}}$1 models.

The paper derives a closed-form expression for the adversarial perturbation threshold $y_{\text{model}}$2 at which signal retrieval fails at zero temperature—an explicit quantification of adversarial robustness in the DAM/Hopfield setting.

Figure 5: Monte Carlo simulations giving the critical overlap as a function of data load $y_{\text{model}}$3 and adversarial perturbation size $y_{\text{model}}$4, validating analytic robustness predictions.

RBMs, Lottery Tickets, and Implicit Low-dimensional Learning

The statistical mechanics framework is extended to RBMs, where both the number and structure of hidden units can be systematically controlled and analyzed. When the teacher's patterns are uncorrelated, it is shown—and proven—that student performance is independent of the number of hidden units, in line with earlier conjectures. This factorization property is demonstrated both for binary and real-valued weights, and is numerically confirmed via Monte Carlo experiments.

A central novel observation is that when the student is overparameterized ($y_{\text{model}}$5), only a select subset of hidden units aligns with the teacher’s structure, while redundant units remain in ergodic, frozen disorder—a concrete instance of the lottery ticket hypothesis. The manuscript demonstrates, via targeted pruning and retraining, that these "winning ticket" subnetworks can be identified via magnitude-based selection and, in practice, converge faster than random initializations.

Figure 6: Results of a lottery ticket experiment—winning ticket subnetworks (B) identified by pruning from an overparameterized student converge significantly faster to the true teacher structure than random networks (A).

For correlated teacher patterns, the replica solution predicts and numerically confirms a hierarchy of spontaneous and permutation symmetry breaking phase transitions, in which student representations evolve from extracting common features (SSB), through learning distinguishable combinations (PSB$y_{\text{model}}$6), up to fully aligned retrieval (PSB$y_{\text{model}}$7). Increasing pattern correlation or the number of hidden units lowers the critical data load necessary for successful learning. Importantly, the decorrelation commonly employed for training speedup is shown here to generically increase the sample complexity required for robust learning—a result with direct algorithmic implications.

Dense Associative Memory for Rapid Stable Learning

Building on insights gained in DHNs and RBMs, the work studies a DAM of prototype-based energy functions capable of both supervised and unsupervised classification. Inspired by the observed statistical structure of student-teacher saddle points, a regularization procedure is introduced to stabilize and accelerate DAM training. Weights obtained in small DAMs are proven to lie at saddle points of larger networks’ loss landscapes—a hierarchy exploited to propose a scalable growing-DAM training algorithm, with considerably enhanced convergence.

Empirical Validation and Phase Transition Theory

All major analytic predictions are stringently validated via large-scale Monte Carlo experiments on synthetic and real-data tasks; finite-size effects and algorithmic metastabilities are analyzed. Notably, many phase transition boundaries, e.g., paramagnetic–retrieval, and the thresholds for adversarial success (feature-to-prototype phase boundary) are in precise correspondence with theory, supporting the underlying replica symmetric and symmetry breaking mean-field approaches.

Implications, Limitations, and Future Directions

This work provides deep, quantitatively precise understanding of the fundamental trade-offs between memorization, learning, robustness, and overparameterization in neural systems under a statistical mechanics lens:

• Adversarial robustness: Demonstrated as a phase property, robust regimes (prototype/exemplar retrieval, high-$y_{\text{model}}$8 models, large sample loads) are sharply delineated; formulaic adversarial thresholds can guide principled network design.
• Low-dimensional learning: The low-dimensional manifold structure of effective learning is made explicit in analytic order parameters, suggesting avenues for neural pruning, architecture search, and fast fine-tuning.
• Generalization of the lottery ticket hypothesis: The network redundancy and sparse subnetwork selection phenomena are analytically tractable, with direct analogies in modern overparameterized deep architectures.
• Practical hyperparameter guidance: The dependence of memorization and learning phases on interaction orders $y_{\text{model}}$9, temperature, and data load $p$0 provides actionable recommendations for model specification, regularization, and curriculum design.

The limitations include the focus on mean-field models, binary or Gaussian priors, and the teacher-student (Bayesian) setting with usually limited to one or few teachers. There is still significant work required to extend these results to arbitrarily deep and structured architectures and match them quantitatively to modern neural language or vision models.

Conclusion

By establishing exact and approximate phase diagrams, analytic thresholds, and replicable simulation results, this paper demonstrates that dense neural networks’ learning, memorization, and robustness properties can be comprehensively understood and predicted via statistical mechanics. The work closes several open theoretical questions regarding implicit low-dimensional learning, adversarial vulnerability, and parameter redundancy, and lays a rigorous foundation for leveraging these phenomena in robust, scalable artificial intelligence systems. Future research may extend these concepts to deeper, more structured networks and use this mathematical apparatus to design architectures with tailored statistical properties.