Free energy landscape of Dense Associative Memory

Abstract: Using large deviations theory, we solve and obtain a general expression for the free energy functional for a broad class of associative memories, including dense associative memories. We illustrate the method by reproducing classical results for the Hopfield model. For a finite number of patterns, we derive the temperature-dependent free energy functional for dense associative memories featuring polynomial interactions and Log-Sum-Exponential (LSE) activation. We also evaluate the disorder-averaged ground-state energy of these systems in the extensive limit. Our analytical framework reveals how memory retrieval depends on the initial state in higher-order dense networks, and gives the exact full-retrieval threshold for the LSE model. This method provides a systematic procedure for analyzing diverse, complex architectures in associative memory.

• The paper develops a tilted large-deviations framework that produces exact, temperature-dependent free energy functionals for energy-based associative memories with arbitrary interaction functions.
• The analysis shows that polynomial DenseAMs with order k>2 retain a persistent nonretrieval minimum at zero overlap, making retrieval highly dependent on initialization even as successful retrieval becomes more accurate.
• For the Log-Sum-Exponential model, the paper derives the exact full-retrieval threshold α_c(λ)=λ(1−λ/2) for λ≤λ*, where exponentially many stored patterns can still be retrieved without error.​​​​

Overview

The paper develops a large-deviations framework for computing the free energy functional of a general class of energy-based associative memories, including dense associative memories (DenseAMs) with polynomial interactions of arbitrary order and the Log-Sum-Exponential (LSE) model. The central contribution is an exact, temperature-dependent free energy functional $\mathcal{F}(\mathbf{m})$ expressed in terms of the overlap vector $\mathbf{m} = (m^1,\dots,m^P)$ between a spin configuration and the $P$ stored patterns. The framework reproduces classical replica results for the Hopfield model, extends them to $k>2$ interactions where no prior free energy expression was available, and yields the exact full-retrieval threshold $\alpha_c(\lambda)$ for the LSE model with exponentially many stored patterns.

Method: tilted large deviations

Neurons are Ising spins $s_i = \pm 1$; patterns $\boldsymbol{\xi}^\mu$ are drawn i.i.d. from a binary distribution. For a general Hamiltonian $H = -N f(\mathbf{m})$, with $m^\mu = \frac{1}{N}\sum_i \xi_i^\mu s_i$, the authors first obtain the rate function $I_0(\mathbf{m})$ of the joint distribution of overlaps via a $\mathbf{m} = (m^1,\dots,m^P)$0-dimensional Laplace transform and saddle-point evaluation, where the cumulant generating function is $\mathbf{m} = (m^1,\dots,m^P)$1. The Gibbs-weighted distribution is then handled with the tilted large deviation principle, giving

$\mathbf{m} = (m^1,\dots,m^P)$2

and the fixed-point condition reduces the saddle-point inversion to the self-consistency equation

$\mathbf{m} = (m^1,\dots,m^P)$3

with the free energy functional

$\mathbf{m} = (m^1,\dots,m^P)$4

This avoids the $\mathbf{m} = (m^1,\dots,m^P)$5-dimensional inversion of $\mathbf{m} = (m^1,\dots,m^P)$6 that becomes intractable at large $\mathbf{m} = (m^1,\dots,m^P)$7, and—unlike the Hubbard–Stratonovich transformation—handles higher-order interactions directly.

Polynomial DenseAMs: exact free energy and ground-state landscape

For the $\mathbf{m} = (m^1,\dots,m^P)$8-th order Hamiltonian $\mathbf{m} = (m^1,\dots,m^P)$9, the framework yields closed-form fixed-point equations and free energy. For $P$0 the results match Amit, Gutfreund, and Sompolinsky's replica calculation, and the fixed-point equation agrees with the stochastic-dynamics treatment of Rooke et al. For $P$1 the free energy functional is new. For a single pattern, transitions are continuous for $P$2 and first order for $P$3, with transition temperatures $P$4 for $P$5 respectively.

In the extensive limit $P$6, with cross-overlaps treated as Gaussian noise of variance $P$7, the disorder-averaged ground-state energy (zero-temperature landscape of gradient descent) is

$P$8

with fixed points satisfying $P$9; for $k>2$0 this reproduces Amit et al.'s result with $k>2$1.

A key structural finding concerns higher-order networks: for $k>2$2 the curvature at $k>2$3 satisfies $k>2$4 for all $k>2$5, so the $k>2$6 state remains a local minimum indefinitely, while a nonzero retrieval minimum $k>2$7 appears only below a threshold $k>2$8 at which it becomes global. Retrieval therefore depends strongly on the initial state: if the initial configuration lies in the $k>2$9 basin, the pattern is never retrieved at zero noise, for any $\alpha_c(\lambda)$0. The authors note that escaping this basin requires a mechanism such as the stochastic noise considered by Rooke et al. As $\alpha_c(\lambda)$1 grows, the $\alpha_c(\lambda)$2 basin shrinks even though $\alpha_c(\lambda)$3 (e.g., $\alpha_c(\lambda)$4 for $\alpha_c(\lambda)$5), so higher order improves retrieval fidelity but restricts the set of successful initial states. The paper also distinguishes the transition thresholds $\alpha_c(\lambda)$6 from the conventional capacity threshold defined by an allowed error fraction—for polynomial models these do not coincide because $\alpha_c(\lambda)$7 at the transition (e.g., $\alpha_c(\lambda)$8 vs. $\alpha_c(\lambda)$9 for $s_i = \pm 1$0).

LSE model: exact full-retrieval threshold

For the LSE Hamiltonian with interaction strength $s_i = \pm 1$1, the authors introduce the auxiliary quantity $s_i = \pm 1$2. Using the tilted LDP with $s_i = \pm 1$3 and Gaussian cross-overlaps of variance $s_i = \pm 1$4, they obtain $s_i = \pm 1$5, with an extremum at $s_i = \pm 1$6. Retrieval is possible only in the phase $s_i = \pm 1$7; in the complementary phase the model reduces to the standard Hopfield model, whose $s_i = \pm 1$8 capacity is irrelevant against exponentially many patterns, so no retrieval occurs.

In the $s_i = \pm 1$9 phase and the $\boldsymbol{\xi}^\mu$0 limit, the ground-state landscape becomes $\boldsymbol{\xi}^\mu$1, with a unique fixed point at $\boldsymbol{\xi}^\mu$2: retrieval is error free. Equating $\boldsymbol{\xi}^\mu$3 gives the exact full-retrieval threshold

$\boldsymbol{\xi}^\mu$4

with $\boldsymbol{\xi}^\mu$5, reproducing the threshold recently derived via the random energy model by Lucibello and Mézard. Notably, unlike polynomial DenseAMs, the transition threshold and the retrieval threshold coincide for LSE because $\boldsymbol{\xi}^\mu$6 exactly at and below the transition.

Limitations and open questions

The formalism is derived for binary neurons; extension to continuous or other discrete variables is asserted but not demonstrated. The extensive-limit analysis of the polynomial case treats cross-overlap contributions as Gaussian with variance $\boldsymbol{\xi}^\mu$7, an assumption valid by the central limit theorem in the stated scaling but not verified beyond it. The LSE treatment assumes an exponential number of stored patterns, though the method is claimed to apply to arbitrary $\boldsymbol{\xi}^\mu$8. The initial-state dependence of gradient descent at $\boldsymbol{\xi}^\mu$9 raises the open question of what dynamical mechanisms (e.g., finite-temperature stochasticity) suffice to escape the persistent $H = -N f(\mathbf{m})$0 basin, and how the basin geometry scales with $H = -N f(\mathbf{m})$1 and $H = -N f(\mathbf{m})$2.

Conclusion

The paper provides a systematic large-deviations procedure for computing exact free energy functionals of associative memory models with arbitrary cost functions. It recovers classical Hopfield results, supplies the previously unknown free energy landscape for polynomial DenseAMs of any order, and establishes that higher-order networks exhibit persistent initial-state dependence in retrieval, while for the LSE model it derives the exact full-retrieval capacity threshold $H = -N f(\mathbf{m})$3. The framework extends naturally to other architectures within the class of energy-based memories defined by $H = -N f(\mathbf{m})$4.