Testing the Role of Diagonal Interactions in High-Order Hopfield Models via Dynamical Mean-Field Theory

Abstract: High-order extensions of the Hopfield model are known to exhibit dramatically enhanced storage capacity at equilibrium, while their dynamical retrieval properties remain less well understood. In our previous work, we carried out a dynamical mean-field theory (DMFT) analysis of the Krotov--Hopfield-type dense associative memory and found that the transition between successful and failed retrieval is accompanied by pronounced slow dynamics. As a consequence, the effective basin of attraction observed in numerical simulations extends well beyond that predicted by equilibrium statistical mechanics. A natural hypothesis is that this discrepancy originates from diagonal (self-interaction) contributions in the Krotov--Hopfield model, which generate a large number of lower-order interaction terms and may induce glassy relaxation near the retrieval boundary. To test this hypothesis, we analyze an alternative high-order associative memory model, namely the Abbott--Arian-type $p$-body Hopfield model, in which such diagonal contributions are absent by construction. Using dynamical mean-field theory, we derive an effective single-site process together with closed macroscopic equations governing the retrieval dynamics. Our analysis reveals that both slow dynamics and a substantial enlargement of the apparent basin of attraction persist even in this model. These results indicate that the dynamical slowdown near the retrieval boundary cannot be attributed primarily to diagonal self-interaction effects, but instead originates from intrinsic properties of high-order interactions.

• The paper demonstrates that slow glassy dynamics and enlarged retrieval basins are intrinsic to high-order p-body interactions, independent of diagonal self-interaction effects.
• It employs a DMFT path-integral formulation to reduce the high-dimensional system to an effective non-Markovian scalar process governing retrieval dynamics.
• Numerical simulations validate the DMFT predictions by revealing extended metastable states and notable discrepancies with static replica theory near critical capacity.

Dynamical Slowdown and Diagonal Interactions in High-Order Hopfield Models: A DMFT Analysis

Overview

This paper conducts a rigorous investigation into the retrieval dynamics of high-order Hopfield models, with specific attention to the Abbott–Arian-type $p$-body Hopfield model, using dynamical mean-field theory (DMFT) (2604.03115). The central question is whether the slow, glassy dynamics and the enlarged retrieval basins observed near critical capacity in prior studies of the Krotov–Hopfield variant can be attributed primarily to diagonal (self-interaction) terms—terms that generate lower-order effective interactions—or if these dynamical phenomena are intrinsic to high-order interactions themselves.

High-Order Hopfield Model Formulations

Two major high-order Hopfield model formulations are considered:

• Krotov–Hopfield (KH) model: Incorporates diagonal (self-interaction) terms due to the inclusion of repeated indices in $p$-body interactions, producing a mixture of lower- and higher-order effective interactions.
• Abbott–Arian (AA) model: Strictly excludes diagonal interactions by construction, enforcing distinct indices in every interaction term and thus offers a clean $p$-body model for disentangling the effects of diagonal contributions.

The authors focus their DMFT-based analysis and numerical experiments exclusively on the AA-type model to determine if glassy dynamics and basin enlargement are present even without diagonal terms.

Analytical Approach: Dynamical Mean-Field Theory

A path-integral formulation of DMFT is employed to derive the effective single-site stochastic process that governs retrieval. The core dynamical variable is the overlap $m(t)$ with a stored pattern, and dynamics are tracked under synchronous, zero-temperature updates.

Key features of the DMFT derivation include:

• Systematic elimination of diagonal contributions in the interaction sums via Hermite polynomial expansions.
• Reduction of the high-dimensional system to an effective scalar process with non-Markovian colored noise and history-dependent self-interaction kernels.
• Closed-form macroscopic equations for $m(t)$, correlation functions, and associated covariances.

By analyzing the Abbott–Arian model in this exacting fashion, the work isolates dynamical effects of strictly high-order interactions, independent of diagonal-induced lower-order terms.

Numerical Results: Retrieval Dynamics and Basins

Overlap Evolution and Dynamical Transition

Direct comparison between DMFT predictions and finite-size Monte Carlo simulations demonstrates close agreement in the evolution of the order parameter $m(t)$. Retrieval transitions between successful and failed phases are characterized by dramatic slowdowns near the dynamical threshold, with pronounced metastable plateaus and nontrivial relaxation timescales.

Figure 1: Time evolution of the overlap $m(t)$ for $p=3$ over several loading levels $\alpha$; DMFT prediction vs direct simulation.

Basins of Attraction and Capacity Comparisons

By systematically varying initial overlap $m(0)$ and load $p$0, the authors produce phase diagrams of retrieval success rates after fixed finite times, mapping out the effective basins of attraction. Results reveal several notable features:

• The dynamical basin of attraction near critical capacity is substantially larger than that predicted by equilibrium (replica) theory.
• There exists an extended region in parameter space where trajectories have not yet relaxed to failure (zero overlap) even though static analysis predicts no retrieval phase, indicating extremely slow relaxation.

Figure 2: Retrieval outcome as a function of initial overlap and loading level for $p$1; the gradual transitions delineate slow dynamics near the retrieval boundary.

Finite-Time Effects and Dynamical-Static Discrepancies

Longer observation times ($p$2) reduce the region of apparent retrieval, shifting observed retrieval-failure transitions toward the static (replica) limits, but do not eliminate the discrepancy. Even for $p$3, retrieval can persist well beyond the predicted storage capacity.

Figure 3: Final overlap $p$4 as a function of load $p$5 for fixed $p$6 and $p$7 over different time horizons $p$8; static (RS, 1RSB) thresholds are indicated.

Implications and Theoretical Insights

Absence of Diagonal-Term Dependence

A critical claim of the paper is that glassy retrieval dynamics, slow relaxation near the critical point, and dynamical basin enlargement persist in the AA-type model, where diagonal interactions are rigorously excluded. That is, these phenomena are intrinsic to high-order $p$9-body interaction structure and cannot be ascribed solely to the presence of diagonal-induced lower-order or self-interaction effects. The findings contradict hypotheses positing a dominant diagonal-term effect, shifting the focus to the generic complexity of high-order associative memory dynamics.

Rugged Energy Landscapes and Glassiness

Both numerical and analytical results support the picture of a highly rugged energy landscape with an abundance of metastable states and long-lived traps near the retrieval boundary, echoing classic results for glassy systems. The pronounced finite-time discrepancies with static (replica or 1RSB) predictions imply that true dynamical convergence is extremely slow, and equilibrium capacity may only be approached on timescales inversely related to system size and interaction order.

Reconciliation with Replica-Symmetry Breaking

The findings motivate further investigation into the connections and divergences between DMFT-based dynamical approaches and replica theory, particularly higher-step RSB analyses. The slow relaxation and apparent discrepancies suggest that nonergodic, glassy states dominate at high $p$0 and near critical load.

Broader Impact and Future Work

The conclusions extend to the design of high-capacity neural associative memories, highlighting robust dynamical limitations beyond static storage bounds. Future research avenues include refining DMFT to incorporate higher RSB steps, extending analysis to finite temperature or asynchronous update rules, and exploring the effect in other high-order network models.

Conclusion

The study provides a comprehensive DMFT-based analysis of the retrieval dynamics in the high-order AA-type Hopfield model, demonstrating that slow dynamics and enlarged attraction basins are not artifacts of diagonal self-interactions but stem from the intrinsic complexity of high-order associative memory models. These results refine our understanding of the interplay between static capacity, basins of attraction, and relaxation dynamics in neural network memory systems, underlining the necessity of dynamical, rather than purely static, perspectives for assessing performance in high-capacity network architectures.