Basin-Preserving Discretizations of Modern Hopfield Retrieval Dynamics: Energy Cells, Dissipation, and the Attention Limit

Abstract: The retrieval dynamics of a modern Hopfield network is the gradient flow of a log-sum-exp energy, while the attention update is its exact difference-of-convex minimization step. We study which time discretizations preserve not only energy decay and equilibria but also basins of attraction. We introduce energy cells, connected components of sublevel sets containing one attractor and no other critical point. Our main theorem shows that every finite energy cell below the escape energy is contained simultaneously in the basin of the continuous flow, every relaxed attention map $Ψ_θ=(1-θ)\,\mathrm{id}+θ\,\mathrm{attention}$ for $0<θ<2$, and implicit Euler throughout its uniqueness regime. A parameter-uniform unit-curvature majorant yields unconditional dissipation and a monotone interpolation of each discrete step. We also derive explicit local contraction bounds near well-separated patterns, with a certified optimal slight overrelaxation; characterize proximal tunneling and overshoot beyond the preservation regimes; compare first-order error constants; establish an order barrier for scalar reparametrizations of the relaxed family; construct a second-order scalar-auxiliary-variable scheme; and extend cell preservation to damped difference-of-convex iterations in Bregman geometry, including a certified overrelaxed window under bounded asymmetry. Nine numerical campaigns test the bounds and failure mechanisms. In two-dimensional basin experiments, all observed disagreements between continuous and discrete retrieval occur above the attractor-specific numerically inferred escape level.

• The paper introduces **energy cells**, topological structures underlying each basin of attraction that are preserved by discretized Hopfield retrieval dynamics, ensuring high accuracy in retrieval tasks.
• Theoretical and experimental evidence demonstrates an overrelaxation parameter leading to faster convergence rates for retrieval tasks, providing practical insights on updating the softmax algorithm.
• The study identifies specific failure modes such as proximal tunneling and overrelaxation collapse, guiding researchers in safely implementing Hopfield dynamics.

The paper "Basin-Preserving Discretizations of Modern Hopfield Retrieval Dynamics: Energy Cells, Dissipation, and the Attention Limit" develops a numerical-analysis theory of the gradient-flow interpretation of modern Hopfield retrieval (2608.21304). Its central question is deliberately sharper than the usual stability criteria applied to time discretizations: not whether a scheme dissipates energy or preserves equilibria, but whether it preserves the assignment of queries to stored memories. The answer is a common certified basin core, described entirely by the topology of sublevel sets of the log-sum-exp energy, that is shared by the continuous flow and by an entire one-parameter family of discretizations simultaneously.

Framework and the damped attention family

Retrieval in the modern Hopfield network is the gradient flow of $$E(x) = \tfrac12\|x\|^2 - \operatorname{lse}_\beta(X^{\top}x)$$, whose exact difference-of-convex (DC) minimization step is the attention update $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$. The structural fact underlying everything else is a global quadratic majorization of curvature exactly one,

$$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$

valid uniformly in $\beta$, $N$, and pattern geometry, even though the Hessian of $E$ is unbounded below as $\beta$ grows. This asymmetry—curvature bounded above by one, unbounded from below—is exactly the DC structure of the energy, and it lets every unrelaxed gradient step inherit descent guarantees of a 1-smooth function without $E$ being 1-smooth.

Three of the four basic discretizations (explicit Euler, convex splitting, exponential integrator) collapse into the single family $\Psi_\theta(x) = (1-\theta)x + \theta X p(x)$ under reparametrizations $\theta_{\mathrm{CS}} = \Delta t/(1+\Delta t)$ and $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$0; attention is the endpoint $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$1, reached as $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$2 for both semi-implicit schemes. Fixed points of $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$3 coincide exactly with critical points of $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$4 for all $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$5. The collapse is specific to the quadratic convex part; for sparse or Fenchel–Young variants the schemes are genuinely distinct.

Unconditional dissipation and local contraction

The majorant yields unconditional per-step dissipation $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$6 for every $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$7, with no restriction on $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$8, patterns, or initialization; the inequality is an identity up to the Bregman remainder of the log-sum-exp. The same holds set-valuedly for implicit Euler at arbitrary step size, with uniqueness requiring $x \mapsto X\operatorname{softmax}(\beta X^{\top}x)$9. Real analyticity of $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$0 upgrades these to convergence of full iterates to a single critical point via standard Kurdyka–Łojasiewicz machinery.

Locally, near a well-separated pattern $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$1 with margin $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$2, softmax concentration yields a quantity $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$3 exponentially small in $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$4 controlling everything: strong convexity on the retrieval ball, contraction factors $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$5, and the certified overrelaxed optimum $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$6 with factor $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$7, strictly better than the attention factor $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$8. The implicit Euler local branch contracts with $$E(y) \le E(x) + \langle\nabla E(x), y-x\rangle + \tfrac12\|y-x\|^2,$$9 for arbitrary $\beta$0, and its $\beta$1 limit is the deep-equilibrium formulation rather than attention.

Two caveats temper this theory, which the author reports as part of the result. First, the certificate $\beta$2 is conservative: in experiments its slack over the observed spectral radius never dropped below $\beta$3, exceeding four orders of magnitude for correlated overcomplete patterns, so plain attention beat the certified overrelaxation in observed iterations at every tested $\beta$4 despite $\beta$5 being certified-optimal and exactly tight. Second, within the warm-started Picard hierarchy, closed-form factors $\beta$6 show that no allocation of inner iterations certifies a better per-softmax-evaluation factor than attention's bound.

Energy cells and basin preservation

The main theorem introduces energy cells: connected components of sublevel sets $\beta$7 containing an attractor and no other critical point, up to the attractor-specific escape energy $\beta$8. The cell-preservation theorem states that every finite cell below the escape energy lies simultaneously in the basin of the flow, of every $\beta$9 for $N$0, and of implicit Euler throughout its uniqueness regime, with no further condition on data or parameters. The mechanism is elementary: evaluating the majorant along the segment between consecutive iterates gives a monotone interpolation lemma ($N$1 keeps the bracket nonnegative); cells being connected components cannot be exited by such an interpolant; precompactness plus vanishing gradient identifies the unique limit inside the cell. No Łojasiewicz argument is needed. In the well-separated regime a sandwich estimate places each cell between concentric balls centered at the attractor whose radii differ by $N$2.

Two failure modes delimit the statement precisely. The global proximal map tunnels out of non-global cells beyond the explicit threshold $N$3, after which non-global local minimizers cease to be fixed points—as a memory this scheme answers the wrong question, though as a global optimizer it reaches low energy in one step. Explicit Euler turns the attractor into a repeller beyond $N$4, giving the basin empty interior by a Baire-category argument on the real-analytic map; this recovers the classical unit-step instability of synchronous Hopfield updates in quantitative form and shows the safety of attention ($N$5) has uniform margin in $N$6.

Temporal accuracy, second-order schemes, and cost

An order barrier shows no scalar reparametrization $N$7 reaches second order under a non-degeneracy hypothesis proved for two linearly independent patterns (conjectured generic for $N$8). Among first-order members, however, the error constants differ sharply: ETD's leading error is carried entirely by the softmax curvature, hence exponentially small on retrieval balls—at $N$9 its trajectory error was $E$0 versus $E$1 for explicit Euler, a factor of order $E$2, making first-order ETD more accurate than the second-order SAV scheme at every tested step size.

The second-order construction is a scalar-auxiliary-variable (SAV) Crank–Nicolson scheme with a resplit energy ensuring the auxiliary variable is globally defined without truncation. It admits a unique solution computable at one softmax evaluation per step, obeys the exact modified-energy law $E$3 for every step size, and converges at second order uniformly in $E$4. The caveat is substantive: only the modified energy dissipates exactly, and numerics confirm the true energy can increase by order-one amounts at large steps. A discrete-gradient benchmark dissipates the true energy exactly to roundoff at second order, but costs roughly 200–300 evaluation units per step in the reference implementation against 1 for SAV. Neither is known to admit a monotone interpolant; achieving second order, exact true-energy dissipation, a monotone interpolant, linear implicitness, and one softmax evaluation per step simultaneously remains open.

Bregman generalization

Under Legendre-type assumptions on the convex part, the damped DCA family in dual coordinates satisfies an exact Bregman identity expressing energy decrease as a sum of three nonpositive terms. Overrelaxation survives under bounded Bregman asymmetry, narrowing the window to $E$5 with $E$6 sufficient; the quadratic case recovers $E$7. Cell preservation transfers verbatim along mirror segments, including a certified overrelaxed window—the basin statement being, to the author's knowledge, absent from prior damped-DCA analyses, which concern descent and convergence only. Nonsmooth normalized models (indicator-function convex parts) fall outside this framework.

Numerical evidence

Nine campaigns test each falsifiable prediction. Campaign A confirms the dissipation inequality is numerically sharp: minimum observed-to-certified decrease ratios of 1.0000 across 38,000 pairs and 27 configurations spanning correlation and overcompleteness, confirming the unit-curvature majorant captures worst-case curvature exactly. Campaign E is the sharpest test of the central claim: comparing high-precision integration of the flow against $E$8 on a $E$9 grid produced 12,727 disagreeing nodes across four values of $\beta$0, and not one lay below the attractor-specific numerically inferred escape level. Disagreement fractions ranged from $\beta$1 at $\beta$2 to $\beta$3 at $\beta$4, hugging separatrices at moderate $\beta$5 and spreading deep into basins only near the collapse threshold—exactly where the theory permits discrepancy.

Limitations and open questions

Several boundaries are stated plainly. Above the escape energy the results localize but do not bound basin deformation; a measure-theoretic estimate of the symmetric difference as a function of $\beta$6 and $\beta$7 is open, with saddle stable manifolds as the organizing objects. The tunneling analysis leaves unresolved the intermediate window $\beta$8, where Campaign D found no tunneling despite neither result applying, suggesting the true threshold is governed by proximal-envelope geometry. The certificate constant $\beta$9 degrades precisely in the correlated overcomplete regime where modern Hopfield layers operate, and sharpening it is identified as a target. Escape levels used in Campaign E rest on a numerical, not certified, census of critical points. The genericity hypothesis behind the order barrier is unproven for $E$0, the nonsmooth extension of cell preservation is deferred, and no claim of algorithmic lower bounds is made beyond certified worst-case factors within the analyzed hierarchies.

Conclusion

The paper establishes that a well-defined part of each basin of the modern Hopfield retrieval flow—an energy cell below the escape energy—is preserved exactly, simultaneously, by the continuous dynamics, by relaxed attention at any $E$1, and by implicit Euler in its uniqueness regime, through a single structural inequality valid uniformly in inverse temperature. Failure modes are quantified explicitly (proximal tunneling, overshoot collapse), accuracy and cost are ranked per softmax evaluation, and nine numerical campaigns show the certified core respected without exception while all discrepancies occur above the escape level. The remaining gaps—quantitative basin deformation, the second-order combination problem, certificate sharpness, and the nonsmooth setting—are clearly posed rather than obscured.