Hopfield-Resnet: Energy-Based Deep Architectures

• The paper introduces a Hopfield network architecture with residual connections trained by equilibrium propagation, achieving 93.92% CIFAR-10 accuracy.
• It employs residual interactions directly in the energy function, enhancing stability and enabling nearly twice the depth compared to previous EP studies.
• The design leverages local learning properties suited for neuromorphic hardware and offers a competitive alternative to backpropagation-based models.

Searching arXiv for the target paper and closely related equilibrium propagation work. Hopfield-Resnet is a deep, convolutional Hopfield network trained by equilibrium propagation (EP) in which residual (skip) connections and a clipped $\mathrm{ReLU}$ nonlinearity are used to scale EP to substantially deeper architectures (P et al., 30 Sep 2025). It was introduced in "Scaling Equilibrium Propagation to Deeper Neural Network Architectures" (P et al., 30 Sep 2025) as an architectural response to a recurring limitation of EP: prior studies had been restricted to networks containing only dense layers or relatively small architectures with a few convolutional layers followed by a final dense layer, with a significant gap in accuracy relative to similarly sized feedforward networks trained with backpropagation. In this formulation, residual interactions are incorporated directly into the Hopfield energy, and the resulting model supports networks with nearly twice the number of layers reported in prior works; the flagship configuration, Hopfield-Resnet13, achieves $93.92\%$ accuracy on CIFAR-10, approximately $3.5\%$ higher than the previous best reported EP result and comparable to a Resnet13 trained with backpropagation (P et al., 30 Sep 2025).

1. Conceptual basis and motivation

Hopfield-Resnet is defined within the equilibrium propagation framework, where learning is performed by contrasting equilibria of an energy-based dynamical system rather than by standard reverse-mode differentiation (P et al., 30 Sep 2025). The model is a layered Hopfield network with neuronal state variables $\{s^n\}_{n=0}^{N-1}$, where each $s^n$ is a tensor for convolutional layers or a vector for dense layers, and with parameters $\theta$ comprising kernels, matrices, and biases. The central architectural innovation is the integration of residual connections into this Hopfield formalism, so that skip edges contribute explicit pairwise interaction terms to the energy rather than being treated as an external feedforward modification.

The stated motivation is twofold. First, EP has attractive locality properties: parameter updates depend on locally available pre- and post-synaptic states at equilibrium, which is presented as well-suited for neuromorphic hardware (P et al., 30 Sep 2025). Second, depth had been a practical obstacle for EP-trained convolutional networks. The introduction of residual pathways and clipped activations is therefore aimed at improving stability of the relaxation dynamics and enabling substantially deeper architectures under centered equilibrium propagation (CEP) (P et al., 30 Sep 2025).

A plausible implication is that Hopfield-Resnet should be understood not merely as a residual CNN trained with an alternative objective, but as a specific energy-based construction in which residual structure alters both the equilibrium states and the local learning signals.

2. Energy function, state dynamics, and fixed points

For a baseline convolution-pooling-dense stack, the energy is defined as a sum of pairwise interaction terms between adjacent layers:

$$E(x, s; \theta) = \sum_{n=0}^{N_{\mathrm{conv}}-1} \langle s^{n+1}, \mathcal{P}( W_{n+1} \star s^n ) \rangle + \sum_{n=N_{\mathrm{conv}}}^{N_{\mathrm{tot}}-1} s^{n+1\top} W_{n+1} s^n + \text{bias/regularization terms},$$

where $W_{n+1}$ are convolution kernels or dense matrices, $\star$ denotes convolution, $\mathcal{P}$ is the pooling operator, and $93.92\%$0 is the spatial/channelwise inner product (P et al., 30 Sep 2025). In Hopfield-Resnet, residual edges are added through

$93.92\%$1

so that the total energy is $93.92\%$2 (P et al., 30 Sep 2025).

The free-phase dynamics minimize this energy by relaxing the states according to the gradient flow

$93.92\%$3

with fixed points $93.92\%$4 satisfying

$93.92\%$5

(P et al., 30 Sep 2025). In discrete time, layerwise updates apply an activation $93.92\%$6 to pre-activations. For convolutional layers without residuals, the bidirectional Hopfield update is

$93.92\%$7

and for dense layers,

$93.92\%$8