Training a Predictive Coding Network on ImageNet using Equilibrium Propagation

Abstract: Equilibrium Propagation (EP) is a physics-based training framework that has primarily been employed in energy-based models, including continuous Hopfield networks, nonlinear resistive networks and coupled phase oscillators. However, EP's practical applications have so far remained limited to relatively small-scale problems. Predictive coding networks (PCNs), another class of energy-based models rooted in computational neuroscience, are typically trained with a specialized algorithm and have likewise not yet been demonstrated at large scale. In this work, we develop an EP-based training method for PCNs which combines the centered variant of EP with a novel equilibration scheme for PCNs. Using this approach, we train a 10-layer convolutional PCN (VGG10) on full-size ImageNet, achieving 13.23\% test error rate on the top-5 classification task, close to the 12.2\% backpropagation baseline. To our knowledge, this is the first demonstration of both PCNs and EP-based training at ImageNet scale. These results significantly extend the scalability of both approaches and suggest that the primary challenges in scaling EP in other physical systems may come more from the computational properties of these systems than from inherent limitations of the EP framework.

• The paper demonstrates that nudging-based equilibrium propagation enables training of deep predictive coding networks at full ImageNet scale with competitive error rates to backpropagation.
• It showcases efficient finite difference schemes, including centered and random perturbations, that enhance gradient estimation and reduce computational cost.
• The experimental insights reveal that EP’s performance depends on optimal perturbation strength and iterations while offering flexible cost function choices for complex tasks.

Training Predictive Coding Networks at Scale via Equilibrium Propagation

Overview

The paper "Training a Predictive Coding Network on ImageNet using Equilibrium Propagation" (2606.03584) systematically advances the scalability and empirical evaluation of EP-based training in Predictive Coding Networks (PCNs), showing for the first time competitive performance at full ImageNet scale. The authors rigorously investigate the interaction between perturbation schemes and finite difference approximations in EP, demonstrating that nudging-based perturbations with centered and random schemes enable efficient training of a deep convolutional PCN (VGG10) on ImageNet—the largest yet experiment for both PCNs and EP-based learning.

Equilibrium Propagation for Energy-Based Models

EP is positioned as a contrastive learning algorithm for energy-based models governed by variational principles. By introducing a perturbation ($\beta$) into the system's energy via a cost function, EP extracts approximate gradients from differences in equilibrium states under varied boundary conditions. Critically, the framework generalizes to arbitrary cost functions and applies across a spectrum of physical and modeled systems, but previous practical implementations have been confined to modest-scale tasks. The scalability challenge arises from requirements for numerical minimization (energy function optimization to equilibrium), gradient quality (approximation error from finite differences and perturbation strength), and the uncertain expressivity of EP-compatible architectures.

Predictive Coding Networks as Neuromorphic Benchmarks

PCNs, analytically derived from belief updating under layerwise Gaussian assumptions, represent an energy-based formulation of feedforward ANNs where inference corresponds to minimization of a sum of prediction errors. The flexibility of PCNs allows for direct comparison with BP, as their feedforward equilibria recover conventional neural network outputs, but most prior work has relied on clamping-based learning techniques. EP-based nudging offers richer gradient estimation and aligns the learning objective with explicit cost function minimization, enabling scaling to deeper and more complex architectures.

Experimental Insights: EP in Large-Scale Vision

The authors deploy two architectures: VGG5 (for MNIST, CIFAR-10/100, ImageNet32) and VGG10 (for full ImageNet), examining a breadth of hyperparameters:

• Perturbation method: Nudging (energy perturbation via cost function) versus clamping (output forced to target).
• Finite difference scheme: Forward, backward, centered, random.
• Nudging strength ($\beta$) and iterations ($K$).
• Cost function: MSE and Cross-Entropy.

On CIFAR-100 and ImageNet32, nudging-based EP with centered and random schemes substantially outperforms clamping approaches, particularly as dataset complexity escalates (Figure 1).

Figure 1: Test error rates on CIFAR-100 exemplify competitive performance across perturbation and finite difference schemes; nudging-based schemes plateau at low error rates even with increased $K$ or $\beta$.

The sensitivity analysis (Figure 2) establishes a regime where EP's accuracy is robust to choices in nudging strength and nudge-phase iterations, provided perturbations are sufficient to propagate error signals fully but not excessively.

Figure 2: VGG5 on CIFAR100, demonstrating insensitivity of final error rate to $\beta$ and $K$, provided $K$ exceeds minimal threshold for equilibrium.

Full ImageNet experiments substantiate the central claim: a VGG10 PCN trained with centered EP reaches a top-5 error rate of 13.23%, nearly matching BP (12.2%), with random EP competitive (14.02%). Notably, the random scheme is advantageous in PCNs due to the vanishing gradient at the free equilibrium; it requires only one phase, halving compute cost.

Figure 3: Top-1 test error rate for VGG10 on ImageNet, illustrating the convergence profile and small performance gap between centered EP and BP.

Architectural and Algorithmic Variations

The study includes deployment of skip connections (VGG10Skip), confirming EP’s applicability to ResNet-type structures, albeit with increased nudge-phase complexity. Modified projected gradient descent (mod-PGD) further refines the nudge phase efficiency and convergence stability, especially on datasets with deeper architectures.

Figure 4: Nudge-phase equilibration using mod-PGD with $\beta=0.05$ for VGG10, depicting stabilization after sufficient iterations and validating $K=10$ as an empirically sound choice.

Comparative Evaluation and Practical Implications

Experimental comparison across multiple datasets (Table 1 in the paper) consistently finds EP competitive with BP, with performance determined more by cost function, initialization, and batch size than by the learning algorithm itself. The ability to choose arbitrary cost functions with nudging-based EP is demonstrated to be crucial on complex datasets like ImageNet32, where cross-entropy outperforms MSE and clamping.

The paper convincingly argues, via empirical and architectural comparisons, that perceived scalability limitations of EP are not intrinsic to EP itself but rather emanate from the computational properties and expressivity of the underlying physical or simulated systems.

Figure 5: Schematic of the nudging-based perturbation process in EP for PCNs, showing propagation of error signals through layers and flexibility in cost function selection.

Theoretical Implications and Future Directions

The work shifts the narrative regarding contrastive, physics-inspired training frameworks, demonstrating that EP can serve as a viable alternative to BP in large-scale, real-world tasks. The empirical evidence supports deployment in Stochastic Gradient Descent contexts, and positions PCNs as effective benchmarks for scaling energy-based training in both modeled and hardware systems.

Future directions include more nuanced analysis of EP in hardware deployment, especially as architectures become deeper (ResNet variants) and nudge-phase optimization becomes more computationally intense. The findings suggest potential for integrating EP in neuromorphic platforms, provided physical constraints can accommodate accelerated equilibrium finding. The random scheme’s single-phase gradient extraction, effective at scale, invites mechanistic exploration in other model classes where energy derivative vanishes at equilibrium.

Conclusion

This paper systematically extends both the empirical and theoretical landscape of EP and PCNs in large scale vision classification, achieving near parity with BP on full-size ImageNet and elucidating optimal configurations for practical deployment. The results suggest EP’s scalability is primarily limited by the system in which it is deployed, not by the algorithmic framework itself, and reinforce the critical role of cost function selection in high-performance energy-based learning.