Benchmarking Predictive Coding Networks -- Made Simple

Abstract: In this work, we tackle the problems of efficiency and scalability for predictive coding networks (PCNs) in machine learning. To do so, we propose a library, called PCX, that focuses on performance and simplicity, and use it to implement a large set of standard benchmarks for the community to use for their experiments. As most works in the field propose their own tasks and architectures, do not compare one against each other, and focus on small-scale tasks, a simple and fast open-source library and a comprehensive set of benchmarks would address all these concerns. Then, we perform extensive tests on such benchmarks using both existing algorithms for PCNs, as well as adaptations of other methods popular in the bio-plausible deep learning community. All this has allowed us to (i) test architectures much larger than commonly used in the literature, on more complex datasets; (ii)~reach new state-of-the-art results in all of the tasks and datasets provided; (iii)~clearly highlight what the current limitations of PCNs are, allowing us to state important future research directions. With the hope of galvanizing community efforts towards one of the main open problems in the field, scalability, we release code, tests, and benchmarks. Link to the library: https://github.com/liukidar/pcx

• The paper introduces PCX, an open-source library that streamlines the training and benchmarking of predictive coding networks using JAX for enhanced speed.
• The paper benchmarks PCNs on standard computer vision tasks, achieving competitive results on datasets such as CIFAR100 and Tiny ImageNet.
• The paper’s analysis highlights scalability challenges in PCNs, urging further development in optimization and energy propagation techniques.

An Analysis of "Benchmarking Predictive Coding Networks -- Made Simple"

This paper addresses the fundamental issues surrounding the efficiency and scalability of Predictive Coding Networks (PCNs) in the context of machine learning. The authors introduce PCX, an open-source library designed to facilitate the training of PCNs, and provide a substantial suite of benchmarks. This framework aims to standardize tasks and architectures in the field, ensuring comparability across different research efforts.

Key Contributions

The paper is structured around three central contributions: tool development, benchmarking, and analysis.

• Tool Development: PCX is designed to streamline the process of implementing and experimenting with PCNs. It leverages JAX for computational efficiency, offering a syntax that parallels common deep learning frameworks like PyTorch, thus reducing the learning curve for practitioners. A notable feature of the library is its support for JAX's Just-In-Time (JIT) compilation, enhancing the execution speed of PC networks considerably.
• Benchmarking: By establishing a common framework, the authors enable a direct comparison of results across disparate studies. The benchmark suite covers standard computer vision tasks such as image classification and generation. The proposed models and datasets range in complexity, providing a gradient for researchers to test their algorithms, from simple feedforward networks to more intricate convolutional models.
• Analysis: This portion of the work provides a comparative study that includes various hyperparameters and PC algorithms across multiple tasks. The study benchmarks standard PC, incremental PC (iPC), PC with Langevin dynamics, and nudged PC algorithms. Notably, the paper claims state-of-the-art performance for PCNs on several complex datasets like CIFAR100 and Tiny ImageNet, positioning PCNs as viable alternatives to backpropagation in these contexts.

Theoretical and Practical Implications

The research elucidates critical insights into the scalability challenges facing PCNs. Despite achieving promising results, the paper acknowledges that further advancements are necessary to match the scalability witnessed in traditional backpropagation-based approaches. This includes addressing energy propagation issues within deep networks, which currently present a barrier to effectively training larger models.

A practical outcome of the work is the demonstration of PC as a biologically plausible alternative to backpropagation, particularly in its local computation structure. The algorithms evaluated display comparable performance on smaller tasks and show competitive results on more demanding datasets. The work suggests that future efforts should focus on improving these outcomes, particularly in scaling up architectures akin to ResNets.

Future Directions

Future research is expected to focus on several areas highlighted by this paper. There is a clear need for enhancements in:

• Optimization Techniques: The study points out that while Adam proves effective for weight optimization, there is an operational instability observed in wider networks. This underlines the need for optimization techniques tailored to PCNs' unique training dynamics.
• State Initialization and Energy Propagation: The analysis indicates a disproportion in energy distribution within the layers of PCNs. Addressing this could lead to better PCN depth scalability.
• Algorithmic Variations and Extensions: The results with nudging and Monte Carlo PC suggest that algorithmic innovations can lead to performance boosts, but they also highlight that such strategies require more comprehensive exploration.

Overall, this paper presents PCX not only as a tool but as a catalyst to foster collaborative standardization and scalability in PCN research. The insights derived from this work lay a solid groundwork for expanding PCNs' applicability in machine learning, aiming toward alignment with modern demands in robustness and computational efficiency.