Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures (1807.04587v2)

Abstract: The backpropagation of error algorithm (BP) is impossible to implement in a real brain. The recent success of deep networks in machine learning and AI, however, has inspired proposals for understanding how the brain might learn across multiple layers, and hence how it might approximate BP. As of yet, none of these proposals have been rigorously evaluated on tasks where BP-guided deep learning has proved critical, or in architectures more structured than simple fully-connected networks. Here we present results on scaling up biologically motivated models of deep learning on datasets which need deep networks with appropriate architectures to achieve good performance. We present results on the MNIST, CIFAR-10, and ImageNet datasets and explore variants of target-propagation (TP) and feedback alignment (FA) algorithms, and explore performance in both fully- and locally-connected architectures. We also introduce weight-transport-free variants of difference target propagation (DTP) modified to remove backpropagation from the penultimate layer. Many of these algorithms perform well for MNIST, but for CIFAR and ImageNet we find that TP and FA variants perform significantly worse than BP, especially for networks composed of locally connected units, opening questions about whether new architectures and algorithms are required to scale these approaches. Our results and implementation details help establish baselines for biologically motivated deep learning schemes going forward.

• The paper introduces modified variants of TP and FA that eliminate conventional backpropagation in key layers to enhance neural plausibility.
• It rigorously compares performance on fully-connected versus locally-connected networks, emphasizing the critical role of weight sharing.
• The study reveals that while TP and FA variants perform well on simple datasets like MNIST, they significantly underperform on complex datasets like ImageNet.

Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

The paper "Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures" explores the potential of biologically-inspired algorithms, such as Target Propagation (TP) and Feedback Alignment (FA), to effectively scale in deep learning tasks. Inspired by the limitations of the backpropagation algorithm (BP) in biological contexts, the paper evaluates these biologically-motivated algorithms on datasets and architectures that demand substantial learning capacity, including MNIST, CIFAR-10, and ImageNet.

Overview of Key Contributions

1. Algorithms:
   • The research investigates several biologically-motivated algorithms, particularly focusing on TP and FA, and their variants. Notably, variants of Difference Target Propagation (DTP) are introduced, aimed at eliminating the backpropagation component in the penultimate layer, addressing neural plausibility concerns.
2. Architectural Testing:
   • The performance of TP and FA algorithms is evaluated both in fully-connected and locally-connected networks. The latter architecture seeks to mimic the locality of biological neural networks without relying on convolutional weight sharing, pivotal for artificial networks' success but less biologically plausible.
3. Scalability on Datasets:
   • The paper rigorously tests these algorithms across different datasets, starting from simpler datasets like MNIST, moving to more complex ones like CIFAR-10, and ultimately to the significantly more challenging ImageNet. This progression allows a relative comparison of these algorithms' scalability.

Key Findings and Implications

• Performance Comparison: The results indicate that biologically-motivated algorithms often fall short of traditional backpropagation, especially as the complexity of datasets increases. While some TP and FA variants achieve competitive performance on simpler tasks like MNIST, they significantly underperform on more challenging datasets such as ImageNet.
• The Limitations of TP and FA: On datasets requiring deeper architecture and complex representation, such as ImageNet, the biologically-inspired approaches failed to match BP's performance, underscoring their limitations in scalability and practicality in their current form.
• Importance of Weight Sharing: The comparison between convolutional and locally-connected architectures elucidated the importance of weight sharing, prevalent in convolutional layers, as a critical factor in achieving state-of-the-art results in large-scale image recognition tasks beyond the mere algorithms used.

Conclusions and Future Directions

Given the findings, the paper suggests several pathways forward. First, the need to enhance existing algorithms or develop new ones that align with biological plausibility while scaling effectively is evident. Despite the inherent obstacles, advancing this line of research may offer insights into brain inspiration for machine learning and AI development. Secondly, incorporating additional biologically relevant factors, like various learning mechanisms and evolutionary priors, might unlock potential benefits in these algorithms.

The paper serves as a foundational baseline in the exploration of biologically-motivated deep learning algorithms, highlighting the necessity for continued research to bridge the performance gap between biologically inspired learning methods and traditional BP. Future studies might explore hybrid algorithms or further modifications to TP and FA to align more closely with biological constraints while achieving robust performance comparable to backpropagation on large-scale, challenging datasets.

In essence, while TP and FA provide incremental insights into how learning might proceed in neural systems, their current implementations are not yet sufficient to rival the computational power and effectiveness of BP, especially in complex tasks. However, continuing to explore these avenues holds promise for both advancing AI and understanding the biological learning processes.