Direct Feedback Alignment Provides Learning in Deep Neural Networks (1609.01596v5)

Abstract: Artificial neural networks are most commonly trained with the back-propagation algorithm, where the gradient for learning is provided by back-propagating the error, layer by layer, from the output layer to the hidden layers. A recently discovered method called feedback-alignment shows that the weights used for propagating the error backward don't have to be symmetric with the weights used for propagation the activation forward. In fact, random feedback weights work evenly well, because the network learns how to make the feedback useful. In this work, the feedback alignment principle is used for training hidden layers more independently from the rest of the network, and from a zero initial condition. The error is propagated through fixed random feedback connections directly from the output layer to each hidden layer. This simple method is able to achieve zero training error even in convolutional networks and very deep networks, completely without error back-propagation. The method is a step towards biologically plausible machine learning because the error signal is almost local, and no symmetric or reciprocal weights are required. Experiments show that the test performance on MNIST and CIFAR is almost as good as those obtained with back-propagation for fully connected networks. If combined with dropout, the method achieves 1.45% error on the permutation invariant MNIST task.

• The paper introduces direct feedback alignment (DFA), using fixed random feedback connections to train each hidden layer independently without symmetric weight constraints.
• The methodology achieves competitive results on benchmarks like MNIST, reaching a test error of 1.45% when combined with dropout techniques.
• The study advocates DFA as a biologically plausible alternative to back-propagation, inspiring new architectures that are computationally efficient and adaptable.

Direct Feedback Alignment: Towards Biologically Plausible Learning in Deep Neural Networks

The paper "Direct Feedback Alignment Provides Learning in Deep Neural Networks" presents an investigation into the viability of an alternative to the conventional back-propagation (BP) method for training artificial neural networks. This approach, termed direct feedback alignment (DFA), builds upon the principles of feedback alignment (FA), which proposes that the weight symmetry required by BP is not a strict necessity for effective learning. Instead, fixed random feedback connections can suffice to train hidden layers.

Key Methodology and Theoretical Insights

Traditionally, BP has been the dominant algorithm for training neural networks due to its ability to efficiently compute gradients necessary for updating weights. The back-propagation of error gradients layer by layer is, however, not biologically plausible and necessitates symmetric reverse-path weights, which may not naturally occur in biological systems. FA relaxes this constraint by demonstrating that random feedback weights can still guide networks towards learning optimal solutions, leveraging the network's intrinsic capability to adjust to the feedback structure.

DFA, as detailed in this paper, extends FA by connecting output errors directly to each hidden layer through fixed random feedback weights. These feedback pathways operate independently of intermediary layers, allowing each hidden layer to receive learning signals that do not depend on the weights of subsequent layers. This aligns with biological learning principles where neurons do not rely on symmetric pathways between forward and backward connections.

The theoretical underpinning of DFA is supported by Theorem \ref{theorem_fa}, demonstrating that non-zero random feedback is sufficient for descent directions conducive to learning. Furthermore, the method is robust to initial conditions, as DFA systems exhibit a capacity to progress towards zero error from zero-initialized weights.

Experimentation and Results

The paper validates the DFA approach through a set of rigorous experiments on benchmark datasets such as MNIST and CIFAR-10/100. In testing conditions, DFA achieves competitive performance relative to BP and FA. Specifically, notable results include a test error of 1.45% on MNIST when DFA is combined with dropout techniques, suggesting that this method remains viable even in deep architectures.

While test performance marginally trails BP, DFA's ability to drive the training error to zero across multiple deep network configurations underscores its potential as a biologically inspired alternative. Notably, DFA successfully trains networks with long pathways, which BP occasionally struggles with when using basic initialization methods.

Implications and Future Directions

The implications of DFA are multifaceted. The algorithm represents a step towards reconciling the discrepancy between artificial neural network training methods and hypothesized biological learning mechanisms, offering a conceptual framework more reminiscent of neural signal processing in the brain. Practically, the elimination of symmetric weight dependencies and disconnected feedback paths could inspire new neural network architectures and training algorithms that are less computationally intense and more adaptable to varied hardware implementations.

Looking forward, further exploration is warranted to refine DFA performance on challenging datasets and extend its applicability to more complex architectures, such as those used in state-of-the-art computer vision and natural language processing systems. Future research might also consider integrating DFA with unsupervised and reinforcement learning techniques to expand its capabilities beyond supervised learning paradigms.

In conclusion, this paper presents DFA as a promising methodology for neural network training, emphasizing the greater flexibility and biological inspiration it offers compared to conventional methods, and providing a robust foundation for future enhancements within biologically plausible machine learning domains.