Adding Gradient Noise Improves Learning for Very Deep Networks (1511.06807v1)

Abstract: Deep feedforward and recurrent networks have achieved impressive results in many perception and language processing applications. This success is partially attributed to architectural innovations such as convolutional and long short-term memory networks. The main motivation for these architectural innovations is that they capture better domain knowledge, and importantly are easier to optimize than more basic architectures. Recently, more complex architectures such as Neural Turing Machines and Memory Networks have been proposed for tasks including question answering and general computation, creating a new set of optimization challenges. In this paper, we discuss a low-overhead and easy-to-implement technique of adding gradient noise which we find to be surprisingly effective when training these very deep architectures. The technique not only helps to avoid overfitting, but also can result in lower training loss. This method alone allows a fully-connected 20-layer deep network to be trained with standard gradient descent, even starting from a poor initialization. We see consistent improvements for many complex models, including a 72% relative reduction in error rate over a carefully-tuned baseline on a challenging question-answering task, and a doubling of the number of accurate binary multiplication models learned across 7,000 random restarts. We encourage further application of this technique to additional complex modern architectures.

• The paper demonstrates that adding annealed Gaussian noise to gradients enables successful training of deep networks, achieving up to a 72% error reduction in challenging tasks.
• It shows that the method improves robustness and convergence in 20-layer fully-connected networks and other architectures despite poor parameter initialization.
• Empirical results across models like End-To-End Memory Networks and Neural GPUs highlight the technique’s practical benefits in reducing overfitting and enhancing performance.

Adding Gradient Noise Improves Learning for Very Deep Networks

The paper "Adding Gradient Noise Improves Learning for Very Deep Networks," authored by Arvind Neelakantan and colleagues, explores a straightforward and low-overhead technique to enhance the learning process of deep neural networks by introducing gradient noise. This method, characterized by its simplicity and ease of implementation, presents a strategic augmentation to conventional optimization techniques.

Overview

The paper explores the challenges of optimizing deep networks, particularly those with complex architectures such as Neural Turing Machines and Memory Networks, which are employed in tasks like question answering and algorithm learning. The authors present their approach of injecting annealed Gaussian noise into the gradient during the training phase as a promising solution to these challenges.

This technique demonstrates notable benefits, including a reduction in training loss and the mitigation of overfitting. A salient outcome is the method's ability to enable successful training of a 20-layer deep fully-connected network using standard gradient descent, even from poor initializations. The reported results are compelling: a 72% relative reduction in error rate in a challenging question-answering task and a significant performance boost across 7,000 random restarts for binary multiplication models.

Experimental Highlights

The researchers conducted a series of experiments across various deep architectures:

• Deep Fully-Connected Networks: In experiments on MNIST with 20-layer networks, the addition of gradient noise improved average and best test accuracies, showcasing resilience to poor initialization and challenging gradient landscapes.
• End-To-End Memory Networks: The technique provided gains in both training and validation errors, particularly benefiting scenarios where models are trained without warmstarting.
• Neural Programmer: Significant improvements were observed on tasks involving single and multiple columns in tables. Models trained with gradient noise showed better robustness against varying initialization conditions.
• Neural Random-Access Machines (NRAM): For tasks like finding the k-th element in a linked list, the addition of noise bolstered training robustness, significantly increasing the rate of successful model runs.
• Convolutional Gated Recurrent Networks (Neural GPUs): The findings are particularly striking in the binary multiplication task, where incorporating gradient noise yielded better performance across numerous random trials, evidenced by a more than twofold increase in the number of models achieving less than 1% error.

Implications and Future Directions

The research contributes a practical tool for improving optimization in very deep networks, potentially impacting a wide range of applications involving complex model architectures. While the empirical evidence is robust, it invites further formal analysis into the underlying mechanisms by which gradient noise facilitates better exploration of the parameter space.

The ease of implementation makes this technique a valuable addition to the arsenal of neural network practitioners, particularly when dealing with optimization difficulties inherent in deep networks. Future work could explore combining gradient noise with other optimization strategies, potentially leading to even more effective algorithms for training deep learning models in challenging domains.

Conclusion

In conclusion, the paper delivers a rigorous exploration of gradient noise as a mechanism to enhance learning in very deep networks. With robust empirical support, it opens avenues for further research and application development in deep neural networks, positioning gradient noise as a viable technique for practitioners facing optimization hurdles in complex architectures.