Selfless Sequential Learning (1806.05421v5)

Abstract: Sequential learning, also called lifelong learning, studies the problem of learning tasks in a sequence with access restricted to only the data of the current task. In this paper we look at a scenario with fixed model capacity, and postulate that the learning process should not be selfish, i.e. it should account for future tasks to be added and thus leave enough capacity for them. To achieve Selfless Sequential Learning we study different regularization strategies and activation functions. We find that imposing sparsity at the level of the representation (i.e.~neuron activations) is more beneficial for sequential learning than encouraging parameter sparsity. In particular, we propose a novel regularizer, that encourages representation sparsity by means of neural inhibition. It results in few active neurons which in turn leaves more free neurons to be utilized by upcoming tasks. As neural inhibition over an entire layer can be too drastic, especially for complex tasks requiring strong representations, our regularizer only inhibits other neurons in a local neighbourhood, inspired by lateral inhibition processes in the brain. We combine our novel regularizer, with state-of-the-art lifelong learning methods that penalize changes to important previously learned parts of the network. We show that our new regularizer leads to increased sparsity which translates in consistent performance improvement %over alternative regularizers we studied on diverse datasets.

Insights into Selfless Sequential Learning

The paper "Selfless Sequential Learning" investigates the nuances of sequential learning, commonly known as lifelong learning (LLL), where tasks are learned in a sequence without retaining access to data from previously learned tasks. The crux of this research lies in selflessly allocating model capacity for upcoming tasks, preventing catastrophic forgetting, a common issue in traditional deep learning paradigms.

Key Concepts and Hypotheses

• Fixed Model Capacity: The research posits that learning in a fixed model capacity environment can closely resemble the learning processes of a biological brain, which is also capacity-limited.
• Sparsity in Representation: The paper argues that encouraging sparsity in neuron activations (representation level) is significantly more beneficial than focusing on parameter sparsity.
• Neural Inhibition and Lateral Inhibition: Inspired by the biological brain, lateral inhibition is used to achieve representation sparsity through neural inhibition techniques. This curtails interference between learning sequences by limiting activation overlap.

Novel Contributions

The major contribution of this work is the introduction of a regularization technique termed Sparse coding through Local Neural Inhibition and Discounting (SLNID). The SLNID regularizer penalizes neurons that are active concurrently, thus encouraging sparse activations, which helps preserve capacity for future tasks:

• Local Neighborhood Inhibition: SLNID applies inhibition locally within a layer, allowing complex tasks that require richer, more widespread activations to express sufficiently strong representations.
• Neuron Importance Discounting: SLNID integrates neuron importance measures to discount inhibition on neurons critical to previously learned tasks. This novel approach aids in protecting learned representations from interference by subsequent tasks.

Experimental Validation

Experiments performed on permuted MNIST, CIFAR-100, and Tiny Imagenet datasets demonstrate the efficacy of SLNID over other regularization techniques. SLNID shows superior performance across multiple tasks by maintaining capacity and minimizing task interference. The empirical results significantly underscore the beneficial impact on both single-task and multi-task performance without prior knowledge of subsequent tasks.

Implications and Future Directions

The findings imply potential ways of enhancing neural networks' efficiency by emulating biological learning processes. These insights pave the way for more adaptive AI systems, where dynamic task sequencing can occur with minimal retraining and lower computational overhead. This lays the foundation for future development of advanced LLL methods that feasibly tackle more diverse and complex sequences of tasks. Moreover, the incorporation of neuron importance models can refine the preservation of crucial task information over multiple learning rounds.

Conclusion

The "Selfless Sequential Learning" paper advances understanding of sequential learning within capacity-bound environments. The introduction of SLNID as a neural inhibition-inspired regularization technique is a constructive step toward more adaptive, efficient learning models capable of sequential task updates without recourse to fixed allocation strategies. The paper not only enhances theoretical understanding but also offers robust groundwork for practical application and further exploration within AI by mimicking more organic learning methodologies.