Using Hindsight to Anchor Past Knowledge in Continual Learning (2002.08165v2)

Abstract: In continual learning, the learner faces a stream of data whose distribution changes over time. Modern neural networks are known to suffer under this setting, as they quickly forget previously acquired knowledge. To address such catastrophic forgetting, many continual learning methods implement different types of experience replay, re-learning on past data stored in a small buffer known as episodic memory. In this work, we complement experience replay with a new objective that we call anchoring, where the learner uses bilevel optimization to update its knowledge on the current task, while keeping intact the predictions on some anchor points of past tasks. These anchor points are learned using gradient-based optimization to maximize forgetting, which is approximated by fine-tuning the currently trained model on the episodic memory of past tasks. Experiments on several supervised learning benchmarks for continual learning demonstrate that our approach improves the standard experience replay in terms of both accuracy and forgetting metrics and for various sizes of episodic memories.

Using Hindsight to Anchor Past Knowledge in Continual Learning

The paper "Using Hindsight to Anchor Past Knowledge in Continual Learning" by Chaudhry et al. addresses the critical issue of catastrophic forgetting in continual learning. This problem arises when neural networks, as they adapt to new data distributions over time, lose previously acquired knowledge. Traditional approached this challenge by relying heavily on experience replay techniques, which involve re-training on past data samples stored in a small episodic memory. However, these methods do not fully bridge the performance gap between sequential learning and the oracle scenario where all data is available simultaneously.

In this work, the authors propose a novel augment to experience replay, called Hindsight Anchor Learning (HAL), that introduces an anchoring objective. This involves a bilevel optimization framework ensuring that while updating the model on new tasks, predictions remain invariant on strategically chosen 'anchor points’ from past tasks. These points are determined via a gradient-based optimization method to maximize retained knowledge by learning their forgetting loss, defined as the change in prediction before and after fine-tuning.

Methodology and Technical Contributions

HAL employs nested optimization. The primary update aims to minimize the current task's loss while accounting for past task anchors’ stability. The process can be detailed as follows:

1. Temporary Update: Models receive a preliminary parameter update based on both current task data and samples from episodic memory.
2. Anchoring: A nested step ensures consistency in predictions at learned anchor points across tasks by embedding forgetting losses into the objective.
3. Anchor Point Selection: Crucially, these anchors are optimized iteratively, once training for a task completes, through fine-tuning temporary models using past tasks’ episodic memory to simulate future forgetting.

Experimental Evaluation

Chaudhry et al. perform extensive experiments on benchmarks like Permuted MNIST, Rotated MNIST, Split CIFAR-100, and Split miniImageNet. They demonstrate HAL's superiority by reporting up to a 7.5% improvement in accuracy and a 23% reduction in forgetting compared to standard experience replay methods. Moreover, the results hold consistently across varied sizes of episodic memory, illustrating HAL’s robustness and efficiency.

Multi-task and clone-and-finetune scenarios serve as upper-bound baselines, showcasing HAL achieves closer performance to these ideal scenarios than prior methods, making it a practical step forward in continual learning.

Implications and Future Directions

This work implies significant advancements for real-world AI applications that encounter non-stationary data distributions, such as robotics and autonomous driving, where constant model updates are crucial. The improved knowledge retention capabilities mean models can operate effectively despite continuous learning demands.

Future work could explore dynamic anchor adaptation as tasks evolve, potential automatization of anchor selection thresholds, and extending this approach to unsupervised or reinforcement learning environments where data distributions may change more drastically and unpredictably.

Overall, HAL introduces a strategic and effective method for addressing catastrophic forgetting, making it a potent addition to the continual learning toolkit. Its concept of leveraging hindsight for knowledge stabilization could serve as an inspiration for adjacent areas in AI research.