Remembering for the Right Reasons: Explanations Reduce Catastrophic Forgetting (2010.01528v2)

Abstract: The goal of continual learning (CL) is to learn a sequence of tasks without suffering from the phenomenon of catastrophic forgetting. Previous work has shown that leveraging memory in the form of a replay buffer can reduce performance degradation on prior tasks. We hypothesize that forgetting can be further reduced when the model is encouraged to remember the \textit{evidence} for previously made decisions. As a first step towards exploring this hypothesis, we propose a simple novel training paradigm, called Remembering for the Right Reasons (RRR), that additionally stores visual model explanations for each example in the buffer and ensures the model has "the right reasons" for its predictions by encouraging its explanations to remain consistent with those used to make decisions at training time. Without this constraint, there is a drift in explanations and increase in forgetting as conventional continual learning algorithms learn new tasks. We demonstrate how RRR can be easily added to any memory or regularization-based approach and results in reduced forgetting, and more importantly, improved model explanations. We have evaluated our approach in the standard and few-shot settings and observed a consistent improvement across various CL approaches using different architectures and techniques to generate model explanations and demonstrated our approach showing a promising connection between explainability and continual learning. Our code is available at \url{https://github.com/SaynaEbrahimi/Remembering-for-the-Right-Reasons}.

• The paper demonstrates that integrating explanation loss to align saliency maps across tasks effectively reduces catastrophic forgetting.
• It extends traditional experience replay by incorporating a dual-memory system for both raw data and model explanations.
• Empirical evaluations reveal that RRR improves predictive accuracy and explanation fidelity, especially with Grad-CAM on ResNet18 architectures.

Exploring the Nexus of Explainability and Continual Learning: Introducing RRR for Remembering with the Right Reasons

Overview

Continual Learning (CL) stands as a pivotal challenge in the development of AI systems capable of learning from sequential tasks without succumbing to catastrophic forgetting. A novel approach, Remembering for the Right Reasons (RRR), is introduced to enhance the robustness of models against forgetting by leveraging explainability—the saliency maps of predictions. By ensuring that models remember the underlying reasons for their predictions across tasks, RRR not only addresses forgetting but also enriches model explanations, proposing a fertile ground for future research at the intersection of explainability and continual learning.

Remembering for the Right Reasons (RRR)

Conceptual Framework

RRR operates under the hypothesis that a model's retention of previously learned tasks can be significantly improved by maintaining consistency in the explanations for its predictions. To implement this, RRR extends the conventional experience replay mechanisms by incorporating a dual memory system: one for the raw samples ($\mathcal{M}^{\text{rep}$) and another for storing the corresponding saliency maps ($\mathcal{M}^{\text{RRR}$). These saliency maps capture the "reasons" — the visual evidence — based on which decisions were made. The core of RRR is an explanation loss formulated to minimize the disparity between current and initially stored saliency maps, thereby anchoring the model's explanatory focus over successive learning phases.

Integration with Existing CL Approaches

The versatility of RRR is underscored by its applicability across a spectrum of CL strategies, from memory-based to regularization-based methods. Its efficacy is evaluated using popular models and techniques for generating model explanations, including vanilla backpropagation, SmoothGrad, Guided Backpropagation, and Grad-CAM, across different network architectures. Notably, the incorporation of RRR does not necessitate task IDs at test time, making it a practical enhancement to current CL methodologies.

Experimental Insights

Quantitative Improvements

Empirical results demonstrate RRR's capacity to mitigate forgetting and improve the accuracy of model predictions in both standard and few-shot class incremental learning settings. Tests conducted on CIFAR100, ImageNet100, and Caltech-UCSD Birds 200 datasets reveal consistent performance enhancements across varying CL approaches and explanation techniques, with Grad-CAM combined with ResNet18 architecture showing particular promise due to its efficient memory usage and effectiveness in preserving saliency-based explanations.

Enhanced Model Explanations

The qualitative analysis of saliency maps before and after the application of RRR reveals a marked improvement in the stability and relevance of the visual explanations over successive tasks. Models trained with RRR exhibit a stronger adherence to the initial "reasons" for their decisions, as demonstrated by more focused attention on relevant image regions, even as new tasks are introduced. This sustained explanation fidelity suggests that RRR not only aids in retaining knowledge but also in maintaining interpretability and trustworthiness of model predictions.

Looking Forward

Theoretical and Practical Implications

The introduction of RRR opens several avenues for future exploration. Theoretically, it prompts a reevaluation of the role of explainability in enhancing model robustness against forgetting, suggesting that the way a model learns to "see" may be as crucial as what it learns. Practically, the improvement in model explanations has implications for the interpretability and accountability of AI systems, particularly in applications requiring transparency about the decision-making process.

Future Directions

Further research could explore the optimization of memory usage and computational efficiency in storing and retrieving saliency maps, as well as the adaptability of RRR to a wider range of tasks and real-world scenarios. Another promising direction is the investigation of other forms of explanations beyond visual saliency maps, potentially extending RRR's applicability to non-visual tasks and modalities.

Conclusion

RRR represents a significant step forward in addressing catastrophic forgetting in continual learning by intertwining the principles of explainability. By encouraging models to remember the right reasons for their predictions, RRR not only enhances performance across sequential tasks but also elevates the quality and stability of model explanations. This dual benefit underscores the potential of integrating explainability into the foundational architecture of AI systems, paving the way for more resilient and interpretable models.