Online Continual Learning in Image Classification: An Empirical Survey (2101.10423v4)

Abstract: Online continual learning for image classification studies the problem of learning to classify images from an online stream of data and tasks, where tasks may include new classes (class incremental) or data nonstationarity (domain incremental). One of the key challenges of continual learning is to avoid catastrophic forgetting (CF), i.e., forgetting old tasks in the presence of more recent tasks. Over the past few years, many methods and tricks have been introduced to address this problem, but many have not been fairly and systematically compared under a variety of realistic and practical settings. To better understand the relative advantages of various approaches and the settings where they work best, this survey aims to (1) compare state-of-the-art methods such as MIR, iCARL, and GDumb and determine which works best at different experimental settings; (2) determine if the best class incremental methods are also competitive in domain incremental setting; (3) evaluate the performance of 7 simple but effective trick such as "review" trick and nearest class mean (NCM) classifier to assess their relative impact. Regarding (1), we observe iCaRL remains competitive when the memory buffer is small; GDumb outperforms many recently proposed methods in medium-size datasets and MIR performs the best in larger-scale datasets. For (2), we note that GDumb performs quite poorly while MIR -- already competitive for (1) -- is also strongly competitive in this very different but important setting. Overall, this allows us to conclude that MIR is overall a strong and versatile method across a wide variety of settings. For (3), we find that all 7 tricks are beneficial, and when augmented with the "review" trick and NCM classifier, MIR produces performance levels that bring online continual learning much closer to its ultimate goal of matching offline training.

An Empirical Survey of Online Continual Learning in Image Classification

The paper "Online Continual Learning in Image Classification: An Empirical Survey" provides a comprehensive comparative analysis of various methods and techniques in the field of online continual learning (CL). The paper explores the challenges faced when learning from streaming data, focusing particularly on preventing catastrophic forgetting (CF) in image classification tasks. The authors conduct experiments using multiple datasets, including class incremental and domain incremental setups, to address varying data characteristics.

Key Findings and Methodological Analysis

1. Method Comparison Across Incremental Learning Scenarios:
   • The survey compares several state-of-the-art methods such as Maximally Interfered Retrieval (MIR), Incremental Classifier and Representation Learning (iCaRL), and GDumb.
   • The paper highlights that iCaRL maintains competitiveness when the memory buffer is constrained, leveraging a specialized nearest class mean (NCM) classifier to mitigate class imbalance effects.
   • GDumb, notable for its simplistic approach of using a balanced memory buffer, shows remarkable performance in medium-sized datasets but falters in domain incremental settings.
   • MIR stands out as a robust method across multiple scenarios, emphasizing its capability to adapt to both class and domain incremental challenges effectively.
2. Empirical Evaluation and Theoretical Implications:
   • The work identifies CF as a pivotal challenge primarily arising from the recency bias towards learning new classes. This bias is exacerbated by the imbalance in class representations within the memory.
   • The authors confirm the efficacy of replay mechanisms and the importance of direct memory sample utilization, showcasing that methods employing knowledge distillation alone struggle in larger scale tasks.
   • The experimental findings also suggest that while regularization techniques like EWC++ aim to prevent forgetting, they often introduce gradient stability issues, limiting their effectiveness.
3. Tricks for Enhanced Performance:
   • A notable contribution is the evaluation of simple yet effective tricks that augment memory-based methods. These include techniques such as the "review" trick and the use of NCM classifiers, which help bring performance closer to offline learning benchmarks.
   • The "review" trick, in particular, offers significant improvements by fine-tuning models at key training stages to balance class distributions in memory.
4. Implications for Future AI Developments:
   • The adaptability and efficiency of MIR reflect its potential as a versatile CL method suitable for diverse and realistic environments, aligning with the growing interest in deploying CL systems on resource-constrained edge devices.
   • The necessity for raw-data-free approaches underscores a shift towards more privacy-preserving CL solutions, which can work without storing sensitive image data.
   • The paper points out meta-learning as a promising direction for future research, offering frameworks that enhance model generalization and robustness across a broad set of tasks.

Conclusion and Future Outlook

The paper provides a crucial empirical evaluation of current methodologies, highlighting the importance of adapting both new and existing models to address class and domain nonstationarities in online continual learning setups. While the current best-performing methods rely heavily on memory storage and replay strategies, the future may see a convergence towards more sophisticated architectures that seamlessly balance stability and plasticity. This paper serves as a resourceful guide for researchers aiming to develop advanced CL systems capable of operating under practical constraints and dynamically changing environments. As AI systems become more prevalent in edge computing contexts, these insights will be invaluable in shaping the future landscape of continual learning.