A Bag of Tricks for Few-Shot Class-Incremental Learning (2403.14392v2)

Abstract: We present a bag of tricks framework for few-shot class-incremental learning (FSCIL), which is a challenging form of continual learning that involves continuous adaptation to new tasks with limited samples. FSCIL requires both stability and adaptability, i.e., preserving proficiency in previously learned tasks while learning new ones. Our proposed bag of tricks brings together six key and highly influential techniques that improve stability, adaptability, and overall performance under a unified framework for FSCIL. We organize these tricks into three categories: stability tricks, adaptability tricks, and training tricks. Stability tricks aim to mitigate the forgetting of previously learned classes by enhancing the separation between the embeddings of learned classes and minimizing interference when learning new ones. On the other hand, adaptability tricks focus on the effective learning of new classes. Finally, training tricks improve the overall performance without compromising stability or adaptability. We perform extensive experiments on three benchmark datasets, CIFAR-100, CUB-200, and miniIMageNet, to evaluate the impact of our proposed framework. Our detailed analysis shows that our approach substantially improves both stability and adaptability, establishing a new state-of-the-art by outperforming prior works in the area. We believe our method provides a go-to solution and establishes a robust baseline for future research in this area.

• The paper introduces a unified framework combining eight influential techniques to enhance model stability and adaptability in few-shot class-incremental learning.
• It applies stability tricks like supervised contrastive loss and pre-assigned prototypes alongside adaptability methods such as incremental fine-tuning to mitigate forgetting.
• Empirical results on CIFAR-100, CUB-200, and miniImageNet demonstrate state-of-the-art performance and highlight future directions for computational efficiency improvements.

A Comprehensive Framework for Few-Shot Class-Incremental Learning

Introduction

Few-Shot Class-Incremental Learning (FSCIL) poses a significant challenge in the domain of continual learning, necessitating a delicate balance between maintaining proficiency in previously learned tasks (stability) and adapting to newly introduced classes with minimal examples (adaptability). The paper "A Bag of Tricks for Few-Shot Class-Incremental Learning" by Shuvendu Roy et al. addresses this challenge by proposing a unified framework comprised of eight highly influential techniques, categorized into stability, adaptability, and training tricks. This framework is empirically validated across three benchmark datasets, demonstrating its superiority over existing methods and establishing a new state-of-the-art in FSCIL.

Methodology

Stability Tricks

The introduced stability tricks focus on enhancing the separation between class embeddings to mitigate forgetting of previously learned classes. Techniques such as supervised contrastive loss, pre-assigning prototypes, and incorporating pseudo-classes are employed. These tricks collectively work towards increasing inter-class distance and decreasing intra-class variance, significantly improving model stability in the face of new classes.

Adaptability Tricks

To complement stability, adaptability tricks aim at refining the model's capacity to learn novel classes effectively. Incremental fine-tuning and SubNet tuning are the two primary strategies adopted. Incremental fine-tuning selectively tunes parts of the encoder for new tasks, while SubNet tuning freezes a sub-network, identified to encapsulate crucial features for previous tasks, thus allowing for focused learning on new information.

Training Tricks

The paper also introduces training-specific techniques to boost overall performance without compromising on the balance between stability and adaptability. These include using a larger encoder, adding a pre-training step with self-supervised learning, and incorporating an additional learning signal (rotation prediction task). These tricks yield a richer feature representation, consequently enhancing both adaptability and stability.

Experimental Results

Extensive experiments demonstrate that the proposed framework significantly outperforms existing methods, achieving notable improvements in accuracy across CIFAR-100, CUB-200, and miniImageNet datasets. Ablation studies underscore the critical impact of stability and adaptability tricks on the model's performance, with stability tricks having the most substantial effect. Additionally, scalability experiments on ImageNet-1K indicate the framework's effectiveness in handling a large number of classes.

Discussion and Future Directions

The paper presents an elaborate paper on improving FSCIL through a strategic collection of techniques, meticulously designed to enhance stability and adaptability. These improvements are critical in applications where models continuously learn from new, scarce data without forgetting previously acquired knowledge. While the framework sets a new benchmark in FSCIL, a discernible performance gap between base and novel classes persists, suggesting room for future research in achieving a more balanced performance. Additionally, the computational cost associated with some of the introduced tricks invites further exploration into more efficient yet effective methodologies.

In conclusion, "A Bag of Tricks for Few-Shot Class-Incremental Learning" introduces a robust and comprehensive framework that significantly advances the state-of-the-art in FSCIL. The research opens numerous avenues for future work, including exploring more computationally efficient techniques and further reducing the performance discrepancy between base and novel classes.