Constrained Few-shot Class-incremental Learning (2203.16588v1)

Abstract: Continually learning new classes from fresh data without forgetting previous knowledge of old classes is a very challenging research problem. Moreover, it is imperative that such learning must respect certain memory and computational constraints such as (i) training samples are limited to only a few per class, (ii) the computational cost of learning a novel class remains constant, and (iii) the memory footprint of the model grows at most linearly with the number of classes observed. To meet the above constraints, we propose C-FSCIL, which is architecturally composed of a frozen meta-learned feature extractor, a trainable fixed-size fully connected layer, and a rewritable dynamically growing memory that stores as many vectors as the number of encountered classes. C-FSCIL provides three update modes that offer a trade-off between accuracy and compute-memory cost of learning novel classes. C-FSCIL exploits hyperdimensional embedding that allows to continually express many more classes than the fixed dimensions in the vector space, with minimal interference. The quality of class vector representations is further improved by aligning them quasi-orthogonally to each other by means of novel loss functions. Experiments on the CIFAR100, miniImageNet, and Omniglot datasets show that C-FSCIL outperforms the baselines with remarkable accuracy and compression. It also scales up to the largest problem size ever tried in this few-shot setting by learning 423 novel classes on top of 1200 base classes with less than 1.6% accuracy drop. Our code is available at https://github.com/IBM/constrained-FSCIL.

• The paper presents C-FSCIL, a novel approach that combines a frozen meta-learned feature extractor, a fixed-size trainable layer, and dynamic external memory for efficient incremental learning.
• It leverages hyperdimensional embedding and innovative loss functions to maintain quasi-orthogonal class representations and reduce interference among new and existing classes.
• Empirical results on CIFAR100, miniImageNet, and Omniglot demonstrate scalability to 1623 classes with less than 1.6% accuracy decay, underscoring its practical efficiency.

Constrained Few-shot Class-incremental Learning: A Comprehensive Analysis

The paper "Constrained Few-shot Class-incremental Learning" presents a novel approach named C-FSCIL, aiming to tackle the challenging problem of expanding classification capabilities in constrained environments. This method addresses the pressing issue of class-incremental learning where models need to learn new classes incrementally without forgetting the previously learned classes, a problem exacerbated by constraints such as limited training samples per class and fixed computational resources.

Architectural Overview

C-FSCIL cleverly leverages a combination of three core components: a frozen meta-learned feature extractor, a trainable fixed-size fully connected layer, and a dynamically growing external memory (EM). This architecture is designed to respect memory and computational constraints by limiting memory growth to a linear scale relative to the number of classes and keeping the learning cost of new classes constant. Notably, the feature extractor remains static post meta-learning, thus retaining foundational knowledge even as new classes are encountered.

A notable innovation is the use of hyperdimensional embedding within the vector space. This allows the system to maintain minimal interference between classes despite an expanding class set, by assigning each class a quasi-orthogonal vector representation. This strategic embedding, enriched by novel loss functions, ensures robust class differentiation.

Three Modes of Learning

C-FSCIL introduces three modes of updating memory and training that provide a balance between classification accuracy and computational cost:

1. Mode 1: Averaged Prototypes - This mode utilizes averaged prototypes from new data, stored directly in the EM without any gradient updates. It is efficient computationally, as it does not require backpropagation, making it suitable for real-time applications.
2. Mode 2: Retraining on Bipolarized Prototypes - Here, class prototypes are bipolarized to enhance their separability, followed by limited retraining of the fully connected layer using stored global class activations. This mode uses a few iterations to refine classification boundaries, improving accuracy beyond Mode 1.
3. Mode 3: Retraining on Nudged Prototypes - This mode is the most computationally intense and fine-tunes prototypes by nudging them to achieve optimal quasi-orthogonality, followed by extensive retraining. It tends to offer the best performance when the class space grows substantially, as evidenced by the minimal accuracy drop across large dataset expansions.

Empirical Validation

The effectiveness of C-FSCIL is demonstrated on CIFAR100, miniImageNet, and Omniglot datasets. These experiments reveal that even with minimal retraining (Mode 1), the system can outperform existing baselines in terms of accuracy. Specifically, C-FSCIL expands to 1623 classes on Omniglot with less than 1.6% accuracy decay when compared to baseline methods, exemplifying its scalable representation learning potential. Importantly, C-FSCIL maintains a competitive edge over state-of-the-art systems even under compression, further highlighting its adaptability and memory efficiency.

Implications and Outlook

The proposed C-FSCIL framework presents significant theoretical and practical advancements in class-incremental learning. The exploitation of hyperdimensional computing paves the way for embedding locally interpretable representations within a scalable global structure, a shift that holds intriguing implications for smart devices and real-time learning systems. This model could serve as a foundation for developing more refined AI systems capable of learning in dynamic and resource-constrained environments, potentially extending the applicability to other domains such as continual reinforcement learning or complex sensory data processing.

The research opens avenues for further innovations, mainly focused on reducing the computational footprint of training processes further and enhancing the robustness of the model under adversarial conditions. Future development could explore hardware-specific optimizations or integration with neuromorphic computing architectures, which might capitalize on the lower-dimensional efficient computing paradigms presented.

C-FSCIL exemplifies a promising stride towards bridging the gap between theoretical research in class-incremental learning and its practical implementation in constrained environments, fostering innovation in the continual learning capabilities of intelligent systems.