Semantic Drift Compensation for Class-Incremental Learning (2004.00440v1)

Abstract: Class-incremental learning of deep networks sequentially increases the number of classes to be classified. During training, the network has only access to data of one task at a time, where each task contains several classes. In this setting, networks suffer from catastrophic forgetting which refers to the drastic drop in performance on previous tasks. The vast majority of methods have studied this scenario for classification networks, where for each new task the classification layer of the network must be augmented with additional weights to make room for the newly added classes. Embedding networks have the advantage that new classes can be naturally included into the network without adding new weights. Therefore, we study incremental learning for embedding networks. In addition, we propose a new method to estimate the drift, called semantic drift, of features and compensate for it without the need of any exemplars. We approximate the drift of previous tasks based on the drift that is experienced by current task data. We perform experiments on fine-grained datasets, CIFAR100 and ImageNet-Subset. We demonstrate that embedding networks suffer significantly less from catastrophic forgetting. We outperform existing methods which do not require exemplars and obtain competitive results compared to methods which store exemplars. Furthermore, we show that our proposed SDC when combined with existing methods to prevent forgetting consistently improves results.

• The paper introduces Semantic Drift Compensation to counteract catastrophic forgetting in deep networks.
• It leverages embedding networks to iteratively estimate and correct feature drift without storing exemplars.
• Empirical results on CIFAR100 and ImageNet-Subset reveal substantial improvements over traditional classification methods.

Semantic Drift Compensation for Class-Incremental Learning

The paper "Semantic Drift Compensation for Class-Incremental Learning" addresses the challenge of catastrophic forgetting in deep networks when they are trained sequentially across tasks. The authors focus on optimizing embedding networks rather than traditional classification networks for class-incremental learning (class-IL). This approach leverages the intrinsic capability of embedding networks to accommodate new classes without architectural modifications, thus presenting a potential solution to an inherently complex issue.

Current solutions to incremental learning primarily focus on classification networks, requiring modifications such as adding new weights for additional classes. In contrast, embedding networks natively facilitate the inclusion of new classes as they map data to embedding spaces, encapsulating semantic dissimilarities. The paper introduces a novel technique called Semantic Drift Compensation (SDC), which estimates and corrects the drift of features experienced due to continual learning, especially when no exemplars from previous tasks are used.

The empirical evaluation demonstrates significant advantages. The implementation on fine-grained datasets like CIFAR100 and ImageNet-Subset reveals that embedding networks suffer markedly less from catastrophic forgetting, compared to their classification counterparts. Contrary to the classical methods which accumulate drift over tasks, the SDC approach approximates semantic drift for previous tasks based on the drift observed in current task data, thereby enhancing the robustness of incremental learning processes.

The authors compare several regularization strategies like E-LwF, E-EWC, and E-MAS, modified for embedding networks, demonstrating substantial improvements. In particular, their results, when combined with the proposed SDC, show noticeable advancements, outperforming existing methods that do not store exemplars and rival even those that do.

From a theoretical standpoint, the paper suggests that embedding networks offer a promising avenue for overcoming limitations associated with traditional classification networks in class-incremental settings. The Semantic Drift Compensation technique provides an interesting alternative to the paradigm of preserving model performance on earlier tasks. With the rise of data privacy and security concerns, methodologies like SDC that do not depend on exemplar storage hold significant future potential.

The paper further outlines the potential of combining SDC with traditional methods designed to mitigate forgetting, indicating a complementary relationship which could be explored in future work. Practical implications include more efficient model updates in environments with limited data retention capabilities, an increasingly critical aspect in real-world applications constrained by privacy regulations.

Overall, this work sets the stage for further explorations into embedding-centric approaches to continual learning, particularly the interpretation and leverage of data semantics through improved drift estimation techniques, presenting a valued contribution to the contemporary discourse on adaptive AI systems.