SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning (2106.11562v3)

Abstract: This paper introduces a solid state-of-the-art baseline for a class-incremental semantic segmentation (CISS) problem. While the recent CISS algorithms utilize variants of the knowledge distillation (KD) technique to tackle the problem, they failed to fully address the critical challenges in CISS causing the catastrophic forgetting; the semantic drift of the background class and the multi-label prediction issue. To better address these challenges, we propose a new method, dubbed SSUL-M (Semantic Segmentation with Unknown Label with Memory), by carefully combining techniques tailored for semantic segmentation. Specifically, we claim three main contributions. (1) defining unknown classes within the background class to help to learn future classes (help plasticity), (2) freezing backbone network and past classifiers with binary cross-entropy loss and pseudo-labeling to overcome catastrophic forgetting (help stability), and (3) utilizing tiny exemplar memory for the first time in CISS to improve both plasticity and stability. The extensively conducted experiments show the effectiveness of our method, achieving significantly better performance than the recent state-of-the-art baselines on the standard benchmark datasets. Furthermore, we justify our contributions with thorough ablation analyses and discuss different natures of the CISS problem compared to the traditional class-incremental learning targeting classification. The official code is available at https://github.com/clovaai/SSUL.

• The paper introduces SSUL-M, a method that anticipates unknown classes via saliency maps to enhance incremental semantic segmentation.
• It employs model freezing and pseudo-labeling to prevent catastrophic forgetting while ensuring precision in multi-label predictions.
• The integration of a tiny exemplar memory significantly boosts mIoU scores across incremental tasks on standard benchmark datasets.

Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning

The paper "SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning" presents a novel approach to Class-Incremental Semantic Segmentation (CISS), a significant challenge within machine learning that strives to enable systems to learn new classes incrementally without forgetting previously acquired knowledge. The authors propose a method named SSUL-M which addresses known issues within CISS, particularly the semantic drift of the background class and the multi-label prediction problem.

Contributions and Methodology

1. Unknown Class Definition: The paper introduces the concept of defining unknown classes within the background class. This approach anticipates future classes by utilizing saliency map detectors to identify potential unknowns. This strategy increases the ability to maintain plasticity — the capacity to learn new information — by isolating unknown class objects even if their specific labels are not currently available in the learning phase.
2. Model Freezing and Pseudo-labeling: To combat catastrophic forgetting — where the introduction of new data leads to the loss of existing knowledge — SSUL-M freezes the backbone network and past classifiers once initial learning is secured. It employs binary cross-entropy loss alongside pseudo-labeling, effectively maintaining stability — the ability to retain prior knowledge when new information is learned. This methodology deviates significantly from classic knowledge distillation approaches and shows promising results.
3. Exemplar Memory Utilization: For the first time in CISS, tiny exemplar memory is introduced, storing a small portion of training data to further enhance both plasticity and stability. This inclusion diverges from traditional practices by using the memory beyond preventing the forgetting of past classes; it improves the mean Intersection-over-Union (mIoU) scores for currently learned classes.

Results

The comprehensive experimental evaluations demonstrate the superiority of SSUL-M in achieving better performance compared to recent state-of-the-art approaches on standard benchmark datasets like Pascal VOC 2012 and ADE20K. Notably, SSUL-M showcases distinct advantages as the number of incremental tasks increases. The approach altered traditional incremental learning paradigms, particularly in the context of handling semantic drift and prediction precision, by adopting model strategies distinct from prior segmentation and classification methods.

Implications and Future Directions

The implications of SSUL-M extend beyond immediate improvements in mIoU scores. The inventive utilization of unknown class modeling and exemplar memory suggest new paradigms for managing evolving datasets without substantial retraining. Furthermore, the methodology invites additional exploration into how saliency detectors can be more effectively integrated with segmentation tasks, particularly for datasets where object definitions are ambiguous.

Future research may benefit from exploring the integration of SSUL-M techniques with other complex learning environments. Potential challenges remain, such as optimizing unknown label definitions in datasets with intricate class hierarchies and assessing how model freezing may impact the adaptability of systems in rapidly changing environments.

In summary, SSUL-M represents a significant and practical advancement in the domain of semantic segmentation within incremental learning settings. Its contributions pave the way for more effective and robust learning methods that hold their relevance in both current and evolving machine learning landscapes.