A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application (2310.14277v2)

Abstract: Continual learning, also known as incremental learning or life-long learning, stands at the forefront of deep learning and AI systems. It breaks through the obstacle of one-way training on close sets and enables continuous adaptive learning on open-set conditions. In the recent decade, continual learning has been explored and applied in multiple fields especially in computer vision covering classification, detection and segmentation tasks. Continual semantic segmentation (CSS), of which the dense prediction peculiarity makes it a challenging, intricate and burgeoning task. In this paper, we present a review of CSS, committing to building a comprehensive survey on problem formulations, primary challenges, universal datasets, neoteric theories and multifarious applications. Concretely, we begin by elucidating the problem definitions and primary challenges. Based on an in-depth investigation of relevant approaches, we sort out and categorize current CSS models into two main branches including data-replay and data-free sets. In each branch, the corresponding approaches are similarity-based clustered and thoroughly analyzed, following qualitative comparison and quantitative reproductions on relevant datasets. Besides, we also introduce four CSS specialities with diverse application scenarios and development tendencies. Furthermore, we develop a benchmark for CSS encompassing representative references, evaluation results and reproductions, which is available at~\url{https://github.com/YBIO/SurveyCSS}. We hope this survey can serve as a reference-worthy and stimulating contribution to the advancement of the life-long learning field, while also providing valuable perspectives for related fields.

• The paper introduces continual semantic segmentation as a framework for incremental learning that mitigates catastrophic forgetting and semantic drift.
• It compares data-replay and data-free methods, outlining their trade-offs in handling storage limitations and privacy concerns.
• Quantitative insights on benchmarks like Pascal VOC and ADE20K validate effective strategies and guide future research directions.

A Survey on Continual Semantic Segmentation: Theory, Challenge, Method and Application

The paper provides an extensive overview of Continual Semantic Segmentation (CSS), a crucial area within the broader context of continual learning, which has captured significant interest over the past decade due to its applicability in various domains, particularly computer vision. CSS aims to enhance deep learning models' ability to learn incrementally, adapting progressively to new data, tasks, or environments without losing knowledge obtained from prior experiences—a challenge commonly referenced as catastrophic forgetting.

Theoretical Underpinnings

Continual learning (CL), synonymous with incremental or life-long learning, is grounded in cognitive neuroscience, particularly concerning memory and forgetting mechanisms. The stability-plasticity dilemma—a central theme in CL—is explored in the CSS context, seeking a balance between maintaining previously acquired knowledge (stability) and integrating new information (plasticity). The paper underscores CSS's unique challenges, such as its dense prediction nature, which complicates tasks like classifying pixels into meaningful semantic components.

Key Challenges

CSS faces two predominant challenges: catastrophic forgetting and semantic drift. Catastrophic forgetting occurs when a model's performance on previously learned tasks deteriorates as it learns new tasks. Semantic drift, particularly prominent in CSS, refers to the progressive shift in background class semantics across incremental learning steps, leading to classifier bias and performance degradation.

Methodological Approaches

The paper categorizes existing CSS methodologies into data-replay and data-free methods. Data-replay methods, including exemplar-replay and generative-replay approaches, utilize stored or synthesized past data to mitigate forgetting. However, they face issues such as storage burdens and privacy concerns. Data-free methods, on the other hand, do not rely on past data storage, employing strategies like knowledge distillation and contrastive learning to retain previous knowledge while integrating new tasks.

Quantitative Insights

The paper provides quantitative comparisons across various settings like class-incremental and domain-incremental CSS using standard benchmarks such as Pascal VOC 2012 and ADE20K datasets. Techniques like SSUL and PLOP demonstrate effective trade-offs between stability and plasticity, with considerable focus on managing the semantic drift and robust feature representation that characterizes CSS challenges.

Practical Implications and Future Prospects

CSS methodologies hold significant potential across diverse applications, including autonomous driving, remote sensing, and medical diagnostics, where they can facilitate continuous adaptation and learning from dynamic data streams. Future research directions emphasize enhancing brain-inspired architectures for improved neuro-science-aligned adaptability and integrating large foundation models' capabilities for better generalization. Strategies such as strengthening interpretability and model efficiency, especially in resource-constrained environments like edge computing, are highlighted as promising avenues for CSS advancements.

Conclusion

The comprehensive survey underscores the critical role of continual learning methodologies in advancing semantic segmentation tasks, driving AI systems towards more sophisticated, adaptive, and application-specific models. While the paper provides substantial insights and compiles recent progress, it also lays a foundational framework for future investigations into CSS's nuanced challenges and potential implementations.