Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception (2407.18145v2)

Abstract: Semantic segmentation models are typically trained on a fixed set of classes, limiting their applicability in open-world scenarios. Class-incremental semantic segmentation aims to update models with emerging new classes while preventing catastrophic forgetting of previously learned ones. However, existing methods impose strict rigidity on old classes, reducing their effectiveness in learning new incremental classes. In this work, we propose Taxonomy-Oriented Poincar\'e-regularized Incremental-Class Segmentation (TOPICS) that learns feature embeddings in hyperbolic space following explicit taxonomy-tree structures. This supervision provides plasticity for old classes, updating ancestors based on new classes while integrating new classes at fitting positions. Additionally, we maintain implicit class relational constraints on the geometric basis of the Poincar\'e ball. This ensures that the latent space can continuously adapt to new constraints while maintaining a robust structure to combat catastrophic forgetting. We also establish eight realistic incremental learning protocols for autonomous driving scenarios, where novel classes can originate from known classes or the background. Extensive evaluations of TOPICS on the Cityscapes and Mapillary Vistas 2.0 benchmarks demonstrate that it achieves state-of-the-art performance. We make the code and trained models publicly available at http://topics.cs.uni-freiburg.de.

• The paper introduces TOPICS, a framework using hyperbolic geometry to integrate new classes while retaining prior knowledge in segmentation tasks.
• It employs novel regularization losses in the Poincaré model to maintain structured feature spaces and reduce catastrophic forgetting.
• Evaluations on Cityscapes and Mapillary Vistas 2.0 show state-of-the-art performance, enhancing open-world perception in autonomous driving.

Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception

The paper "Taxonomy-Aware Continual Semantic Segmentation in Hyperbolic Spaces for Open-World Perception" by Julia Hindel et al. addresses critical challenges in Class-Incremental Semantic Segmentation (CISS) for autonomous driving. The authors propose a novel framework, Taxonomy-Oriented Poincare-regularized Incremental-Class Segmentation (TOPICS), which ensures the learning and retention of new and old classes with a taxonomy-based approach leveraging hyperbolic space.

Key Contributions

One of the paper's primary contributions is the development of the TOPICS framework. By exploiting the geometric properties of hyperbolic space, this framework allows semantic segmentation models to evolve and integrate new classes without catastrophic forgetting of previously learned classes. Such capabilities are critical in open-world scenarios typical of autonomous vehicles, where new categories of objects can emerge unpredictably from either known classes or background noise.

Hyperbolic Space Utility: The Poincare model is used to project features in a hyperbolic space, which inherently supports efficient hierarchical structuring due to the equidistance of hyperbolic nodes. The models show robust performance improvements by positioning classes at appropriate levels of a taxonomic hierarchy, facilitating a balance between rigidity (preservation of old classes) and plasticity (learning new classes).

Regularization Losses: The authors augment the learning process with two novel regularization losses designed for hyperbolic spaces. These losses help preserve the equidistant feature space and manage class relations, mitigating the severe risk of forgetfulness in incremental learning tasks.

Evaluation and Results

The authors conducted extensive evaluations of their model using Cityscapes and Mapillary Vistas 2.0 datasets, covering a range of CISS scenarios from both known classes and background. The evaluations indicate that TOPICS achieves state-of-the-art performance, outperforming existing baselines by significant margins in multiple setups. Notably, the framework displays significantly improved generalization metrics for novel classes suggesting enhanced plasticity.

Implications and Future Directions

The implications of this research extend beyond its immediate application in autonomous driving. By leveraging hierarchical taxonomies in hyperbolic spaces, the framework sets a precedent for handling expanding knowledge bases in various AI applications where adaptability to new information is essential.

Practical Implications: For real-world applications in autonomous driving, this solution offers a means to continuously improve perception systems without the necessity of retraining from scratch, making the process computationally efficient and fostering more adaptive AI systems.

Theoretical Implications: This work emphasizes the benefits of hyperbolic modeling for incremental learning, encouraging future investigations into other areas where hyperbolic geometries could equally support structured and scalable learning.

The paper opens new pathways for research to explore the optimization of taxonomy trees dynamically and investigate alternative geometric spaces that could potentially enhance hierarchical learning in machine perception systems. As AI systems increasingly operate in complex, unstructured environments, the importance of such frameworks will only continue to grow.