Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes (2111.12264v6)

Abstract: State-of-the-art (SOTA) anomaly segmentation approaches on complex urban driving scenes explore pixel-wise classification uncertainty learned from outlier exposure, or external reconstruction models. However, previous uncertainty approaches that directly associate high uncertainty to anomaly may sometimes lead to incorrect anomaly predictions, and external reconstruction models tend to be too inefficient for real-time self-driving embedded systems. In this paper, we propose a new anomaly segmentation method, named pixel-wise energy-biased abstention learning (PEBAL), that explores pixel-wise abstention learning (AL) with a model that learns an adaptive pixel-level anomaly class, and an energy-based model (EBM) that learns inlier pixel distribution. More specifically, PEBAL is based on a non-trivial joint training of EBM and AL, where EBM is trained to output high-energy for anomaly pixels (from outlier exposure) and AL is trained such that these high-energy pixels receive adaptive low penalty for being included to the anomaly class. We extensively evaluate PEBAL against the SOTA and show that it achieves the best performance across four benchmarks. Code is available at https://github.com/tianyu0207/PEBAL.

• The paper introduces PEBAL, integrating pixel-wise abstention learning with energy-based models to accurately differentiate inlier and anomaly pixels.
• It employs adaptive penalty factors and the AnomalyMix augmentation to reduce false positives and negatives while maintaining segmentation precision.
• Experimental results on four benchmarks, including LostAndFound, show significant improvements with an AP of 78.29% and an FPR95 of 0.81.

Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation in Urban Driving

The paper introduces a novel approach, known as Pixel-wise Energy-biased Abstention Learning (PEBAL), to enhance anomaly segmentation in urban driving scenes. The primary motivation for this advancement is to address the inefficacy of existing state-of-the-art (SOTA) methods that either rely on classification uncertainty or reconstruction models, both of which have limitations in real-time applications such as autonomous driving.

Problem Statement

Anomaly segmentation in urban driving scenarios is critical for road safety, as undetected anomalies like unexpected road obstacles can lead to accidents. Traditional methods, which associate high classification uncertainty with anomaly and use reconstruction models, face significant challenges. The association of uncertainty with anomalies can cause false positives and negatives, whereas reconstruction models can be inefficient for real-time processing.

Proposed Method: PEBAL

The PEBAL method integrates two advanced learning models: Pixel-wise Abstention Learning (PAL) and Energy-Based Models (EBM). This integration is pivotal for determining inlier and outlier distributions at the pixel level without biasing the classification towards outliers. PEBAL's key innovation lies in its ability to adaptively learn penalty factors that influence whether pixels should abstain from classification into pre-defined inlier classes or be classified as anomalies.

1. Pixel-wise Abstention Learning (PAL): This model adapts the penalty factor based on pixel-level energy scores, facilitating the accurate classification of anomalies without misclassifying challenging inliers.
2. Energy-Based Models (EBM): These models partition pixel-level energy scores, enabling the establishment of a distinct energy gap between normal and anomaly pixels. The adaptive estimation of these energy scores is critical for efficient anomaly classification.

Training Procedure

PEBAL's training involves a nuanced joint training strategy of the PAL and EBM models. By fine-tuning only the last classification block, it maintains the segmentation accuracy for inlier classes, thereby balancing computational efficiency with classification precision. The paper introduces the AnomalyMix strategy, a novel augmentation technique that enhances the diversity of anomaly exposure during training by dynamically combining inlier scenes with synthetic anomaly objects sourced from external datasets (e.g., COCO), allowing a broader range of anomaly scenarios to be effectively learned.

Experimental Results

The paper reports extensive evaluations of PEBAL across four benchmarks, including Fishyscapes and LostAndFound datasets. The results indicate superior performance compared to existing methods, particularly in reducing false positive and negative rates, while achieving high accuracy in detecting small and distant anomalous objects. Notably, PEBAL achieves a significant reduction in false positive rate (FPR95) and improves the average precision (AP) across all evaluated datasets. For instance, on the LostAndFound test set, PEBAL's AP of 78.29% and FPR95 of 0.81 showcase its effectiveness.

Practical Implications and Future Work

PEBAL's implementation can enhance autonomous driving systems by providing a robust solution to detect and handle anomalies effectively in real-time. The approach's independence from full network retraining and its ability to work efficiently with the last layer fine-tuning make it highly applicable for integration into existing systems.

Future work could explore the expansion of PEBAL to other computer vision domains where real-time anomaly detection is critical, as well as the exploration of unsupervised learning techniques to further improve anomaly detection performance without relying heavily on annotated outlier datasets.

In summary, PEBAL represents a significant step forward in the field of anomaly segmentation for urban driving scenes, providing a more precise, efficient, and adaptable solution compared to previous models. This work sets the stage for further advances in anomaly segmentation, particularly in applications demanding high degrees of accuracy and reliability, such as autonomous vehicles.