Metric Learning for Novelty and Anomaly Detection (1808.05492v1)

Abstract: When neural networks process images which do not resemble the distribution seen during training, so called out-of-distribution images, they often make wrong predictions, and do so too confidently. The capability to detect out-of-distribution images is therefore crucial for many real-world applications. We divide out-of-distribution detection between novelty detection ---images of classes which are not in the training set but are related to those---, and anomaly detection ---images with classes which are unrelated to the training set. By related we mean they contain the same type of objects, like digits in MNIST and SVHN. Most existing work has focused on anomaly detection, and has addressed this problem considering networks trained with the cross-entropy loss. Differently from them, we propose to use metric learning which does not have the drawback of the softmax layer (inherent to cross-entropy methods), which forces the network to divide its prediction power over the learned classes. We perform extensive experiments and evaluate both novelty and anomaly detection, even in a relevant application such as traffic sign recognition, obtaining comparable or better results than previous works.

Metric Learning for Novelty and Anomaly Detection

This paper by Masana et al. addresses the domain of out-of-distribution (OOD) detection by distinguishing between two crucial aspects: novelty detection and anomaly detection. The researchers propose a novel approach leveraging metric learning as an alternative to traditional methods predominantly based on cross-entropy loss, which relies on the softmax layer. Their work aims to overcome the inherent limitation of softmax-based networks that tend to make overconfident predictions on OOD samples.

Summary of Approach

The authors differentiate novelty detection, which deals with images of related but unseen classes, from anomaly detection, which pertains to images entirely unrelated to the training set. They identify a gap where most existing research focuses on anomaly detection usually optimized using cross-entropy loss. In contrast, this paper proposes metric learning methods to accomplish effective OOD detection without the confidence misjudgments induced by the output normalization in softmax layers.

Experimental Design and Results

The paper presents an extensive series of experiments:

1. Benchmark Datasets: The authors evaluate their methodology using standard datasets such as MNIST, SVHN, and CIFAR-10 and focus especially on assessing performance in novelty and anomaly detection tasks. The experiments demonstrate that metric learning provides a more nuanced embedding space conducive to accurate OOD assessment.
2. Real-world Applications: The authors also conduct experiments on the Tsinghua Traffic Sign dataset to simulate practical OOD detection settings. This work identifies challenging scenarios in recognizing unseen classes in traffic sign recognition systems, underlining the method's applicability to real-world tasks beyond controlled experimental settings.

The results showcase that metric learning, when compared to state-of-the-art methods such as ODIN and CC-AG, provides either comparable or superior performance in both novelty and anomaly detection tasks. Metric learning based on contrastive loss allows for embeddings where in-distribution classes cluster coherently, while OOD samples are clearly detached.

Implications and Future Directions

The primary implication of this research is the enhanced capability for OOD detection systems to distinguish between novel and anomalous inputs effectively, reducing false positives in real-world applications such as autonomous vehicles' perception modules. The emphasis on utilizing metric learning also opens up new avenues for further exploration on how embeddings for OOD detection could be refined with more sophisticated loss functions and training paradigms that integrate diverse forms of OOD samples.

Future development in this arena could extend towards creating more discriminative metric learning losses and exploring relationships between different OOD dataset types to enhance generalization. Additionally, leveraging semi-supervised or unsupervised approaches can further refine OOD detection strategies, especially in dynamically evolving environments or applications requiring continuous learning.

Overall, the paper marks a significant contribution by demonstrating metric learning's potential to advance OOD detection capabilities, outlining not just robust practical results but also offering a theoretical lens through which OOD detection can be approached more holistically.