Open-Set Recognition: a Good Closed-Set Classifier is All You Need? (2110.06207v2)

Abstract: The ability to identify whether or not a test sample belongs to one of the semantic classes in a classifier's training set is critical to practical deployment of the model. This task is termed open-set recognition (OSR) and has received significant attention in recent years. In this paper, we first demonstrate that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We find that this relationship holds across loss objectives and architectures, and further demonstrate the trend both on the standard OSR benchmarks as well as on a large-scale ImageNet evaluation. Second, we use this correlation to boost the performance of a maximum logit score OSR 'baseline' by improving its closed-set accuracy, and with this strong baseline achieve state-of-the-art on a number of OSR benchmarks. Similarly, we boost the performance of the existing state-of-the-art method by improving its closed-set accuracy, but the resulting discrepancy with the strong baseline is marginal. Our third contribution is to present the 'Semantic Shift Benchmark' (SSB), which better respects the task of detecting semantic novelty, in contrast to other forms of distribution shift also considered in related sub-fields, such as out-of-distribution detection. On this new evaluation, we again demonstrate that there is negligible difference between the strong baseline and the existing state-of-the-art. Project Page: https://www.robots.ox.ac.uk/~vgg/research/osr/

• The paper demonstrates that boosting closed-set classifier accuracy inherently improves open-set detection performance, as shown by a strong Pearson correlation.
• The methodology introduces the maximum logit score (MLS) as a robust alternative to softmax probabilities, achieving superior results across multiple OSR benchmarks.
• The paper presents the Semantic Shift Benchmark (SSB) to isolate semantic novelty detection, paving the way for unified evaluation strategies in OSR research.

Evaluating Open-Set Recognition and Closed-Set Classifiers

The paper presented by Vaze et al. addresses a fundamental challenge pertinent to deploying machine learning models: open-set recognition (OSR). This task involves discerning whether a given test sample belongs to any of the known semantic classes encountered during training or if it represents an unseen class. This classical problem has significant implications in real-world applications, where classifying out-of-distribution (OoD) samples accurately is crucial. The authors revisit the prevalent notion in the open-set literature that distinct methods are required to address OSR, postulating instead that improvements in closed-set classification accuracy can naturally extend to OSR performance.

Central to this work is the hypothesis that enhancing a classifier’s closed-set accuracy intrinsically boosts its ability to perform in an open-set scenario. This hypothesis is evaluated across different datasets, architectures, and loss functions, suggesting that the relationship between closed and open-set performances is highly correlated. This implies that advancing closed-set classification techniques, predominantly used across various image datasets, could offer substantial benefits to OSR tasks.

Key Findings

1. Correlation Between Performances: The paper extensively demonstrates that strong predictor models trained under closed-set conditions can inherently serve the purpose of open-set detection. Through statistical analysis, the high Pearson Product-Moment correlation observed between closed-set accuracy and OSR performance substantiates the claim that the two are interlinked.
2. Maximizing Logits for OSR: The authors introduce the maximum logit score (MLS) as an alternative open-set indicator to normalized softmax probabilities, enhancing detection capability by retaining feature magnitude information that softmax tends to normalize out. Their results reveal substantial improvements over traditional benchmarks, significantly exceeding previously reported baseline figures and state-of-the-art results across four out of six OSR benchmark datasets.
3. Semantic Shift Benchmark (SSB): Recognizing the limitations of existing OSR benchmarks in terms of scale and semantic definition, the paper proposes the SSB. This benchmark suite evaluates models on datasets with clear semantic classes, aiming to isolate the OSR challenge from potential confounding factors such as low-level distribution shifts. The SSB includes datasets like CUB for fine-grained visual categorization, intended to provide a more nuanced understanding of semantic novelty detection.
4. Analyses of Architecture Performance: Evaluations on different model architectures for large-scale datasets like ImageNet reveal that the positive correlation between the classifiers' closed and open-set performance persists in more varied settings. This insight suggests that improvements in model architecture that elevate closed-set performance could inevitably enhance open-set detection capabilities.

Implications and Future Directions

The implications of these findings are significant for both theoretical research and practical applications. The assertion that optimizing closed-set classifiers may be adequate for open-set recognition presents a streamlined approach that leverages existing improvements in image classification for OSR tasks. The promising results from using MLS advocate for potential refinement in evaluation methodologies, as they offer an efficient, conceptually simple baseline that competes with more complex, method-specific approaches.

Moving forward, the work invites speculation about future research directions. As open-set recognition continues to evolve, further exploration of architectures that inherently balance the trade-off between closed-set accuracy and semantic generalization could be immensely valuable. Additionally, expanding the open-set problem into a broader framework that encompasses anomalies and outliers while carefully delineating related tasks such as OoD detection could too prove beneficial for constructing more robust AI systems.

In conclusion, this paper provides a detailed reevaluation of OSR approaches, suggesting a paradigm shift that leverages the full potential of closed-set classifiers. As open-set recognition applications become increasingly consequential in deploying AI systems safely and effectively, continuing to bridge closed-set advancements with OSR research remains a promising avenue for exploration.