---
title: Open-Set Recognition Overview
url: https://www.emergentmind.com/topics/open-set-recognition
type: topic
---

# Open-Set Recognition Overview

Open-Set Recognition (OSR) is the supervised learning paradigm for systems where, at inference time, samples from classes absent at training may appear. An OSR system must both classify inputs from known classes and reliably reject or flag instances from unknown classes, minimizing "open space risk"—the tendency to assign known labels to samples outside the support of the training set. This challenge is pervasive in realistic machine learning deployments across domains such as vision, speech, healthcare, and cybersecurity, where exhaustively sampling all possible classes is infeasible. OSR frameworks address this by developing decision rules, representations, and training objectives explicitly designed to model and constrain the classifier's generalization in the presence of unknown unknowns.

## 1. Formal Foundations and Risk Formulation

Open-set recognition extends closed-set classification by introducing an explicit reject option and by balancing empirical risk against open space risk. Let $\mathcal{C}_{KC} = \{Y_1,\ldots,Y_N\}$ denote the known (training) classes, and $\mathcal{C}_{UC} = \{Y_{N+1},\ldots,Y_{N+\Omega}\}$ the unknown classes that may occur at test time. For a recognition function $f$, the open space risk quantifies the measure of positively labeled space outside training support,
\[
R_{O}(f) = \frac{\int_{O} f(\mathbf{x})\,d\mathbf{x}}{\int_{S_{O}} f(\mathbf{x})\,d\mathbf{x}},
\]
where $O$ is open space, and $S_{O} = O \cup S_{\text{train}}$ is the total support. The design of OSR algorithms centers on objectives of the form
\[
\min_{f}\;R_{\mathrm{emp}}(f) + \lambda R_{O}(f),\quad \lambda \ge 0,
\]
with $R_{\mathrm{emp}}(f)$ the empirical (closed-set) risk. Thresholding strategies—whether on probability, distance, or meta-statistics—instantiate the rejection mechanism, with the precise threshold $\tau$ dictating the tradeoff between known-class accuracy and the rate of unknown detection [2312.08785, 2109.00893].

## 2. Taxonomy of Methods

OSR research is organized along several axes: discriminative versus generative; distance/prototype versus calibration; and whether the system is inductive or transductive. The following families are prominent:

| Method Class     | Key Principle          | Example Algorithms                              |
|------------------|-----------------------|-------------------------------------------------|
| Discriminative   | Calibrated boundaries | 1-vs-rest SVM/PISVM, OpenMax, DOC               |
| Distance-based   | Prototypes/Clusters   | NNO, NNDR, Gaussian-prototype, clustering+SVDD  |
| EVT-based        | Statistical tails     | W-SVM, EVM, OpenMax, SROSR                      |
| Generative       | Synthetic unknowns    | GAN-based (G-OpenMax, ARPL), VAE, AAE-C2AE      |
| Hybrid           | Joint objectives      | CROSR, clustering+classification hybrids        |

**Discriminative approaches** include calibrated SVMs [W-SVM, EVM], one-vs-rest deep sigmoids (DOC [2103.10230]), OpenMax with EVT calibration of penultimate features, and collective thresholding [2312.15571, 2312.08785].

**Distance- and prototype-based** methods learn explicit feature centroids (NCM/NNO, CPN, GCPL) or model class densities with Gaussian or more general distributions, rejecting by distance or likelihood [2312.15571].

**Extreme Value Theory (EVT)** motivates tail-modeling of SVM margins or deep-network activation distances. OpenMax and SROSR fuse EVT-calibrated scores for open-set confidence [1705.02431, 2312.15571].

**Generative approaches** either synthesize pseudo-unknown "hard negatives" (GAN-based G-OpenMax, ARPL), model the latent structure of knowns (VAE, AAE, CPGM), or perform outlier exposure with background data [2203.13238, 2008.05129].

**Hybrid and clustering-based** pipelines combine DNN representation learning with clustering (e.g., DBSCAN+SVDD, GMM+KOSNN), or integrate contrastive or reconstruction signals for open-set separability [2312.15571, 2312.08785, 2207.02158].

## 3. Technical Approaches and Instantiations

### 3.1 Thresholding and Calibration

SVM and DNN outputs can be calibrated for OSR via EVT (fitting Weibull or GPD tails to class margins or feature distances), yielding a per-class open-set membership estimate. For example, SROSR computes sparse reconstruction errors per class and fuses EVT-modeled right-tails of matched and non-matched residuals, declaring "unknown" if the fused score exceeds a threshold [1705.02431]. OpenMax computes the distances from sample activations to class means, fits Weibull models on these, and recalibrates logits to introduce an "unknown" slot [2312.08785, 2109.00893, 2312.15571].

### 3.2 Distance- and Prototype-based Rejection

Many algorithms replace the softmax classifier with a metric-based head, where the likelihood of each class is inversely proportional to the feature distance from class prototypes or anchor vectors. Mahalanobis or von Mises-Fisher metrics are common, with rejection based on a distance threshold or likelihood ratio [2503.08049, 2404.10370]. Chi-square–based inclusion probabilities and background-class regularization exploit auxiliary background (surrogate unknown) data to push feature representations outside known-class domains [2207.10287].

### 3.3 Generative and Hybrid Models

**Conditional Probabilistic Generative Models** (CPGM) combine VAE or AAE frameworks with discriminative class-conditional priors, learning to reconstruct knowns and distinguish unknowns by density and reconstruction error [2008.05129]. **Gaussian Mixture VAEs** (GMVAE) further model intra-class subclusters, using distance- or uncertainty-based rejection after projecting into learned latent manifolds [2006.02003, 2201.02923].

**Augmentation-based similarity learning** uses distribution-shifting transformations (e.g., $90^\circ$ rotations) to generate pseudo-unseen data, matches similarities via meta-classification losses, and allocates dedicated "unknown" slots in the prediction layer [2203.13238]. **Contrastive methods** (DCTAU) improve boundaries by associating $K$ pseudo-unknown (Target-Aware Universum, TAU) classes to each known, training with dual contrastive loss to ensure class-equal treatment and feature surgicality [2401.17654].

### 3.4 Model-Agnostic and Ensemble Techniques

Gradient-based representations quantify how much an input would require re-tuning of a pretrained classifier to reach an arbitrary label; this gradient norm reliably distinguishes off-manifold unknowns, allowing for open-set detectors independent of base network architecture [2206.08229]. One-vs-rest ensemble architectures replace the softmax with per-class sigmoids, enabling more selective boundaries [2103.10230].

### 3.5 Open-Set Recognition with Random Forests

For non-DNN settings, open-set rejection is enabled via metric learning on Random Forest proximity matrices, mapping to a Mahalanobis space learned via sparse GP regression, and then applying a K-nearest output statistics plus EVT thresholding [2408.02684]. This yields known-vs-unknown separation superior to earlier KOSNN/OSNN or threshold-on-leaf metrics.

## 4. Evaluation Protocols and Metrics

Standard benchmarks derive from canonical datasets by withholding some classes as unknowns at test. Representative splits are:

| Dataset      | Known / Unknown Split (example) |
|--------------|--------------------------------|
| MNIST        | 6 / 4                           |
| CIFAR-10     | 6 / 4, or grouped 2 vs 6        |
| CIFAR+10     | 4 CIFAR-10 known, 10 CIFAR-100 unknown |
| SVHN         | 6 / 4                           |
| TinyImageNet | 20 / 180                        |

Metrics include:

- **AUROC**: Area under the ROC curve of true positive rate (known) vs. false positive rate (unknown) as threshold varies.
- **OSCR**: Open-Set Classification Rate—mean of correctly classified known and correctly rejected unknown at varying thresholds.
- **Macro-F1**: F1-score treating "unknown" as an additional class.
- **Closed-set accuracy** at specific operating points (e.g., 95% known recall).

These are always evaluated with respect to the (K+1)-class or (K+M)-class target, including the unknown.

## 5. Open Issues and Research Directions

Challenges persist in threshold selection (whether per-class or global), scaling to high "openness"; incremental and lifelong OSR with dynamic updates to the set of knowns; computational overheads of generative/EVT methods; adversarial vulnerability (unknowns crafted to evade rejection); and colloquially, "near-distribution" failures (open classes that closely mimic knowns) [2312.08785, 2109.00893, 2312.15571].

Recent work identifies representation structure as a critical component: higher-diversity, better-separated, and more compact features improve unknown rejection [2404.10370]. Jacobian norm analysis demonstrates that inter-class separation increases representation sensitivity outside known support, creating a measurable gap leveraged for detection [2209.11436].

Future lines include: meta- or self-adaptive thresholding (online tuning), joint clustering-classification frameworks, temporal/streaming models for the emergence of new classes, the use of auxiliary data or side-channel information for disambiguation, and rigorous unified benchmarks and protocols for new modalities.

## 6. Vision-Language Models and Open-Set Recognition

Contrary to the intuition that vision-language models (VLMs) such as CLIP function as open-set recognizers due to internet-scale pretraining, empirical analysis shows that their closed-set assumption is reinstated at test time via the finite query set. Open-set errors are prevalent when true labels are absent from the query set. Simple uncertainty proxies (softmax confidence, max cosine similarity, entropy) and negative embedding augmentation are insufficient: improvements in open-set rejection come at a steep cost in closed-set task performance, and open-set precision/recall trade-offs cannot be effectively managed through naive expansion of the query set. This demonstrates the necessity of dedicated open-set methods for VLMs, such as learned unknown tokens or uncertainty estimation tailored to multimodal settings [2403.16528].

## 7. Empirical Insights and State-of-the-Art

Recent methods have advanced state-of-the-art open-set performance, with AUROC and OSCR gains exceeding previous baselines on both standard and fine-grained tasks. Examples:

| Method    | Protocol    | AUROC (%)           | OSCR (%)           |
|-----------|-------------|---------------------|--------------------|
| Smooth SupCon Ensemble [2404.10370] | CIFAR-10   | 94.95              | 90.3               |
| DCTAU [2401.17654]      | CIFAR+10    | 98.5               | 97.2               |
| CSSR [2207.02158]       | CIFAR+50    | 96.2               | —                  |
| OPG [2203.13238]        | CIFAR+10    | 96.2               | —                  |
| SROSR [1705.02431]      | MNIST       | F1 > 0.90 at 60% openness | —           |

These results indicate convergence towards representation-driven and contrastive approaches, with advantages in both detection performance and practical interpretability. Limitations remain in cross-domain, high-openness, and adversarially rich settings, motivating further research into continuous, scalable, and risk-sensitive OSR.

---

**References:**
- [1705.02431], [2203.13238], [1903.04774], [2312.08785], [2206.08229], [2103.10230], [2207.02158], [2008.05129], [2201.02923], [2109.00893], [2312.15571], [2503.08049], [2207.10287], [2401.17654], [2006.02003], [2209.11436], [2408.02684], [2404.10370], [2403.16528]

Source: https://www.emergentmind.com/topics/open-set-recognition