Open-Set Recognition Systems

• Open-set recognition systems are machine learning models designed to classify known categories while effectively rejecting unknown samples.
• They utilize diverse methods including thresholded discriminative models, prototype and distance-based strategies, and generative reconstruction techniques.
• Empirical evaluations on benchmarks like CIFAR-10 and TinyImageNet demonstrate high AUROC performance and controlled open-space risk.

Open-set recognition (OSR) systems are machine learning models designed to address the reality that, at deployment, classifiers will inevitably encounter samples from categories absent during training. Unlike closed-set classifiers, which presume a known, finite class set at both train and test time, open-set recognition models must not only discriminate among the known classes but also detect and reject truly unknown categories. The formal goal of OSR is to minimize errors on known-class test samples while bounding the so-called open-space risk: the likelihood of erroneously assigning known labels to points lying far from all known training data. This paradigm is critical across domains where prior knowledge of all possible classes is unattainable, such as autonomous navigation, security, surveillance, and real-world image understanding (Geng et al., 2018, Sun et al., 2023, Mahdavi et al., 2021).

1. Foundational Problem Formulation and Open-space Risk

The canonical setup defines a feature space $X \subseteq \mathbb{R}^d$ and a set of known classes $\mathcal{Y}_K = \{1, \ldots, C\}$ available at training time. At test time, inputs may also belong to unknown classes $\mathcal{Y}_U$, disjoint from $\mathcal{Y}_K$. The open-space risk $R_{\mathcal{O}}(f)$ measures the extent to which a classifier $f$ labels points in the “open space” (regions far from any known training instance) as a known class (Geng et al., 2018): $$R_{\mathcal{O}}(f) = \frac{\int_{\mathcal{O}} f(x) \, dx}{\int_{S_0} f(x) \, dx}$$ where $f(x) = 1$ if $x$ is labeled as known, $S_0$ is a bounded region containing all data, and $\mathcal{O}$ is the open region.

The OSR learner seeks to minimize a combined risk: $$R_\mathrm{OSR}(f) = R_\mathrm{closed}(f) + \lambda\,R_{\mathcal{O}}(f)$$ where $R_\mathrm{closed}$ is empirical risk on known data, and $\lambda > 0$ is a regularization parameter (Xu, 2024, Geng et al., 2018). The degree of "openness" is quantified as

$\text{Openness} = 1 - \sqrt{\frac{2 C}{C + U}}$

where $C$ is the number of training classes, and $U$ is the number of unknown classes seen at testing (Geng et al., 2018, Sun et al., 2023).

Two primary objectives distinguish OSR from closed-set classification:

1. High accuracy among the known classes.
2. High recall of the “unknown” label when samples do not belong to any known class.

2. Methodological Landscape and Algorithmic Taxonomy

Open-set recognition encompasses a diverse suite of methodologies (Geng et al., 2018, Sun et al., 2023, Mahdavi et al., 2021, Sun et al., 2023):

• Thresholded Discriminative Models: The earliest OSR approaches focus on adapting discriminative classifiers (e.g., SVM, neural nets) with calibrated thresholds or specialized output layers to enforce compact positively-labeled regions and explicit reject options. Examples include OpenMax (EVT-calibrated softmax) (Sun et al., 2023, Geng et al., 2018), one-vs-rest networks (Jang et al., 2021), and softmax thresholding (Vaze et al., 2021).
• Feature/Prototype/Distance-based Methods: These construct explicit class prototypes or anchor points (e.g., mean feature vectors, vMF means (Bahavan et al., 11 Mar 2025)) and classify based on feature distances, often rejecting points lying beyond class-specific or global thresholds. Margin-based models further penalize intra-class dispersion and enforce mutual separation among class centroids (Cho et al., 2022, Huang et al., 2022, Sun et al., 2023).
• Generative and Reconstruction-based Models: Generative methods synthesize or model unknown regions via autoencoders, VAEs, or GANs. The underlying rationale is that unfamiliar samples yield large reconstruction errors or low log-likelihoods in learned latent spaces (Huang et al., 2022, Cao et al., 2020, Zhang et al., 2020, Sun et al., 2020). Instance-generation based approaches use GANs to explicitly hallucinate open-space samples (Geng et al., 2018, Sun et al., 2023). Variants like GMVAE (Cao et al., 2020) or CPGM (Sun et al., 2020) integrate clustering in latent space for improved separation.
• Statistical (EVT-based) Calibration: Extreme Value Theory is widely applied to calibrate decision score tails, facilitating robust per-class or per-sample confidence estimation and data-driven threshold selection (Zhang et al., 2017, Geng et al., 2018, Sun et al., 2023).
• Clustering and Nonparametric Bayesian Methods: Some models perform joint clustering of test and training data, forgoing fixed thresholds entirely and discovering both known and novel classes in a batch setting through hierarchical Dirichlet processes (Geng et al., 2018).
• Self-supervised and Contrastive Representation Learning: Recent advances highlight the importance of feature diversity and supervised contrastive losses, often yielding richer representations better suited for OSR (Xu, 2024, Bahavan et al., 11 Mar 2025, Li et al., 2024).

3. Training Objectives, Decision Rules, and Uncertainty Quantification

Training paradigms in OSR enforce compactness for known classes and either minimize open-space risk directly or maximize the margin to unknowns:

• Cross-Entropy with Reject Option: Supervised cross-entropy is augmented with explicit unknown classes or thresholded softmax/posterior scores. Some models utilize intra-class splitting to simulate unknowns from atypical known samples, training an $(N+1)$-way network (Schlachter et al., 2019).
• Contrastive and Prototype Losses: Losses based on supervised contrastive learning, often with temperature scaling, compactify class clusters and punish class collision (Xu, 2024, Bahavan et al., 11 Mar 2025). Feature diversity (measured via KL-divergence or cluster compactness) is found to correlate with OSR success.
• Reconstruction/Criterion-based Losses: Autoencoder and VAE losses supply an anomaly criterion (reconstruction error, conditional likelihood) that facilitates per-class “distance to manifold” scoring (Huang et al., 2022, Cao et al., 2020, Sun et al., 2020).
• EVT-Modeled Tail Losses: For models relying on calibrated score tails, the threshold is chosen such that the open-space risk is finite or meets a desired operating characteristic (e.g., fixed FPR or Youden index) (Zhang et al., 2017, Júnior et al., 2016, Geng et al., 2018).

At inference, decision rules generally follow one of two templates:

1. Compute a class-wise confidence (from logits, prototype alignment, Mahalanobis or vMF distance, or reconstruction likelihood).
2. Accept the predicted class if the confidence passes a calibrated threshold; otherwise, reject as “unknown.” Batch-mode models eschew thresholds, instead assigning new clusters when test instances fail to match any known-class component (Geng et al., 2018).

4. Empirical Evaluation Protocols and Findings

Open-set recognition is assessed on held-out splits or cross-dataset settings where known and unknown classes are non-overlapping (Sun et al., 2023, Mahdavi et al., 2021):

• Benchmark Datasets: Common protocols include MNIST, CIFAR-10/100, SVHN, TinyImageNet, and numerous fine-grained datasets (CUB-200, Cars, FGVC-Aircraft) (Sun et al., 2023, Xu, 2024, Geng et al., 2018).
• Metrics:
  • Closed-set Accuracy: Standard correct classification on known classes.
  • AUROC: Area under the ROC curve for distinguishing known vs unknown; central to threshold-agnostic performance claims (Xu, 2024, Sun et al., 2023).
  • Macro-F1: Balances precision/recall over known and “unknown” predictions (Huang et al., 2022).
  • Open-Set Classification Rate (OSCR): Plots correct-known-class accuracy vs. unknown detection rate across thresholds (Xu, 2024, Sun et al., 2023).
  • Openness Sensitivity: Performance is reported as a function of openness (fraction of unknowns at test) (Geng et al., 2018).

Empirically, representation learning and feature diversity (i.e., the diversity and compactness of class clusters in feature space) are strong determinants of OSR performance. For example, augmenting cross-entropy with supervised contrastive or Mixup/label smoothing losses yields state-of-the-art AUROC on challenging splits and is computationally more efficient than elaborate generative approaches (Xu, 2024, Bahavan et al., 11 Mar 2025). Modern OSR systems such as SphOR (Bahavan et al., 11 Mar 2025), DCTAU (Li et al., 2024), and CSSR (Huang et al., 2022) report AUROC exceeding 94% on CIFAR-10 (22% openness), and robustness to high openness scenarios.

5. Algorithmic Developments: Design Patterns and Modularity

Recent OSR systems emphasize the following design patterns:

• Combining Closed- and Open-set Heads: Augmenting a standard classification head (softmax or distance-based) with a set of light-weight one-vs-all or binary detectors for each class yields tight, category-aware decision regions and superior separation of out-of-distribution targets (Safaei et al., 2022, Jang et al., 2021).
• Contrastive Learning and Feature Ensembling: Supervised contrastive objectives train models to capture both “easy” and “hard” features via temperature modulation or model ensembling, leading to significant OSR performance boosts (Xu, 2024, Bahavan et al., 11 Mar 2025).
• Class-specific Manifold Modeling: Utilizing class-specific autoencoders or mixture modeling in latent space provides a flexible mechanism to capture intra-class variation, enabling improved detection of unknowns while preserving closed-set generalization (Huang et al., 2022, Cao et al., 2020, Sun et al., 2020).
• Efficient Regularization: Outlier exposure (injecting “known unknowns” during training as regularizers), background-class regularization, or dual contrastive learning (e.g., TAU in DCTAU) address the class and instance-imbalance inherent to “open” space and avoid distribution collapse (Cho et al., 2022, Li et al., 2024).
• Statistical or Bayesian nonparametric calibration: Fully Bayesian co-clustering strategies eliminate threshold selection altogether, batch-discovering new classes and tightly controlling open-space risk (Geng et al., 2018).

6. Practical Implications, Limitations, and Ongoing Challenges

Comprehensive open-set evaluation reveals several persistent challenges:

• Threshold Selection and Calibration: Despite advances, many OSR systems rely on empirical thresholding and held-out pseudo-unknowns for calibration, with class- or data-dependent sensitivity (Geng et al., 2018, Sun et al., 2023).
• Open-space Coverage: Current models may still leave “holes” in the open-space, vulnerable to both adversarial and semantically valid unknowns. EVT-based calibrations and prototype/contrastive methods mitigate but do not eliminate this issue.
• Closed-set vs. Open-set Trade-offs: Strong closed-set classifiers empirically display high open-set performance when paired with sound novelty scoring (e.g., maximum-logit or Mahalanobis scoring), but decision boundaries can still trade off known-class accuracy for low open-space risk (Vaze et al., 2021).
• Inherent Imbalance and Unseen Novelty: Modeling the vast unknown space remains fundamentally constrained by the lack of representative negative samples. Techniques that synthesize class-conditional “pseudo-unknowns” or use target-aware universa (e.g., TAU) partially alleviate this (Li et al., 2024).
• Scalability and Computational Cost: Contrastive and generative techniques can incur high computational cost; efficient prototype and vMF-based losses (e.g., SphOR) offer orders-of-magnitude speedups with competitive OSR performance (Bahavan et al., 11 Mar 2025).

A plausible implication is that future OSR research should emphasize self-/transductive learning on test batches (Sun et al., 2023), adaptive threshold selection, incorporation of semantic side information, and continual/open-world learning to dynamically assimilate newly encountered classes (Geng et al., 2018, Sun et al., 2023). Bridging the gap between strong closed-set generalization and robust open-space modeling remains central to the field.

7. Representative Methods and Quantitative Benchmarks

Method	CIFAR-10 AUROC	TinyImageNet AUROC	Macro-F1 (cross-dataset)	Key Ingredient(s)
OpenMax	0.695	0.576	0.668	EVT-calibrated Softmax
C2AE [2019]	0.895	0.748	0.801	Class-conditioned AE + EVT
CSSR [2022]	0.913	0.823	0.929	Class-specific AE Manifolds
ARPL+CS [2021]	0.910	0.782	0.870	Reciprocal-point margin loss
OpenHybrid [2020]	0.950	0.793	0.757	Flow + joint embedding
SphOR [2025]	0.947	0.810	0.936	vMF + Mixup + prototype repul
DCTAU [2024]	--	0.836	>0.93	Target-aware universum + DC
Smooth SupCon [2024]	0.940	0.866	>0.94	SupCon + diverse features

These results confirm that feature diversity, prototype-based regularization, and manifold-aware losses collectively yield state-of-the-art open-set detection and classification. Nevertheless, further progress will require advances in uncertainty quantification, adaptive calibration, and seamless handling of evolving class spaces (Xu, 2024, Huang et al., 2022, Sun et al., 2023, Geng et al., 2018).