Open Set Recognition Mechanisms

• Open set recognition mechanisms are frameworks that allow models to classify known inputs and identify unseen classes by controlling open space risk.
• They employ methods like threshold-based scoring, generative modeling, and prototype mining to differentiate between known and unknown data.
• Hybrid approaches combining discriminative and generative techniques enhance robustness, as shown by high AUROC scores in benchmark evaluations.

Open set recognition (OSR) mechanisms enable models to perform classification not only over a set of known classes but also to detect when an input originates from a class unseen during training, i.e., an unknown. This detection is essential for real-world deployments where class priors cannot be exhaustively enumerated. The challenge is to tightly control “open space risk”—the region of feature space far from known-data support—to avoid misclassifying unknowns with high confidence. Modern OSR mechanisms span threshold-based scoring, generative modeling, prototype metric learning, reconstruction-based strategies, hybrid models, and transductive adaptation, with recent developments in probabilistic modeling, deep hybrid architectures, and feature diversity.

1. OSR Formulation and Risk Principles

Open set recognition generalizes the classical closed-set assumption. Given a labeled training set with classes $\mathcal{K}$ and test set containing $\mathcal{K}\cup\mathcal{U}$ (unknowns), a robust classifier $f$ must

• Accurately assign samples from $\mathcal{K}$;
• Reject any sample from $\mathcal{U}$ as “unknown.”

Bendale & Boult introduced the concept of open space risk $R_O(f)$,

$$R_O(f) = \int_{x\in\mathcal O} \mathbb I[f(x)\in \mathcal{K}]\, dx$$

where $\mathcal O$ is the open (unseen) region of feature space. A state-of-the-art OSR system minimizes joint risk

$R(f) = R_{\rm emp}(f) + \lambda R_O(f)$

where $R_{\rm emp}$ is empirical (known-class) risk and $\lambda$ controls conservativeness (Sun et al., 2023).

Operationally, nearly all OSR mechanisms reduce to:

• Compute per-class confidence scores $\{S_k(x)\}$,
• If $\max_k S_k(x) < \tau$: declare “unknown,”
• else: return class $k^* = \arg\max_k S_k(x)$, with the choice of $S$, the decision rule, and threshold $\tau$ central to performance.

2. Score-Based and Generative OSR Mechanisms

2.1 Threshold-Based Scoring and OpenMax

A foundational OSR strategy is thresholding posterior scores (e.g., softmax, MSP, max-logit, entropy) (Sun et al., 2023, Miller et al., 25 Mar 2024, Liu et al., 2022):

• Softmax+Threshold: $S(x) = \max_k p_k(x)$. Unknown if $S(x) < \tau$.
• OpenMax: Adjusts softmax via per-class activation vectors and Weibull modeling of distance tails. The open set probability is siphoned from known-class scores depending on extremal tail likelihood (Zhang et al., 2020, Liu et al., 2022).

Method	Decision Basis	Unknown Detection
Softmax+τ	$\max_k p_k(x)$	$<\tau$
OpenMax	Weibull-calibrated	max is “unknown”

2.2 Generative and Density Models

Generative models reject samples unlikely under known-class (conditional) density models:

• Conditional Probabilistic Generative Models (CPGM) (Sun et al., 2020): VAEs/AAEs with class-conditional latent Gaussians $\mathcal N(\mu_k, I)$, combining reconstruction loss and $D_{KL}$ regularizers. At inference, low likelihood under all $p(z|k)$ or high reconstruction error triggers rejection.
• Hybrid Density Models: The OpenHybrid framework (Zhang et al., 2020) employs end-to-end joint learning of an embedding $z=E(x)$, a classifier, and a normalizing flow $F(z)$ to model $p_{\rm flow}(z)$, using likelihood thresholding:

$\text{Unknown if } \log p_{\rm flow}(E(x)) < \tau$

Joint training prevents the flow from spuriously assigning high likelihood to OOD samples.

• Gaussian Mixture VAEs (GMVAE) (Cao et al., 2020, Cao et al., 2022): Use class-conditioned or mixture priors; rejection is via nearest centroid distance or normalized uncertainty.

2.3 Reconstruction- and Prototype-Based

Autoencoder-based and prototype-based mechanisms model the known-class “region” via reconstruction or embedding distance:

• Class-Specific Semantic Reconstruction (CSSR) (Huang et al., 2022): A per-class AE is trained to reconstruct semantic backbone features $z$; min-class reconstruction error below threshold indicates known, otherwise unknown.
• Robust Prototype Mining (PMAL) (Lu et al., 2022): Explicit mining of high-quality, diverse class prototypes and embedding optimization with point-to-set margin loss. OSR is thresholding distance to nearest prototype set.
• P-ODN (Shu et al., 2019): Jointly trained prototypes and radii; robust triplet thresholding of class-prototype distance scores for acceptance, rejection, or assignment.

3. Algorithms for Resilient Open Space Carving

3.1 Sparse Representation and EVT

Sparse representation-based OSR (SROSR) uses class reconstruction errors via SRC and extreme value modeling:

• Reconstruction error tails (matched, non-matched) are modeled by generalized Pareto distributions (GPD).
• Hypothesis testing on GPD CDFs fuses tail probabilities into an overall open set score for rejection decision (Zhang et al., 2017).

3.2 OSSVM: Bounded Support Vector Machines

OSSVM imposes a $+\lambda b$ term on the SVM primal, which, with RBF kernels, ensures bounded positively-labeled region iff $b<0$:

• If all class decision functions have $b_k<0$, OVA OSSVM never labels regions far from known-data as known, bounding open-space risk by construction (Júnior et al., 2016).

3.3 Plug-and-Play Generative Models

GeMOS (Vendramini et al., 2021) fits generative models (PCA, GMM, OCSVM) on pooled CNN features per class:

• At inference, only accept prediction from class $c^*$ if the generative likelihood under $c^*$ exceeds threshold; otherwise declare unknown.

4. Hybrid and Advanced OSR Strategies

• Hybrid Discriminative–Generative Models (Zhang et al., 2020): End-to-end joint objectives specifically couple discriminative and density modeling, improving class separation and OOD likelihood suppression.
• Human Perception Informed OSR (Huang et al., 2022): Incorporates human reaction time via psychophysical loss, using reaction-time/decision-depth consistency to expand margin between known and novel classes, especially in few-shot and ambiguous conditions.
• Multi-Attribute and Multi-Label OSR (Saranrittichai et al., 2022, Zhao et al., 2023): Extending OSR to multi-attribute settings and multi-label outputs introduces challenges related to shortcut learning and cross-attribute correlation. Confidence scores for each attribute or label provide independent axes for open set detection but are vulnerable to spurious correlations unless specifically controlled.

5. OSR for Vision-Language and Domain-Adapted Models

Modern OSR extends to vision-LLMs, domain-adaptation, and open-world scenarios:

• Vision-Language OSR (Miller et al., 25 Mar 2024): Despite open-vocabulary training sets, VLMs remain vulnerable to closed-set errors due to finite query sets; predictive uncertainty (softmax, entropy) and negative embeddings mitigate open-set errors, while brute-force query expansion degrades performance.
• Open Set Domain Recognition (He et al., 2021): Combines attention-based graph convolution (GCN) for class transfer weights with semantic matching optimization for domain adaptation. Loss terms align source and target features with refined “known” and “unknown” classifier vectors using ontology-induced knowledge graphs.

6. Thresholding and Decision Rules

Effective OSR universally relies on robust threshold selection:

• Validation-based thresholding on held-out knowns or known/unknown mixtures is typical; per-class or percentile-based heuristics are widespread (Huang et al., 2022, Zhang et al., 2020, Cao et al., 2022).
• Decision rules include
  • Max softmax margin (score < $\tau$),
  • Minimum likelihood/membership to known clusters (likelihood < $\tau$),
  • Tail probability fusion or ensemble statistics,
  • Triplet or margin-based acceptance/rejection with class-specific statistics.

7. Empirical Performance and Open Challenges

Representative evaluation protocols use AUROC, macro-F1, and OSCR, across both standard and cross-dataset splits. Recent flow-based, generative, and prototype-mining methods consistently surpass earlier threshold-based and EVT calibrations, demonstrating AUROCs exceeding 0.95 on MNIST, SVHN, CIFAR-10 protocols (Zhang et al., 2020, Lu et al., 2022, Huang et al., 2022, Cao et al., 2020). "Diversity-encouraged" feature learning, via temperature-scheduled contrastive learning, further improves OSR by expanding the margin between knowns and unknowns (Xu, 16 Apr 2024).

Open problems include:

• Adaptive thresholding and class-wise calibration,
• Robustness against adversarial/shortcut attribute correlations in multi-attribute recognition,
• OSR extension to large-scale, multi-modal (e.g., VLM) and domain-adapted settings,
• Reducing performance trade-off between known-class fidelity and open space rejection.

Cross-dataset generalization, explicit modeling of feature diversity, and the integration of human-in-the-loop metrics (reaction time, psychophysics) represent current frontiers for open set recognition research (Huang et al., 2022, Xu, 16 Apr 2024, Sun et al., 2023).