Open-Set Recognition Protocols

• Open-set recognition protocols are evaluation frameworks that partition classes into known and unknown to enable models to classify familiar inputs and reject novel ones.
• They employ diverse methodologies, including standard, cross-dataset, and large-scale benchmarks, with metrics like AUROC, OSCR, and macro-F1 to quantify performance.
• Tailored for applications such as autonomous vision, intrusion detection, and clinical decision support, these protocols guide threshold tuning and model calibration for real-world safety.

Open-set recognition protocols define principled methodologies for evaluating classifiers in the presence of “unknown” classes—categories that are not represented in the training set but may occur during deployment. Contrasting closed-set evaluation, which assumes complete knowledge of all possible label categories, open-set protocols explicitly quantify a model’s ability to both classify among known classes and to detect or reject inputs from unseen, novel classes. This dual goal is fundamental for systems deployed in dynamic or safety-critical environments, including autonomous vision, intrusion detection, medical decision support, and large-scale open-vocabulary learning.

1. Formalization of the Open-Set Problem

Let the space of all possible classes be partitioned into known classes $\mathcal{K} = \{1, \ldots, K\}$ available at training time, and unknown classes $\mathcal{U}$, which may appear only during inference. Given a test sample $x$, an open-set recognizer $f$ must assign it to a $k\in\mathcal{K}$ or declare $x\in\mathcal{U}$ (“unknown”, “reject”).

A quantitative measure of problem difficulty is the openness, defined as: $$\mathrm{openness} = 1 - \sqrt{\frac{2|\mathcal{K}|}{|\mathcal{K}| + |\mathcal{T}|}}$$ where $|\mathcal{T}|$ is the number of total test classes (including both known and unknown). Openness $=0$ corresponds to classic closed-set recognition, while larger values increase the proportion of unknowns and thus the frequency of required rejections (Zhang et al., 2017, Sun et al., 2023).

Protocols specify:

• Which classes are designated as “known” (train-time) and “unknown” (test-time only)
• How data is split and whether unknowns are semantically or distributionally close to knowns
• The evaluation metrics for both closed-set and open-set performance

2. Core Evaluation Protocols

2.1 Standard-Dataset Protocol

In this approach, a standard multi-class dataset is split into subsets of known and unknown classes. The classifier is trained exclusively on the known classes. At test time, both known and unknown class samples are mixed; the model must assign inputs to a known label or reject as unknown (Sun et al., 2023). Typical splits include:

• MNIST: 6 known / 4 unknown
• TinyImageNet: 20 known / 180 unknown

Thresholds for unknown detection are tuned via cross-validation, often using held-out known classes or synthetic unknowns. Metrics include closed-set accuracy, AUROC, and open-set classification rates (OSCR).

2.2 Cross-Dataset Protocol

Here, the unknown samples are drawn from an entirely separate dataset, introducing covariate and domain shift in addition to semantic novelty. The classifier is trained on known classes from one dataset, with unknowns coming from out-of-domain sources (e.g., CIFAR-10 vs. LSUN), testing generalization beyond semantic class boundaries (Sun et al., 2023).

2.3 Large-Scale Benchmark Protocols

Contemporary works define scalable, semantically structured benchmarks using large taxonomies (e.g., WordNet/ImageNet). For example, protocols partition ILSVRC-2012 into:

• Known: all dog breeds
• Negative (“known-unknown”): other mammals
• Unknown: non-animal objects

Protocols P₁, P₂, P₃ vary the semantic and visual distance among K/N/U, mimicking real-world deployment where negatives may be available for training/validation but true unknowns arise only at test time (Palechor et al., 2022, Bisgin et al., 2024).

2.4 Domain-Specific Open-Set Protocols

Certain fields, such as face or intrusion recognition, employ problem-adapted protocols:

• Face: Partitioning “Labeled Faces in the Wild” into known, known-unknown (persons seen but not enrolled), and unknown-unknown (never seen) for open-set identification (Günther et al., 2017)
• Intrusion: Fine-grained exploit-level splits and evaluation on operational cost-weighted metrics (Cruz et al., 2017)

2.5 Vision-LLM (VLM) Protocols

Recent open-set protocols tailored for vision–LLMs (VLMs) test rejection capability when presented with images whose correct class is not in the query set, using standardized splits and calibration of PR-curve metrics across closed and open regimes (Miller et al., 2024).

3. Metrics and Thresholding Strategies

A core challenge is the joint evaluation of classification accuracy and rejection performance. Commonly used metrics include:

• Known-class accuracy (closed-set):

$$\mathrm{ACC}_k = \frac{\mathrm{TP_{known}}}{\mathrm{TP_{known}} + \mathrm{FN_{known}}}$$

• Open-set accuracy (accept known, reject unknown):

$\mathcal{U}$0

• ROC/AUROC: Area under TPR vs. FPR curve as rejection threshold varies
• Open-Set Classification Rate (OSCR): Plots Correct Classification Rate (CCR) for knowns versus FPR for unknowns as threshold varies (Sun et al., 2023, Palechor et al., 2022, Bisgin et al., 2024)
• Macro-F1: Averaged over C+1 classes (each known class plus the unknown class)
• Cost-weighted accuracy: Used in scenarios where misclassifying unknowns carries differing operational consequences; $\mathcal{U}$1 models this trade-off (Cruz et al., 2017)

Thresholds for rejection are commonly calibrated on a validation set of known classes (and optionally available negatives) to fix FPR, maximize macro-F1, or optimize combined metrics such as OSCR area.

4. Protocol Construction and Dataset Organization

Protocols are formalized by explicit dataset construction, ensuring reproducibility and fine control over semantic structure. For large-scale benchmarks (Palechor et al., 2022, Bisgin et al., 2024):

• Parse class taxonomy (e.g., WordNet)
• Define Known, Negative, and Unknown class IDs
• Stratified splitting into train/val/test to preserve class distributions
• Test set contains both known, negatives, and truly unseen unknowns

In specialized domains:

• Face: Disjoint sets for enrollment, known-unknowns, and unknown-unknowns, supporting evaluation across multiple open-set regimes (Günther et al., 2017)
• Healthcare: Known/unknown drug cocktails by FDA approval, per-encounter splits, and extensive validation/test cycles to robustly estimate macro-F1 (Cao et al., 2022)

Best-practice guidelines include averaging over multiple random class splits to mitigate selection bias, holding out validation splits for thresholding, and reporting results over several levels of openness.

5. Comparative Analysis and Methodological Impact

Open-set evaluation protocols jointly reward classification among knowns and robust rejection of unknowns. Empirical findings consistently demonstrate:

• Adding explicit “negative” (known-unknown) samples in training improves rejection of similar-but-nonsembled classes but does not ensure strong performance on completely unseen unknowns (Palechor et al., 2022, Bisgin et al., 2024)
• Fine-grained and cross-domain unknowns increase open-set difficulty, with all protocols showing a degradation in detection performance as semantic similarity between knowns and unknowns increases (Bisgin et al., 2024)
• Calibration and thresholding on validation negatives (as opposed to arbitrary fixed thresholds) are critical for stable operational deployment (Sun et al., 2023, Cao et al., 2022)
• Protocols offering cost-weighted or macro-F1 optimization facilitate practical deployment in settings where false rejections and false acceptances have asymmetric consequences (Cruz et al., 2017, Cao et al., 2022)

The introduction of hybrid training/post-processing protocols, such as combining entropic open-set losses with post-hoc unknown modeling (OpenMax, EVM, PROSER), yields improved rejection without sacrificing closed-set accuracy, but none universally resolves the challenge of fine-grained or truly open-world unknowns (Bisgin et al., 2024).

6. Protocol-Specific Insights and Recommendations

• For standard benchmark development, use controlled within-dataset splits to enable direct method comparison and optimize AUROC/OSCR (Sun et al., 2023)
• Realistic deployment evaluation requires protocol design that mimics operational data shift and incorporates “known-unknowns” if available (Palechor et al., 2022)
• In vision–LLMs, increasing the finite query set size does not mitigate open-set confusion; protocols must reject samples outside the candidate set, and metrics such as AuPR, P@95R, and R@95P provide nuanced performance gradients (Miller et al., 2024)
• For clinical or safety-critical applications, fix thresholds based on known validation data, report macro-F1 including an aggregate unknown class, and monitor early stopping by open-set calibration metrics (Cao et al., 2022)
• In intrusion/fine-grained settings, protocol construction should reflect the granularity of operational unknowns, with key attention paid to the balance between closed-set accuracy and novel attack rejection (Cruz et al., 2017)

The increasing maturity of open-set recognition protocols, combined with the release of deterministic data splits and evaluation code, is moving the field toward standardized, reproducible measurement of both recognition and rejection performance in the open world.