---
title: Open-Set Recognition Protocols
url: https://www.emergentmind.com/topics/open-set-recognition-protocols
type: topic
---

# Open-Set Recognition Protocols

Open-set recognition protocols define principled methodologies for evaluating classifiers in the presence of “unknown” classes—categories that are not represented in the training set but may occur during deployment. Contrasting closed-set evaluation, which assumes complete knowledge of all possible label categories, open-set protocols explicitly quantify a model’s ability to both classify among known classes and to detect or reject inputs from unseen, novel classes. This dual goal is fundamental for systems deployed in dynamic or safety-critical environments, including autonomous vision, intrusion detection, medical decision support, and large-scale open-vocabulary learning.

## 1. Formalization of the Open-Set Problem

Let the space of all possible classes be partitioned into known classes $\mathcal{K} = \{1, \ldots, K\}$ available at training time, and unknown classes $\mathcal{U}$, which may appear only during inference. Given a test sample $x$, an open-set recognizer $f$ must assign it to a $k\in\mathcal{K}$ or declare $x\in\mathcal{U}$ (“unknown”, “reject”).

A quantitative measure of problem difficulty is the openness, defined as:
\[
\mathrm{openness} = 1 - \sqrt{\frac{2|\mathcal{K}|}{|\mathcal{K}| + |\mathcal{T}|}}
\]
where $|\mathcal{T}|$ is the number of total test classes (including both known and unknown). Openness $=0$ corresponds to classic closed-set recognition, while larger values increase the proportion of unknowns and thus the frequency of required rejections [1705.02431, 2312.15571].

Protocols specify:
- Which classes are designated as “known” (train-time) and “unknown” (test-time only)
- How data is split and whether unknowns are semantically or distributionally close to knowns
- The evaluation metrics for both closed-set and open-set performance

## 2. Core Evaluation Protocols

### 2.1 Standard-Dataset Protocol

In this approach, a standard multi-class dataset is split into subsets of known and unknown classes. The classifier is trained exclusively on the known classes. At test time, both known and unknown class samples are mixed; the model must assign inputs to a known label or reject as unknown [2312.15571]. Typical splits include:
- MNIST: 6 known / 4 unknown
- TinyImageNet: 20 known / 180 unknown

Thresholds for unknown detection are tuned via cross-validation, often using held-out known classes or synthetic unknowns. Metrics include closed-set accuracy, AUROC, and open-set classification rates (OSCR).

### 2.2 Cross-Dataset Protocol

Here, the unknown samples are drawn from an entirely separate dataset, introducing covariate and domain shift in addition to semantic novelty. The classifier is trained on known classes from one dataset, with unknowns coming from out-of-domain sources (e.g., CIFAR-10 vs. LSUN), testing generalization beyond semantic class boundaries [2312.15571].

### 2.3 Large-Scale Benchmark Protocols

Contemporary works define scalable, semantically structured benchmarks using large taxonomies (e.g., WordNet/ImageNet). For example, protocols partition ILSVRC-2012 into:
- Known: all dog breeds
- Negative (“known-unknown”): other mammals
- Unknown: non-animal objects

Protocols P₁, P₂, P₃ vary the semantic and visual distance among K/N/U, mimicking real-world deployment where negatives may be available for training/validation but true unknowns arise only at test time [2210.06789, 2406.09112].

### 2.4 Domain-Specific Open-Set Protocols

Certain fields, such as face or intrusion recognition, employ problem-adapted protocols:
- Face: Partitioning “Labeled Faces in the Wild” into known, known-unknown (persons seen but not enrolled), and unknown-unknown (never seen) for open-set identification [1705.01567]
- Intrusion: Fine-grained exploit-level splits and evaluation on operational cost-weighted metrics [1703.02244]

### 2.5 Vision-Language Model (VLM) Protocols

Recent open-set protocols tailored for vision–language models (VLMs) test rejection capability when presented with images whose correct class is not in the query set, using standardized splits and calibration of PR-curve metrics across closed and open regimes [2403.16528].

## 3. Metrics and Thresholding Strategies

A core challenge is the joint evaluation of classification accuracy and rejection performance. Commonly used metrics include:

- **Known-class accuracy (closed-set):**
  \[
  \mathrm{ACC}_k = \frac{\mathrm{TP_{known}}}{\mathrm{TP_{known}} + \mathrm{FN_{known}}}
  \]
- **Open-set accuracy** (accept known, reject unknown):
  \[
  \mathrm{ACC_{open}} = \frac{\mathrm{TP} + \mathrm{TN}}{\mathrm{TP} + \mathrm{FN} + \mathrm{FP} + \mathrm{TN}}
  \]
- **ROC/AUROC:** Area under TPR vs. FPR curve as rejection threshold varies
- **Open-Set Classification Rate (OSCR):** Plots Correct Classification Rate (CCR) for knowns versus FPR for unknowns as threshold varies [2312.15571, 2210.06789, 2406.09112]
- **Macro-F1:** Averaged over C+1 classes (each known class plus the unknown class)
- **Cost-weighted accuracy**: Used in scenarios where misclassifying unknowns carries differing operational consequences; $\mathrm{ACC}_\beta$ models this trade-off [1703.02244]

Thresholds for rejection are commonly calibrated on a validation set of known classes (and optionally available negatives) to fix FPR, maximize macro-F1, or optimize combined metrics such as OSCR area.

## 4. Protocol Construction and Dataset Organization

Protocols are formalized by explicit dataset construction, ensuring reproducibility and fine control over semantic structure. For large-scale benchmarks [2210.06789, 2406.09112]:
- Parse class taxonomy (e.g., WordNet)
- Define Known, Negative, and Unknown class IDs
- Stratified splitting into train/val/test to preserve class distributions
- Test set contains both known, negatives, and truly unseen unknowns

In specialized domains:
- Face: Disjoint sets for enrollment, known-unknowns, and unknown-unknowns, supporting evaluation across multiple open-set regimes [1705.01567]
- Healthcare: Known/unknown drug cocktails by FDA approval, per-encounter splits, and extensive validation/test cycles to robustly estimate macro-F1 [2201.02923]

Best-practice guidelines include averaging over multiple random class splits to mitigate selection bias, holding out validation splits for thresholding, and reporting results over several levels of openness.

## 5. Comparative Analysis and Methodological Impact

Open-set evaluation protocols jointly reward classification among knowns and robust rejection of unknowns. Empirical findings consistently demonstrate:
- Adding explicit “negative” (known-unknown) samples in training improves rejection of similar-but-nonsembled classes but does not ensure strong performance on completely unseen unknowns [2210.06789, 2406.09112]
- Fine-grained and cross-domain unknowns increase open-set difficulty, with all protocols showing a degradation in detection performance as semantic similarity between knowns and unknowns increases [2406.09112]
- Calibration and thresholding on validation negatives (as opposed to arbitrary fixed thresholds) are critical for stable operational deployment [2312.15571, 2201.02923]
- Protocols offering cost-weighted or macro-F1 optimization facilitate practical deployment in settings where false rejections and false acceptances have asymmetric consequences [1703.02244, 2201.02923]

The introduction of hybrid training/post-processing protocols, such as combining entropic open-set losses with post-hoc unknown modeling (OpenMax, EVM, PROSER), yields improved rejection without sacrificing closed-set accuracy, but none universally resolves the challenge of fine-grained or truly open-world unknowns [2406.09112].

## 6. Protocol-Specific Insights and Recommendations

- For standard benchmark development, use controlled within-dataset splits to enable direct method comparison and optimize AUROC/OSCR [2312.15571]
- Realistic deployment evaluation requires protocol design that mimics operational data shift and incorporates “known-unknowns” if available [2210.06789]
- In vision–language models, increasing the finite query set size does not mitigate open-set confusion; protocols must reject samples outside the candidate set, and metrics such as AuPR, P@95R, and R@95P provide nuanced performance gradients [2403.16528]
- For clinical or safety-critical applications, fix thresholds based on known validation data, report macro-F1 including an aggregate unknown class, and monitor early stopping by open-set calibration metrics [2201.02923]
- In intrusion/fine-grained settings, protocol construction should reflect the granularity of operational unknowns, with key attention paid to the balance between closed-set accuracy and novel attack rejection [1703.02244]

The increasing maturity of open-set recognition protocols, combined with the release of deterministic data splits and evaluation code, is moving the field toward standardized, reproducible measurement of both recognition and rejection performance in the open world.

Source: https://www.emergentmind.com/topics/open-set-recognition-protocols