Incremental Open Set Recognition

• Incremental Open Set Recognition (IOSR) is a paradigm that classifies known classes while detecting and managing unknown classes in evolving data streams.
• It employs techniques like online detection, clustering, and replay-based updates to address open-space risk and mitigate catastrophic forgetting.
• Research in IOSR focuses on adaptive thresholding, metric learning, and generative replay to enhance performance in non-stationary and resource-constrained settings.

Incremental Open Set Recognition (IOSR) denotes a learning regime in which a recognition system must simultaneously classify currently known classes, detect samples from unseen classes, structure or otherwise handle those unknowns, and update itself as new classes are incorporated over time. Across the literature, closely related formulations appear under names such as open-world recognition, streaming open set recognition, incremental open-set learning, open-set class-incremental learning, open-set graph class-incremental learning, and incremental open-set domain adaptation. A recurring core requirement is that the model operate under non-stationarity while controlling open-space or over-occupied-space effects and catastrophic forgetting, often under memory or computation constraints (Liu et al., 25 Mar 2026).

1. Conceptual scope and relation to adjacent paradigms

IOSR extends standard open-set recognition (OSR) beyond static settings. In classical OSR, training is performed on a fixed set of known classes, and test data may contain previously unseen classes that must be rejected rather than compulsorily classified as known. The OSR surveys formalize this as recognition over known classes with an explicit reject option for unknown classes, motivated by open space risk and the practical impossibility of enumerating all possible categories during training (Mahdavi et al., 2021).

What distinguishes IOSR is the incremental or streaming requirement. In one formulation, data arrive as a stream $(\mathbf{x}_t, y_t)$, and the system must decide whether each sample belongs to known classes or unknown classes while the classifier and the associated clustering or metric components are incrementally updated (Barcina-Blanco et al., 2024). In another formulation, the model is trained over a sequence of tasks, and at evaluation time the known classes are those seen up to the current task while the unknown classes are exactly the classes that will become known at the next phase (Yang et al., 8 Sep 2025). Open-world recognition expresses a similar idea through online learning, in which new labeled classes are added when they appear, while unknown detection remains active throughout deployment (Rosa et al., 2016).

IOSR also differs from conventional class-incremental learning. Closed-set incremental learning typically assumes that arriving classes are already labeled and known to the learner when they appear. IOSR instead requires an additional stage in which samples from unseen classes are first detected as unknown, and in many formulations subsequently clustered, pseudo-labeled, or manually validated before being absorbed into the classifier (Liu et al., 25 Mar 2026). Relative to transductive or semi-supervised OSC, IOSR further adds temporal persistence: the model is expected to evolve from one open-set phase to the next rather than solve a single pool-based unknown-detection problem (Yang et al., 2020).

2. Formal problem formulations

Several formalizations of IOSR appear in the literature, differing mainly in modality and supervision assumptions. In an open-world RF-based UAV setting, the class universe is divided into initially known classes $\mathcal{C}_0$ and deployment-time unknown classes $\mathcal{C}_u$, with $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$ and $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$. A replay memory $\mathcal{M}$ is constrained by $|\mathcal{M}| \le M_{\max}$, and the objective is to maximize performance on old and new classes under both memory and computation budgets (Liu et al., 25 Mar 2026). In that setting, unknowns are detected, clustered into candidate new classes, and then incrementally integrated, so that the effective known set grows as $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$ (Liu et al., 25 Mar 2026).

Streaming OSR adopts a sequential formulation in which $y_t \in \mathcal{C}_{KC} \cup \mathcal{C}_{UC}$, with $\mathcal{C}_{KC}\cap \mathcal{C}_{UC}=\emptyset$. The system must decide online whether $\mathcal{C}_0$0 belongs to known classes or unknown classes and, if known, predict the correct label. This formulation is explicitly linked to resilient AI, concept drift, non-stationarity, and evolving data, and it frames over-occupied space as the central failure mode of closed-set streaming classifiers (Barcina-Blanco et al., 2024).

Online open world recognition provides a supervised online variant. At time $\mathcal{C}_0$1, the system has a known label set $\mathcal{C}_0$2, predicts $\mathcal{C}_0$3, then receives the true label $\mathcal{C}_0$4, updates the model and the metric, and adds new classes whenever $\mathcal{C}_0$5 is not already in $\mathcal{C}_0$6 (Rosa et al., 2016). This formulation foregrounds incremental metric learning and incremental threshold estimation as necessary ingredients of open-world learning.

In few-shot class-incremental learning with the open-set hypothesis, the data arrive as disjoint sessions $\mathcal{C}_0$7, with a large base session and few-shot novel sessions. The base session is trained as a $\mathcal{C}_0$8-way classifier, reserving an explicit unknown class for future categories. The corresponding objective combines empirical classification risk on known data with open-space risk over unknown space (Cui et al., 2022). A related graph formulation, open-set graph class-incremental learning, defines a sequence of node-classification tasks $\mathcal{C}_0$9, where each task introduces disjoint new classes, while test nodes are partitioned into known-class nodes and unknown-class nodes that have never been seen in training up to task $\mathcal{C}_u$0 (Chen et al., 23 Jul 2025).

A domain-adaptive variant, incremental open-set domain adaptation, assumes one labeled source domain $\mathcal{C}_u$1, unlabeled target domains $\mathcal{C}_u$2, unequal domain distributions, a shared set of known classes $\mathcal{C}_u$3, and target-private unknown classes. Only the current domain is accessible at each time-stamp, and inference is performed over all seen target domains without revealing their identities (Rakshit et al., 2024). This suggests that IOSR is not tied to a single-domain or fully supervised setting; rather, the shared structure is incremental learning under open-set conditions.

3. Canonical pipeline: detection, discovery, and incremental integration

A common IOSR pipeline contains four stages: structured representation learning, unknown detection, unknown structuring, and incremental model update. One explicit formulation begins with semantic embedding learning, where a shared encoder maps inputs to a feature space designed to exhibit intra-class compactness and inter-class separation; this shared space is then used for both open-set recognition and incremental learning (Liu et al., 25 Mar 2026). OpenIncrement articulates the same principle in terms of a representation-quality ratio $\mathcal{C}_u$4, emphasizing that incremental learning procedures can distort feature geometry in ways that directly degrade distance-based OSR (Xu et al., 2023).

Unknown detection is typically implemented through either statistical rejection or similarity-based rejection. Mahalanobis-distance OSR models each known class as a Gaussian in embedding space and rejects a sample if its nearest-class distance exceeds a per-class threshold set by the three-sigma rule (Liu et al., 25 Mar 2026). Streaming OSR instead computes pseudo-probabilities from distances to cluster centroids and uses normalized entropy $\mathcal{C}_u$5 as an unknownness score, with high entropy indicating a likely unknown (Barcina-Blanco et al., 2024). Online open world recognition uses confidence functions derived from learned metrics, including RBF-style confidence $\mathcal{C}_u$6, with online threshold updates for novelty detection (Rosa et al., 2016). Distance-based KNN similarity to exemplars has also been used as the OSR score in deep class-incremental learning, with thresholding on a normalized per-class similarity statistic (Xu et al., 2023).

Unknown structuring is often handled by clustering. In RF-based IOSR, unknown samples are accumulated in a buffer $\mathcal{C}_u$7, standardized, optionally compressed by PCA if the embedding dimension exceeds 64, and then clustered by K-Means or GMM. The number of new classes is selected through a hybrid elbow plus composite-validity criterion involving Silhouette score, Calinski–Harabasz index, Davies–Bouldin index, and explained variance (Liu et al., 25 Mar 2026). In authorship attribution, unknown texts detected by an ensemble outlier detector are clustered with algorithms including K-Means, BIRCH, DBSCAN, and Spectral clustering; dominant clusters are then instantiated as new authors and used to retrain the classifier (Leo et al., 2019). The streaming OSC extension of S2OSC uses a pool-based filtering stage to identify distinct out-of-class instances, trains a holistic classifier using labeled known data, labeled unknown-superclass data, and unlabeled data, and then performs incremental updates with memory and manual labeling of newly detected unknowns (Yang et al., 2020).

Incremental integration is commonly performed through replay or exemplar-based updates. One replay-based RF IOSR loop assembles mixed training batches from newly discovered classes and a bounded replay memory $\mathcal{C}_u$8, optimizes a cross-entropy loss over the expanded label set, recomputes class statistics $\mathcal{C}_u$9, and updates memory under per-class caps (Liu et al., 25 Mar 2026). OpenIncrement similarly uses supervised contrastive learning, relation-based knowledge distillation on distances and angles, and exemplar memory updated by isometric sampling, then trains a linear classifier on the frozen encoder features (Xu et al., 2023). In online open world recognition, new classes are incorporated by initializing a new class mean or a new local ball and updating the metric and thresholds online (Rosa et al., 2016). In all of these variants, the core operation is the same: unknowns are not only rejected, but eventually transformed into known classes through a bounded incremental update.

4. Representative methodological families and domain-specific instantiations

IOSR methods can be organized by how they represent known classes and how they reserve space for future unknowns. Prototype-based and metric-based methods are one major family. Online open world recognition extends NCM, NNO, and NBC through online metric learning, incremental updates of class means or local balls, and online threshold estimation, arguing that local and online learning are important to capture the full dynamics of open world recognition (Rosa et al., 2016). OpenIncrement uses a deep encoder plus exemplar memory, but still relies on distance-based OSR in the learned embedding space (Xu et al., 2023). Recent IOSR work in angular space goes further by fixing Equiangular Tight Frame prototypes, treating some prototypes as inactive, and encouraging unknown representations to align around these inactive directions so that future class insertion produces less representation drift (Yang et al., 8 Sep 2025).

A second family couples representation learning with explicit modeling of unknown regions. In few-shot class-incremental learning with the open-set hypothesis, Hyper-RPL uses reciprocal points to represent extra-class space and integrates Euclidean and hyperbolic distances in a Poincaré-ball formulation. The base session is trained as an open-set classifier, while a novel branch learns later classes with distillation and hyperbolic metric learning (Cui et al., 2022). Retentive Angular Representation Learning similarly reserves representational capacity for future classes, but does so via inactive ETF prototypes, a virtual-intrinsic interactive training strategy, and stratified rectification for old/new and positive/negative imbalance (Yang et al., 8 Sep 2025).

A third family relies on clustering and transductive or semi-supervised refinement. S2OSC filters distinct out-of-class instances from an unlabeled test pool using a score $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$0, where $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$1 is prediction uncertainty and $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$2 is distance to known class centers, then trains a holistic classifier with labeled known data, labeled unknown-superclass data, and remaining unlabeled data in a semi-supervised paradigm with consistency and distillation (Yang et al., 2020). The incremental extension I-S2OSC processes a stream $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$3, performs OSC on each new pool, manually labels detected unknowns, updates the model with replay under an explicit forgetting-control constraint, and maintains a bounded memory $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$4 (Yang et al., 2020). Streaming OSR as formulated for data streams likewise combines an incremental classifier with an incremental clustering model and an optional UC→KC consolidation module, although that consolidation stage is conceptual rather than fully evaluated (Barcina-Blanco et al., 2024).

A fourth family uses generative replay and modality-specific open-set objectives. Open-set graph class-incremental learning introduces a prototypical conditional variational autoencoder to synthesize node embeddings for old classes, a mixing-based strategy to generate OOD samples, and a prototypical hypersphere classification loss that anchors ID embeddings to class prototypes while repelling OOD embeddings away from all prototypes (Chen et al., 23 Jul 2025). Incremental open-set domain adaptation constructs a pseudo source domain from random noise with MDCGAN, then adapts that pseudo source to the current target domain with a multi-output open-set domain adaptation network and OSBP-style losses, thereby combining generative replay with open-set adaptation across domain streams (Rakshit et al., 2024). In open-set video face recognition, OSDe-SVM combines a fixed ArcFace feature encoder with an Open-Set Dynamic Ensembles of SVM, EVT-based sequence-level recognition, diversity-based pruning, and self-healing to support low-labelled streaming operation (Lopez-Lopez et al., 2020). These domain-specific systems differ operationally, but they instantiate the same IOSR pattern under different constraints.

5. Evaluation protocols and empirical behavior

IOSR evaluation typically separates known-class classification, unknown detection, clustering quality, and forgetting. Open-set surveys emphasize metrics such as accuracy on known classes, AUROC for known-versus-unknown discrimination, macro-F1 over known plus unknown, and OSCR, which measures the trade-off between correct classification of knowns and false acceptance of unknowns (Sun et al., 2023). Streaming OSR explicitly uses KC-Acc, UC-Acc, open macro F1, and AUROC, selecting thresholds by the Youden Index and reporting Wilcoxon signed-rank tests over varying missing-class ratios $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$5 (Barcina-Blanco et al., 2024).

Representative IOSR systems reveal different trade-offs. In RF-based IOSR on a real-world UAV RF dataset with 24 UAV types across multiple bands, the scenario uses 18 known classes and 6 unknown classes, STFT spectrograms as inputs, and reports strong diagonal dominance in open-set confusion matrices, clustering purity around $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$6, and replay-sensitive incremental performance (Liu et al., 25 Mar 2026). In that study, without replay old-class accuracy falls to $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$7 while new-class accuracy is $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$8; with $\mathcal{C}_u \cap \mathcal{C}_0 = \emptyset$9 replay samples per old class, old-class accuracy becomes $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$0 and new-class accuracy is $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$1; with $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$2 replay samples per old class, old-class accuracy remains $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$3 and new-class accuracy is $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$4 (Liu et al., 25 Mar 2026). This suggests that, at least in that RF setting, very small rehearsal can be sufficient to stabilize old knowledge.

Streaming OSR shows a different profile. On isoGauss datasets at $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$5, static and incremental closed-set classifiers attain KC-Acc of about $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$6 but UC-Acc near $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$7 and $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$8, whereas the hybrid streaming OSR framework reaches UC-Acc around $\mathcal{C}_u \cup \mathcal{C}_0 = \mathcal{C}_t$9 at the cost of KC-Acc around $\mathcal{M}$0; at $\mathcal{M}$1, sOSR attains UC-Acc around $\mathcal{M}$2 and AUROC around $\mathcal{M}$3, while F1 becomes comparable to static and incremental methods (Barcina-Blanco et al., 2024). The interpretation given there is that sOSR better mitigates over-occupied space, but often sacrifices KC accuracy and depends strongly on clustering quality and threshold selection (Barcina-Blanco et al., 2024).

OpenIncrement, evaluated on CIFAR-100 and Tiny ImageNet, reports both incremental classification accuracy and OSR AUROC. With 2000 exemplars on CIFAR-100, iCaRL achieves $\mathcal{M}$4, DER $\mathcal{M}$5, and OpenIncrement $\mathcal{M}$6 incremental classification accuracy; for OSR AUROC on the same setting, CE+ResKD attains $\mathcal{M}$7, CE+RKD $\mathcal{M}$8, OpenIncrement $\mathcal{M}$9, and a jointly trained supervised-contrastive model $|\mathcal{M}| \le M_{\max}$0 (Xu et al., 2023). The authors relate these gains to lower feature distortion and better preservation of the intra-class compactness and inter-class separation needed by distance-based OSR (Xu et al., 2023).

In text-based open-set incremental learning, a single cycle of open-set detection, clustering, and retraining on Victorian and CCAT-50 authorship datasets yields large gains in accuracy and F1 relative to pre-trained models. For example, on Victorian with 5 classes the model moves from $|\mathcal{M}| \le M_{\max}$1 accuracy and $|\mathcal{M}| \le M_{\max}$2 F1 to $|\mathcal{M}| \le M_{\max}$3 accuracy and $|\mathcal{M}| \le M_{\max}$4 F1 after the open-set step, while on CCAT-50 with 10 classes it moves from $|\mathcal{M}| \le M_{\max}$5 accuracy and $|\mathcal{M}| \le M_{\max}$6 F1 to $|\mathcal{M}| \le M_{\max}$7 accuracy and $|\mathcal{M}| \le M_{\max}$8 F1 (Leo et al., 2019). The same work introduces Incremental Class Accuracy (ICA), combining homogeneity, completeness, and unknown-identification accuracy for a cluster; on the one-new-class setting, ICA declines as the initial number of known classes increases, indicating that class growth makes reliable cluster-to-class promotion harder (Leo et al., 2019).

Finally, recent benchmark-driven IOSR work such as RARL reports state-of-the-art performance under explicit IOSR protocols on CIFAR100 and TinyImageNet, with gains over strong CIL baselines such as LUCIR in both closed-set accuracy and open-set metrics. On CIFAR100 Base 20 + 8 Steps, for instance, RARL improves average ACC from $|\mathcal{M}| \le M_{\max}$9 to $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$0, average AUROC from $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$1 to $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$2, and average OSCR from $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$3 to $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$4 relative to LUCIR (Yang et al., 8 Sep 2025). These results support the view that maintaining a representation geometry suitable for both retention and unknown rejection is central to IOSR.

6. Limitations, controversies, and research directions

Several limitations recur across the literature. A first is dependence on representation quality. RF-based IOSR explicitly notes that if the learned embedding is not well-structured, both Mahalanobis thresholds and clustering quality can degrade (Liu et al., 25 Mar 2026). OpenIncrement similarly argues that feature distortion accumulated during incremental updates undermines distance-based OSR assumptions, and RARL frames representation drift as the central reason why OSR decision boundaries become hard to maintain over time (Xu et al., 2023). This suggests that IOSR is not merely a matter of attaching a reject option to a continual learner; it requires continual preservation of geometry.

A second limitation concerns the modeling of unknowns. Some methods treat all unknowns as a single cluster or superclass for evaluation convenience, while others argue that this is suboptimal because unknowns are inherently multi-modal. OGCIL explicitly rejects the idea of assigning all unknown samples to one cluster and instead models them as outliers through prototype-aware rejection regions (Chen et al., 23 Jul 2025). Streaming OSR leaves the UC→KC consolidation mechanism unspecified in practice, highlighting the unresolved challenge of deciding when a cluster of unknowns represents a genuine new class rather than drift (Barcina-Blanco et al., 2024). In few-shot IOSR-like settings, the open-set mechanism is often only trained for the base classes, leaving open the question of how to maintain OSR over all later novel classes (Cui et al., 2022).

A third issue is thresholding and calibration. Papers about OSR note that many methods depend on constant global thresholds, even though there is no prior knowledge about unknown classes and threshold robustness degrades as the problem evolves (Mahdavi et al., 2021). Streaming OSR identifies the fixed entropy threshold $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$5 as a major weakness, and video face IOSR likewise uses a fixed Weibull threshold $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$6 without adaptive tuning (Barcina-Blanco et al., 2024). Online open world recognition is notable for treating threshold estimation itself as an incremental learning problem, updating $$\mathcal{C}_0 \leftarrow \mathcal{C}_0 \cup \mathcal{C}_{\text{new}}$$7 or its Hoeffding-bound variant over time (Rosa et al., 2016). A plausible implication is that adaptive, class-specific, and phase-aware thresholding remains one of the most important unsolved IOSR subproblems.

A fourth issue is the level of supervision assumed when converting unknowns into knowns. Some systems assume that true labels arrive after prediction, as in online open world recognition (Rosa et al., 2016). Others rely on clustering plus manual or semi-automatic validation, as in RF-based IOSR and I-S2OSC (Liu et al., 25 Mar 2026). Still others, such as the authorship-attribution framework, instantiate new classes from clusters and retrain from scratch, thereby reducing forgetting by reusing all earlier data but sacrificing scalability (Leo et al., 2019). These differences reflect a genuine methodological divide rather than a settled standard: IOSR spans supervised online class revelation, semi-supervised cluster promotion, and unsupervised pool-based discovery.

Current research directions therefore converge on a few themes: stronger streaming clustering and adaptive hyperparameter selection in data streams (Barcina-Blanco et al., 2024); generative replay without raw-data storage for graphs and domain streams (Chen et al., 23 Jul 2025); representation spaces that reserve capacity for future classes via inactive prototypes or open-set hypotheses (Yang et al., 8 Sep 2025); and unified pipelines in which OSR, novel class discovery, and incremental learning are optimized jointly rather than bolted together post hoc (Liu et al., 25 Mar 2026). Taken together, the literature presents IOSR as an open-world learning problem in which recognition, rejection, discovery, and retention must be solved as a single coupled system rather than as separable modules.