Novel Category Discovery: Methods & Challenges

• Novel Category Discovery (NCD) is a machine learning framework that transfers semantic knowledge from labeled datasets to identify and cluster completely unseen classes.
• It employs both two-stage and one-stage methods, balancing supervised representation learning and unsupervised pseudo-labeling to form coherent semantic groups.
• Key challenges include accurate pseudo-label generation, class-number estimation, and handling domain shifts and class imbalances in diverse applications.

Searching arXiv for recent and foundational papers on Novel Category Discovery to ground the article. Using the arXiv search tool to retrieve relevant NCD papers. arXiv search query: "Novel Class Discovery survey UNO tabular explainable" Novel Category Discovery (NCD) is the problem of discovering previously unseen classes in an unlabeled set by transferring knowledge from a labeled set of disjoint but related classes. In the standard formulation, the labeled set is $D_l = \{(x_i, y_i)\}_{i=1}^{n_l}$ with $y_i \in \mathcal{Y}_l$, the unlabeled set is $D_u = \{x_j\}_{j=1}^{n_u}$, and the label spaces satisfy $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$. The objective is to learn a representation $f_\theta$ and a decision rule $g_\phi$ that transfer semantic structure from $D_l$ to $D_u$, so that the unlabeled data can be partitioned into coherent novel categories; depending on the setting, the number of novel classes $K_u = |\mathcal{Y}_u|$ may be known or unknown (Troisemaine et al., 2023). What began largely as an image-centric, transductive clustering problem now spans generalized category discovery, open-world recognition, cross-domain transfer, explainability, tabular data, graphs, video, and 3D semantic segmentation (He et al., 26 Sep 2025).

1. Formal problem statement and setting variants

The canonical NCD setting assumes that all unlabeled samples come from novel classes only. In that form, the learner uses supervision from known classes to induce a transferable representation, then clusters the unlabeled set into $K_u$ categories. A common practical realization is to embed each sample as $y_i \in \mathcal{Y}_l$0, cluster the unlabeled embeddings, and learn a clustering head $y_i \in \mathcal{Y}_l$1 that maps $y_i \in \mathcal{Y}_l$2 to novel-class indices (Troisemaine et al., 2023).

Two assumptions vary across the literature. First, $y_i \in \mathcal{Y}_l$3 may be treated as known or unknown. Many methods assume it is known, while others attempt to estimate it using overparameterized output heads, probe classes, constrained $y_i \in \mathcal{Y}_l$4-means, or model selection procedures (Troisemaine et al., 2023). Second, evaluation may be transductive or inductive. Transductive protocols cluster the same unlabeled pool seen during training, whereas inductive variants require the learned model to generalize to previously unseen unlabeled instances (Troisemaine et al., 2023).

NCD is also distinguished from closely related settings. In semi-supervised learning, labeled and unlabeled data share the same label space; in NCD, the spaces are disjoint. In open-set recognition or out-of-distribution detection, the model detects unknowns but does not necessarily organize them into semantically meaningful clusters. Generalized Category Discovery (GCD) relaxes the core NCD assumption by allowing unlabeled data to contain both known and novel classes; in that sense, NCD is a special case in which test data are only novel (Troisemaine et al., 2023).

2. Methodological families and learning objectives

A standard taxonomy separates NCD methods into two-stage and one-stage approaches. Two-stage pipelines first learn a transferable representation or similarity function from labeled data, or from self-supervision on $y_i \in \mathcal{Y}_l$5, and then cluster or self-label the unlabeled set. One-stage methods instead optimize supervised losses on $y_i \in \mathcal{Y}_l$6 and unsupervised or pseudo-label objectives on $y_i \in \mathcal{Y}_l$7 jointly in a shared embedding space (Troisemaine et al., 2023).

Early representative methods such as CCN, KCL, and MCL learn a binary same/different predictor on labeled classes, generate pairwise pseudo-labels on unlabeled data, and train a $y_i \in \mathcal{Y}_l$8-way clustering head. DTC adapts Deep Embedded Clustering to NCD by pretraining on known classes, projecting unlabeled data into the learned latent space, and refining clusters via prototypes. AutoNovel combines supervised cross-entropy on labeled data, binary cross-entropy on unlabeled pairwise relations, and consistency regularization. NCL augments this line with supervised and unsupervised contrastive terms, while OpenMix uses MixUp between labeled and unlabeled data to strengthen pseudo-labels (Troisemaine et al., 2023).

A widely cited unification of the one-stage paradigm is UNO, which uses a single softmax over the combined known and novel label space. If $y_i \in \mathcal{Y}_l$9 are logits from the labeled head and $D_u = \{x_j\}_{j=1}^{n_u}$0 from the novel head, UNO applies a unified temperature-softmax $D_u = \{x_j\}_{j=1}^{n_u}$1 and a single cross-entropy to both ground-truth labels and pseudo-labels. For labeled samples, targets are padded as $D_u = \{x_j\}_{j=1}^{n_u}$2; for unlabeled samples, they are padded as $D_u = \{x_j\}_{j=1}^{n_u}$3. Pseudo-labels are generated by entropy-regularized optimal transport with Sinkhorn-Knopp and used in a swapped multi-view prediction objective (Fini et al., 2021).

Other methods depart from explicit pseudo-labeling. One line constrains the empirical first-order and second-order statistics of predicted novel-class probabilities to match a multinoulli prior, adding entropy and consistency terms so that semantic clustering emerges from distributional constraints rather than instancewise labels (Hasan et al., 2023). Another line emphasizes prototypes. Adaptive Prototype Learning combines DINO-based representation learning, nearest-prototype assignment, exponential-moving-average prototype updates, and pair-wise angular separation to improve category discrimination before an offline pseudo-labeling stage (Zhang et al., 2022). Spacing Loss shapes the latent space by transporting prototypes toward equidistant anchors derived from multidimensional scaling, enforcing intra-class compactness and inter-class dispersion; it can operate as a standalone two-stage method or as a plug-in regularizer for one-stage methods (Joseph et al., 2022).

Across these families, a recurrent abstract objective takes the form

$D_u = \{x_j\}_{j=1}^{n_u}$4

where $D_u = \{x_j\}_{j=1}^{n_u}$5 may represent entropy minimization, pairwise binary cross-entropy, prototype fitting, or spectral objectives, and $D_u = \{x_j\}_{j=1}^{n_u}$6 may include supervised or unsupervised contrastive terms (Troisemaine et al., 2023).

3. Pseudo-labels, class-number estimation, and validation without novel labels

Pseudo-labeling is central to most NCD systems, but it also introduces confirmation bias. A common formulation defines a confident set $D_u = \{x_j\}_{j=1}^{n_u}$7 and optimizes

$D_u = \{x_j\}_{j=1}^{n_u}$8

or pairwise pseudo-labels $D_u = \{x_j\}_{j=1}^{n_u}$9 derived from similarity thresholds or ranking statistics (Troisemaine et al., 2023). The accuracy of these assignments largely determines whether one-stage joint learning converges to semantic clusters or amplifies early errors.

The number of novel classes is a second fault line. Many methods assume $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$0 is known, but that assumption is often unrealistic. A practical tabular-data study explicitly targets the case in which “no prior knowledge of the novel classes is available” and proposes hyperparameter tuning by adapting $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$1-fold cross-validation and hiding some known classes in each fold. It further argues that methods with too many hyperparameters are likely to overfit these hidden classes, introduces a simple deep NCD model with only essential components, reports that the model’s latent space can be used to reliably estimate the number of novel classes, and adapts $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$2-means and Spectral Clustering to leverage known-class information (Troisemaine et al., 2023).

Open-world variants also address class-number estimation directly. OpenNCD maintains an overcomplete set of prototypes, progressively groups them using a similarity metric based on common representing instances, and uses the stabilized number of prototype groups as an estimate of the total class count; the final $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$3 yields an estimate of $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$4, and thus $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$5 (Liu et al., 2023). By contrast, explainable concept-bottleneck approaches such as xNCD remain in the standard NCD regime that assumes $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$6 is known (Uddin et al., 5 Jul 2026).

Validation without labels on $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$7 remains a practical difficulty. Surveyed strategies include probe-class splits carved from the labeled set, k-means model selection, agreement across heads or augmentations, and prototype compactness or separability criteria (Troisemaine et al., 2023). The need for such procedures reflects a broader point: in realistic deployments, model selection itself is part of the NCD problem rather than an afterthought.

4. Evaluation protocols, metrics, and benchmarks

The standard protocol holds out some classes from a fully labeled dataset to construct the unlabeled novel set. Evaluation then measures how well predicted clusters align with the withheld labels. The most common metric is clustering accuracy with optimal permutation: $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$8 where $\mathcal{Y}_u \cap \mathcal{Y}_l = \emptyset$9 is the predicted cluster and $f_\theta$0 is obtained by Hungarian matching. Normalized Mutual Information is typically reported as

$f_\theta$1

and Adjusted Rand Index measures partition agreement corrected for chance (Troisemaine et al., 2023).

Benchmarks have historically centered on MNIST, CIFAR-10, CIFAR-100, and ImageNet subsets, with fine-grained datasets such as CUB, Cars, and Aircraft used to stress subtle inter-class distinctions (Troisemaine et al., 2023). More recent evaluation distinguishes task-aware and task-agnostic settings. In task-aware evaluation, known samples are routed to known-class heads and novel samples to novel-class heads. In task-agnostic evaluation, a single head must discriminate across the full label space. UNO reported strong gains in both regimes, including 93.4 task-agnostic test accuracy on CIFAR-10 and 73.2 on CIFAR-100-20, illustrating the value of a unified softmax across known and novel classes (Fini et al., 2021).

Benchmark construction itself has become an object of study. A large-scale ImageNet benchmark based on the ENTITY-30 hierarchy keeps the number of labeled and unlabeled classes fixed while varying semantic similarity between them. In that benchmark, each split uses 90 labeled and 30 unlabeled classes, and NCD performance tracks semantic similarity closely. For example, UNO on one unlabeled split achieved 83.9 with a high-similarity labeled set, 81.0 with a medium-similarity labeled set, and 77.2 with a low-similarity labeled set; the paper further introduces transfer leakage as a formal measure of semantic relatedness and shows that it correlates strongly with NCD accuracy (Li et al., 2022).

A recent survey distills broader empirical trends: large-scale pretrained backbones, hierarchical and auxiliary cues, and curriculum-style training are beneficial for category discovery, while label assignment, class-number estimation, and scaling to complex multi-object scenarios remain unresolved bottlenecks (He et al., 26 Sep 2025).

5. Modalities, derived settings, and emerging formulations

Although the early literature was dominated by image classification, NCD is now a multi-modal open-world framework. In tabular data, TabularNCD extends the paradigm using VIME self-supervised pretraining and a two-head architecture, while later work shows that NCD can be tuned under realistic conditions without access to novel labels and can be coupled with adapted $f_\theta$2-means and Spectral Clustering (Troisemaine et al., 2023, Troisemaine et al., 2023). An interactive interface for tabular NCD further emphasizes rule-based interpretability and human-in-the-loop analysis through decision trees and latent-space visualization (Troisemaine et al., 2023).

Graph-structured data motivate NC-NCD, which removes old nodes after supervised pretraining and performs task-agnostic evaluation with a single joint classifier over old and new node classes. Its SWORD framework combines pairwise self-training for novel classes with prototype replay and feature-level distillation to preserve old-class performance without retaining old nodes (Hou et al., 2024). Video action recognition motivates NEV-NCD, a single-stage method that enforces disjointness through negative learning, confident assignments through entropy regularization, and near-equipartition through variance regularization; in the reported experiments it reaches approximately 70% clustering accuracy over unlabeled actions and improves performance under novel viewpoints when view-invariance constraints are added (Hasan et al., 2023).

3D perception has introduced additional geometric complexity. For point-cloud semantic segmentation, SNOPS combines online clustering, uncertainty-aware filtering, multi-head over-clustering, and semantic distillation from a 3D foundation model, and substantially outperforms a direct 2D-to-3D adaptation of EUMS on SemanticKITTI, SemanticPOSS, and S3DIS (Riz et al., 2023). For open-world semantic segmentation in RGB-D scenes, NeurNCD replaces explicit 3D segmentation maps with an Embedding-NeRF, aggregates semantic embedding and entropy in a volumetric space, and clusters segment descriptors with Markov Clustering; it reports 51.29 mIoU on NYUv2 and on Replica exceeds sparse-label NeRF baselines on known classes while additionally segmenting novel categories without scene-level labels (Wang et al., 6 Jun 2025).

The setting itself has diversified. OpenNCD addresses open-world semi-supervised discovery with unknown numbers of novel classes through progressive prototype grouping (Liu et al., 2023). DCCL extends the logic of NCD to GCD by alternating dynamic conception generation and dual-level contrastive learning on class-, super-class-, and sub-class-level relations (Pu et al., 2023). Few-Shot Novel Category Discovery replaces transductive unlabeled-pool assumptions with episodic support-query tasks, using Semi-supervised Hierarchical Clustering and Uncertainty-aware K-means Clustering to switch between known-class identification and novel-class clustering as query evidence accumulates (Li et al., 13 May 2025).

Interpretability has also become an explicit design target. xNCD performs representation learning and pseudo-label assignment directly in a semantic concept space aligned with vision-language priors. It argues that routing discovery through a concept bottleneck yields human-readable cluster-level signatures and instance-level concept evidence while preserving strong performance; under task-agnostic evaluation it reports 92.63 overall accuracy on CIFAR-10, close to UNO’s 93.4, and improves CIFAR-100 overall accuracy from 73.2 to 76.45 while providing intrinsic explanations (Uddin et al., 5 Jul 2026).

6. Misconceptions, controversies, and open problems

A persistent misconception is that labeled supervision is always beneficial. Several studies dispute this. When semantic similarity between labeled and unlabeled classes is high, supervised knowledge helps. When semantic similarity is low, it can be suboptimal or even harmful relative to self-supervised knowledge. On the ImageNet hierarchy benchmark, the same work that introduced transfer leakage showed that low-similarity labeled sets degrade NCD performance, and a companion analysis framed this effect through transfer flow and pseudo transfer flow, arguing that supervised knowledge may hurt when labeled and unlabeled distributions are semantically distant (Li et al., 2022, Li et al., 2023).

Another unresolved issue is domain shift. Standard NCD usually assumes labeled and unlabeled data come from the same domain, but cross-domain NCD can become ill-posed if domain-specific style is preserved in the representation. The cross-domain study on Exclusive Style Removal formalizes this point through a solvability theorem and an impossibility theorem: if the style distributions differ across domains, style information can destroy the intersection of separable transformations required for discovery. Its proposed plug-in style-removal module improves multiple baselines on OfficeHome and DomainNet40 by enforcing exclusivity between content and style features (Wang et al., 2024).

A further tension concerns balance assumptions. Balanced pseudo-label assignments, Sinkhorn transport constraints, and equipartition-style regularizers often stabilize training and prevent collapse, but they may conflict with long-tailed or skewed novel-class distributions. UNO explicitly notes that balanced assignments may clash with class imbalance, the moment-constraint approach motivates itself partly by class-imbalance failures in pseudo-labeling, and NEV-NCD’s fair-dice variance target is acknowledged to weaken under strong imbalance (Fini et al., 2021, Hasan et al., 2023, Hasan et al., 2023).

Finally, interpretability and realism remain open fronts. Most classical NCD methods discover clusters in opaque latent spaces whose coordinates have no explicit meaning. xNCD treats this opacity as a practical limitation for scientific and safety-critical settings, but its concept bottleneck still assumes $f_\theta$3 is known and, by its own account, may incur a modest accuracy–interpretability trade-off (Uddin et al., 5 Jul 2026). More broadly, a recent survey argues that the field still lacks robust solutions for label assignment, class-number estimation, and large-scale open-world scenes with complex object structure (He et al., 26 Sep 2025).

Taken together, these developments suggest that NCD is no longer best viewed as a narrow clustering problem. It is increasingly a family of open-world inference problems centered on transfer, uncertainty, semantic relatedness, and model selection under weak supervision. A plausible implication is that future progress will depend less on a single universal loss and more on setting-aware combinations of representation learning, label assignment, calibration, and semantic structure.