VLM-NCD:Novel Class Discovery with Vision-Based Large Language Models

Abstract: Novel Class Discovery aims to utilise prior knowledge of known classes to classify and discover unknown classes from unlabelled data. Existing NCD methods for images primarily rely on visual features, which suffer from limitations such as insufficient feature discriminability and the long-tail distribution of data. We propose LLM-NCD, a multimodal framework that breaks this bottleneck by fusing visual-textual semantics and prototype guided clustering. Our key innovation lies in modelling cluster centres and semantic prototypes of known classes by jointly optimising known class image and text features, and a dualphase discovery mechanism that dynamically separates known or novel samples via semantic affinity thresholds and adaptive clustering. Experiments on the CIFAR-100 dataset show that compared to the current methods, this method achieves up to 25.3% improvement in accuracy for unknown classes. Notably, our method shows unique resilience to long tail distributions, a first in NCD literature.

• The paper presents VLM-NCD, a framework that fuses visual and textual features to improve the discovery of novel classes.
• It employs a two-stage pipeline with prototype-guided clustering and adaptive semi-supervised k-means to enhance inter-class separability.
• Empirical results show up to a 25.3% accuracy improvement for unknown classes, demonstrating the benefits of multimodal semantic alignment.

Vision-LLMs for Novel Class Discovery: The VLM-NCD Framework

Introduction

Novel Class Discovery (NCD) in visual recognition reflects the demand for systems capable of discovering and categorizing unknown visual concepts from unlabelled data, leveraging prior knowledge from labelled classes. Existing methods generally rely on deep visual feature clustering, but are hampered by limited generalizability to novel categories and susceptibility to long-tail distribution bias. "VLM-NCD: Novel Class Discovery with Vision-Based LLMs" (2512.10262) proposes a multimodal framework integrating vision-LLMs (VLMs) with prototype-guided clustering, explicitly targeting these deficiencies. This framework leverages large-scale pretrained visual-linguistic representations and semantic knowledge from LLMs, specifically via CLIP, for enhanced multimodal semantic alignment and dynamic clustering.

Methodology

The VLM-NCD framework introduces a multimodal feature fusion mechanism where visual and textual features are concantenated to yield joint representations, robustly improving inter-class separability. The methodology employs a two-stage pipeline. Initially, cluster centers and semantic prototypes of known classes are modeled by joint supervised optimization of both image and text features. Subsequently, novel class samples are dynamically discovered using a dual-phase mechanism: (i) separation of known/novel samples by semantic affinity thresholds derived from text similarity, and (ii) adaptive semi-supervised k-means clustering for the residual unlabelled data.

The framework utilizes CLIP's dual encoder architecture. For each image in the dataset, the system retrieves top-k relevant text captions from large-scale databases such as Conceptual Captions or LAION using cross-modal similarity. These text embeddings, mean pooled, are concatenated with the visual embeddings for each image, and the resulting joint features are clustered. Notably, the clustering step is performed via a semi-supervised k-means algorithm constrained to assign labelled data to their correct clusters, while novel classes are initialized by k-means++.

Analysis and Empirical Evaluation

Ablation studies validate that multimodal features provide a non-trivial advantage over purely visual representations. Specifically, replacing DINO-based visual encoders with CLIP, and supplementing visual features with retrieved text descriptors, results in consistently higher accuracy, especially for the 'new' (novel) class subset. The model exhibits pronounced resilience to long-tail effects, a previously unsolved challenge in the NCD literature.

To illustrate the benefit of joint feature spaces, t-SNE comparisons are provided:

Figure 2: t-SNE projections on CIFAR-10 comparing clustering boundaries with image-only versus image+text features.

When only visual features are used, class clusters exhibit noticeable overlap. The inclusion of text guidance results in more pronounced cluster separation, confirming the hypothesis that multimodal prior knowledge substantially tightens semantic clustering boundaries. The accuracy improvement is benchmarked at up to 25.3% for unknown classes compared to baseline approaches.

Additionally, the choice of text database and the number of top-k retrieved captions impact performance. Optimal results on CIFAR-100 are reported with moderate k and with Conceptual Captions 12M, but accuracy saturates or decreases with excessive descriptor augmentation.

Implications and Theoretical Considerations

VLM-NCD demonstrates that language-conditioned prototypes derived from LLM/VLMs not only improve feature separability but materially impact clustering quality for previously unseen categories. This indicates the pivotal role of multimodal pretraining in open-world recognition: the semantic priors encoded in caption corpora provide essential supervision for addressing data imbalance and semantic ambiguity inherent to long-tail novel classes.

The framework's lightweight implementation—requiring no end-to-end retraining and utilizing frozen pretrained CLIP weights—underscores the practicality of vision-LLM deployment in resource-constrained or scalable open recognition settings.

From a theoretical standpoint, the results suggest that multimodal alignment expands the effective intrinsic dimensionality of the representation space, facilitating improved partitioning for class discovery tasks. The two-stage separation-discovery protocol efficiently disentangles foreground-background and inter-class semantics without explicit prompt engineering, addressing prompt inapplicability for truly open novel class regimes.

Future Directions

Practical extensions include refining retrieval strategies to optimize contextual coverage and semantic relevance, and transfer evaluation on fine-grained or domain-agnostic datasets to further test generalization. The integration of more advanced caption generators or instruction-following LLMs could potentially serve as next-stage semantic augmenters. Additionally, dynamic selection of descriptor sets or adaptive k-schemes may provide further improvements, particularly under continuously changing class priors or evolving dataset distributions.

Conclusion

"VLM-NCD: Novel Class Discovery with Vision-Based LLMs" (2512.10262) provides strong empirical and algorithmic evidence for the value of multimodal vision-language joint representations and semantically guided prototype clustering in the NCD setting. The approach yields substantial accuracy improvements, most notably for novel and long-tail classes, and sets the stage for continued advances in open-world, minimally supervised visual recognition systems powered by LLM/VLM pretraining.