Generalized Category Discovery under Domain Shifts: From Vision to Vision-Language Models

Abstract: Generalized Category Discovery (GCD) aims to categorize unlabelled instances from both known and unknown classes by transferring knowledge from labelled data of known classes. Existing methods assume all data comes from a single domain, yet real-world unlabelled data often exhibits domain shifts alongside semantic shifts. We study GCD under domain shifts and propose three frameworks that adapt foundation models, ranging from self-supervised vision models to vision-LLMs. (i) HiLo disentangles domain and semantic features through multi-level feature extraction and mutual information minimization, combined with PatchMix augmentation and curriculum sampling. (ii) HLPrompt extends HiLo with semantic-aware spatial prompt tuning to suppress background and domain noise. (iii) VLPrompt leverages vision-LLMs via factorized textual prompts and cross-modal consistency regularization. The three methods share core design principles while operating on different foundation backbones, making them suitable for different deployment scenarios. Extensive experiments on synthetic corruptions and real-world multi-domain shifts demonstrate consistent improvements over strong baselines. Project page: https://visual-ai.github.io/hilo/

• The paper introduces three novel frameworks (HiLo, HLPrompt, VLPrompt) that disentangle domain and semantic features to enhance category discovery under complex domain shifts.
• It leverages self-supervised vision transformers, MI minimization, and PatchMix augmentation to improve clustering robustness and mitigate domain-induced noise.
• Empirical results demonstrate significant cross-domain gains and set a foundation for future extensions to multi-modal and nonparametric learning approaches.

Generalized Category Discovery under Domain Shifts: Methodological and Empirical Analysis

Problem Formulation and Theoretical Context

The paper addresses the Generalized Category Discovery (GCD) problem under domain shifts—a scenario where the unlabelled data pool may contain both known and unknown classes and exhibit systematic domain-level variations resulting from environmental, sensor, or acquisition differences. The underlying objective is to cluster all unlabelled data into meaningful categories using knowledge transferred from labelled examples of known classes and domains, in a setting where neither domain adaptation nor domain generalization methods are trivially applicable.

Formally, the problem is instantiated as learning a classifier $f: \mathcal{X} \rightarrow \mathcal{Y}_{\text{base} \cup \mathcal{Y}_{\text{novel}}$ from labelled data $(\mathbf{x}_i, y_i)$ in domain $\omega^a$ and unlabelled data from both $\omega^a$ and new domains $\Omega^b$, spanning classes $\mathcal{Y}_{\text{base} \cup \mathcal{Y}_{\text{novel}}$. The challenges are multifold: semantic discovery, domain robustness, and domain-semantic disentanglement.

The theoretical analysis emphasizes the complexity increment due to domain heterogeneity. The sample complexity grows linearly with the number of domains and classes; this is formalized by the multiplicative lower bound $N_u = \Omega((|\mathcal{Y}_{\text{base}| + |\mathcal{Y}_{\text{novel}|) \cdot d \cdot |\Omega| / \epsilon^2)$, showing the intrinsic difficulty of the GCD task as domains proliferate (2605.00906).

Methodological Framework

Three frameworks are proposed, each adapting foundation models to the GCD under domain shift: HiLo, HLPrompt, and VLPrompt.

HiLo: Feature Disentanglement and Robust Clustering

HiLo operates on self-supervised vision transformers (DINO), extracting domain features from early layers and semantic features from deep layers, enforcing independence via mutual information minimization (MI). PatchMix augmentation is used to synthesize intermediate-domain representations by mixing patch tokens, regularizing decision boundaries and improving inter-domain interpolation. Curriculum sampling systematically increases training difficulty by gradually introducing unlabelled new-domain data, stabilizing optimization and providing better initialization for robust semantic learning.

MI minimization is realized through Jensen-Shannon estimators, achieving domain-invariant semantic embeddings (2605.00906). PatchMix applies mixup at the token level, weighted by patch-level attention. The curriculum sampling schedule is dynamically adjusted via pseudo-domain labels obtained from semi-supervised k-means in domain feature space.

HLPrompt: Semantic-Aware Spatial Prompt Tuning

HLPrompt extends HiLo by adapting prompt-based input transformations, guiding visual attention towards foreground objects and suppressing domain-specific background noise. Semantic-aware spatial prompts are injected only into patches identified as foreground by unsupervised Normalized Cut (NCut) segmentation applied to ViT tokens. Alternating optimization decouples prompt learning from model parameter updates, improving convergence behavior and preventing gradient interference.

This approach leverages the foreground localization properties of DINO attention, which, empirically, concentrates on object-centric regions, making NCut masks reliable across domains.

Figure 1: Visualization of last-layer [CLS] attention maps and NCut-based foreground masks produced by DINOv3 ViT and CLIP ViT on DomainNet-GCD, demonstrating object localization and the efficacy of semantic-aware prompting.

VLPrompt: Vision-LLM Adaptation

VLPrompt adapts CLIP-based vision-LLMs to GCD with domain shifts. It utilizes factorized textual prompts, combining task-level context embeddings and category embeddings—initialized randomly for novel categories—to enable category discovery without explicit class vocabularies. Cross-modal consistency regularization, combined with PatchMix, ensures alignment between mixed visual and text representations.

Boundary-based spatial prompt tuning is used for CLIP’s vision encoder, injecting prompts at patch boundaries without disturbing global attention distribution necessary for cross-modal alignment. Alternating optimization schedules prompt and textual embedding updates for stable adaptation.

Figure 2: Comparison of fine-tuning strategies for CLIP backbone parameters demonstrates the effect of prompt tuning and input modulation on GCD performance across domains.

Ablation and Empirical Analysis

Comprehensive ablations are provided for all three methods, quantifying the incremental contribution of each design. MI minimization significantly improves performance on shifted domains, as evidenced in ablation metrics; PatchMix further regularizes inter-domain classification boundaries; curriculum sampling stabilizes learning of hard samples. HLPrompt’s spatial prompting yields substantial gains in shifted domains (e.g., +9.7% on ‘Painting’ in DomainNet-GCD) when compared to HiLo, confirming the value of input-level noise suppression.

The analysis of feature hierarchy confirms that extracting domain and semantic features from different transformer levels is critical; attempts to use only shallow or deep layers for both result in catastrophic failure.

Figure 3: The performance impact of replacing DINOv1 backbone with DINOv3 and the further gains achieved by spatial prompt tuning, highlighting backbone strength and architectural synergies in GCD robustness.

VLPrompt’s vision-language alignment loss is essential; removing it causes the largest drops in shifted domains. Boundary SPT is more reliable for CLIP, as semantic-aware SPT can disturb global attention and is less compatible with CLIP’s alignment geometry.

Figure 4: Ablation study contrasting post-backbone and post-projector features for classification, showing HiLo and HLPrompt performance consistency in GCD robustness across domains.

Implications and Future Directions

The work demonstrates strong empirical improvements across both synthetic (SSB-C, corruption-type domain shifts) and realistic (DomainNet-GCD, style-level domain shifts) benchmarks. HiLo provides a principled vision-only solution for domains with explicit control requirements and interpretability; HLPrompt augments this with prompt-based attention localization; VLPrompt leverages vision-language alignment for extreme cross-domain gaps.

The methods, however, rely on domain-class independence and require a priori knowledge of the total number of categories. Relaxation of these assumptions (via joint domain-label modeling, or nonparametric estimation) and further extension to multi-modal paradigms are open directions. The approach is theoretically justified: MI minimization yields tighter generalization bounds, and curriculum sampling improves optimization landscape.

Recent evidence indicates that VLMs are robust beyond memorization, retaining transfer performance even with aggressive train-test pruning [mayilvahanan2024forgotten]. Nonetheless, careful evaluation remains critical.

Conclusion

The paper provides a rigorous and practical suite of frameworks for GCD under domain shifts, with theoretical and empirical evidence supporting the core claim: explicit disentanglement, intermediate-domain augmentation, and prompt-based adaptation substantially enhance the robustness and discoverability of categories in heterogeneous open-world data. The integration of spatial and cross-modal prompting within foundation models establishes an effective template for future development in domain-agnostic open-world learning (2605.00906).