Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have

Abstract: We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.

• The paper introduces FINO, a label-free framework that adapts vision models using existing metadata, eliminating the need for task-specific labels.
• The paper presents a metadata-guided self-supervised adaptation strategy that employs both discrete and continuous metadata to counter domain shifts effectively.
• The paper demonstrates FINO's robust performance across diverse scientific domains, achieving or exceeding specialized SOTA results with minimal labeled data.

Adapting Vision Foundation Models Without Labels Using Metadata

Introduction

This paper introduces FINO (FIne tuning with NO labels), a unified framework for adapting vision foundation models (VFMs) to specialized scientific domains solely using metadata, eschewing the need for explicit task-specific labels. The context is that VFMs pre-trained on large-scale, natural image distributions often perform suboptimally in specialized domains such as biomedical microscopy, earth observation, and medical imaging, due to distribution shifts not addressable by classical fine-tuning or unsupervised adaptation. Metadata—high-cardinality, often continuous annotations that accompany scientific datasets—provides a virtually cost-free, domain-specific supervisory signal. FINO leverages both discrete and continuous metadata for robust, scalable, label-free model adaptation that is resistant to domain shifts and batch effects.

Figure 1: The FINO pipeline leverages discrete and continuous metadata as weak supervision, guiding a self-supervised adaptation of a foundation model backbone and achieving or exceeding the performance of fully supervised fine-tuning and specialized SOTA methods.

Methodology: Metadata-Guided Self-Supervised Adaptation

FINO integrates self-supervised objectives (DINO, iBOT) with flexible, scalable metadata-guided loss functions and targeted regularization. It explicitly formalizes two sets of metadata:

• Informative ($\mathcal{M}_+$): Metadata corresponding to meaningful axes of variation (e.g., antibody identity in microscopy, geographic subregion).
• Spurious ($\mathcal{M}_-$): Nuisance metadata associated with acquisition variations or batch effects.

Discrete Metadata are handled by prototype-based contrastive objectives, leveraging exponentially moving average (EMA)-updated class prototypes for arbitrary cardinalities. Continuous Metadata are regressed via lightweight MLP heads. The model encourages encoding of informative metadata (backpropagating the standard loss), while suppressing encoding of spurious metadata using gradient reversal. SIGReg regularizes the representation space to avoid feature collapse.

The overall adaptation proceeds in two phases: (1) training loss heads with a frozen backbone; (2) unfreezing and fine-tuning the backbone with adaptively balanced gradients across all loss branches.

Figure 2: The method applies prototype-based contrastive losses for discrete metadata, leveraging EMA-updated prototypes to efficiently guide the model according to arbitrary, high-cardinality metadata factors.

Empirical Results Across Scientific Domains

Evaluation spans four domains: Human Protein Atlas (HPA, microscopy), FMoW (remote sensing), iWildCam (ecology), and MIMIC-CXR (chest X-ray). The same FINO configuration is applied across all domains without hyperparameter tuning. Main findings include:

• Consistent Superiority Over Baselines: FINO outperforms supervised fine-tuning, unsupervised domain adaptation (e.g., DANN, CORAL), and pure SSL adaptation on all tested benchmarks.
• Exceeding Specialized SOTA: FINO matches or exceeds specialized, domain-specific SOTA methods, including top Kaggle solutions for HPA and robust self-supervision approaches for MIMIC-CXR, despite using less task-specific information.
• Low-Label Efficiency: With only 1–10% labeled data for lightweight probing, FINO maintains high accuracy (>50% F1 on HPA), outperforming both fine-tuning and group DRO, which degrade rapidly with reduced supervision.

Table 1 (referenced in the paper) and additional ablations demonstrate that the EMA prototype mechanism for discrete metadata is crucial; replacing it with MLP substantially degrades performance.

Metadata Selection and Interpretability

A central contribution of FINO is a principled, empirically validated framework for assigning metadata to informative or spurious roles, with strong evidence that most metadata naturally fall into one regime. Notably, dependencies in metadata structure (e.g., in HPA the co-entanglement of plate, cell line, and antibody identity) necessitate caution; positive guidance is preferred for entangled factors, and suppressing them in isolation can be detrimental.

FINO's architecture is robust to these choices, as shown by sensitivity analysis (Figure 2), and offers interpretability via probe-based UMAP embeddings. FINO representations align with target variables (e.g., protein localization) and suppress spurious clustering along nuisance axes (e.g., cell line).

Figure 3: UMAP of OpenCell protein-level feature embeddings obtained with FINO, colored by subcellular localization, confirms that FINO adapts representations to capture biologically relevant target structure.

Transferability and Generalization

FINO-adapted representations transfer robustly across datasets within the same domain, yielding improved out-of-domain performance on datasets with differing class definitions, acquisition modalities, or annotation protocols. For instance, HPA-adapted features outperform prior self-supervised methods on OpenCell (protein-level ARI), and FMoW-adapted backbones improve land-cover segmentation on FLAIR-Hub.

Theoretical and Practical Implications

• Task-Agnostic Robustness: FINO decouples representation adaptation from tasks, preserving the model's generality and resisting the collapse typical of label-driven fine-tuning under distribution shift.
• Backbone Agnosticism: Ablation with SigLIP2 initialization confirms FINO's approach is general and independent of the specific VFM backbone.
• Domain-Specific SOTA Without Supervision: Substantially reduces annotation requirements, opening the possibility to deploy foundation models in low-label regimes or rapidly expanding domains.

Caveats: The method depends on reliable, unbiased metadata and requires domain expertise in assigning factors to informative/spurious categories. The model may propagate dataset biases if metadata are not appropriately selected.

Conclusion

FINO establishes a practical, theoretically sound approach to label-free domain adaptation for vision foundation models. By leveraging freely available metadata as weak supervision, it sets a new standard for robust, scalable, and task-agnostic representation learning in scientific imaging and beyond. This framework invites future theoretical analysis on the limits of metadata guidance (e.g., for adversarial or confounded metadata) and practical extensions to multimodal or spatiotemporally structured datasets.