Assess the coverage of downstream evaluation concepts within multimodal pretraining datasets

Determine the extent to which large-scale multimodal pretraining datasets used for models such as CLIP and Stable Diffusion contain the downstream concepts employed during so-called "zero-shot" evaluation across classification, retrieval, and text-to-image generation tasks, in order to assess the meaningfulness of reported "zero-shot" generalization claims.

Background

The paper questions the validity of "zero-shot" generalization claims for multimodal models by pointing out that it is not established how well pretraining datasets cover the concepts targeted during evaluation. This motivates their data-centric analysis of concept frequencies across several widely used pretraining corpora and many downstream tasks.

Although the authors conduct an extensive empirical study relating concept frequency to performance, the abstract explicitly frames the unknown about coverage as a motivating unresolved issue, emphasizing the need to quantify the overlap between pretraining data and evaluation concepts.

References

However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation.

— No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance  (2404.04125 - Udandarao et al., 2024) in Abstract

Foundation-model pretraining overlap is unknown.

— From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model  (2609.29283 - Zhou et al., 24 Sep 2026) in Section 6, Discussion: What Does “General-Purpose” Require?, paragraph “Current transfer evidence has specific limits”

For the other checkpoints, undocumented membership remains unresolved; the available evidence supports neither confirmed independence nor confirmed contamination.

— Beyond Benchmark Scores: Auditing Medical Vision-Language Models for Chest X-Ray Tuberculosis Screening  (2609.21763 - Akhtar et al., 18 Sep 2026) in Section 3.5, “Supervision and exposure require separate qualifications”