Identify the key to genuine zero-shot generalization under large-scale training

Ascertain the mechanisms or training strategies that enable multimodal models trained at large scale (e.g., CLIP and Stable Diffusion) to achieve true zero-shot generalization without relying on exponentially increasing pretraining concept frequency, thereby overcoming the observed log-linear sample inefficiency.

Background

Across 34 models and multiple pretraining datasets, the authors find a persistent log-linear scaling between concept frequency and downstream performance, indicating exponential data requirements for linear gains and challenging the notion of "zero-shot" generalization.

They explicitly conclude that despite extensive analysis and controls, the fundamental driver of genuine zero-shot generalization in large-scale multimodal training remains unidentified, framing a central open problem for future work.