Fake it till you make it: Learning transferable representations from synthetic ImageNet clones (2212.08420v2)

Abstract: Recent image generation models such as Stable Diffusion have exhibited an impressive ability to generate fairly realistic images starting from a simple text prompt. Could such models render real images obsolete for training image prediction models? In this paper, we answer part of this provocative question by investigating the need for real images when training models for ImageNet classification. Provided only with the class names that have been used to build the dataset, we explore the ability of Stable Diffusion to generate synthetic clones of ImageNet and measure how useful these are for training classification models from scratch. We show that with minimal and class-agnostic prompt engineering, ImageNet clones are able to close a large part of the gap between models produced by synthetic images and models trained with real images, for the several standard classification benchmarks that we consider in this study. More importantly, we show that models trained on synthetic images exhibit strong generalization properties and perform on par with models trained on real data for transfer. Project page: https://europe.naverlabs.com/imagenet-sd/

Review of "Fake it till you make it: Learning transferable representations from synthetic ImageNet clones"

This paper proposes a comprehensive examination of synthetic image generation using advanced diffusion models, specifically focusing on the creation of ImageNet clones through Stable Diffusion (SD). The primary objective is to ascertain whether synthetic images can replace real ones in training highly accurate image classification models. The researchers investigate this by training models on datasets comprised exclusively of synthetic images corresponding to ImageNet classes and evaluating their performance across several domains including ImageNet validation, out-of-distribution tasks, and transfer learning scenarios.

Synthetic Datasets and Prompt Engineering

The creation of synthetic datasets is a multistep process requiring extensive prompt engineering to generate high-quality images depicting ImageNet classes. The paper evaluates various prompt configurations, from simple class names to those with appended hypernyms or semantic descriptions, aiming to resolve issues like semantic ambiguity and lack of visual diversity. The synthetic datasets developed include ImageNet-100-SD, a subset of ImageNet-1K selected classes, and ImageNet-1K-SD, encompassing the entire ImageNet-1K dataset.

Through prompt variation and adjustments to Stable Diffusion's internal parameters such as guidance scale, the researchers addressed semantic and diversity issues inherent in the synthetic data. For example, adding hypernyms and class descriptions improved semantic accuracy and sometimes augmented image diversity. Results indicate that setting a lower guidance scale and generating images in diverse backgrounds positively impacts the diversity of synthetic datasets.

Performance Evaluation and Analysis

The performances of classification models trained solely on synthetic datasets were evaluated on both the classes they were trained on and novel ones—testing their generalization capacity on unseen distributions and domains. Notably, the paper reports surprisingly high performance for models trained on synthetic images in fine-grained classification tasks, closing the gap with baseline models trained on real images and achieving considerable generalization across diverse domains.

In terms of transfer learning, models trained with synthetic data exhibited performance comparable to their real-image counterparts when applied to various benchmarks, indicating the models' strong ability to generalize learned features. The models demonstrated resilience in instances such as ImageNet-Sketch and ImageNet-R, suggesting potential advantages of incorporating synthetic imagery for robustness to domain shifts.

Discussion and Implications

The work highlights the viability of synthetic datasets as a potent resource for training generic visual models while considerably reducing the cost and labor associated with curating real datasets. Despite the promising outcomes, the paper acknowledges challenges related to semantic accuracy, visual domain shifts, and biases both intrinsic to the training data and the generation process itself. These challenges underscore the necessity for more sophisticated methods of prompt engineering and conscious evaluation of model biases.

The research points towards future avenues, suggesting synthetic imagery's potential to enable infinite scaling and exploring their use as a complement to real datasets for enhanced domain adaptation or generalization. The societal implications, particularly concerning data moderation and ethical considerations, remain critical factors, highlighting the need for continued scrutiny.

In conclusion, the paper posits synthetic images as a valuable alternative for training visual models, offering insights into efficient data generation strategies and potential improvements in representation learning. It sets a precedent for leveraging large-scale generative models in various computer vision tasks, showing a promising horizon for synthetic datasets as a practical solution for transferable representation learning.