SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (2410.05057v1)

Abstract: Data curation is the problem of how to collect and organize samples into a dataset that supports efficient learning. Despite the centrality of the task, little work has been devoted towards a large-scale, systematic comparison of various curation methods. In this work, we take steps towards a formal evaluation of data curation strategies and introduce SELECT, the first large-scale benchmark of curation strategies for image classification. In order to generate baseline methods for the SELECT benchmark, we create a new dataset, ImageNet++, which constitutes the largest superset of ImageNet-1K to date. Our dataset extends ImageNet with 5 new training-data shifts, each approximately the size of ImageNet-1K itself, and each assembled using a distinct curation strategy. We evaluate our data curation baselines in two ways: (i) using each training-data shift to train identical image classification models from scratch (ii) using the data itself to fit a pretrained self-supervised representation. Our findings show interesting trends, particularly pertaining to recent methods for data curation such as synthetic data generation and lookup based on CLIP embeddings. We show that although these strategies are highly competitive for certain tasks, the curation strategy used to assemble the original ImageNet-1K dataset remains the gold standard. We anticipate that our benchmark can illuminate the path for new methods to further reduce the gap. We release our checkpoints, code, documentation, and a link to our dataset at https://github.com/jimmyxu123/SELECT.

• The paper introduces SELECT, a benchmark that systematically evaluates data curation strategies for image classification using the extended ImageNet++ dataset.
• It employs two evaluation methods: training models from scratch and fine-tuning pretrained self-supervised representations to compare different curation approaches.
• The findings highlight that while synthetic and CLIP-based retrieval techniques show promise, they require further refinement to meet the standard set by ImageNet-1K.

The paper "SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification" presents an important advancement in understanding data curation practices for image classification tasks. Data curation is recognized as the process of collecting and organizing data samples to facilitate effective learning. Although it is a crucial aspect of machine learning, there has been limited systematic analysis of different curation strategies on a large scale. This paper addresses this gap by introducing SELECT, the first comprehensive benchmark dedicated to evaluating various data curation methodologies.

The researchers introduce ImageNet++, a significant extension to the well-known ImageNet-1K, which serves as the core of the SELECT benchmark. ImageNet++ incorporates five additional training-data shifts, each mirroring ImageNet-1K in size but curated with distinct methodologies. This extension allows for a robust comparison of different curation strategies under controlled conditions.

Key elements of the research include:

1. Benchmark Evaluation: The authors propose two primary evaluation methods:
   • Training identical image classification models from scratch across various data shifts.
   • Fitting the additional data to a pretrained self-supervised representation.
2. Comparative Insights: The paper reveals interesting trends in data curation, particularly evaluating newer methods like synthetic data generation and retrieval based on CLIP embeddings. Despite their competitive performance in specific scenarios, the original ImageNet-1K curation strategy remains the benchmark standard.
3. Implications for Future Research: The paper suggests that while recent methods show promise, there is room for further refinement to meet or surpass the existing ImageNet-1K results. SELECT is intended to drive innovation in data curation practices by providing a rigorous evaluation framework and a shared dataset for the research community to build upon.

The release of the accompanying code, documentation, and dataset further underscores the authors' commitment to fostering ongoing research and collaboration in the field. The GitHub repository offers resources to facilitate replication and extension of their findings, aiming to enhance the overall understanding and development of data curation strategies.