DataPerf: Benchmarks for Data-Centric AI Development (2207.10062v4)

Abstract: Machine learning research has long focused on models rather than datasets, and prominent datasets are used for common ML tasks without regard to the breadth, difficulty, and faithfulness of the underlying problems. Neglecting the fundamental importance of data has given rise to inaccuracy, bias, and fragility in real-world applications, and research is hindered by saturation across existing dataset benchmarks. In response, we present DataPerf, a community-led benchmark suite for evaluating ML datasets and data-centric algorithms. We aim to foster innovation in data-centric AI through competition, comparability, and reproducibility. We enable the ML community to iterate on datasets, instead of just architectures, and we provide an open, online platform with multiple rounds of challenges to support this iterative development. The first iteration of DataPerf contains five benchmarks covering a wide spectrum of data-centric techniques, tasks, and modalities in vision, speech, acquisition, debugging, and diffusion prompting, and we support hosting new contributed benchmarks from the community. The benchmarks, online evaluation platform, and baseline implementations are open source, and the MLCommons Association will maintain DataPerf to ensure long-term benefits to academia and industry.

• The paper introduces a suite of five benchmarks that evaluate data selection, debugging, acquisition, and adversarial testing, promoting data-centric AI.
• It demonstrates that focusing on dataset quality can significantly improve model performance without changing model architectures.
• The work catalyzes future research on active learning and data valuation, ensuring more robust, fair, and efficient AI systems.

DataPerf: Benchmarks for Data-Centric AI Development

The paper "DataPerf: Benchmarks for Data-Centric AI Development" presents a suite of benchmarks aimed at a paradigm shift in machine learning research. The central theme revolves around shifting focus from model-centric to data-centric development to address the limitations observed in the reliance on large, often static datasets like ImageNet or SQuAD. This focus intends to reduce the accuracy, bias, and fragility that result in real-world applications when the underlying data issues are ignored.

Key Contributions

DataPerf introduces five benchmarks that address various components of machine learning pipelines:

1. Selection for Speech and Vision: These tasks target the effective selection of training samples from noisy multilingual and image datasets, respectively. These benchmarks emphasize the quality over quantity, crucial for low-resource settings and efficient data usage.
2. Debugging for Vision: This benchmark addresses the identification of erroneous data points to minimize manual correction and focus attention on subset samples that impact the most on model performance.
3. Data Acquisition: It simulates a marketplace scenario where participants optimize the purchase of datasets for specific tasks within budget constraints. This addresses the practical challenge of dataset content opacity and manually identifying useful data.
4. Adversarial Nibbler: A unique challenge aimed at identifying failure modes in generative AI models like Stable Diffusion. This problem-solving targets "unknown unknowns" in the safety filters of text-to-image models.

Implications and Speculation

Data-centric approaches introduced in DataPerf allow for significant improvements in datasets' generalization capabilities without the need to modify the model architecture. This framework highlights the necessity for better evaluation tools that can assess dataset quality, pushing the frontier of AI research beyond traditional model performance checks. With comprehensive, open-source benchmarks, DataPerf provides a platform for long-term progress via iterative development—users iterating on datasets rather than models.

The implications are noteworthy: integrating data debugging and selection strategies into standard practices could lead to more robust ML systems that perform consistently in diverse conditions, promoting models that are fairer and more adaptable. As AI systems become integral in decision-making processes, enhanced dataset-centric methodologies will ensure these systems are reliable and unbiased.

Looking ahead, DataPerf sets the stage for future developments in AI, with an expectation for more community-led benchmarks and contributions to further drive ML research in handling data effectively. The work paves the way for new active learning methodologies and emphasizes the continuing need for research in data valuation—how we assess the utility of data and prioritize data acquisition—areas that remain ripe for exploration.

Through the establishment of a standardized suite of benchmarks, ongoing challenges, and the facilitation of collaborative efforts amongst researchers, DataPerf serves not only as a benchmark repository but as a catalyst for the advancement of data-centric techniques in AI research.