A Label is Worth a Thousand Images in Dataset Distillation

Abstract: Data $\textit{quality}$ is a crucial factor in the performance of machine learning models, a principle that dataset distillation methods exploit by compressing training datasets into much smaller counterparts that maintain similar downstream performance. Understanding how and why data distillation methods work is vital not only for improving these methods but also for revealing fundamental characteristics of "good" training data. However, a major challenge in achieving this goal is the observation that distillation approaches, which rely on sophisticated but mostly disparate methods to generate synthetic data, have little in common with each other. In this work, we highlight a largely overlooked aspect common to most of these methods: the use of soft (probabilistic) labels. Through a series of ablation experiments, we study the role of soft labels in depth. Our results reveal that the main factor explaining the performance of state-of-the-art distillation methods is not the specific techniques used to generate synthetic data but rather the use of soft labels. Furthermore, we demonstrate that not all soft labels are created equal; they must contain $\textit{structured information}$ to be beneficial. We also provide empirical scaling laws that characterize the effectiveness of soft labels as a function of images-per-class in the distilled dataset and establish an empirical Pareto frontier for data-efficient learning. Combined, our findings challenge conventional wisdom in dataset distillation, underscore the importance of soft labels in learning, and suggest new directions for improving distillation methods. Code for all experiments is available at https://github.com/sunnytqin/no-distillation.

• The paper demonstrates that soft labels are the primary driver behind effective dataset distillation, outperforming complex synthetic image generation.
• It introduces a simple baseline pairing real images with probabilistic labels from expert models to rival state-of-the-art techniques.
• The research proposes an empirical scaling law linking data budget and label quality, emphasizing structured semantic information in effective learning.

Understanding the Role of Soft Labels in Dataset Distillation

The paper "A Label is Worth a Thousand Images in Dataset Distillation" provides a profound exploration into the mechanisms that underlie the effectiveness of dataset distillation methods, particularly emphasizing the role of soft labels. Researchers have long sought ways to compress training datasets into smaller synthetic counterparts that maintain comparable performance. This paper argues that the pivotal factor is not the data compression techniques themselves but rather the strategic use of soft labels.

The paper commences by addressing the widely acknowledged observation that data quality often takes precedence over sheer data quantity in improving the performance of machine learning models. This is particularly relevant when considering large datasets for training. The work in this paper upholds that dataset distillation could illuminate what constitutes "good" training data by way of examining what information must be retained during the distillation process. A bold take here is that existing top-performing distillation methods, known for employing soft labels, inadvertently highlight that the label quality could significantly overshadow the nuanced strategies used in generating synthetic data.

A key inquiry posed by the authors is whether the input data or the label plays a more crucial role in facilitating data-efficient learning. They proposed a simple baseline involving randomly sampled real dataset images paired with soft, probabilistic labels generated by pretrained expert models. This method was surprisingly found to rival state-of-the-art (SOTA) techniques, challenging the need for complex synthetic image generation in dataset distillation.

The paper extensively evaluates existing distillation techniques across multiple datasets, including ImageNet-1K, TinyImageNet, CIFAR-10, and CIFAR-100. It concludes that soft labels are indispensable in achieving commendable distillation outcomes. In particular, synthetic image generation contributes less to the successes traditionally attributed to SOTA methods. This revelation prompts practitioners to rethink investments in compute and research efforts aimed at synthetic data generation alone.

In studying the constitution of effective soft labels, the paper identifies the importance of structured semantic information — knowledge like distributional similarities across related classes — encoded therein. The research indicates that optimal soft label quality varies with the data budget used in distillation, suggesting that the labels must be tailored to the available image budget. An empirical scaling law is introduced to describe the trade-offs between dataset size and label information richness, thereby establishing a Pareto frontier for data-efficient learning.

Moreover, the work draws connections between dataset distillation and knowledge distillation (KD) by illustrating that pre-trained expert knowledge, encapsulated in soft labels, allows for more efficient learning, particularly at smaller data budgets. Notably, the paper proposes that optimized distillation may involve integrating the structured information directly, improving on the methods explored within KD contexts.

In summary, this paper implies a significant shift in perspective for dataset distillation: researchers should focus on advancing the informational content of labels rather than on sophisticated image synthesis techniques. The investigation culminates with the assertion that embedding meaningful expert knowledge through soft labels could act as a universal mechanism to achieve data-efficient learning. Future research might focus on refining methods that simultaneously optimize image and label creation to further enhance distillation techniques. This paradigm could significantly evolve machine learning practices in both theoretical exploration and practical applications where data efficiency is a priority.