Dataset Distillation by Matching Training Trajectories (2203.11932v1)

Abstract: Dataset distillation is the task of synthesizing a small dataset such that a model trained on the synthetic set will match the test accuracy of the model trained on the full dataset. In this paper, we propose a new formulation that optimizes our distilled data to guide networks to a similar state as those trained on real data across many training steps. Given a network, we train it for several iterations on our distilled data and optimize the distilled data with respect to the distance between the synthetically trained parameters and the parameters trained on real data. To efficiently obtain the initial and target network parameters for large-scale datasets, we pre-compute and store training trajectories of expert networks trained on the real dataset. Our method handily outperforms existing methods and also allows us to distill higher-resolution visual data.

• The paper presents a novel dataset distillation technique by matching long-range training trajectories from expert models.
• It employs pre-computed expert trajectories and an efficient batching strategy to manage large datasets like ImageNet.
• Results show significant improvements, achieving 46.3% accuracy on CIFAR-10 with one image per class, indicating scalable and efficient training.

Dataset Distillation by Matching Training Trajectories: An Expert Overview

The paper "Dataset Distillation by Matching Training Trajectories" presents a novel approach to dataset distillation, specifically focusing on creating an efficient yet effective distilled dataset that rivals models trained on the full dataset. The core innovation lies in optimizing a synthetic dataset to guide a neural network's training trajectory similarly to one that is trained on real data. This approach addresses limitations present in previous efforts, particularly those constrained to toy datasets and short-range behavior, by facilitating the application of distillation techniques to more substantial, real-world datasets.

Traditional dataset distillation techniques typically face challenges when applied to high-resolution datasets, such as ImageNet, primarily due to the computational burden and complexity involved in managing large, real-world datasets. The authors enhance the scalability and efficacy of dataset distillation by proposing a method that utilizes pre-computed expert training trajectories. These expert trajectories are derived from models trained on real datasets and are used to guide synthetic datasets to optimize the parameter space similarly to the real dataset, achieving similar model capabilities.

Methodology

The proposed methodology involves several key steps:

• Expert Trajectories: Networks are trained on the full dataset to acquire expert trajectories, which are pre-recorded sequences of network parameters that serve as benchmarks.
• Long-Range Parameter Matching: The constituent contribution of the work is the alignment of long-range parameter dynamics instead of single-step approximations. By placing a trained model repeatedly on synthetic datasets alongside recorded trajectories, parameters are adjusted to close the performance gap iteratively.
• Memory Management: Given the potential memory overhead, particularly with large distilled datasets, the authors introduce an effective batching strategy to mitigate memory consumption while ensuring robust learning of complex data dynamics.

Strong Results and Claims

Experiments conducted across multiple datasets, including CIFAR-10, CIFAR-100, Tiny ImageNet, and subsets of ImageNet, demonstrate that the methodology attains performance improvements over previous techniques. Specifically, the method achieves notable accuracy with only one image per class by leveraging its long-range trajectory matching, which outperforms state-of-the-art methods such as DSA and DM. For instance, on CIFAR-10, the approach reaches 46.3% accuracy with merely one image per class, a significant enhancement over existing benchmarks.

Implications and Future Directions

Practically, the proposed method holds potential for accelerating model training in resource-limited settings, supporting more efficient neural architecture search, and enhancing continual learning. Moreover, the ability to generate high-fidelity synthetic datasets for larger data resolutions (e.g., ImageNet) signifies an important step toward real-world applicability.

Theoretically, this contribution advances the understanding of dataset distillation by emphasizing the role of long-range behavior in network training. It suggests that distillation optimizing trajectories over extended training periods could more effectively capture crucial learning dynamics than existing short-range methods.

Looking forward, the implications of this work suggest several avenues for further development:

1. Scalability Refinement: Exploring the application of distributed computing to further enhance scalability when distilling larger datasets.
2. Cross-Model Generalization: Investigating how models distilled using this method generalize across varying architectures and domain applications is a promising direction.
3. Energy-efficient ML: Given its potential impact on reducing training data requirements, exploring energy efficiency as a key metric could prove beneficial for developing sustainable ML practices.

In conclusion, dataset distillation by matching training trajectories presents a significant advancement in creating minimized datasets that do not compromise on model performance. By bridging the gap between synthetic and real data-driven model capabilities, this approach opens up new possibilities for large-scale, practical applications in dataset distillation.