Dataset Condensation with Differentiable Siamese Augmentation (2102.08259v2)

Abstract: In many machine learning problems, large-scale datasets have become the de-facto standard to train state-of-the-art deep networks at the price of heavy computation load. In this paper, we focus on condensing large training sets into significantly smaller synthetic sets which can be used to train deep neural networks from scratch with minimum drop in performance. Inspired from the recent training set synthesis methods, we propose Differentiable Siamese Augmentation that enables effective use of data augmentation to synthesize more informative synthetic images and thus achieves better performance when training networks with augmentations. Experiments on multiple image classification benchmarks demonstrate that the proposed method obtains substantial gains over the state-of-the-art, 7% improvements on CIFAR10 and CIFAR100 datasets. We show with only less than 1% data that our method achieves 99.6%, 94.9%, 88.5%, 71.5% relative performance on MNIST, FashionMNIST, SVHN, CIFAR10 respectively. We also explore the use of our method in continual learning and neural architecture search, and show promising results.

• The paper presents Differentiable Siamese Augmentation as a novel method to condense datasets into synthetic counterparts using gradient matching loss.
• It leverages differentiable data augmentation to maintain semantic consistency between real and synthetic data through a Siamese framework.
• Experiments on CIFAR10 and CIFAR100 demonstrate up to a 7% accuracy improvement even using only 1% of the original data, highlighting its efficiency.

Dataset Condensation with Differentiable Siamese Augmentation

The paper "Dataset Condensation with Differentiable Siamese Augmentation" addresses the computational inefficiencies associated with large-scale datasets used in training state-of-the-art deep networks. The primary focus is on condensing these extensive datasets into significantly more compact synthetic counterparts, thereby reducing computational demands while maintaining minimal performance degradation. The authors introduce a method termed as Differentiable Siamese Augmentation (DSA), which synergizes data augmentation with synthetic data condensation, leading to superior model training results with reduced data.

Methodology

At the core of the proposed method is the notion of leveraging data augmentation during the synthesis of smaller, synthetic datasets. Conventional approaches to data augmentation are non-differentiable, but this paper introduces differentiable transformations that maintain gradient flow, enabling the effective transfer of knowledge from the real data to the synthetic data through optimization of a gradient matching loss. The Siamese augmentation strategy plays a pivotal role, wherein identical transformations are applied to both real and synthetic data in tandem, preserving semantic equivariance between them.

Results and Evaluation

The authors report significant improvements in performance across various image classification benchmarks, demonstrating the efficacy of DSA over previous state-of-the-art methods. Notably, the results showed a remarkable 7% improvement on the CIFAR10 and CIFAR100 datasets. The experiments were conducted using varying architectures and datasets, including MNIST, FashionMNIST, and SVHN, showcasing the robustness and adaptability of the proposed approach. The results convey that even with a mere 1% of the original data volume, models trained using synthetically condensed datasets achieved competitive accuracy levels.

Implications and Future Work

The practical implications of this research are substantial, particularly in scenarios where computational resources are a constraint. By enabling efficient model training with reduced data volumes, the approach offers a pathway towards sustainable AI model development. The insights offered could also pave the way for advances in areas like few-shot learning, continual learning, and neural architecture search, where data efficiency is critical.

Theoretical implications suggest potential investigations into new types of data augmentations and synthetic data generation strategies that further bridge the gap between training on full datasets and their condensed versions. Moreover, future work could explore domain-specific applications where dataset condensation provides strategic advantages, potentially in fields where data privacy and security concerns are prevalent.

Conclusion

Overall, the paper successfully tackles the challenge of dataset condensation by integrating data augmentation and synthetic data generation in a unique and effective manner. The proposed DSA method not only enhances model training efficiency but also sheds light on the importance of transferable data transformations in synthetic data generation. The advancements reported in this paper contribute meaningfully to ongoing efforts in optimizing AI workflows for better performance and resource utilization.