Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Curriculum Dataset Distillation (2405.09150v1)

Published 15 May 2024 in cs.CV

Abstract: Most dataset distillation methods struggle to accommodate large-scale datasets due to their substantial computational and memory requirements. In this paper, we present a curriculum-based dataset distillation framework designed to harmonize scalability with efficiency. This framework strategically distills synthetic images, adhering to a curriculum that transitions from simple to complex. By incorporating curriculum evaluation, we address the issue of previous methods generating images that tend to be homogeneous and simplistic, doing so at a manageable computational cost. Furthermore, we introduce adversarial optimization towards synthetic images to further improve their representativeness and safeguard against their overfitting to the neural network involved in distilling. This enhances the generalization capability of the distilled images across various neural network architectures and also increases their robustness to noise. Extensive experiments demonstrate that our framework sets new benchmarks in large-scale dataset distillation, achieving substantial improvements of 11.1\% on Tiny-ImageNet, 9.0\% on ImageNet-1K, and 7.3\% on ImageNet-21K. The source code will be released to the community.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (85)
  1. Squeeze, recover and relabel: Dataset condensation at imagenet scale from a new perspective. In NeurIPS, 2023.
  2. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
  3. Dataset condensation with gradient matching. In ICLR, 2021.
  4. Dataset distillation by matching training trajectories. In CVPR, 2022.
  5. Dataset condensation via efficient synthetic-data parameterization. In ICML, 2022.
  6. Sparse parameterization for epitomic dataset distillation. In NeurIPS, 2023.
  7. Towards lossless dataset distillation via difficulty-aligned trajectory matching. arXiv preprint arXiv:2310.05773, 2023.
  8. Condensed composite memory continual learning. In IJCNN, 2021.
  9. Remember the past: Distilling datasets into addressable memories for neural networks. In NeurIPS, 2022.
  10. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In ICML, 2020.
  11. Learning multiple layers of features from tiny images. Technical report, 2009.
  12. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
  13. Inceptionism: Going deeper into neural networks. 2015.
  14. Understanding deep image representations by inverting them. In CVPR, 2015.
  15. Inverting visual representations with convolutional networks. In CVPR, 2016.
  16. Dreaming to distill: Data-free knowledge transfer via deepinversion. In CVPR, 2020.
  17. A fast knowledge distillation framework for visual recognition. In ECCV, 2022.
  18. Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
  19. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972, 2021.
  20. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
  21. Mlp-mixer: An all-mlp architecture for vision. In NeurIPS, 2021.
  22. Dc-bench: Dataset condensation benchmark. In NeurIPS, 2022.
  23. Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In WACV, 2023.
  24. Model inversion attacks that exploit confidence information and basic countermeasures. In SIGSAC, 2015.
  25. Model inversion attacks against collaborative inference. In ACSAC, 2019.
  26. The knowledge within: Methods for data-free model compression. In CVPR, 2020.
  27. Contrastive model invertion for data-free knolwedge distillation. In IJCAI, 2021.
  28. Zeroq: A novel zero shot quantization framework. In CVPR, 2020.
  29. Diversifying sample generation for accurate data-free quantization. In CVPR, 2021.
  30. Intraq: Learning synthetic images with intra-class heterogeneity for zero-shot network quantization. In CVPR, 2022.
  31. Always be dreaming: A new approach for data-free class-incremental learning. In ICCV, 2021.
  32. Few-shot class-incremental learning via entropy-regularized data-free replay. In ECCV, 2022.
  33. Curriculum learning. In ICML, 2009.
  34. Self-paced learning for latent variable models. In NeurIPS, 2010.
  35. Generative adversarial nets. In NeurIPS, 2014.
  36. Teppei Suzuki. Teachaugment: Data augmentation optimization using teacher knowledge. In CVPR, 2022.
  37. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020.
  38. Dream: Efficient dataset distillation by representative matching. In ICCV, 2023.
  39. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
  40. A convnet for the 2020s. In CVPR, 2022.
  41. Densely connected convolutional networks. In CVPR, 2017.
  42. Efficientnetv2: Smaller models and faster training. In ICML, 2021.
  43. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
  44. Designing network design spaces. In CVPR, 2020.
  45. Shufflenet v2: Practical guidelines for efficient cnn architecture design. In ECCV, 2018.
  46. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):5314–5321, 2022.
  47. Deep residual learning for image recognition. In CVPR, 2016.
  48. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
  49. On divergence measures for bayesian pseudocoresets. In NeurIPS, 2022.
  50. Dataset distillation via factorization. In NeurIPS, 2022.
  51. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
  52. Dataset distillation with infinitely wide convolutional networks. In NeurIPS, 2021.
  53. Dataset distillation using neural feature regression. In NeurIPS, 2022.
  54. Efficient dataset distillation using random feature approximation. In NeurIPS, 2022.
  55. Dataset distillation with convexified implicit gradients. In ICML, 2023.
  56. Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In ICML, 2021.
  57. Dataset condensation with contrastive signals. In ICML, 2022.
  58. Minimizing the accumulated trajectory error to improve dataset distillation. In CVPR, 2023.
  59. Scaling up dataset distillation to imagenet-1k with constant memory. In ICML, 2023.
  60. Sequential subset matching for dataset distillation. In NeurIPS, 2023.
  61. Cafe: Learning to condense dataset by aligning features. In CVPR, 2022.
  62. Bo Zhao and Hakan Bilen. Synthesizing informative training samples with gan. In NeurIPS Workshop, 2022.
  63. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.10494, 2022.
  64. Improved distribution matching for dataset condensation. In CVPR, 2023.
  65. Datadam: Efficient dataset distillation with attention matching. In ICCV, 2023.
  66. Dataset distillation in large data era. arXiv preprint arXiv:2311.18838, 2023.
  67. Generalized large-scale data condensation via various backbone and statistical matching. arXiv preprint arXiv:2311.17950, 2023.
  68. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. arXiv preprint arXiv:2312.03526, 2023.
  69. Max Welling. Herding dynamical weights to learn. In ICML, 2009.
  70. Super-samples from kernel herding. In UAI, 2010.
  71. Scalable training of mixture models via coresets. In NeurIPS, 2011.
  72. icarl: Incremental classifier and representation learning. In CVPR, 2017.
  73. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018.
  74. Active learning by acquiring contrastive examples. In EMNLP, 2021.
  75. Adversarial active learning for deep networks: a margin based approach. arXiv preprint arXiv:1802.09841, 2018.
  76. Imagenet classification with deep convolutional neural networks. 2012.
  77. mixup: Beyond empirical risk minimization. In ICLR, 2018.
  78. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
  79. Differentiable augmentation for data-efficient gan training. NeurIPS, 2020.
  80. Training generative adversarial networks with limited data. NeurIPS, 2020.
  81. Learning the easy things first: Self-paced visual category discovery. In CVPR, 2011.
  82. A self-paced multiple-instance learning framework for co-saliency detection. In ICCV, 2015.
  83. Petru Soviany. Curriculum learning with diversity for supervised computer vision tasks. In ICML Workshop, 2020.
  84. Curriculum dropout. In ICCV, 2017.
  85. Curriculum by smoothing. In NeurIPS, 2020.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets