Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Boosting the Cross-Architecture Generalization of Dataset Distillation through an Empirical Study (2312.05598v2)

Published 9 Dec 2023 in cs.LG

Abstract: The poor cross-architecture generalization of dataset distillation greatly weakens its practical significance. This paper attempts to mitigate this issue through an empirical study, which suggests that the synthetic datasets undergo an inductive bias towards the distillation model. Therefore, the evaluation model is strictly confined to having similar architectures of the distillation model. We propose a novel method of EvaLuation with distillation Feature (ELF), which utilizes features from intermediate layers of the distillation model for the cross-architecture evaluation. In this manner, the evaluation model learns from bias-free knowledge therefore its architecture becomes unfettered while retaining performance. By performing extensive experiments, we successfully prove that ELF can well enhance the cross-architecture generalization of current DD methods. Code of this project is at \url{https://github.com/Lirui-Zhao/ELF}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Flexible Dataset Distillation: Learn Labels Instead of Images. In Neural Information Processing Systems Workshop (NeurIPSW).
  2. JAX: composable transformations of Python+NumPy programs.
  3. Dataset distillation by matching training trajectories. In Computer Vision and Pattern Recognition (CVPR).
  4. Bidirectional Learning for Offline Infinite-width Model-based Optimization. In Neural Information Processing Systems (NeurIPS).
  5. Private Set Generation with Discriminative Information. In Neural Information Processing Systems (NeurIPS).
  6. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI).
  7. Clancey, W. J. 1979. Transfer of Rule-Based Expertise through a Tutorial Dialogue. Ph.D. diss., Dept. of Computer Science, Stanford Univ., Stanford, Calif.
  8. Clancey, W. J. 1983. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. In Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83), 556–560. Menlo Park, Calif: IJCAI Organization.
  9. Clancey, W. J. 1984. Classification Problem Solving. In Proceedings of the Fourth National Conference on Artificial Intelligence, 45–54. Menlo Park, Calif.: AAAI Press.
  10. Clancey, W. J. 2021. The Engineering of Qualitative Models. Forthcoming.
  11. DC-BENCH: Dataset Condensation Benchmark. In Neural Information Processing Systems (NeurIPS).
  12. Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory. In International Conference on Machine Learning (ICML).
  13. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR).
  14. Remember the Past: Distilling Datasets into Addressable Memories for Neural Networks. In Neural Information Processing Systems (NeurIPS).
  15. Privacy for Free: How does Dataset Condensation Help Privacy? In International Conference on Machine Learning (ICML).
  16. Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation. In Conference on Computer Vision and Pattern Recognition (CVPR).
  17. Blackboard Systems. Reading, Mass.: Addison-Wesley.
  18. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning (ICML).
  19. Dynamic Few-Shot Visual Learning without Forgetting Spyros Gidaris. In Computer Vision and Pattern Recognition (CVPR).
  20. Rich feature hierarchies for accurate object detection and semantic segmentation. In International Conference on Computer Vision (ICCV).
  21. Strategic explanations for a diagnostic consultation system. International Journal of Man-Machine Studies, 20(1): 3–19.
  22. Strategic Explanations in Consultation—Duplicate. The International Journal of Man-Machine Studies, 20(1): 3–19.
  23. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR).
  24. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML).
  25. Dataset Condensation via Efficient Synthetic-Data Parameterization. In International Conference on Machine Learning (ICML).
  26. Learning multiple layers of features from tiny images. Technical report, Citeseer.
  27. Tiny imagenet visual recognition challenge. CS 231N.
  28. Dataset condensation with latent space knowledge factorization and sharing. arXiv preprint arXiv:2208.10494.
  29. Wide neural networks of any depth evolve as linear models under gradient descent. In Neural Information Processing Systems (NeurIPS).
  30. Dataset condensation with contrastive signals. In International Conference on Machine Learning (ICML).
  31. A Comprehensive Survey to Dataset Distillation. arXiv preprint arXiv:2301.05603.
  32. Dataset Distillation for Medical Dataset Sharing. In AAAI Conference on Artificial Intelligence Workshop (AAAIW).
  33. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV).
  34. Dataset Distillation via Factorization. In Neural Information Processing Systems (NeurIPS).
  35. Fully convolutional networks for semantic segmentation. In Computer Vision and Pattern Recognition (CVPR).
  36. Efficient Dataset Distillation using Random Feature Approximation. In Neural Information Processing Systems (NeurIPS).
  37. Gradient-Based Hyperparameter Optimization Through Reversible Learning. In International Conference on Machine Learning (ICML).
  38. NASA. 2015. Pluto: The ’Other’ Red Planet. https://www.nasa.gov/nh/pluto-the-other-red-planet. Accessed: 2018-12-06.
  39. Dataset Meta-Learning from Kernel Ridge-Regression. In International Conference on Learning Representations (ICLR).
  40. Dataset Distillation with Infinitely Wide Convolutional Networks. In Neural Information Processing Systems (NeurIPS).
  41. Pytorch: An imperative style, high-performance deep learning library. In Neural Information Processing Systems (NeurIPS).
  42. Faster r-cnn: Towards real-time object detection with region proposal networks. In Neural Information Processing Systems (NeurIPS).
  43. Kornia: an open source differentiable computer vision library for pytorch. In Winter Conference on Applications of Computer Vision (WACV).
  44. Rice, J. 1986. Poligon: A System for Parallel Problem Solving. Technical Report KSL-86-19, Dept. of Computer Science, Stanford Univ.
  45. Robinson, A. L. 1980a. New Ways to Make Microcircuits Smaller. Science, 208(4447): 1019–1022.
  46. Robinson, A. L. 1980b. New Ways to Make Microcircuits Smaller—Duplicate Entry. Science, 208: 1019–1026.
  47. Distilled Replay: Overcoming Forgetting through Synthetic Samples. In International Joint Conference on Artificial Intelligence Workshop (IJCAIW).
  48. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV).
  49. Data Distillation: A Survey. arXiv preprint arXiv:2301.04272.
  50. Infinite Recommendation Networks: A Data-Centric Approach. In Neural Information Processing Systems (NeurIPS).
  51. Sample Condensation in Online Continual Learning. In International Joint Conference on Neural Networks (IJCNN).
  52. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations (ICLR).
  53. Soft-Label Dataset Distillation and Text Dataset Distillation. In International Joint Conference on Neural Networks (IJCNN).
  54. Going deeper with convolutions. In Computer Vision and Pattern Recognition (CVPR).
  55. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022.
  56. Attention Is All You Need. arXiv:1706.03762.
  57. CAFE: Learning to Condense Dataset by Aligning Features. In Computer Vision and Pattern Recognition (CVPR).
  58. Dataset distillation. arXiv preprint arXiv:1811.10959.
  59. Werbos, P. 1990. Backpropagation through time: what it does and how to do it. Institute of Electrical and Electronics Engineers (IEEE).
  60. Condensed Composite Memory Continual Learning. In International Joint Conference on Neural Networks (IJCNN).
  61. Dataset Distillation: A Comprehensive Review. arXiv preprint arXiv:2301.07014.
  62. Dataset condensation with Differentiable Siamese Augmentation. In International Conference on Machine Learning (ICML).
  63. Dataset Condensation with Gradient Matching. In International Conference on Learning Representations (ICLR).
  64. Synthesizing Informative Training Samples with GAN. In Neural Information Processing Systems Workshop (NeurIPSW).
  65. Dataset Condensation with Distribution Matching. In Winter Conference on Applications of Computer Vision (WACV).
  66. Dataset Distillation using Neural Feature Regression. In Neural Information Processing Systems (NeurIPS).

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now