Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
153 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A Note on Generalization in Variational Autoencoders: How Effective Is Synthetic Data & Overparameterization? (2310.19653v3)

Published 30 Oct 2023 in stat.ML, cs.CV, and cs.LG

Abstract: Variational autoencoders (VAEs) are deep probabilistic models that are used in scientific applications. Many works try to mitigate this problem from the probabilistic methods perspective by new inference techniques or training procedures. In this paper, we approach the problem instead from the deep learning perspective by investigating the effectiveness of using synthetic data and overparameterization for improving the generalization performance. Our motivation comes from (1) the recent discussion on whether the increasing amount of publicly accessible synthetic data will improve or hurt currently trained generative models; and (2) the modern deep learning insights that overparameterization improves generalization. Our investigation shows how both training on samples from a pre-trained diffusion model, and using more parameters at certain layers are able to effectively mitigate overfitting in VAEs, therefore improving their generalization, amortized inference, and robustness performance. Our study provides timely insights in the current era of synthetic data and scaling laws.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (45)
  1. Self-consuming generative models go mad. arXiv preprint arXiv:2307.01850, 2023.
  2. Synthetic data from diffusion models improves imagenet classification. arXiv preprint arXiv:2304.08466, 2023.
  3. End-to-end optimized image compression. In International Conference on Learning Representations, 2017.
  4. Variational inference: A review for statisticians. Journal of the American Statistical Association, 2017.
  5. A training algorithm for optimal margin classifiers. In Fifth Annual Workshop on Computational Learning Theory, 1992.
  6. Inference suboptimality in variational autoencoders. In International Conference on Machine Learning, 2018.
  7. Robustbench: a standardized adversarial robustness benchmark. In Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  8. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
  9. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  10. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, 2017.
  11. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  12. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
  13. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
  14. Training generative adversarial networks with limited data. In Advances in Neural Information Processing Systems, 2020.
  15. Elucidating the design space of diffusion-based generative models. In Advances in Neural Information Processing Systems, 2022.
  16. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
  17. Learning multiple layers of features from tiny images. 2009.
  18. On information and sufficiency. The Annals of Mathematical Statistics, 22, 1951.
  19. Alleviating adversarial attacks on variational autoencoders with mcmc. In Advances in Neural Information Processing Systems, 2022.
  20. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 1998.
  21. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, 2023.
  22. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
  23. Iterative amortized inference. In International Conference on Machine Learning, 2018.
  24. Do deep generative models know what they don’t know? In International Conference on Learning Representations, 2018.
  25. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
  26. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
  27. PixelCNN++: Improving the pixelCNN with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017.
  28. Improving gans using optimal transport. In International Conference on Learning Representations, 2018.
  29. Amortized inference regularization. In Advances in Neural Information Processing Systems, 2018.
  30. The curse of recursion: Training on generated data makes models forget. arXiv preprint arxiv:2305.17493, 2023.
  31. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
  32. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  33. Consistency models. In International Conference on Machine Learning, 2023.
  34. A note on the evaluation of generative models. In International Conference on Learning Representations, 2016.
  35. Stablerep: Synthetic images from text-to-image models make strong visual representation learners. arXiv preprint arXiv:2306.00984, 2023.
  36. Neural discrete representation learning. In Advances in Neural Information Processing Systems, 2017.
  37. Better diffusion models further improve adversarial training. In International Conference on Machine Learning, 2023.
  38. Multiscale structural similarity for image quality assessment. In The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, 2003.
  39. On the quantitative analysis of decoder-based generative models. In International Conference on Learning Representations, 2017.
  40. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017.
  41. Trading information between latents in hierarchical variational autoencoders. In International Conference on Learning Representations, 2023.
  42. Image data augmentation for deep learning: A survey. arXiv preprint arXiv:2204.08610, 2022.
  43. An introduction to neural data compression. Foundations and Trends® in Computer Graphics and Vision, 2023.
  44. Generalization gap in amortized inference. In Advances in Neural Information Processing Systems, 2022.
  45. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now