2000 character limit reached
Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks
Published 14 Mar 2022 in cs.LG, math.PR, and stat.ML | (2203.07379v2)
Abstract: Given any deep fully connected neural network, initialized with random Gaussian parameters, we bound from above the quadratic Wasserstein distance between its output distribution and a suitable Gaussian process. Our explicit inequalities indicate how the hidden and output layers sizes affect the Gaussian behaviour of the network and quantitatively recover the distributional convergence results in the wide limit, i.e., if all the hidden layers sizes become large.
- Luigi Ambrosio, Nicola Gigli and Giuseppe Savaré “Metric measure spaces with Riemannian Ricci curvature bounded from below” In Duke Mathematical Journal 163.7 Duke University Press, 2014, pp. 1405–1490
- Devanshu Agrawal, Theodore Papamarkou and Jacob Hinkle “Wide Neural Networks with Bottlenecks are Deep Gaussian Processes” In Journal of Machine Learning Research 21.175, 2020, pp. 1–66 URL: http://jmlr.org/papers/v21/20-017.html
- Thomas Bonis “Stein’s method for normal approximation in Wasserstein distances with application to the multivariate central limit theorem” In Probab. Theory Relat. Fields 178.3, 2020, pp. 827–860 DOI: 10.1007/s00440-020-00989-4
- Anastasia Borovykh “A Gaussian Process perspective on Convolutional Neural Networks” arXiv: 1810.10798 In arXiv:1810.10798 [cs, stat], 2019 URL: http://arxiv.org/abs/1810.10798
- “Large-width functional asymptotics for deep Gaussian neural networks”, 2020 URL: https://openreview.net/forum?id=0aW6lYOYB7d
- “A review on neural networks with random weights” In Neurocomputing 275 Elsevier, 2018, pp. 278–287
- Mark K. Cowan “battlesnake/neural” original-date: 2013-08-12T23:46:47Z, 2022 URL: https://github.com/battlesnake/neural
- “Kernel methods for deep learning” In Advances in neural information processing systems 22, 2009
- Ronen Eldan, Dan Mikulincer and Tselil Schramm “Non-asymptotic approximations of neural networks by Gaussian processes” In Conference on Learning Theory, 2021, pp. 1754–1775 PMLR
- “On the rate of convergence in Wasserstein distance of the empirical measure” In Probability theory and related fields 162.3-4 Springer, 2015, pp. 707–738
- “Pot: Python optimal transport” In The Journal of Machine Learning Research 22.1 JMLRORG, 2021, pp. 3571–3578
- Ian Goodfellow, Yoshua Bengio and Aaron Courville “Deep Learning” http://www.deeplearningbook.org MIT Press, 2016
- “Generative Adversarial Nets” In Advances in Neural Information Processing Systems 27 Curran Associates, Inc., 2014 URL: https://proceedings.neurips.cc/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html
- Adrià Garriga-Alonso, Carl Edward Rasmussen and Laurence Aitchison “Deep Convolutional Networks as shallow Gaussian Processes”, 2018 URL: https://openreview.net/forum?id=Bklfsi0cKm
- J Leo Hemmen and Tsuneya Ando “An inequality for trace ideals” In Communications in Mathematical Physics 76.2 Springer, 1980, pp. 143–148
- Boris Hanin “Random Neural Networks in the Infinite Width Limit as Gaussian Processes” arXiv: 2107.01562 In arXiv:2107.01562 [cs, math, stat], 2021 URL: http://arxiv.org/abs/2107.01562
- “Exact posterior distributions of wide Bayesian neural networks”, 2020 URL: https://openreview.net/forum?id=s279nFs8NNT
- Adam Klukowski “Rate of Convergence of Polynomial Networks to Gaussian Processes” In Conference on Learning Theory, 2022, pp. 701–722 PMLR
- Yann LeCun, Yoshua Bengio and Geoffrey Hinton “Deep learning” Number: 7553 Publisher: Nature Publishing Group In Nature 521.7553, 2015, pp. 436–444 DOI: 10.1038/nature14539
- “Deep Neural Networks as Gaussian Processes”, 2018 URL: https://openreview.net/forum?id=B1EA-M-0Z
- “Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent” In Advances in Neural Information Processing Systems 32 Curran Associates, Inc., 2019 URL: https://proceedings.neurips.cc/paper/2019/hash/0d1a9651497a38d8b1c3871c84528bd4-Abstract.html
- “Gaussian Process Behaviour in Wide Deep Neural Networks”, 2018 URL: https://openreview.net/forum?id=H1-nGgWC-
- Song Mei, Andrea Montanari and Phan-Minh Nguyen “A mean field view of the landscape of two-layer neural networks” In Proceedings of the National Academy of Sciences 115.33, 2018, pp. E7665–E7671 DOI: 10.1073/pnas.1806579115
- Takuo Matsubara, Chris.J. Oates and F. Briol “The Ridgelet Prior: A Covariance Function Approach to Prior Specification for Bayesian Neural Networks” In J. Mach. Learn. Res. 22, 2020, pp. 157:1–157:57 URL: https://api.semanticscholar.org/CorpusID:223956904
- Meenal V Narkhede, Prashant P Bartakke and Mukul S Sutaone “A review on weight initialization strategies for neural networks” In Artificial intelligence review 55.1 Springer, 2022, pp. 291–322
- Radford M. Neal “Priors for Infinite Networks” In Bayesian Learning for Neural Networks, Lecture Notes in Statistics New York, NY: Springer, 1996, pp. 29–53 DOI: 10.1007/978-1-4612-0745-0˙2
- “Neural Tangents: Fast and Easy Infinite Neural Networks in Python” In International Conference on Learning Representations, 2020 URL: https://github.com/google/neural-tangents
- Phan-Minh Nguyen and Huy Tuan Pham “A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networks” arXiv: 2001.11443 In arXiv:2001.11443 [cond-mat, stat], 2021 URL: http://arxiv.org/abs/2001.11443
- “Computational optimal transport: With applications to data science” In Foundations and Trends® in Machine Learning 11.5-6 Now Publishers, Inc., 2019, pp. 355–607
- Stefano Peluchetti, Stefano Favaro and Sandra Fortini “Stable behaviour of infinitely wide deep neural networks” ISSN: 2640-3498 In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics PMLR, 2020, pp. 1137–1146 URL: https://proceedings.mlr.press/v108/peluchetti20b.html
- Frank Rosenblatt “The perceptron: a probabilistic model for information storage and organization in the brain.” In Psychological review 65.6 American Psychological Association, 1958, pp. 386
- Daniel A Roberts, Sho Yaida and Boris Hanin “The principles of deep learning theory” Cambridge University Press Cambridge, MA, USA, 2022
- Filippo Santambrogio “Optimal transport for applied mathematicians” In Birkäuser, NY 55.58-63 Springer, 2015, pp. 94
- Terrence J. Sejnowski “The Deep Learning Revolution” Google-Books-ID: 9xZxDwAAQBAJ MIT Press, 2018
- Cédric Villani “Optimal transport: old and new” Springer, 2009
- CK Williams and Carl Edward Rasmussen “Gaussian processes for machine learning, vol. 2, no. 3” Cambridge, MA, USA: MIT Press, 2006
- Greg Yang “Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes” In Advances in Neural Information Processing Systems 32 Curran Associates, Inc., 2019 URL: https://proceedings.neurips.cc/paper/2019/hash/5e69fda38cda2060819766569fd93aa5-Abstract.html
- Greg Yang “Wide feedforward or recurrent neural networks of any architecture are gaussian processes” In Advances in Neural Information Processing Systems 32, 2019
- “Tensor programs iib: Architectural universality of neural tangent kernel training dynamics” In International Conference on Machine Learning, 2021, pp. 11762–11772 PMLR
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.