Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
167 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimal rates of approximation by shallow ReLU$^k$ neural networks and applications to nonparametric regression (2304.01561v3)

Published 4 Apr 2023 in stat.ML and cs.LG

Abstract: We study the approximation capacity of some variation spaces corresponding to shallow ReLU$k$ neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU$k$ neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning H\"older functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. A general approximation lower bound in Lpsuperscript𝐿𝑝L^{p}italic_L start_POSTSUPERSCRIPT italic_p end_POSTSUPERSCRIPT norm, with applications to feed-forward neural networks. arXiv: 2206.04360, 2022.
  2. Neural network learning: Theoretical foundations. Cambridge University Press, 2009.
  3. Francis Bach. Breaking the curse of dimensionality with convex neural networks. Journal of Machine Learning Research, 18(19):1–53, 2017.
  4. Andrew R. Barron. Universal approximation bounds for superpositions of a sigmoidal function. IEEE Transactions on Information Theory, 39(3):930–945, 1993.
  5. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463–482, 2002.
  6. Spectrally-normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, pages 6240–6249. 2017.
  7. Nearly-tight VC-dimension and Pseudodimension bounds for piecewise linear neural networks. Journal of Machine Learning Research, 20(63):1–17, 2019.
  8. Understanding neural networks with reproducing kernel Banach spaces. Applied and Computational Harmonic Analysis, 62:194–236, 2023.
  9. On deep learning as a remedy for the curse of dimensionality in nonparametric regression. The Annals of Statistics, 47(4):2261–2285, 2019.
  10. Approximation of zonoids by zonotopes. Acta mathematica, 162:73–141, 1989.
  11. George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4):303–314, 1989.
  12. Approximation theory and harmonic analysis on spheres and balls, volume 23. Springer, 2013.
  13. Constructive approximation, volume 303. Springer Science & Business Media, 1993.
  14. Optimal nonlinear approximation. Manuscripta Mathematica, 63(4):469–478, 1989.
  15. Zeev Ditzian. Measures of smoothness on the sphere. In Frontiers in Interpolation and Approximation, pages 75–91. Chapman and Hall/CRC, 2006.
  16. Theory of deep convolutional neural networks II: Spherical analysis. Neural Networks, 131:154–162, 2020.
  17. Charles Fefferman. Whitney’s extension problem for Cmsuperscript𝐶𝑚C^{m}italic_C start_POSTSUPERSCRIPT italic_m end_POSTSUPERSCRIPT. Annals of Mathematics, 164(1):313–359, 2006.
  18. Charles Fefferman. Extension of Cm,ωsuperscript𝐶𝑚𝜔C^{m,\omega}italic_C start_POSTSUPERSCRIPT italic_m , italic_ω end_POSTSUPERSCRIPT-smooth functions by linear operators. Revista Matemática Iberoamericana, 25(1):1–48, 2009.
  19. Fitting smooth functions to data. American Mathematical Society, 2020.
  20. Size-independent sample complexity of neural networks. Information and Inference: A Journal of the IMA, 9(2):473–504, 2020.
  21. David Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation, 100(1):78–150, 1992.
  22. Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 4(2):251–257, 1991.
  23. Approximation bounds for norm constrained neural networks with applications to regression and GANs. Applied and Computational Harmonic Analysis, 65:249–278, 2023.
  24. Approximation by combinations of ReLU and squared ReLU ridge functions with l1superscript𝑙1l^{1}italic_l start_POSTSUPERSCRIPT 1 end_POSTSUPERSCRIPT and l0superscript𝑙0l^{0}italic_l start_POSTSUPERSCRIPT 0 end_POSTSUPERSCRIPT controls. IEEE Transactions on Information Theory, 64(12):7649–7656, 2018.
  25. Adaptive regression estimation with multilayer feedforward neural networks. Journal of Nonparametric Statistics, 17(8):891–913, 2005.
  26. On the rate of convergence of fully connected deep neural network regression estimates. The Annals of Statistics, 49(4):2231–2249, 2021.
  27. Deep learning. Nature, 521(7553):436–444, 2015.
  28. Probability in Banach spaces: isoperimetry and processes. Springer, 1991.
  29. Universal consistency of deep convolutional neural networks. IEEE Transactions on Information Theory, 68(7):4610–4617, 2022.
  30. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465–5506, 2021.
  31. Uniform approximation rates and metric entropy of shallow neural networks. Research in the Mathematical Sciences, 9(3):46, 2022.
  32. On the degree of approximation by manifolds of finite pseudo-dimension. Constructive Approximation, 15(2):291–300, 1999.
  33. Yuly Makovoz. Random approximants and neural networks. Journal of Approximation Theory, 85(1):98–109, 1996.
  34. Rates of approximation by ReLU shallow neural networks. Preprint, 2023.
  35. Jiří Matoušek. Improved upper bounds for approximation by zonotopes. Acta Mathematica, 177(1):55–73, 1996.
  36. Convergence rates for single hidden layer feedforward networks. Neural Networks, 7(1):147–158, 1994.
  37. Hrushikesh N. Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural computation, 8(1):164–177, 1996.
  38. Foundations of machine learning. MIT Press, 2018.
  39. Adaptive approximation and generalization of deep neural network with intrinsic dimensionality. Journal of Machine Learning Research, 21(174):1–38, 2020.
  40. Norm-based capacity control in neural networks. In Proceedings of the 28th Conference on Learning Theory, pages 1376–1401. 2015.
  41. A PAC-Bayesian approach to spectrally-normalized margin bounds for neural networks. In 6th International Conference on Learning Representations, 2018.
  42. A function space view of bounded norm infinite width ReLU nets: The multivariate case. In 8th International Conference on Learning Representations, 2020.
  43. What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM Journal on Mathematics of Data Science, 4(2):464–489, 2022.
  44. Near-minimax optimal estimation with shallow ReLU neural networks. IEEE Transactions on Information Theory, 69(2):1125–1140, 2023.
  45. Equivalence of approximation by convolutional neural networks and fully-connected networks. Proceedings of the American Mathematical Society, 148(4):1567–1581, 2020.
  46. Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta Numerica, 8:143–195, 1999.
  47. Gilles Pisier. Remarques sur un résultat non publié de B. Maurey. Séminaire d’Analyse fonctionnelle (dit” Maurey-Schwartz”), pages 1–12, 1981.
  48. Kh. P. Rustamov. On equivalence of different moduli of smoothness on the sphere. Trudy Matematicheskogo Instituta im. V. A. Steklova, 204:274–304, 1993.
  49. How do infinite width bounded norm networks look in function space? In Proceedings of the 32nd Conference on Learning Theory, pages 2667–2690. 2019.
  50. Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4):1875–1897, 2020.
  51. Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014.
  52. Deep network approximation characterized by number of neurons. Communications in Computational Physics, 28(5):1768–1811, 2020.
  53. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313–321, 2020.
  54. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations of Computational Mathematics, pages 1–57, 2022.
  55. Characterization of the variation spaces corresponding to shallow neural networks. Constructive Approximation, 2023.
  56. Charles J. Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics, 10(4):1040–1053, 1982.
  57. Martin J. Wainwright. High-dimensional statistics: A non-asymptotic viewpoint, volume 48. Cambridge University Press, 2019.
  58. Information-theoretic determination of minimax rates of convergence. The Annals of Statistics, 27(5):1564–1599, 1999.
  59. Yunfei Yang. Learning distributions by generative adversarial networks: approximation and generalization. PhD thesis, The Hong Kong University of Science and Technology, 2022.
  60. Approximation in shift-invariant spaces with deep ReLU neural networks. Neural Networks, 153:269–281, 2022.
  61. Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.
  62. Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Proceedings of the 31st Conference on Learning Theory, pages 639–649. 2018.
  63. Ding-Xuan Zhou. Theory of deep convolutional neural networks: Downsampling. Neural Networks, 124:319–327, 2020a.
  64. Ding-Xuan Zhou. Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2):787–794, 2020b.
Citations (13)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now