Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Nonlinear functional regression by functional deep neural network with kernel embedding (2401.02890v1)

Published 5 Jan 2024 in stat.ML and cs.LG

Abstract: With the rapid development of deep learning in various fields of science and technology, such as speech recognition, image classification, and natural language processing, recently it is also widely applied in the functional data analysis (FDA) with some empirical success. However, due to the infinite dimensional input, we need a powerful dimension reduction method for functional learning tasks, especially for the nonlinear functional regression. In this paper, based on the idea of smooth kernel integral transformation, we propose a functional deep neural network with an efficient and fully data-dependent dimension reduction method. The architecture of our functional net consists of a kernel embedding step: an integral transformation with a data-dependent smooth kernel; a projection step: a dimension reduction by projection with eigenfunction basis based on the embedding kernel; and finally an expressive deep ReLU neural network for the prediction. The utilization of smooth kernel embedding enables our functional net to be discretization invariant, efficient, and robust to noisy observations, capable of utilizing information in both input functions and responses data, and have a low requirement on the number of discrete points for an unimpaired generalization performance. We conduct theoretical analysis including approximation error and generalization error analysis, and numerical simulations to verify these advantages of our functional net.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Sobolev Spaces. Elsevier, 2003.
  2. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2020.
  3. Francis Bach. On the equivalence between kernel quadrature rules and random feature expansions. The Journal of Machine Learning Research, 18(1):714–751, 2017.
  4. Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks. The Journal of Machine Learning Research, 20(1):2285–2301, 2019.
  5. Reproducing Kernel Hilbert Spaces in Probability and Statistics. Springer Science & Business Media, 2011.
  6. Principal components analysis of sampled functions. Psychometrika, 51(2):285–311, 1986.
  7. Model reduction and neural networks for parametric PDEs. The SMAI Journal of Computational Mathematics, 7:121–157, 2021.
  8. Spline estimators for the functional linear model. Statistica Sinica, pages 571–591, 2003.
  9. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911–917, 1995.
  10. Online gradient descent algorithms for functional data learning. Journal of Complexity, 70:101635, 2022.
  11. Deep neural networks for rotation-invariance approximation and learning. Analysis and Applications, 17(05):737–772, 2019.
  12. Moo K Chung. Statistical and Computational Methods in Brain Image Analysis. CRC press, 2013.
  13. On the mathematical foundations of learning. Bulletin of the American Mathematical Society, 39(1):1–49, 2002.
  14. Learning Theory: An Approximation Theory Viewpoint, volume 24. Cambridge University Press, 2007.
  15. Learning to synthesize: robust phase retrieval at low photon counts. Light: Science & Applications, 9(1):36, 2020.
  16. Constructive Approximation, volume 303. Springer Science & Business Media, 1993.
  17. Optimal regression rates for SVMs using Gaussian kernels. Electronic Journal of Statistics, 7:1–42, 2013.
  18. Gregory E Fasshauer. Positive definite kernels: past, present and future. Dolomites Research Notes on Approximation, 4:21–63, 2011.
  19. Nonparametric Functional Data Analysis: Theory and Practice, volume 76. New York: Springer, 2006.
  20. A Distribution-Free Theory of Nonparametric Regression, volume 1. Springer, 2002.
  21. David Haussler. Decision theoretic generalizations of the PAC model for neural net and other learning applications. Information and Computation, 100(1):78–150, 1992.
  22. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421–435, 2021.
  23. On universal approximation and error bounds for fourier neural operators. The Journal of Machine Learning Research, 22(1):13237–13312, 2021.
  24. Error estimates for DeepONets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
  25. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
  26. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020.
  27. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, 2021.
  28. Theory of deep convolutional neural networks III: Approximating radial functions. Neural Networks, 144:778–790, 2021.
  29. Approximating functions with multi-features by deep convolutional neural networks. Analysis and Applications, pages 1–33, 2022.
  30. Neural networks for functional approximation and system identification. Neural Computation, 9(1):143–159, 1997.
  31. New error bounds for deep ReLU networks using sparse grids. SIAM Journal on Mathematics of Data Science, 1(1):78–92, 2019.
  32. Integral autoencoder network for discretization-invariant learning. The Journal of Machine Learning Research, 23(286):1–45, 2022.
  33. Allan Pinkus. N-widths in Approximation Theory, volume 7. Springer Science & Business Media, 2012.
  34. Numerical Recipes in C. Cambridge University Press, 1992.
  35. Evaluation and development of deep neural networks for image super-resolution in optical microscopy. Nature Methods, 18(2):194–202, 2021.
  36. Image inpainting based on deep learning: A review. Displays, 69:102028, 2021.
  37. scikit-fda: a Python package for functional data analysis. arXiv preprint arXiv:2211.02566, 2022.
  38. Functional Data Analysis. New York: Springer, 2005.
  39. Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. MIT press, 2006.
  40. Nonparametric mixed effects models for unequally sampled noisy curves. Biometrics, 57(1):253–259, 2001.
  41. Functional multi-layer perceptron: a non-linear tool for functional data analysis. Neural Networks, 18(1):45–60, 2005.
  42. Functional data analysis with multi layer perceptrons. In Proceedings of the 2002 International Joint Conference on Neural Networks. IJCNN’02 (Cat. No. 02CH37290), volume 3, pages 2843–2848. IEEE, 2002.
  43. Representation of functional data in neural networks. Neurocomputing, 64:183–210, 2005.
  44. Johannes Schmidt-Hieber. Nonparametric regression using deep neural networks with ReLU activation function. The Annals of Statistics, 48(4):1875–1897, 2020.
  45. Bernard W Silverman. Smoothed functional principal components analysis by choice of norm. The Annals of Statistics, 24(1):1–24, 1996.
  46. A Hilbert space embedding for distributions. In Algorithmic Learning Theory, pages 13–31. Springer, 2007.
  47. Approximation of nonlinear functionals using deep ReLU networks. arXiv preprint arXiv:2304.04443, 2023.
  48. Approximation of smooth functionals using deep ReLU networks. Neural Networks, 2022. Minor revision.
  49. Support Vector Machines. Springer Science & Business Media, 2008.
  50. Taiji Suzuki. Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality. In International Conference on Learning Representations, 2019.
  51. Matus Telgarsky. Benefits of depth in neural networks. In Conference on Learning Theory, pages 1517–1539. PMLR, 2016.
  52. Deep learning on image denoising: An overview. Neural Networks, 131:251–275, 2020.
  53. Kernel Smoothing. CRC press, 1994.
  54. Holger Wendland. Scattered Data Approximation, volume 17. Cambridge university press, 2004.
  55. Functional data analysis for sparse longitudinal data. Journal of the American Statistical Association, 100(470):577–590, 2005.
  56. Deep learning for functional data analysis with adaptive basis layers. In International Conference on Machine Learning, pages 11898–11908. PMLR, 2021.
  57. Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103–114, 2017.
  58. Dmitry Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Conference on Learning Theory, pages 639–649. PMLR, 2018.
  59. Ding-Xuan Zhou. Universality of deep convolutional neural networks. Applied and Computational Harmonic Analysis, 48(2):787–794, 2020.
  60. Functional linear regression for discretely observed data: from ideal to reality. Biometrika, 2022.
Citations (4)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets