Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Deep Neural Networks and Finite Elements of Any Order on Arbitrary Dimensions (2312.14276v3)

Published 21 Dec 2023 in math.NA, cs.LG, and cs.NA

Abstract: In this study, we establish that deep neural networks employing ReLU and ReLU$2$ activation functions can effectively represent Lagrange finite element functions of any order on various simplicial meshes in arbitrary dimensions. We introduce two novel formulations for globally expressing the basis functions of Lagrange elements, tailored for both specific and arbitrary meshes. These formulations are based on a geometric decomposition of the elements, incorporating several insightful and essential properties of high-dimensional simplicial meshes, barycentric coordinate functions, and global basis functions of linear elements. This representation theory facilitates a natural approximation result for such deep neural networks. Our findings present the first demonstration of how deep neural networks can systematically generate general continuous piecewise polynomial functions on both specific or arbitrary simplicial meshes.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. Sobolev spaces. Elsevier, 2003.
  2. Finite element exterior calculus: from hodge theory to numerical stability. Bulletin of the American mathematical society, 47(2):281–354, 2010.
  3. Finite element exterior calculus, homological techniques, and applications. Acta numerica, 15:1–155, 2006.
  4. Understanding deep neural networks with rectified linear units. In International Conference on Learning Representations, 2018.
  5. The p and h-p versions of the finite element method, basic principles and properties. SIAM review, 36(4):578–632, 1994.
  6. John Burkardt. The finite element basis for simplices in arbitrary dimensions. Technical report, Technical Report, Florida State University, Department of Scientific Computing, 2013.
  7. Geometric decompositions of the simplicial lattice and smooth finite elements in arbitrary dimension. arXiv preprint arXiv:2111.10712, 2021.
  8. Power series expansion neural network. Journal of Computational Science, 59:101552, 2022.
  9. Philippe G Ciarlet. The finite element method for elliptic problems. SIAM, 2002.
  10. Nonlinear approximation and (deep) relu networks. Constructive Approximation, 55(1):127–172, 2022.
  11. The deep ritz method: a deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1–12, 2018.
  12. Hans Freudenthal. Simplizialzerlegungen von beschrankter flachheit. Annals of Mathematics, pages 580–582, 1942.
  13. Juncai He. On the optimal expressive power of relu dnns and its application in approximation with kolmogorov superposition theorem. arXiv preprint arXiv:2308.05509, 2023.
  14. Relu deep neural networks from the hierarchical basis perspective. Computers & Mathematics with Applications, 120:105–114, 2022.
  15. Relu deep neural networks and linear finite elements. Journal of Computational Mathematics, 38(3):502–527, 2020.
  16. Expressivity and approximation properties of deep neural networks with reluk𝑘{}^{k}start_FLOATSUPERSCRIPT italic_k end_FLOATSUPERSCRIPT activation. In Preparation, 2023.
  17. A construction of c r conforming finite element spaces in any dimension. Foundations of Computational Mathematics, pages 1–37, 2023.
  18. Harold W Kuhn. Some combinatorial lemmas in topology. IBM Journal of research and development, 4(5):518–524, 1960.
  19. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987–1000, 1998.
  20. Better approximations of high dimensional smooth functions by deep neural networks with rectified power units. Communications in Computational Physics, 27(2):379–411, 2019.
  21. Reproducing activation function for deep learning. arXiv preprint arXiv:2101.04844, 2021.
  22. Deep network approximation for smooth functions. SIAM Journal on Mathematical Analysis, 53(5):5465–5506, 2021.
  23. The expressive power of neural networks: A view from the width. In Advances in Neural Information Processing Systems, pages 6231–6239, 2017.
  24. Exponential relu neural network approximation rates for point and edge singularities. Foundations of Computational Mathematics, 23(3):1043–1127, 2023.
  25. Bounds on the approximation power of feedforward neural networks. In International Conference on Machine Learning, pages 3453–3461. PMLR, 2018.
  26. Deep vs. shallow networks: An approximation theory perspective. Analysis and Applications, 14(06):829–848, 2016.
  27. Deep relu networks overcome the curse of dimensionality for generalized bandlimited functions. Journal of Computational Mathematics, 39(6), 2021.
  28. On the number of linear regions of deep neural networks. In Advances in Neural Information Processing Systems, pages 2924–2932, 2014.
  29. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.
  30. Deep relu networks and high-order finite element methods. Analysis and Applications, pages 1–56, 2020.
  31. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686–707, 2019.
  32. Deep network approximation characterized by number of neurons. Communications in Computational Physics, 28(5):1768–1811, 2020.
  33. Neural network approximation: Three hidden layers are enough. Neural Networks, 141:160–173, 2021.
  34. Finite Element Methods. Science Press, Beijing, China, 2013.
  35. Greedy training algorithms for neural networks and applications to pdes. Journal of Computational Physics, 484:112084, 2023.
  36. Matus Telgarsky. Representation benefits of deep feedforward networks. arXiv preprint arXiv:1509.08101, 2015.
  37. Jinchao Xu. Finite neuron method and convergence analysis. Communications in Computational Physics, 28(5), 2020.
  38. Nearly optimal approximation rates for deep super relu networks on sobolev spaces. arXiv preprint arXiv:2310.10766, 2023.
  39. Nearly optimal vc-dimension and pseudo-dimension bounds for deep neural network derivatives. arXiv preprint arXiv:2305.08466, 2023.
  40. Dmitry Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103–114, 2017.
  41. Dmitry Yarotsky. Optimal approximation of continuous functions by very deep relu networks. In Conference on learning theory, pages 639–649. PMLR, 2018.
  42. Dmitry Yarotsky. Elementary superexpressive activations. In International Conference on Machine Learning, pages 11932–11940. PMLR, 2021.
Citations (6)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now