Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetwork (2312.15949v1)

Published 26 Dec 2023 in cs.LG, cs.NA, and math.NA

Abstract: Fast and accurate predictions for complex physical dynamics are a significant challenge across various applications. Real-time prediction on resource-constrained hardware is even more crucial in real-world problems. The deep operator network (DeepONet) has recently been proposed as a framework for learning nonlinear mappings between function spaces. However, the DeepONet requires many parameters and has a high computational cost when learning operators, particularly those with complex (discontinuous or non-smooth) target functions. This study proposes HyperDeepONet, which uses the expressive power of the hypernetwork to enable the learning of a complex operator with a smaller set of parameters. The DeepONet and its variant models can be thought of as a method of injecting the input function information into the target function. From this perspective, these models can be viewed as a particular case of HyperDeepONet. We analyze the complexity of DeepONet and conclude that HyperDeepONet needs relatively lower complexity to obtain the desired accuracy for operator learning. HyperDeepONet successfully learned various operators with fewer computational resources compared to other benchmarks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (46)
  1. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations, 2019. URL https://openreview.net/forum?id=fg2ZFmXFO3.
  2. Prediction of aerodynamic flow fields using convolutional neural networks. Comput. Mech., 64(2):525–545, 2019. ISSN 0178-7675. doi: 10.1007/s00466-019-01740-0. URL https://doi.org/10.1007/s00466-019-01740-0.
  3. Deepm&mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436:110296, 2021.
  4. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911–917, 1995.
  5. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  5939–5948, 2019.
  6. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, 53(7):5113–5155, 2020.
  7. Hyperpinn: Learning parameterized differential equations with physics-informed hypernetworks. In The Symbiosis of Deep Learning and Differential Equations, 2021.
  8. The deep Ritz method: a deep learning-based numerical algorithm for solving variational problems. Commun. Math. Stat., 6(1):1–12, 2018. ISSN 2194-6701. doi: 10.1007/s40304-018-0127-z. URL https://doi.org/10.1007/s40304-018-0127-z.
  9. On the modularity of hypernetworks. Advances in Neural Information Processing Systems, 33:10409–10419, 2020.
  10. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp.  481–490, 2016.
  11. Hypernetworks. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=rkpACe1lx.
  12. Patrik Simon Hadorn. Shift-deeponet: Extending deep operator networks for discontinuous output functions. ETH Zurich, Seminar for Applied Mathematics, 2022.
  13. Approximating continuous functions by relu nets of minimal width. arXiv preprint arXiv:1710.11278, 2017.
  14. Solving pde-constrained control problems using operator learning. arXiv preprint arXiv:2111.04941, 2021.
  15. Solving parametric PDE problems with artificial neural networks. European J. Appl. Math., 32(3):421–435, 2021. ISSN 0956-7925. doi: 10.1017/S0956792520000182. URL https://doi.org/10.1017/S0956792520000182.
  16. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980.
  17. Hypernetwork functional image representation. In International Conference on Artificial Neural Networks, pp.  496–510. Springer, 2019.
  18. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22:Art–No, 2021a.
  19. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481, 2021b.
  20. Fourcastnet: Accelerating global high-resolution weather forecasting using adaptive fourier neural operators. arXiv preprint arXiv:2208.05419, 2022.
  21. Error estimates for deeponets: A deep learning framework in infinite dimensions. Transactions of Mathematics and Its Applications, 6(1):tnac001, 2022.
  22. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=c8P9NQVtmnO.
  23. Operator learning for predicting multiscale bubble growth dynamics. The Journal of Chemical Physics, 154(10):104118, 2021.
  24. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
  25. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218–229, 2021.
  26. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
  27. DeepM&Mnet for hypersonics: predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators. J. Comput. Phys., 447:Paper No. 110698, 24, 2021. ISSN 0021-9991. doi: 10.1016/j.jcp.2021.110698. URL https://doi.org/10.1016/j.jcp.2021.110698.
  28. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.  14214–14223, 2021.
  29. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4460–4470, 2019.
  30. Hrushikesh N Mhaskar. Neural networks for optimal approximation of smooth and analytic functions. Neural computation, 8(1):164–177, 1996.
  31. Machine learning at the network edge: A survey. ACM Computing Surveys (CSUR), 54(8):1–37, 2021.
  32. Neural implicit flow: a mesh-agnostic dimensionality reduction paradigm of spatio-temporal data. arXiv preprint arXiv:2204.03216, 2022.
  33. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  165–174, 2019.
  34. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1711.01297, 2017.
  35. Variable-input deep operator networks. arXiv preprint arXiv:2205.11404, 2022.
  36. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys., 378:686–707, 2019. ISSN 0021-9991. doi: 10.1016/j.jcp.2018.10.045. URL https://doi.org/10.1016/j.jcp.2018.10.045.
  37. Generic bounds on the approximation error for physics-informed (and) operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=bF4eYy3LTR9.
  38. Nomad: Nonlinear manifold decoders for operator learning. arXiv preprint arXiv:2206.03551, 2022.
  39. DGM: a deep learning algorithm for solving partial differential equations. J. Comput. Phys., 375:1339–1364, 2018. ISSN 0021-9991. doi: 10.1016/j.jcp.2018.08.029. URL https://doi.org/10.1016/j.jcp.2018.08.029.
  40. Implicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462–7473, 2020.
  41. PDEBench: An extensive benchmark for scientific machine learning. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=dh_MkX0QfrK.
  42. Svd perspectives for augmenting deeponet flexibility and interpretability. Computer Methods in Applied Mechanics and Engineering, 403:115718, 2023.
  43. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SJgwNerKvB.
  44. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, 7(40):eabi8605, 2021.
  45. Improved architectures and training algorithms for deep operator networks. Journal of Scientific Computing, 92(2):1–42, 2022.
  46. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. J. Comput. Phys., 394:56–81, 2019. ISSN 0021-9991. doi: 10.1016/j.jcp.2019.05.024. URL https://doi.org/10.1016/j.jcp.2019.05.024.
Citations (21)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now