Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
139 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Beyond Regular Grids: Fourier-Based Neural Operators on Arbitrary Domains (2305.19663v4)

Published 31 May 2023 in cs.LG, cs.NA, and math.NA

Abstract: The computational efficiency of many neural operators, widely used for learning solutions of PDEs, relies on the fast Fourier transform (FFT) for performing spectral computations. As the FFT is limited to equispaced (rectangular) grids, this limits the efficiency of such neural operators when applied to problems where the input and output functions need to be processed on general non-equispaced point distributions. Leveraging the observation that a limited set of Fourier (Spectral) modes suffice to provide the required expressivity of a neural operator, we propose a simple method, based on the efficient direct evaluation of the underlying spectral transformation, to extend neural operators to arbitrary domains. An efficient implementation* of such direct spectral evaluations is coupled with existing neural operator models to allow the processing of data on arbitrary non-equispaced distributions of points. With extensive empirical evaluation, we demonstrate that the proposed method allows us to extend neural operators to arbitrary point distributions with significant gains in training speed over baselines while retaining or improving the accuracy of Fourier neural operators (FNOs) and related neural operators.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Direct inversion of the three-dimensional pseudo-polar fourier transform. SIAM Journal on Scientific Computing, 38(2):A1100–A1120, 2016.
  2. The nonuniform discrete Fourier transform and its applications in signal processing. Springer Science and Business Media, 1999. doi: 10.1007/978-1-4615-4925-3.
  3. A parallel non-uniform fast Fourier transform library based on an “exponential of semicircle” kernel. SIAM J. Sci. Comput., 41(5):C479–C504, 2019.
  4. A second-order projection method for the incompressible Navier-Stokes equations. J. Comput. Phys., 85:257–283, 1989.
  5. Beylkin, G. On the fast fourier transform of functions with singularities. Applied and Computational Harmonic Analysis, 2(4):363–381, 1995.
  6. Spherical fourier neural operators: Learning stable dynamics on the sphere, 2023.
  7. Message Passsing Neural PDE solvers. arXiv preprint arXiv:2202.03376, 2022.
  8. DeepM&Mnet: Inferring the electroconvection multiphysics fields based on operator approximation by neural networks. Journal of Computational Physics, 436:110296, 2021.
  9. Cao, S. Choose a transformer: Fourier or galerkin. In 35th conference on neural information processing systems, 2021.
  10. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):911–917, 1995.
  11. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010.11929, 2020. URL https://arxiv.org/abs/2010.11929.
  12. Computing fourier transforms and convolutions on the 2-sphere. Advances in Applied Mathematics, 15(2):202–250, 1994. ISSN 0196-8858. doi: 10.1006/aama.1994.1008.
  13. Fast fourier transforms for nonequispaced data. SIAM Journal on Scientific Computing, 14(6):1368–1393, 1993.
  14. Fast fourier transforms for nonequispaced data ii. Applied and Computational Harmonic Analysis, 2:85–100, 1995.
  15. EarthData, N. Modern-era restrospective analysis for research and applications (merra-2) 2-dimensional data collection., 2021 - 2023.
  16. Multi-scale message passing neural pde solvers. arXiv preprint arXiv:2302.03580, 2023.
  17. Evans, L. C. Partial differential equations, volume 19. American Mathematical Soc., 2010.
  18. A frame theoretic approach to the nonuniform fast fourier transform. SIAM Journal on Numerical Analysis, 52(3):1222–1242, 2014.
  19. Fast algorithms with preprocessing for matrix-vector multiplication problems. Journal of Complexity, 10(4):411–427, 1994a. ISSN 0885-064X. doi: https://doi.org/10.1006/jcom.1994.1021. URL https://www.sciencedirect.com/science/article/pii/S0885064X84710211.
  20. Complexity of multiplication with vectors for structured matrices. Linear Algebra and its Applications, 202:163–192, 1994b. ISSN 0024-3795. doi: https://doi.org/10.1016/0024-3795(94)90189-9. URL https://www.sciencedirect.com/science/article/pii/0024379594901899.
  21. Accelerating the nonuniform fast fourier transform. SIAM Review, 46(3):443–454, 2004. doi: 10.1137/S003614450343200X.
  22. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training, 2024.
  23. Algebraic Methods for Toeplitz-Like Matrices and Operators. Akademie-Verlag, Berlin, 1984.
  24. Adaptive Moving Mesh Methods, volume 174. Springer Science and Business Media, 2010.
  25. Physics informed machine learning. Nature Reviews Physics, pp.  1–19, may 2021. doi: 10.1038/s42254-021-00314-5. URL www.nature.com/natrevphys.
  26. Direct inversion of the nonequispaced fast fourier transform. 11 2018.
  27. Direct inversion of the nonequispaced fast fourier transform. Linear Algebra and its Applications, 575:106–140, 2019.
  28. Fast and direct inverion methods for the multivariate nonequispaced fast fourier transform. 2 2023.
  29. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1–63, 2022.
  30. On universal approximation and error bounds for fourier neural operators. 07 2021a.
  31. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481v3, 2021b.
  32. Kunis, S. Nonequispaced fft generalisation and inversion. PhD. dissertation, Inst. Math., Univ. Lübeck, Lübeck, 2006.
  33. Stability results for scattered data interpolation by trigonometric polynomials. SIAM Journal on Scientific Computing, 29:1403–1419, 2007.
  34. Statistical solutions of the incompressible euler equations. Mathematical Models and Methods in Applied Sciences, 31(02):223–292, Feb 2021. ISSN 1793-6314. doi: 10.1142/s0218202521500068. URL http://dx.doi.org/10.1142/s0218202521500068.
  35. The nonlocal neural operator: Universal approximation, 2023a.
  36. Nonlinear reconstruction for operator learning of pdes with discontinuities. In International Conference on Learning Representations, 2023b.
  37. Fourier neural operator for parametric partial differential equations. CoRR, abs/2010.08895, 2020a. URL https://arxiv.org/abs/2010.08895.
  38. Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp.  6755–6766. Curran Associates, Inc., 2020b.
  39. Multipole graph neural operator for parametric partial differential equations. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp.  6755–6766. Curran Associates, Inc., 2020c.
  40. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.
  41. Fourier neural operator with learned deformations for pdes on general geometries, 2022.
  42. Scalable transformer for pde surrogate modeling, 2023.
  43. Non-equispaced fourier neural solvers for pdes, 2022.
  44. An accurate algorithm for nonuniform fast fourier transforms (NUFFTs). IEEE Microwave and Guided Wave Letters, 8(1):18–20, 1998. doi: 10.1109/75.650975.
  45. DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. CoRR, abs/1910.03193, 2019. URL http://arxiv.org/abs/1910.03193.
  46. Deep learning observables in computational fluid dynamics. Journal of Computational Physics, 410:109339, 2020. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2020.109339. URL https://www.sciencedirect.com/science/article/pii/S0021999120301133.
  47. Iterative surrogate model optimization (ISMO): An active learning algorithm for PDE constrained optimization with deep neural networks. Computer Methods in Applied Mechanics and Engineering, 374:113575, 2021. ISSN 0045-7825. doi: https://doi.org/10.1016/j.cma.2020.113575. URL https://www.sciencedirect.com/science/article/pii/S004578252030760X.
  48. DeepMandMnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators. Preprint, available from arXiv:2011.03349v1, 2020.
  49. A novel sampling theorem on the sphere. IEEE Transactions on Signal Processing, 59(12):5876–5887, December 2011. ISSN 1941-0476. doi: 10.1109/tsp.2011.2166394. URL http://dx.doi.org/10.1109/TSP.2011.2166394.
  50. TorchKbNufft: A high-level, hardware-agnostic non-uniform fast Fourier transform. In ISMRM Workshop on Data Sampling & Image Reconstruction, 2020. Source code available at https://github.com/mmuckley/torchkbnufft.
  51. Pan, V. Y. Structured Matrices and Polynomials: Unified Superfast Algorithms. Birkhauser/Springer, Boston/New York, 2001.
  52. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  53. A fast dvm algorithm for wideband time-delay multi-beam beamformers. the IEEE Transactions on Signal Processing, 70:5913–5925, 2022. doi: 10.1109/TSP.2022.3231182.
  54. Learning mesh-based simulation with graph networks. CoRR, abs/2010.03409, 2020. URL https://arxiv.org/abs/2010.03409.
  55. Variable input deep operator networks. arXiv preprint arXiv:2205.11404, 2022.
  56. Numerical approximation of Partial differential equations, volume 23. Springer, 1994.
  57. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019.
  58. Convolutional neural operators. arXiv preprint arXiv:2302.01178, 2023.
  59. A nonuniform fast fourier transform based on low rank approximation. SIAM Journal on Scientific Computing, 40(1):A529–A547, 2018.
  60. Learning to simulate complex physics with graph networks. CoRR, abs/2002.09405, 2020. URL https://arxiv.org/abs/2002.09405.
  61. Selva, J. Efficient type-4 and type-5 non-uniform fft methods in the one-dimensional case. IET Signal Processing, 12(1):74–81, 2018.
  62. cufinufft: a load-balanced gpu library for general-purpose nonuniform ffts, 2021.
  63. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tmIiMPl4IPa.
  64. Unsupervised deep learning of incompressible fluid dynamics. CoRR, abs/2006.08762, 2020. URL https://arxiv.org/abs/2006.08762.
  65. U-fno – an enhanced fourier neural operator-based deep-learning model for multiphase flow, 2022.
  66. Solving high-dimensional pdes with latent spectral models, 2023.
  67. Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 336:415–447, 2018.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now