Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-Grid Tensorized Fourier Neural Operator for High-Resolution PDEs (2310.00120v1)

Published 29 Sep 2023 in cs.LG

Abstract: Memory complexity and data scarcity have so far prohibited learning solution operators of partial differential equations (PDEs) at high resolutions. We address these limitations by introducing a new data efficient and highly parallelizable operator learning approach with reduced memory requirement and better generalization, called multi-grid tensorized neural operator (MG-TFNO). MG-TFNO scales to large resolutions by leveraging local and global structures of full-scale, real-world phenomena, through a decomposition of both the input domain and the operator's parameter space. Our contributions are threefold: i) we enable parallelization over input samples with a novel multi-grid-based domain decomposition, ii) we represent the parameters of the model in a high-order latent subspace of the Fourier domain, through a global tensor factorization, resulting in an extreme reduction in the number of parameters and improved generalization, and iii) we propose architectural improvements to the backbone FNO. Our approach can be used in any operator learning setting. We demonstrate superior performance on the turbulent Navier-Stokes equations where we achieve less than half the error with over 150x compression. The tensorization combined with the domain decomposition, yields over 150x reduction in the number of parameters and 7x reduction in the domain size without losses in accuracy, while slightly enabling parallelism.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (61)
  1. Solving ill-posed inverse problems using iterative deep neural networks. Inverse Problems, nov 2017.
  2. A spectral fc solver for the compressible navier–stokes equations in general domains i: Explicit time-stepping. Journal of Computational Physics, 230(16):6248–6270, 2011.
  3. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  4. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, pp.  1–21, 2019.
  5. Model reduction and neural networks for parametric pdes. arXiv preprint arXiv:2005.03180, 2020.
  6. The role of internal variability in global climate projections of extreme events. arXiv preprint arXiv:2208.08275, 2022.
  7. Incremental multi-domain learning with network latent tensor factorization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.  10470–10477, 2020a.
  8. Toward fast and accurate human pose estimation via soft-gated skip connections. In 2020 15th IEEE International Conference on Automatic Face & Gesture Recognition, 2020b.
  9. Domain decomposition algorithms. Acta Numerica, 3:61–143, 1994.
  10. Invariant recurrent solutions embedded in a turbulent two-dimensional kolmogorov flow. Journal of Fluid Mechanics, 722:554–595, 2013.
  11. Multi-head attention: Collaborate instead of concatenate. arXiv preprint arXiv:2006.16362, 2020.
  12. Uber die partiellen differenzengleichungen der mathematischen physik. Mathematische annalen, 100(1):32–74, 1928.
  13. Sobolev training for neural networks. Advances in Neural Information Processing Systems, 30, 2017.
  14. Monarch: Expressive structured matrices for efficient and accurate training. In International Conference on Machine Learning, pp. 4690–4721. PMLR, 2022.
  15. The cost-accuracy trade-off in operator learning with neural networks. arXiv preprint arXiv:2203.13181, 2022.
  16. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  17. Meshfreeflownet: A physics-constrained deep continuous space-time super-resolution framework. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.  1–15. IEEE, 2020.
  18. Lawrence C. Evans. Partial differential equations. American Mathematical Society, 2010.
  19. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587, 2021.
  20. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016.
  21. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems, 34:24048–24062, 2021.
  22. Automated multi-stage compression of neural networks. Oct 2019.
  23. Identity mappings in deep residual networks. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pp.  630–645. Springer, 2016.
  24. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448–456. pmlr, 2015.
  25. Spectral learning on matrices and tensors. Found. and Trends® in Mach. Learn., 12(5-6):393–536, 2019.
  26. Compression of deep convolutional neural networks for fast and low power mobile applications. 2016.
  27. Tensor decompositions and applications. SIAM Rev., 51(3):455–500, 2009.
  28. Jean Kossaifi. Tensorly-torch. https://github.com/tensorly/torch, 2021.
  29. Tensorly: Tensor learning in python. Journal of Machine Learning Research (JMLR), 20(26), 2019.
  30. Factorized higher-order CNNs with an application to spatio-temporal emotion estimation. pp.  6060–6069, 2020.
  31. On universal approximation and error bounds for fourier neural operators. Journal of Machine Learning Research, 22(290):1–76, 2021a.
  32. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481, 2021b.
  33. Speeding-up convolutional neural networks using fine-tuned CP-decomposition. 2015.
  34. Ensemble forecasting. Journal of computational physics, 227(7):3515–3539, 2008.
  35. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020a.
  36. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755–6766, 2020b.
  37. Markov neural operators for learning chaotic systems. arXiv preprint arXiv:2106.06898, 2021a.
  38. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021b.
  39. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021c.
  40. A learning-based multiscale method and its application to inelastic impact problems. Journal of the Mechanics and Physics of Solids, 158:104668, 2022.
  41. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
  42. S. F. McCormick. Multigrid methods for variational problems: General theory for the v- cycle. SIAM Journal on Numerical Analysis, 22(4):634–643, 1985.
  43. Tensorizing neural networks. pp.  442–450, 2015.
  44. I. V. Oseledets. Tensor-train decomposition. SIAM J. Sci. Comput., 33(5):2295–2317, September 2011.
  45. Tensor methods in computer vision and deep learning. Proceedings of the IEEE, 109(5):863–890, 2021. doi: 10.1109/JPROC.2021.3074329.
  46. Efficient learning of multiple nlp tasks via collective weight factorization on bert. In Findings of the Association for Computational Linguistics: NAACL 2022, pp.  882–890, 2022.
  47. Tensors for data mining and data fusion: Models, applications, and scalable algorithms. ACM Trans. Intell. Syst. and Technol. (TIST), 8(2):1–44, 2016.
  48. Automatic differentiation in PyTorch. 2017.
  49. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  50. An Introduction to Computational Stochastic PDEs. Texts in Applied Mathematics. Cambridge University Press, United Kingdom, 1 edition, August 2014. ISBN 9780521728522.
  51. Generative adversarial neural operators. arXiv preprint arXiv:2205.03017, 2022a.
  52. U-no: U-shaped neural operators. arXiv preprint arXiv:2204.11127, 2022b.
  53. Tensor decomposition for signal processing and machine learning. Transactions Signal Processing, 65(13):3551–3582, 2017.
  54. Uncertainty in weather and climate prediction. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 369(1956):4751–4767, 2011.
  55. Roger Temam. Infinite-dimensional dynamical systems in mechanics and physics. Applied mathematical sciences. Springer-Verlag, New York, 1988.
  56. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tmIiMPl4IPa.
  57. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
  58. U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022.
  59. Seismic wave propagation and inversion with neural operators. The Seismic Record, 1(3):126–134, 2021.
  60. Accelerated full seismic waveform modeling and inversion with u-shaped neural operators. arXiv preprint arXiv:2209.11955, 2022.
  61. Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 2018. ISSN 0021-9991.
Citations (21)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now