Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
162 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Accelerating Data Generation for Neural Operators via Krylov Subspace Recycling (2401.09516v2)

Published 17 Jan 2024 in cs.LG, cs.AI, cs.NA, and math.NA

Abstract: Learning neural operators for solving partial differential equations (PDEs) has attracted great attention due to its high inference efficiency. However, training such operators requires generating a substantial amount of labeled data, i.e., PDE problems together with their solutions. The data generation process is exceptionally time-consuming, as it involves solving numerous systems of linear equations to obtain numerical solutions to the PDEs. Many existing methods solve these systems independently without considering their inherent similarities, resulting in extremely redundant computations. To tackle this problem, we propose a novel method, namely Sorting Krylov Recycling (SKR), to boost the efficiency of solving these systems, thus significantly accelerating data generation for neural operators training. To the best of our knowledge, SKR is the first attempt to address the time-consuming nature of data generation for learning neural operators. The working horse of SKR is Krylov subspace recycling, a powerful technique for solving a series of interrelated systems by leveraging their inherent similarities. Specifically, SKR employs a sorting algorithm to arrange these systems in a sequence, where adjacent systems exhibit high similarities. Then it equips a solver with Krylov subspace recycling to solve the systems sequentially instead of independently, thus effectively enhancing the solving efficiency. Both theoretical analysis and extensive experiments demonstrate that SKR can significantly accelerate neural operator data generation, achieving a remarkable speedup of up to 13.9 times.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Walter Edwin Arnoldi. The principle of minimized iterations in the solution of the matrix eigenvalue problem. Quarterly of applied mathematics, 9(1):17–29, 1951.
  2. Neural operator-based surrogate solver for free-form electromagnetic inverse design. ACS Photonics, 2023.
  3. Convergence of restarted krylov subspaces to invariant subspaces. SIAM Journal on Matrix Analysis and Applications, 25(4):1074–1109, 2004.
  4. A sparse approximate inverse preconditioner for the conjugate gradient method. SIAM Journal on Scientific Computing, 17(5):1135–1149, 1996.
  5. Daniele Bertaccini et al. Efficient preconditioning for sequences of parametric complex symmetric linear systems. Electronic Transactions on Numerical Analysis, 18:49–64, 2004.
  6. Preconditioner updates applied to cfd model problems. Applied Numerical Mathematics, 58(11):1628–1641, 2008.
  7. Clifford neural layers for pde modeling. arXiv preprint arXiv:2209.04934, 2022.
  8. A flexible generalized conjugate residual method with inner orthogonalization and deflated restarting. SIAM Journal on Matrix Analysis and Applications, 32(4):1212–1235, 2011.
  9. Philippe G Ciarlet. The finite element method for elliptic problems. SIAM, 2002.
  10. Eric de Sturler. Nested krylov methods based on gcr. Journal of Computational and Applied Mathematics, 67(1):15–41, 1996.
  11. Eric De Sturler. Truncation strategies for optimal krylov subspace methods. SIAM Journal on Numerical Analysis, 36(3):864–889, 1999.
  12. Peter Deuflhard. Newton methods for nonlinear problems: affine invariance and adaptive algorithms, volume 35. Springer Science & Business Media, 2005.
  13. Chebfun guide, 2014.
  14. Mark Embree. How descriptive are gmres convergence bounds? arXiv preprint arXiv:2209.01231, 2022.
  15. Deflation and balancing preconditioners for krylov subspace methods applied to nonsymmetric matrices. SIAM Journal on Matrix Analysis and Applications, 30(2):684–699, 2008.
  16. A Gaul and N Schlömer. pynosh: Python framework for nonlinear schrödinger equations. july 2013. URl: https://bitbucket. org/nschloe/pynosh.
  17. André Gaul. Recycling krylov subspace methods for sequences of linear systems. 2014.
  18. Preconditioned recycling krylov subspace methods for self-adjoint problems. arXiv preprint arXiv:1208.0264, 2012.
  19. A framework for deflated and augmented krylov subspace methods. SIAM Journal on Matrix Analysis and Applications, 34(2):495–518, 2013.
  20. Anne Greenbaum. Iterative methods for solving linear systems. SIAM, 1997.
  21. Learning to optimize multigrid pde solvers. In International Conference on Machine Learning, pp. 2415–2423. PMLR, 2019.
  22. Mpi-based implementation of a pcg solver using an ebe architecture and preconditioner for implicit, 3-d finite element analysis. Computers & Structures, 79(5):553–575, 2001.
  23. Physics-informed machine learning: A survey on problems, methods and applications. arXiv preprint arXiv:2211.08064, 2022.
  24. Learning neural pde solvers with convergence guarantees. arXiv preprint arXiv:1906.01200, 2019.
  25. Thomas JR Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
  26. Claes Johnson. Numerical solution of partial differential equations by the finite element method. Courier Corporation, 2012.
  27. Carl T Kelley. Solving nonlinear equations with Newton’s method. SIAM, 2003.
  28. Recycling subspace information for diffuse optical tomography. SIAM Journal on Scientific Computing, 27(6):2140–2166, 2006.
  29. Jacobian-free newton–krylov methods: a survey of approaches and applications. Journal of Computational Physics, 193(2):357–397, 2004.
  30. Data-driven and physics-informed deep learning operators for solution of heat conduction equation with parametric heat source. International Journal of Heat and Mass Transfer, 203:123809, 2023.
  31. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2108.08481, 2021.
  32. Randall J LeVeque. Finite volume methods for hyperbolic problems, volume 31. Cambridge university press, 2002.
  33. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
  34. Incomplete cholesky factorizations with limited memory. SIAM Journal on Scientific Computing, 21(1):24–45, 1999.
  35. Ino: Invariant neural operators for learning complex physical systems with momentum conservation. In International Conference on Artificial Intelligence and Statistics, pp.  6822–6838. PMLR, 2023.
  36. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03193, 2019.
  37. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:114778, 2022.
  38. Nonlinear eigenvalue and frequency response problems in industrial practice. Journal of Mathematics in Industry, 1:1–18, 2011.
  39. Gérard Meurant. On the incomplete cholesky decomposition of a class of perturbed matrices. SIAM Journal on Scientific Computing, 23(2):419–429, 2001.
  40. Igor Moret. A note on the superlinear convergence of gmres. SIAM journal on numerical analysis, 34(2):513–516, 1997.
  41. Ronald B Morgan. Gmres with deflated restarting. SIAM Journal on Scientific Computing, 24(1):20–37, 2002.
  42. Lattice qcd at finite density: an introductory review. Progress of theoretical physics, 110(4):615–668, 2003.
  43. Roy A Nicolaides. Deflation of conjugate gradients with applications to boundary value problems. SIAM Journal on Numerical Analysis, 24(2):355–365, 1987.
  44. Recycling krylov subspaces for sequences of linear systems. SIAM Journal on Scientific Computing, 28(5):1651–1674, 2006.
  45. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202.11214, 2022.
  46. U-no: U-shaped neural operators. arXiv preprint arXiv:2204.11127, 2022.
  47. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561, 2017.
  48. Gmres: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on scientific and statistical computing, 7(3):856–869, 1986.
  49. Yousef Saad. Iterative methods for sparse linear systems. SIAM, 2003.
  50. Weakly-supervised deep learning of heat transport via physics informed loss. arXiv preprint arXiv:1807.11374, 2018.
  51. Gmres convergence for perturbed coefficient matrices, with application to approximate deflation preconditioning. SIAM Journal on Matrix Analysis and Applications, 34(3):1066–1088, 2013.
  52. On the occurrence of superlinear convergence of exact and inexact krylov subspace methods. SIAM review, 47(2):247–272, 2005.
  53. Gordon D Smith. Numerical solution of partial differential equations: finite difference methods. Oxford university press, 1985.
  54. A survey of subspace recycling iterative methods. GAMM-Mitteilungen, 43(4):e202000016, 2020.
  55. Gilbert W Stewart. Matrix Algorithms: Volume II: Eigensystems. SIAM, 2001.
  56. John C Strikwerda. Finite difference schemes and partial differential equations. SIAM, 2004.
  57. James William Thomas. Numerical partial differential equations: finite difference methods, volume 22. Springer Science & Business Media, 2013.
  58. Domain decomposition methods-algorithms and theory, volume 34. Springer Science & Business Media, 2004.
  59. The superlinear convergence behaviour of gmres. Journal of computational and applied mathematics, 48(3):327–341, 1993.
  60. Large-scale topology optimization using preconditioned krylov subspace methods with recycling. International journal for numerical methods in engineering, 69(12):2441–2468, 2007.
  61. U-fno—an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022.
  62. David Young. Iterative methods for solving partial difference equations of elliptic type. Transactions of the American Mathematical Society, 76(1):92–111, 1954.
  63. Introduction to partial differential equations with applications. Courier Corporation, 1986.
  64. A hybrid iterative numerical transferable solver (hints) for pdes based on deep operator network and relaxation methods. arXiv preprint arXiv:2208.13273, 2022.
  65. Artificial intelligence for science in quantum, atomistic, and continuum systems. arXiv preprint arXiv:2307.08423, 2023.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now