Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

NonlinearSolve.jl: High-Performance and Robust Solvers for Systems of Nonlinear Equations in Julia (2403.16341v2)

Published 25 Mar 2024 in math.NA and cs.NA

Abstract: Efficiently solving nonlinear equations underpins numerous scientific and engineering disciplines, yet scaling these solutions for complex system models remains a challenge. This paper presents NonlinearSolve.jl - a suite of high-performance open-source nonlinear equation solvers implemented natively in the Julia programming language. NonlinearSolve.jl distinguishes itself by offering a unified API that accommodates a diverse range of solver specifications alongside features such as automatic algorithm selection based on runtime analysis, support for GPU-accelerated computation through static array kernels, and the utilization of sparse automatic differentiation and Jacobian-free Krylov methods for large-scale problem-solving. Through rigorous comparison with established tools such as Sundials and MINPACK, NonlinearSolve.jl demonstrates unparalleled robustness and efficiency, achieving significant advancements in solving benchmark problems and challenging real-world applications. The capabilities of NonlinearSolve.jl unlock new potentials in modeling and simulation across various domains, making it a valuable addition to the computational toolkit of researchers and practitioners alike.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Solving ordinary differential equations II, volume 375. Springer Berlin Heidelberg New York, 1996.
  2. Runge-Kutta Software with Defect Control for Boundary Value ODEs. SIAM J. Sci. Comput., 17:479–497, 1996.
  3. Multiple shooting techniques revisited. In Numerical Treatment of Inverse Problems in Differential and Integral Equations: Proceedings of an International Workshop, Heidelberg, Fed. Rep. of Germany, August 30—September 3, 1982, pages 74–94. Springer, 1983.
  4. Numerical solution of boundary value problems for ordinary differential equations. SIAM, 1995.
  5. Weidong Zhang. Improved implementation of multiple shooting for BVPs. Computer Science Department, University of Toronto, 2012.
  6. Deep Equilibrium Models. arXiv:1909.01377 [cs, stat], October 2019. URL http://arxiv.org/abs/1909.01377. arXiv: 1909.01377.
  7. Multiscale Deep Equilibrium Models. arXiv:2006.08656 [cs, stat], November 2020. URL http://arxiv.org/abs/2006.08656. arXiv: 2006.08656.
  8. Continuous Deep Equilibrium Models: Training Neural ODEs Faster by Integrating Them to Infinity. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pages 1–9, 2023. 10.1109/HPEC58863.2023.10363574.
  9. Avik Pal. On Efficient Training & Inference of Neural Differential Equations. Master’s thesis, Massachusetts Institute of Technology, 2023.
  10. Julia: A fresh approach to numerical computing. SIAM review, 59(1):65–98, 2017. URL https://doi.org/10.1137/141000671.
  11. Modelingtoolkit: A composable graph transformation system for equation-based modeling, 2021.
  12. Enabling new flexibility in the SUNDIALS suite of nonlinear and differential/algebraic equation solvers. ACM Transactions on Mathematical Software (TOMS), 2022. 10.1145/3539801.
  13. SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers. ACM Transactions on Mathematical Software (TOMS), 31(3):363–396, 2005. 10.1145/1089014.1089020.
  14. User guide for MINPACK-1. Technical report, CM-P00068642, 1980.
  15. Frédéric Devernay. C/C++ Minpack. http://devernay.free.fr/hacks/cminpack/, 2007.
  16. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020. 10.1038/s41592-019-0686-2.
  17. The MathWorks Inc. Optimization Toolbox version: 9.4 (R2022b), 2022. URL https://www.mathworks.com.
  18. PETSc users manual. 2019.
  19. Optimistix: modular optimisation in JAX and Equinox. arXiv preprint arXiv:2402.09983, 2024.
  20. Optim: A mathematical optimization package for Julia. Journal of Open Source Software, 3(24):615, 2018. 10.21105/joss.00615.
  21. Catalyst: Fast and flexible modeling of reaction networks. PLOS Computational Biology, 19(10):1–19, 10 2023. 10.1371/journal.pcbi.1011530. URL https://doi.org/10.1371/journal.pcbi.1011530.
  22. Bruno Cordani. The Kepler Problem: Group Theotretical Aspects, Regularization and Quantization, with Application to the Study of Perturbations, volume 29. Springer Science & Business Media, 2003.
  23. Numerical optimization. Springer, 1999.
  24. Nondiscrete induction and a double step secant method. Mathematica Scandinavica, 46(2):236–250, 1980.
  25. Simple yet highly efficient numerical techniques for systems of nonlinear equations. Computational and Applied Mathematics, 42(1):22, 2023.
  26. Philip Wolfe. Convergence conditions for ascent methods. SIAM review, 11(2):226–235, 1969.
  27. Philip Wolfe. Convergence conditions for ascent methods. II: Some corrections. SIAM review, 13(2):185–188, 1971.
  28. Algorithm 851: CG_DESCENT, a conjugate gradient method with guaranteed descent. ACM Transactions on Mathematical Software (TOMS), 32(1):113–137, 2006.
  29. Line search algorithms with guaranteed sufficient decrease. ACM Transactions on Mathematical Software (TOMS), 20(3):286–307, 1994.
  30. Long Hei. A self-adaptive trust region algorithm. Journal of Computational Mathematics, pages 229–236, 2003.
  31. Ya-xiang Yuan. Recent advances in trust region algorithms. Mathematical Programming, 151:249–281, 2015.
  32. Jinyan Fan. Convergence rate of the trust region method for nonlinear equations under local error bound condition. Computational Optimization and Applications, 34(2):215–227, 2006.
  33. A retrospective trust-region method for unconstrained optimization. Mathematical programming, 123:395–418, 2010.
  34. Michael JD Powell. A hybrid method for nonlinear equations. Numerical methods for nonlinear algebraic equations, pages 87–161, 1970.
  35. Avik Pal. Lux: Explicit parameterization of deep neural networks in julia. GitHub repository, 2022.
  36. Adjoint variable method for transient nonlinear electroquasistatic problems. Electrical Engineering, pages 1–7, 2023.
  37. opez, f. pedregosa, and j.-p. vert, efficient and modular implicit differentiation. arXiv preprint arXiv:2105.15183, 2021.
  38. Steven G Johnson. Notes on adjoint methods for 18.335. Introduction to Numerical Methods, 2006.
  39. Forward-Mode Automatic Differentiation in Julia. arXiv:1607.07892 [cs.MS], 2016. URL https://arxiv.org/abs/1607.07892.
  40. Michael Innes. Don’t Unroll Adjoint: Differentiating SSA-Form Programs. CoRR, abs/1810.07951, 2018. URL http://arxiv.org/abs/1810.07951.
  41. Fashionable modelling with flux. arXiv preprint arXiv:1811.01457, 2018.
  42. Computing large sparse Jacobian matrices using automatic differentiation. SIAM Journal on Scientific Computing, 15(2):285–294, 1994.
  43. Generating efficient derivative code with TAF: Adjoint and tangent linear Euler flow around an airfoil. Future generation computer systems, 21(8):1345–1355, 2005.
  44. Getting Started with ADOL-C. Combinatorial scientific computing, 181:202, 2009.
  45. Sparsity Programming: Automated Sparsity-Aware Optimizations in Differentiable Programming. In Program Transformations for ML Workshop at NeurIPS 2019, 2019.
  46. What color is your Jacobian? Graph coloring for computing derivatives. SIAM review, 47(4):629–705, 2005.
  47. SparseDiffTools.jl. JuliaDiff/SparseDiffTools.jl, 2020.
  48. An enhancement of the bisection method average performance preserving minmax optimality. ACM Transactions on Mathematical Software (TOMS), 47(1):1–24, 2020.
  49. Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation, 19(92):577–593, 1965.
  50. Jan Klement. On using quasi-newton algorithms of the Broyden class for model-to-test correlation. Journal of Aerospace Technology and Management, 6:407–414, 2014.
  51. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on scientific and statistical computing, 7(3):856–869, 1986.
  52. Krylov.jl: A Julia basket of hand-picked Krylov methods. Journal of Open Source Software, 8(89):5187, 2023. 10.21105/joss.05187.
  53. Sivan Toledo. Locality of reference in LU decomposition with partial pivoting. SIAM Journal on Matrix Analysis and Applications, 18(4):1065–1081, 1997.
  54. GitHub - JuliaLinearAlgebra/RecursiveFactorization.jl — github.com. https://github.com/JuliaLinearAlgebra/RecursiveFactorization.jl. [Accessed 16-02-2024].
  55. JuliaGPU/KernelAbstractions. jl: v0. 6.0, 2021.
  56. Efficient GPU-Accelerated Global Optimization for Inverse Problems. In ICLR 2024 Workshop on AI4DifferentialEquations In Science, 2024. URL https://openreview.net/forum?id=nD10o1ge97.
  57. Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization. arXiv preprint arXiv:1201.5885, 2012.
  58. Modeling of galvanostatic charge and discharge of the lithium/polymer/insertion cell. Journal of the Electrochemical society, 140(6):1526, 1993.
  59. Composing modeling and simulation with machine learning in julia. In 2022 Annual Modeling and Simulation Conference (ANNSIM), pages 1–17. IEEE, 2022.
  60. Methods—PETLION: Open-source software for millisecond-scale porous electrode theory-based lithium-ion battery simulations. Journal of The Electrochemical Society, 168(9):090504, 2021.
Citations (3)
View on Semantic Scholar

Summary

  • The paper's main contribution is a unified API enabling runtime algorithm selection and robust solutions for complex nonlinear systems.
  • It demonstrates enhanced speed and efficiency through GPU acceleration, sparse automatic differentiation, and Jacobian-free Krylov methods across 23 test problems.
  • The research validates the solver's practical utility in applications like battery simulations, outperforming traditional frameworks such as Sundials and MINPACK.

NonlinearSolve.jl: Advancements in Solving Nonlinear Systems

The paper under discussion introduces NonlinearSolve.jl, an open-source suite of high-performance solvers for systems of nonlinear equations, implemented in the Julia programming language. This suite is characterized by its robust API, capability for runtime algorithm selection, support for GPU-accelerated computation, and utilization of sparse automatic differentiation and Jacobian-free Krylov methods. A thorough evaluation against established tools like Sundials and MINPACK indicates that NonlinearSolve.jl offers significant advantages in both robustness and efficiency.

Core Contributions and Features

NonlinearSolve.jl is designed to address the pervasive challenge of efficiently solving nonlinear equations, which are central to numerous scientific and engineering disciplines. The paper's contributions include:

  • Unified Solver API: A flexible API allows for rapid experimentation with various solver configurations, including line search methods, trust region schemes, and automatic differentiation backends.
  • Algorithm Selection: The framework includes intelligent polyalgorithmic selection that dynamically chooses solver methods and parameters at runtime to optimize for speed and reliability.
  • GPU-Accelerated Computation: Leveraging static array kernels allows for GPU acceleration, significantly enhancing computation speed.
  • Sparse Automatic Differentiation: The solver exploits sparsity patterns in system Jacobians, reducing computational overhead and facilitating the solution of large-scale problems.
  • Jacobian-Free Krylov Methods: Matrix-free solvers are employed to minimize memory usage and accelerate computation, especially for large systems.

Numerical Experiments and Comparisons

The paper demonstrates the solver’s capabilities through rigorous experimentation. NonlinearSolve.jl reliably solves a suite of 23 standard test problems, outperforming Sundials and MINPACK in terms of speed and robustness. Real-world applications, such as initializing DAE models for battery simulations, further validate its practical utility, outperforming traditional frameworks that fail to converge on challenging instances.

Implications and Future Directions

The robustness and scalability of NonlinearSolve.jl mark it as a valuable tool in the computational toolkit for researchers. Its design and architecture not only solve traditional difficulties in scaling nonlinear equations but also open new avenues for modeling and simulation.

The composable framework allows for the development of domain-specific extensions, making it adaptable to a range of applications from chemical network simulations to machine learning models. The paper outlines several such integrations, indicating the potential for future research and development.

Conclusion

NonlinearSolve.jl sets a high standard for solving nonlinear systems through its versatile, efficient approach. It’s a significant addition to the computational resources available for tackling complex system modeling challenges, offering a path forward for researchers needing robust solutions beyond traditional tools. While not the final word on nonlinear solvers, its innovations provide a solid groundwork for future explorations into more complex, computationally intensive problems in science and industry.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown