Scalable higher-order nonlinear solvers via higher-order automatic differentiation (2501.16895v1)

Abstract: This paper demonstrates new methods and implementations of nonlinear solvers with higher-order of convergence, which is achieved by efficiently computing higher-order derivatives. Instead of computing full derivatives, which could be expensive, we compute directional derivatives with Taylor-mode automatic differentiation. We first implement Householder's method with arbitrary order for one variable, and investigate the trade-off between computational cost and convergence order. We find that the second-order variant, i.e., Halley's method, to be the most valuable, and further generalize Halley's method to systems of nonlinear equations and demonstrate that it can scale efficiently to large-scale problems. We further apply Halley's method on solving large-scale ill-conditioned nonlinear problems, as well as solving nonlinear equations inside stiff ODE solvers, and demonstrate that it could outperform Newton's method.

• The paper introduces scalable higher-order nonlinear solvers that leverage Taylor-mode AD to compute only necessary higher-order directional derivatives, reducing computational expense.
• It implements and analyzes Householder's and Halley's methods for both single-variable and multivariable systems, demonstrating improved convergence over traditional Newton's method.
• Numerical experiments reveal that Halley's method significantly enhances performance on dense, sparse, and stiff ODE problems, offering 5-10% efficiency gains.

Scalable Higher-Order Nonlinear Solvers via Higher-Order Automatic Differentiation

The paper by S. Tan, K. Miao, A. Edelman, and C. Rackauckas introduces advanced methods for solving nonlinear equations using scalable higher-order nonlinear solvers enabled by higher-order automatic differentiation (AD). By focusing on efficiently computing higher-order derivatives, the authors tackle the issues of computational expense commonly associated with traditional methods for nonlinear problem-solving.

Context and Motivation

Solving nonlinear equations is pivotal in multiple domains within mathematics and the sciences, where these equations often arise directly or as intermediates in solving differential equations. Traditional approaches like Newton's method are predominantly used due to their well-established quadratic convergence properties. However, these methods are limited by the reliance on first-order derivatives, prompting research into higher-order solvers that harness derivative information beyond the first order.

Contributions and Methodology

The key contribution of the paper is the implementation of higher-order nonlinear solvers, specifically through methods like Householder's and its derivatives, including Halley's method, utilizing Taylor-mode AD for efficient computation of directional derivatives. Traditional approaches that require full computation of these higher-order derivatives are often computationally prohibitive, especially for large-scale problems. The authors circumvent this by employing efficient AD techniques to compute only the necessary higher-order directional derivatives, developed within the Julia programming language.

Householder's and Halley's Methods

The authors first detail the implementation of Householder's method for single-variable equations and extend the analysis to multivariable systems using Halley's method. Householder's method is a general framework allowing arbitrary convergence orders dependent on the computation of derivatives up to a certain order, providing a flexible balance between computational cost and convergence rate.

Their findings indicate that a second-order variant, Halley's method, yields the most substantial benefits, particularly when generalized to large systems. The method's multi-step nature and reliance on directional derivatives tackle the computational burdens and facilitate efficient scaling up of the nonlinear solvers.

Numerical Experiments and Results

Through a series of numerical experiments, the authors demonstrate that Halley's method can surpass traditional Newton's method in efficiency, particularly in large-scale and ill-conditioned problems often observed in solving PDEs and stiff ODEs. The experiments encompass various scenarios:

• Dense Jacobian Problems: Here, Halley's method retains performance advantages by reducing iteration counts, despite the dense nature of computations.
• Sparse Jacobian Problems: When applied to large-scale ill-conditioned problems, Halley's method, compared to its naive full Hessian counterpart, achieves improved scalability.
• Stiff ODEs: Within implicit solvers for stiff ODEs, the integration of Halley's method exhibits a notable 5-10% increase in efficiency over Newton's method.

Implications and Future Directions

The implications of this research are twofold: practically, for the efficient resolution of nonlinear equations in scientific computing environments, and theoretically, for expanding the toolbox of higher-order AD techniques in numerical analysis. By ushering in scalable, efficient higher-order methods, this work enables further advancement in solving large and complex nonlinear systems, foundational in fields ranging from astrophysics to chemical modeling.

Future research should explore nonlinear problems necessitating iterative solutions for linear subproblems, such as those involving matrix-free operators. This extension could widen the applicability of Halley's method and similar approaches to broader classes of nonlinear problems with enhanced iterative solver strategies.

In conclusion, the authors provide a comprehensive framework for utilizing higher-order AD to empower nonlinear solvers, bridging the gap between theoretical and practical computations for complex scientific challenges.