Adaptive Newton-CG methods with global and local analysis for unconstrained optimization with Hölder continuous Hessian

Abstract: In this paper, we study Newton-conjugate gradient (Newton-CG) methods for minimizing a nonconvex function $f$ whose Hessian is $(H_f,ν)$-Hölder continuous with modulus $H_f&gt;0$ and exponent $ν\in(0,1]$. Recently proposed Newton-CG methods for this problem adopt (i) non-adaptive regularization and (ii) a nested line-search procedure, where (i) often leads to inefficient early progress and the loss of local superlinear convergence, and (ii) may incur high computational cost due to multiple solves of the Newton system per iteration. To address these limitations, we propose two novel Newton-CG algorithms, depending on the availability of $ν$, that adaptively regularize the Newton system by leveraging the auto-conditioning technique to eliminate the nested line search. The proposed algorithms achieve the best-known iteration complexity $${\mathcal O}\big(H_f<sup>{1/(1+ν)}ε<sup>{-(2+ν)/(1+ν)}\big)$$ for finding an $ε$-stationary point and, simultaneously, enjoy local superlinear convergence near nondegenerate local minimizers. Numerical experiments further demonstrate the practical advantages of our algorithms over existing approaches.

• The paper demonstrates adaptive damping through gradient norm scaling to achieve both optimal global complexity and local superlinear convergence.
• It avoids costly line-search procedures by employing a capped CG solver for damped Newton systems, reducing computational overhead.
• Numerical experiments show significant speedups over competing methods in high-dimensional nonconvex problems.

Adaptive Newton-CG Methods for Unconstrained Optimization with Hölder Continuous Hessian

Problem Setting and Motivation

The paper addresses unconstrained nonconvex optimization problems of the form $\min_{x\in\mathbb{R}^n} f(x)$, where $f$ is twice continuously differentiable and its Hessian satisfies a Hölder continuity condition: for some $H_f>0$ and $\nu\in(0,1]$, $\|\nabla^2 f(y)-\nabla^2 f(x)\|\leq H_f\|y-x\|^\nu$ for all $x, y$ in a suitable set. The Lipschitz (i.e., $\nu=1$) and Hölder ($\nu<1$) regimes are both considered, although much of the literature has focused on the former.

Classical Newton-type and cubic regularization methods achieve strong global and local convergence properties under the Lipschitz Hessian assumption, but with significant implementation cost—particularly requiring repeated solution of cubic subproblems. Levenberg-Marquardt-type (quadratic) regularization and Newton-CG procedures offer a reduction in per-iteration cost by solving only damped linear systems. However, existing Newton-CG variants typically use non-adaptive regularization and/or costly line searches to tune the regularization parameter, precluding efficient global behavior and local superlinear convergence, respectively.

In the Hölder-Hessian regime ($\nu<1$), optimal complexity is only achieved by methods that require solving higher-order or inexact regularized subproblems [grapiglia2017regularized, cartis2019universal, cartis2020sharp], or by Newton-CG frameworks with non-adaptive (fixed in terms of $\epsilon$) damping schemes [he2025newton]. However, these approaches do not reconcile the trade-off between global complexity and local (fast) convergence.

Algorithmic Contributions

Two adaptive Newton-CG algorithms are developed, distinguished by whether the exponent $f$0 is known:

1. Algorithm with Known $f$1: The method sets the damping parameter adaptively based on the gradient norm: $f$2. At each iteration, the damping magnitude is determined via an auto-conditioning mechanism, inspired by recent advances in parameter-free first-order optimization [lan2024projected, li2025simple]. The algorithm employs a non-nested line-search, together with a capped CG subroutine for solving the (possibly indefinite) damped Newton system.
2. Universal Algorithm (Unknown $f$3): Here, the algorithm proceeds as if the Hessian were Lipschitz continuous, using $f$4 and adaptively estimating the local regularization parameter $f$5. This approach treats Hölder continuity as "approximately" Lipschitz, introducing a controlled modeling error. The auto-conditioning is carefully modified so that $f$6 does not become unbounded as iterates approach stationary points.

In both cases, the capped CG method is invoked once per iteration, distinguishing between solution and negative curvature directions.

Key distinguishing features:

• Regularization is adaptive—linked to the local geometry through gradient norm scaling.
• Line search over regularization parameters is eliminated, which mitigates inner/outer loop cost.
• The methods are global (no restrictive initialization), and do not require knowledge of $f$7 (the Hölder constant).
• Both local superlinear convergence near nondegenerate minimizers and optimal global complexity are obtained.

Complexity and Convergence Guarantees

Global Complexity:

For both algorithms, the iteration complexity to obtain an $f$8-stationary point (i.e., $f$9) is

$H_f>0$0

This bound is optimal with respect to known lower bounds for second-order methods under Hölder smoothness [cartis2018worst], and matches the best known results for cubic- and higher-order regularized approaches [grapiglia2017regularized, he2025newton].

Local Convergence:

When the objective possesses a non-degenerate local minimizer with $H_f>0$1, both algorithms exhibit local superlinear convergence:

• Known $H_f>0$2: The sequence satisfies $H_f>0$3.
• Universal (unknown $H_f>0$4): A "capture region" analysis ensures that, after entering a suitable neighborhood, all future iterates remain there with a fixed (bounded) regularization parameter, and the rate is $H_f>0$5 in the exponent, i.e., $H_f>0$6.

Notably, the achievement of simultaneous global optimality and local superlinear convergence with only quadratic-regularized (as opposed to cubic or higher-order) subproblems, and in a parameter-free, line-search-free Newton-CG framework, is absent in prior work.

Numerical Results

Extensive numerical experiments are conducted on two families of problems:

• Infeasibility detection models with polynomial loss, which are prototypical nonconvex, nonsmooth optimization benchmarks.
• Single-layer (RePU) neural network training with superquadratic ($H_f>0$7) activation.

The adaptive Newton-CG method (ANCG) consistently outperforms both the non-adaptive parameter-free Newton-CG (HNCG) [he2025newton] and adaptive cubic regularization (ACRN) [grapiglia2017regularized] in wall-clock time, number of linear/cubic subproblems solved, and Hessian-vector products. The advantage increases with problem dimension and nonlinearity. ANCG typically requires less than half the subproblem solves and Hessian-vector products compared to the closest competitor.

Implications and Future Directions

Practical implications:

• The results eliminate the need for computationally costly line search over regularization parameters in Newton-CG approaches, preserving both efficiency and convergence guarantees.
• The framework is attractive for large-scale nonconvex optimization, including machine learning contexts (e.g., deep learning, matrix factorization) where higher-order regularization is expensive, and $H_f>0$8, $H_f>0$9 are unknown or difficult to estimate.
• The universal method bridges the theory-practice gap for second-order nonconvex optimization with only black-box access to gradients and Hessian-vector products.

Theoretical implications:

• The equivalence, in terms of complexity and local rate, between adaptive quadratic-regularized Newton-CG and higher-order regularization frameworks in the Hölder regime, challenges the prevailing notion that higher-order subproblem solvers are necessary for optimality.
• The adaptation of auto-conditioning and parameter-free regularization mechanisms from first- to second-order optimization suggests a promising paradigm for further "universally optimal" algorithms, possibly extending to equality constraint problems, stochastic settings, or weakly convex/non-smooth settings.

Open directions:

• Generalization to structured nonconvex problems (e.g., saddle point, min-max, or conic constrained problems) with only weak or local curvature information.
• Incorporating negative curvature exploitation for certified second-order stationary points, beyond (first-order) $\nu\in(0,1]$0-stationarity.
• Further integration with stochastic and distributed computation frameworks, leveraging Hessian-free approaches.

Conclusion

This work introduces adaptive, line-search-free Newton-CG methods for nonconvex problems with Hölder continuous Hessians. The methods provably achieve optimal global iteration complexity and local superlinear convergence, without requiring any tuning of problem-dependent parameters. Empirical studies demonstrate superior efficiency over both cubic-regularized and existing Newton-CG methods. These results mark a significant refinement in the design and analysis of practical second-order, large-scale nonconvex optimization algorithms.

---

References

Key references include [he2025newton], [grapiglia2017regularized], [cartis2018worst], [lan2024projected], [li2025simple], and [royer2020newton]; see the preprint for the full list.