A Regularized Hessian-Free Inexact Newton-Type Method with Global $\mathcal{O}(k^{-2})$ Convergence

Abstract: We propose a regularized Hessian-free Newton-type method for minimizing smooth convex functions with Lipschitz continuous Hessians. The algorithm constructs an approximate Hessian by finite differences and selects the regularization parameter through an adaptive criterion that ensures sufficient decrease and gradient control. We prove that the method achieves a global $\mathcal{O}(k^{-2})$ convergence rate, matching the best known bound for second-order methods. A modified variant incorporating the exact Hessian when available enjoys local quadratic convergence under standard assumptions. Despite its simplicity, this variant is computationally faster than the \emph{Regularized Newton Method} of Mishchenko (2023) across several convex benchmark problems. Our analysis also provides explicit bounds on the regularization sequence and a worst-case iteration complexity of order $\mathcal{O}(\varepsilon^{-2})$. The proposed framework thus unifies regularized and Hessian-free Newton-type schemes, offering a theoretically sound and practically efficient alternative for smooth convex optimization.