Faithful-Newton Framework: Bridging Inner and Outer Solvers for Enhanced Optimization

Abstract: While Newton-type methods are known for their fast local convergence and strong empirical performance, achieving theoretically favorable global convergence compared to first-order methods remains a key challenge. For instance, surprisingly, for simple strongly convex problems, no straightforward variant of Newton's method matches the global complexity of gradient descent. Although sophisticated variants can improve iteration complexity over gradient descent for various problems, they often involve highly nontrivial subproblems that incur significantly higher per-iteration costs, resulting in a much worse overall operation complexity. These limitations arise from treating the subproblem as an afterthought, either by handling it as a black box, thus permitting complex, highly nontrivial, and nearly unimplementable formulations, or by evaluating subproblem solutions in isolation, without considering their effectiveness in advancing the optimization of the main objective. By tightening the integration between the inner iterations of the subproblem solver and the outer iterations of the optimization algorithm, we introduce simple Newton-type variants, called Faithful-Newton methods, which, in a sense, remain faithful to the overall simplicity of classical Newton's method by retaining simple linear system subproblems. The key conceptual difference, however, is that the quality of the subproblem solution is directly assessed based on its effectiveness in reducing optimality, which in turn enables desirable convergence complexities across a variety of settings. Under standard assumptions, we show that our variants match the worst-case complexity of gradient descent in strongly convex settings, both in iteration and operation complexity, and achieve competitive iteration complexity for general convex problems.