On Second-Order Methods for Bilevel Optimization

Abstract: Bilevel optimization is an indispensable modeling tool for modern machine learning and engineering design. However, the theory and practice for finding second order stationary points in the context of bilevel optimization still remain largely unsettled. Even for bilevel optimization with strongly convex lower-level problem, the hyperfunction it induces is in general nonconvex. Although the Cubic Regularized Newton methods (CRN) famously achieve the optimal $\mathcal{O}(\varepsilon<sup>{-1.5})$ SOSP (second-order stationary point) rate in single-level optimization, it is unclear how to control the accuracy of the hypergradient and hyper-Hessian computations in the context of applying the second-order methods to bilevel problems in order for the overall process to be efficient. In this paper, we set out to answer this question. In particular, we first formulate a double loop CRN baseline that achieves the optimal outer rate but requires repeated lower level solves. Next, we propose a single loop cubic regularized Newton algorithm that combines one lower-level gradient step with one Newton step for the hypergradient, and prove an overall deterministic $\mathcal{O}(\varepsilon<sup>{-1.5})$ total oracle complexity, which is optimal. In addition, we illustrate that some intuitively simple modifications of our method may fail to hold up the convergence result. To the best of our knowledge, this is the first deterministic single loop method for unconstrained NCSC (non-convex upper-level and strongly convex lower-level) bilevel optimization setting that achieves the $\mathcal{O}(\varepsilon<sup>{-1.5})$ optimal convergence rate for finding an $\varepsilon$-SOSP of the hyperfunction.

• The paper introduces novel deterministic second-order algorithms (DLCRN and SLCRN) that achieve optimal O(ε⁻¹.⁵) convergence in unconstrained nonconvex-strongly-convex bilevel optimization.
• It leverages Approximate Implicit Differentiation to accurately estimate hypergradients and hyper-Hessians, balancing inner-loop efficiency with convergence precision.
• Empirical results demonstrate that SLCRN outperforms existing methods in escaping saddle points and converging rapidly on high-dimensional tests.

On Second-Order Methods for Bilevel Optimization

Introduction and Theoretical Context

Bilevel optimization (BLO) is central to hierarchical decision-making models and widely spans applications in machine learning, computational engineering, and operations research. The unconstrained nonconvex-strongly-convex (NCSC) bilevel problem, characterized by a nonconvex upper-level objective and a strongly convex lower-level problem, has become the canonical testbed for theoretical and algorithmic developments. A fundamental challenge is that even under strong convexity in the lower-level objective, the resulting hyperfunction $\Phi(x) = f(x, y^*(x))$ remains nonconvex, and obtaining second-order stationary points (SOSPs) efficiently remains unresolved.

Emergent literature on BLO has largely focused on first-order methods, which provide high-probability guarantees but settle for suboptimal SOSP convergence rates, notably $\mathcal{O}(\varepsilon^{-2}\log\varepsilon^{-1})$ or $$\mathcal{O}(\varepsilon^{-1.75}\log\varepsilon^{-1})$$ [huang2025efficiently, yang2023accelerating]. Second-order approaches—specifically, Cubic Regularized Newton (CRN) methods—are optimal in traditional nonconvex single-level optimization, deterministically achieving $\mathcal{O}(\varepsilon^{-1.5})$ SOSP rates [nesterov2006cubic]. However, adapting CRN to BLO is nontrivial due to the implicit nature of hypergradients and hyper-Hessians, which depend on the solutions of the lower-level problem and require careful approximations.

Methodological Advances: DLCRN and SLCRN Algorithms

The paper introduces two deterministic second-order algorithms for unconstrained NCSC bilevel problems: Double Loop Cubic Regularized Newton (DLCRN) and Single Loop Cubic Regularized Newton (SLCRN). Both leverage the Approximate Implicit Differentiation (AID) framework to estimate hypergradients and hyper-Hessians but differ fundamentally in their inner-loop structure and complexity.

DLCRN follows a classical structure, solving the lower-level problem to a stringent accuracy at each outer iteration. This guarantees hypergradients and hyper-Hessians meet inexact CRN requirements, ensuring optimal $\mathcal{O}(\varepsilon^{-1.5})$ outer convergence, but incurs an additional $$\mathcal{O}(\varepsilon^{-1.5}\log\varepsilon^{-1})$$ inner complexity due to repeated solves.

SLCRN distinguishes itself by requiring only two lower-level updates per outer iteration: a gradient step and a Newton step. The gradient step contracts the tracking error, promoting stability, while the Newton step—used only for hypergradient computation—achieves the higher order accuracy needed for the inexact CRN regime. Crucially, the Newton point is not used for propagation to avoid divergence that is possible with non-globally stable Newton steps. SLCRN thus achieves an overall deterministic $\mathcal{O}(\varepsilon^{-1.5})$ oracle complexity for unconstrained NCSC BLO, matching the theoretical lower bound and surpassing prior methods both in simplicity and efficiency.

Structural Justification and Algorithmic Insights

The architecture of SLCRN is a direct response to the discrepancy in accuracy demands between hypergradient and hyper-Hessian estimation in inexact CRN. Gradient descent alone provides only linear tracking accuracy, insufficient for preserving optimal CRN convergence. By incorporating a Newton refinement for hypergradient evaluation, SLCRN attains local quadratic contraction—essential for the required error scaling.

Variants that omit the Newton step or propagate it as the next iterate are provably and empirically unstable. The Newton step is only valid for localization, not propagation, due to its lack of global convergence guarantees in strongly convex settings.

Figure 1: $f(x_k, y_k)$ (left) and $g(x_k, y_k)$ (right) versus the outer-loop iteration, comparing SLCRN and its Newton-update variant.

Figure 2: Lower-level updates of SLCRN and the Newton-update variant; the Newton-update causes a significant tracking error in the next iteration.

These figures reinforce SLCRN's structural insights, showing rapid convergence with the proposed tracking scheme and divergence when the Newton step is misapplied.

Numerical Results and Empirical Performance

Extensive experiments validate SLCRN against benchmarks including AID-BiO, PBO, iNEON, and PRAHGD on high-dimensional nonconvex-strongly-convex test functions. SLCRN outperforms competitors in both escaping saddle points and locating SOSPs efficiently.

Figure 3: Comparison between SLCRN, PBO, and BO under multiple initializations; SLCRN achieves faster decrease in the function value gap.

Empirical evidence strongly supports the theoretical claims: SLCRN is superior in both computational efficiency and robustness of convergence, requiring minimal parameter tuning and avoiding the practical bottlenecks of nested inner solves.

Practical Implications and Theoretical Significance

SLCRN's deterministic optimal-rate convergence opens new avenues in BLO for applications demanding reliable, rapid escape from saddle points. Notable domains include bilevel hyperparameter optimization, meta-learning, reinforcement learning, and large-scale engineering design. The algorithm's stability and simplicity position it well for real-time and large-scale settings, where repeated inner solves are infeasible.

Fundamentally, the work advances the deterministic theory of second-order optimization in hierarchical settings, establishing that the optimal $\mathcal{O}(\varepsilon^{-1.5})$ rate is attainable for the hyperfunction without resorting to stochastic perturbations or nested solvers. The approach is likely extendable to constrained BLO, minimax, and structured machine learning problems, particularly as higher-order algorithms gain traction in modern nonconvex optimization.

Future Directions

The algorithmic insights of SLCRN suggest several avenues for future development:

• Extension to constrained BLO settings, possibly incorporating barrier or penalty methods for lower-level constraints.
• Febrile adaptation for stochastic and large-scale environments, leveraging variance reduction or Krylov subspace solvers as in single-level CRN variants.
• Application to multi-level or distributed optimization paradigms beyond the traditional bilevel structure.

In the theoretical domain, further analysis of global convergence properties under relaxed assumptions (e.g., PL or Morse conditions in the lower level) could broaden the applicability of second-order deterministic methods in BLO.

Conclusion

This paper provides a rigorous and practical framework for achieving deterministic optimal-rate second-order convergence in unconstrained NCSC bilevel optimization. The introduction of SLCRN—a single loop, efficiently tunable algorithm—addresses longstanding algorithmic bottlenecks and advances both theory and practice in hierarchical nonconvex optimization, laying the groundwork for future research and deployment in diverse machine learning and engineering applications.