Second-Order Bilevel Optimization with Accelerated Convergence Rates

Abstract: This paper studies second-order methods for nonconvex-strongly-convex bilevel optimization. We propose a novel fully second-order bilevel approximation method (FSBA) that achieves an iteration complexity of $\tilde{\mathcal{O}}(ε<sup>{-1.5})$ for finding the $(ε, \mathcal{O}(\sqrtε))$ second-order stationary point of the hyper-objective function. Our results demonstrate that second-order methods can achieve an accelerated convergence rate than first-order methods in bilevel optimization. To address the heavy computational cost associated with the second-order oracle, we introduce a lazy variant of FSBA, called LFSBA, which reuses second-order information across several iterations. We prove that LFSBA exhibits better computational complexity than FSBA by a factor of $\sqrt{d}$, where $d$ is the dimension of the problem. We also apply a similar idea to nonconvex strongly-concave minimax optimization and propose the lazy minimax cubic-regularized Newton (LMCN) method with better computational complexity compared to existing second-order methods.

• The paper introduces FSBA, an inexact cubic-regularized Newton method that finds an approximately second-order stationary point in \tilde{O}(\epsilon^{-1.5}) oracle calls, improving over prior bilevel methods.
• The method optimizes a penalty-based surrogate using warm-started accelerated inner solves, avoiding direct third-derivative computation while controlling gradient and Hessian estimation errors.
• The lazy variant LFSBA periodically refreshes Hessians to reduce high-dimensional computational cost, achieving a dimension-aware improvement while trading off curvature tolerance through a tunable refresh interval.

Problem setting and motivation

The paper studies the nonconvex–strongly-convex bilevel problem $\min_x \varphi(x) := f(x, y^*(x))$ where $y^*(x) = \arg\min_y g(x,y)$, with $g$ strongly convex in $y$. The central observation motivating the work is that existing second-order methods for this problem (AID, ITD, RAHGD, iNEON, etc.) achieve iteration complexities no better than first-order algorithms for nonconvex minimization — typically $\tilde{O}(\epsilon^{-2})$ or, with Hessian-Lipschitz hyper-objective, $\tilde{O}(\epsilon^{-1.75})$ via perturbed accelerated gradient descent. Meanwhile, fully first-order Lagrangian-penalty methods such as F$^2$BA and its accelerated variants already match these rates using only gradient oracles. The authors thus pose a direct question: can second-order oracles yield an improved rate at all?

Their answer is affirmative: they develop FSBA, achieving $\tilde{O}(\epsilon^{-1.5})$ oracle complexity for finding an $(\epsilon, O(\sqrt{\epsilon}))$ second-order stationary point (SOSP) of the hyper-objective $\varphi$, and argue this is near-optimal because nonconvex strongly-concave minimax problems subsume nonconvex minimization, for which Carmon et al.'s lower bound is $y^*(x) = \arg\min_y g(x,y)$0 (2000.02179). This is the paper's strongest claim: second-order information provides a genuine acceleration in bilevel optimization, closing the gap to the known lower bound.

The FSBA method

Rather than differentiating the implicit function $y^*(x) = \arg\min_y g(x,y)$1 directly — which would require third derivatives of $y^*(x) = \arg\min_y g(x,y)$2 to obtain $y^*(x) = \arg\min_y g(x,y)$3 — the algorithm targets the penalty surrogate

$y^*(x) = \arg\min_y g(x,y)$4

For $y^*(x) = \arg\min_y g(x,y)$5, standard results establish that $y^*(x) = \arg\min_y g(x,y)$6 uniformly approximates $y^*(x) = \arg\min_y g(x,y)$7 in value, gradient ($y^*(x) = \arg\min_y g(x,y)$8), and Hessian ($y^*(x) = \arg\min_y g(x,y)$9), that $g$0 and $g$1 are Lipschitz with constants $g$2 and $g$3, and that $g$4 remains strongly convex with condition number $g$5. Hence an SOSP of the surrogate translates into an SOSP of $g$6 after choosing $g$7 appropriately.

Both $g$8 and $g$9 admit estimators depending only on two approximate solutions: $y$0 (via AGD on $y$1) and $y$2 (via AGD on $y$3). A key lemma bounds the estimator errors linearly in $y$4 and $y$5, so the required accuracies $y$6 on the gradient and $y$7 on the Hessian are achieved by warm-started accelerated gradient descent with carefully scheduled inner iteration counts $y$8.

FSBA then performs inexact cubic-regularized Newton steps on the surrogate. Combining the inexact-CRN descent analysis of Luo et al. with the inner-loop error control yields the main guarantee: with $y$9 iterations, the output is an $\tilde{O}(\epsilon^{-2})$0-SOSP of $\tilde{O}(\epsilon^{-2})$1, using $\tilde{O}(\epsilon^{-2})$2 first-order and $\tilde{O}(\epsilon^{-2})$3 second-order oracle calls. Notably, the warm-started AGD inner loops contribute only $\tilde{O}(\epsilon^{-2})$4 per outer iteration in aggregate, thanks to a summation argument combining AM–GM with the CRN step-size bound $\tilde{O}(\epsilon^{-2})$5; without warm starting, inner-loop costs would inflate the overall dependence on $\tilde{O}(\epsilon^{-2})$6.

An inexact variant, IFSBA, avoids forming Hessians explicitly: matrix Chebyshev polynomial approximations replace the inverses of $\tilde{O}(\epsilon^{-2})$7 and $\tilde{O}(\epsilon^{-2})$8 (requiring only HVPs, of order $\tilde{O}(\epsilon^{-2})$9 per use), and the cubic subproblem is solved by a randomized gradient-type method with high-probability guarantees. IFSBA retains the $\tilde{O}(\epsilon^{-1.75})$0 iteration complexity with probability at least $\tilde{O}(\epsilon^{-1.75})$1, at the cost of $\tilde{O}(\epsilon^{-1.75})$2 HVP calls.

Lazy Hessians and LFSBA

Since a second-order oracle costs roughly $\tilde{O}(\epsilon^{-1.75})$3 versus $\tilde{O}(\epsilon^{-1.75})$4 for gradients (Assumption on oracle costing following Doikov et al.), calling it every iteration dominates the total cost $\tilde{O}(\epsilon^{-1.75})$5 when $\tilde{O}(\epsilon^{-1.75})$6 is large. LFSBA computes the approximate Hessian only at snapshot points every $\tilde{O}(\epsilon^{-1.75})$7 iterations and reuses it, generalizing lazy-Hessian cubic regularization (Doikov et al., 2023) to the bilevel setting.

The analysis must absorb an additional drift term $\tilde{O}(\epsilon^{-1.75})$8 from Hessian staleness. The descent lemma acquires a negative correction proportional to $\tilde{O}(\epsilon^{-1.75})$9; telescoping over blocks of length $^2$0 and requiring $^2$1 cancels the accumulated drift via a triple-sum inequality. The resulting guarantee degrades gracefully: LFSBA finds an $^2$2-SOSP with iteration complexity $^2$3 but second-order oracle calls reduced to $^2$4.

Balancing the two terms via AM–GM with $^2$5 gives total computational complexity

$^2$6

an improvement over FSBA by a factor of $^2$7 — significant precisely in the high-dimensional regime where explicit Hessians are expensive. The same lazy strategy applied to nonconvex strongly-concave minimax optimization yields LMCN, which improves on the MCN method of Luo et al. under identical assumptions, achieving $^2$8 computational complexity for an $^2$9-SOSP.

A caveat worth noting: the SOSP tolerance itself degrades with $\tilde{O}(\epsilon^{-1.5})$0 (the $\tilde{O}(\epsilon^{-1.5})$1 factor in the curvature threshold), so laziness trades second-order accuracy for cheaper iterations rather than being free; the tuning of $\tilde{O}(\epsilon^{-1.5})$2 is a genuine trade-off, which the ablation studies confirm empirically.

Experiments

Experiments cover four tasks: (i) a synthetic nonconvex-strongly-concave minimax problem, where LMCN reaches low optimality gaps in fewer oracle calls than PRAGDA, MCN, iMCN, and GDA from both tested initializations; (ii) data hyper-cleaning on breast-cancer, australian, and MNIST at noise rates $\tilde{O}(\epsilon^{-1.5})$3, where FSBA/LFSBA converge faster than ITD, AID-CG, and F$\tilde{O}(\epsilon^{-1.5})$4BA; (iii) hyperparameter tuning for logistic regression on 20 Newsgroups ($\tilde{O}(\epsilon^{-1.5})$5, $\tilde{O}(\epsilon^{-1.5})$6), where IFSBA outperforms baselines in validation loss over wall-clock time; and (iv) 5-way 5-shot few-shot meta-learning on miniImageNet and FC100 against PZOBO and qNBO, where IFSBA attains higher test accuracy within equal running time. Ablations show moderate values of $\tilde{O}(\epsilon^{-1.5})$7 minimize runtime, consistent with the theoretical trade-off, and that performance is robust across wide ranges of the penalty $\tilde{O}(\epsilon^{-1.5})$8 and cubic parameter $\tilde{O}(\epsilon^{-1.5})$9, which are treated as tunable rather than set from worst-case constants.

Limitations and open questions

The paper's framework rests on several assumptions that bound its scope. It requires deterministic exact gradient and Hessian oracles of both $(\epsilon, O(\sqrt{\epsilon}))$0 and $(\epsilon, O(\sqrt{\epsilon}))$1; stochastic and distributed extensions are left open, as is any setting lacking lower-level strong convexity (e.g., PL-based or value-function approaches). The third-derivative Lipschitz assumption on $(\epsilon, O(\sqrt{\epsilon}))$2 (Assumption 1e) is needed for the surrogate analysis and is stronger than what purely first-order methods require. The near-optimality claim inherits the reduction from minimax to nonconvex minimization; whether the specific polynomial dependences on $(\epsilon, O(\sqrt{\epsilon}))$3 (up to $(\epsilon, O(\sqrt{\epsilon}))$4 in the leading term) can be tightened, e.g., by replacing the AGD inner solvers with higher-order methods as the authors suggest, remains unresolved. Finally, the computational-complexity comparison depends on the oracle-costing model in which a Hessian costs exactly $(\epsilon, O(\sqrt{\epsilon}))$5; other accounting regimes would shift the optimal choice of $(\epsilon, O(\sqrt{\epsilon}))$6.

Conclusion

This paper establishes that second-order oracles do confer an acceleration in nonconvex–strongly-convex bilevel optimization: FSBA achieves $(\epsilon, O(\sqrt{\epsilon}))$7 oracle complexity for finding an $(\epsilon, O(\sqrt{\epsilon}))$8-SOSP of the hyper-objective, matching the lower-bound order for the subsuming minimax class and improving on all prior first- and second-order bilevel methods. The lazy variant LFSBA converts this iteration advantage into a dimension-aware computational advantage via periodic Hessian reuse, and the same technique improves the state of the art for second-order minimax methods. The empirical results across hyper-cleaning, hyperparameter tuning, and meta-learning support the practical viability of the approach within its stated assumptions.