A Homogeneous Second-Order Descent Ascent Algorithm for Nonconvex-Strongly Concave Minimax Problems

Abstract: This paper introduces a novel Homogeneous Second-order Descent Ascent (HSDA) algorithm for nonconvex-strongly concave minimax optimization problems. At each iteration, HSDA uniquely computes a search direction by solving a homogenized eigenvalue subproblem built from the gradient and Hessian of the objective function. This formulation guarantees a descent direction with sufficient negative curvature even in near-positive-semidefinite Hessian regimes--a key feature that enhances escape from saddle points. We prove that HSDA finds an $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$-second-order stationary point within $\tilde{\mathcal{O}}(\varepsilon<sup>{-3/2})$ iterations, matching the optimal $\varepsilon$-order iteration complexity among second-order methods for this problem class. To address large-scale applications, we further design an inexact variant (IHSDA) that preserves the single-loop structure while solving the subproblem approximately via a Lanczos procedure. With high probability, IHSDA achieves the same $\tilde{\mathcal{O}}(\varepsilon<sup>{-3/2})$ iteration complexity and attains an $\mathcal{O}(\varepsilon, \sqrt{\varepsilon})$-second-order stationary point, with the total Hessian-vector product cost bounded by $\tilde{\mathcal{O}}(\varepsilon<sup>{-7/4})$. Experiments on synthetic minimax problems and adversarial training tasks confirm the practical effectiveness and robustness of the proposed algorithms.

• The paper proposes the HSDA and its inexact variant IHSDA algorithms for solving nonconvex-strongly concave minimax problems, demonstrating $ ilde{\mathcal{O}(\varepsilon^{-3/2})$ iteration complexity.
• The exact HSDA uses a homogenized eigenvalue direction computation that avoids solving multiple subproblems, in contrast to other second-order methods, while the inexact variant IHSDA employs a Lanczos-based method with efficiently computable accuracy.
• Theoretical and numerical results highlight the superiority of HSDA over competing methods, especially in escaping strict saddle points and handling adversarial training tasks with comparable or better convergence to datasets like MNIST in the setting of a neural network logloss approach.

Overview and problem setting

This paper addresses unconstrained nonconvex-strongly concave minimax problems of the form $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$, where $f$ is strongly concave in $y$ but possibly nonconvex in $x$. The value function is $\mathcal{F}(x):=\max_y f(x,y)$. The authors propose the Homogeneous Second-order Descent Ascent (HSDA) algorithm, together with an inexact Lanczos-based variant (IHSDA), and establish iteration complexity guarantees matching the best known rates for second-order methods on this problem class (2602.14058).

The motivation stems from applications such as adversarial training, distributionally robust optimization, domain adaptation, and AUC maximization. While first-order methods achieve $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$ iterations for an $\varepsilon$-first-order stationary point—with acceleration improving this to $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$—second-order methods can additionally certify curvature and escape strict saddle points. Existing second-order approaches split into cubic-regularization methods (MCN of Luo et al., ICLM of Chen et al.) and trust-region methods (MINIMAX-TR/TRACE of Yao and Xu; GRTR and LMNegCur of Wang and Xu). All of these attain $\mathcal{O}(\varepsilon^{-3/2})$ iteration complexity to reach an $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$-second-order stationary point.

Algorithmic design

HSDA adapts the Homogeneous Second-order Descent Method (HSODM) framework—originally developed for unconstrained nonconvex minimization—to the minimax setting via two components per outer iteration:

Inexact inner maximization. Given $f$0, Nesterov's accelerated gradient ascent with $f$1 steps produces an approximate maximizer $f$2, from which the inexact gradient $f$3 and Schur-complement Hessian approximation $f$4 are formed. The inner step counts are chosen so that both $f$5 and the induced errors $f$6, $f$7 are controlled.

Homogenized eigenvalue direction computation. The search direction solves

$f$8

an $f$9-dimensional extremal-eigenvalue problem whose solution is a unit eigenvector of $y$0 associated with its smallest eigenvalue. If $y$1, the step is $y$2; otherwise it falls back to $y$3. The outer update rescales via $y$4.

A structural advantage over IGRTR/ILMNegCur is emphasized: those methods must solve both a regularized Newton system and an $y$5-dimensional extremal eigenproblem per iteration, whereas HSDA solves only one homogenized eigenvalue problem in a lifted space.

Complexity analysis for exact HSDA

Under strong concavity of $y$6 in $y$7, Lipschitz continuity of the gradient and Hessian blocks, and boundedness below of $y$8, standard results give that $y$9 and $x$0 are Lipschitz continuous, with $x$1 and $x$2.

The analysis proceeds by splitting iterations according to whether $x$3 or not:

• In the former regime, the step length satisfies $x$4, and the optimality conditions of the eigenvalue subproblem imply sufficient decrease: each such iteration decreases $x$5 by at least $x$6 under the parameter choices $x$7, $x$8, $x$9, $\mathcal{F}(x):=\max_y f(x,y)$0.
• In the latter regime, the next iterate satisfies $\mathcal{F}(x):=\max_y f(x,y)$1 and $\mathcal{F}(x):=\max_y f(x,y)$2, i.e., it is already an $\mathcal{F}(x):=\max_y f(x,y)$3-second-order stationary point.

Combining the two regimes via a potential-function argument yields the main theorem: HSDA reaches an $\mathcal{F}(x):=\max_y f(x,y)$4-second-order stationary point within at most $\mathcal{F}(x):=\max_y f(x,y)$5 iterations, matching the optimal $\mathcal{F}(x):=\max_y f(x,y)$6-order among second-order methods for this class. The stopping criterion based on $\mathcal{F}(x):=\max_y f(x,y)$7 is verifiable without explicit Hessian eigendecomposition, which is practically useful.

One assumption worth noting: the descent analysis requires $\mathcal{F}(x):=\max_y f(x,y)$8, which holds automatically under the prescribed parameter choice since $\mathcal{F}(x):=\max_y f(x,y)$9 with small $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$0.

The inexact variant IHSDA

For large-scale problems where computing the smallest eigenpair exactly is prohibitive, IHSDA replaces Step 3 of HSDA by a Lanczos procedure with skewed randomization applied to $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$1, producing a Ritz pair $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$2 with residual $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$3 satisfying controlled accuracy conditions. The algorithm retains the single-loop structure: if $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$4 and the residual component satisfies $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$5, the iterate is certified as second-order stationary; otherwise $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$6 is increased to $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$7 and the Ritz pair recomputed.

The high-probability guarantee rests on two probabilistic facts from the Lanczos-with-skewed-randomization machinery: $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$8 and $\tilde{\mathcal{O}}(\kappa_y^2\varepsilon^{-2})$9 each hold with probability at least $\varepsilon$0. Under these events, the same decrease argument as in the exact case yields a per-iteration decrease of at least $\varepsilon$1, giving the bound $\varepsilon$2 on outer iterations, with overall success probability at least $\varepsilon$3 — equivalently at least $\varepsilon$4 by Bernoulli's inequality when $\varepsilon$5. This success probability depends polynomially on dimension through $\varepsilon$6, requiring roughly $\varepsilon$7.

The total Hessian-vector product cost is bounded by $\varepsilon$8, matching the rate achieved by IGRTR and ILMNegCur while using only one subproblem type per iteration.

Conditioning comparison. A notable theoretical contribution is the conditioning analysis of the inner subproblems. Solving the regularized Newton system $\varepsilon$9 used in gradient-norm-regularized methods requires $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$0 conjugate-gradient-like steps, and $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$1 can blow up as $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$2. By contrast, the Lanczos condition number $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$3 — defined via the gap between the smallest and second-smallest eigenvalues — is provably finite for any $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$4. In the degenerate case $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$5, the ratio of condition numbers is bounded by $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$6, which vanishes when $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$7 with $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$8 bounded away from zero. When $$\tilde{\mathcal{O}}(\sqrt{\kappa_y}\varepsilon^{-2})$$9 the ratio stays order-one, so the homogenized formulation's advantage is most pronounced precisely in the near-stationary regime where Newton-type systems become ill-conditioned. The paper concedes no strict dominance here: in the small-gradient regime the two formulations have comparable conditioning.

Numerical experiments

Experiments compare HSDA/IHSDA against GDA, IMCN, MINIMAX-TRACE, and IGRTR on two tasks.

On a synthetic W-shaped nonconvex-strongly-concave minimax problem containing a strict saddle, GDA stalls near the saddle, while the four second-order methods escape. HSDA shows the fastest decrease, reducing the optimality gap to about $\mathcal{O}(\varepsilon^{-3/2})$0 and the gradient norm to about $\mathcal{O}(\varepsilon^{-3/2})$1 within roughly a dozen iterations from both tested initializations. GRTR converges quickly but oscillates more; MCN progresses smoothly but slower; MINIMAX-TRACE is slowest and most oscillatory with the chosen parameters. These comparisons depend on the specific parameter settings used, and no hyperparameter search protocol across all methods is reported.

For adversarial training on MNIST (a convolutional network trained against $\mathcal{O}(\varepsilon^{-3/2})$2-penalized adversarial perturbations, batch size 64, cross-entropy loss with $\mathcal{O}(\varepsilon^{-3/2})$3), IHSDA reaches approximately 80% test accuracy and attains the lowest objective values among compared methods, with IMCN, IGRTR, and ILMNegCur competitive and GDA remaining below roughly 70% accuracy within the plotted budget. The Lanczos procedure was capped at 80 iterations, indicating that even coarse approximate solutions of the homogenized subproblem suffice in practice.

Limitations and open questions

Several caveats qualify the results. The theory assumes deterministic access to exact gradients and Hessians of $\mathcal{O}(\varepsilon^{-3/2})$4; extension to stochastic or finite-sum settings with subsampled Hessian-vector products is not analyzed, despite the adversarial-training experiment being mini-batch based. The complexity bounds depend on the global constants $\mathcal{O}(\varepsilon^{-3/2})$5, $\mathcal{O}(\varepsilon^{-3/2})$6, $\mathcal{O}(\varepsilon^{-3/2})$7, and the initial gap $\mathcal{O}(\varepsilon^{-3/2})$8, which may be difficult to estimate in practice; the experiments use hand-specified values (e.g., $\mathcal{O}(\varepsilon^{-3/2})$9, $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$0, $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$1). The high-probability guarantee for IHSDA involves a union bound over up to $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$2 Lanczos calls, so the failure probability scales linearly with $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$3. Finally, the framework is restricted to strongly concave inner problems; whether the homogeneous eigenvalue approach extends to merely concave, weakly convex, or fully nonconvex-nonconcave minimax structures remains open.

Conclusion

This paper contributes a second-order descent-ascent method whose search directions arise from a single homogenized extremal-eigenvalue subproblem per iteration. Both the exact and Lanczos-based inexact variants achieve $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$4 iteration complexity for reaching $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$5-second-order stationary points, with IHSDA requiring $\mathcal{O}(\varepsilon,\sqrt{\varepsilon})$6 Hessian-vector products in expectation over randomized eigensolver calls. The conditioning analysis showing that the homogenized subproblem avoids the ill-conditioning of regularized Newton systems near stationary points is a concrete differentiator relative to prior trust-region and cubic-regularization schemes. Numerical results on synthetic saddle-escape problems and MNIST adversarial training support the practical viability of the approach.