AltGDA: Alternating Gradient Descent-Ascent

• AltGDA is a sequential update method for minimax optimization, performing descent in x followed by ascent in y.
• It demonstrates improved convergence and iteration complexity over simultaneous updates in strongly convex–concave and bilinear games.
• Variants extend AltGDA to proximal, stochastic, adaptive, and manifold settings, offering practical benefits in GANs, PCA, and robust models.

Searching arXiv for recent and foundational papers on Alternating Gradient Descent-Ascent. Alternating Gradient Descent–Ascent (AltGDA) is a class of single-loop first-order methods for minimax optimization and game dynamics in which the two blocks are updated sequentially rather than simultaneously. In its canonical form for a smooth two-player problem $\min_x \max_y f(x,y)$, the method performs a descent step in $x$ using $(x_t,y_t)$ and then an ascent step in $y$ using the fresh iterate $x_{t+1}$ and the stale iterate $y_t$. This alternation distinguishes AltGDA from simultaneous GDA and is the source of several theoretical phenomena reported across the literature: bounded regret and recurrent cycling in unconstrained bilinear zero-sum games (Bailey et al., 2019), near-optimal local linear convergence in strongly convex–strongly concave problems (Zhang et al., 2021), improved global iteration complexity over simultaneous updates in smooth strongly-convex-strongly-concave settings (Lee et al., 2024), and single-loop convergence guarantees in nonconvex minimax regimes with proximal, stochastic, adaptive, Riemannian, or zeroth-order variants (Boţ et al., 2020).

1. Canonical formulation and update structure

The standard AltGDA iteration for an unconstrained smooth minimax problem

$$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$

takes the form

$$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$

or, with separate step-sizes,

$$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$

This sequential structure appears verbatim in the strongly convex–strongly concave analysis of Alt-GDA (Zhang et al., 2021) and in the global SCSC analysis comparing alternating and simultaneous updates (Lee et al., 2024).

In bilinear zero-sum games with payoff $f(x_1,x_2)=x_1^\top A x_2$, the alternating update becomes

$x$0

which was introduced as “sequential” or “alternating” play in the finite-regret analysis of Bailey, Gidel, and Piliouras (Bailey et al., 2019). In constrained matrix games, the same alternation is combined with Euclidean projection: $x$1 with convergence measured by the ergodic duality gap (Nan et al., 4 Oct 2025).

A major branch of the literature studies proximal AltGDA for composite objectives. For nonconvex–concave problems of the form

$x$2

the deterministic single-loop updates are

$x$3

with $x$4 (Boţ et al., 2020). The same proximal sequential template recurs in later regularized and accelerated formulations, including Prox-AltGDAm (Chen et al., 2021) and AGDA+ (Zhang et al., 2024).

The defining feature of AltGDA is therefore not a single objective class but the ordering of information flow: the ascent step is evaluated at the updated primal block. This suggests that AltGDA is best understood as an algorithmic pattern spanning Euclidean, constrained, proximal, stochastic, and manifold settings rather than a single fixed method.

2. Bilinear zero-sum games: finite regret, bounded orbits, and recurrence

The 2019 analysis of unconstrained bilinear zero-sum games gave one of the sharpest early characterizations of alternation (Bailey et al., 2019). In the game

$x$5

AltGDA yields an energy–payoff identity for each block: $x$6 and

$x$7

These identities imply a bounded-regret theorem: if Agent 1 plays AltGDA with fixed $x$8, then against any sequence $x$9, her regret after $(x_t,y_t)$0 updates is bounded by

$(x_t,y_t)$1

regardless of how the opponent updates (Bailey et al., 2019). The result is notable because it holds with arbitrary fixed step-size rather than decaying step-sizes.

When both agents use AltGDA, the same paper establishes an exactly conserved “perturbed energy”

$(x_t,y_t)$2

which remains equal to $(x_t,y_t)$3 for all $(x_t,y_t)$4 (Bailey et al., 2019). If $(x_t,y_t)$5, then the iterates remain in a bounded region, and a complementary lower bound shows the joint norm is bounded away from $(x_t,y_t)$6 whenever $(x_t,y_t)$7. The map is also volume-preserving because AltGDA is a composition of two block-triangular maps whose Jacobians each have determinant $(x_t,y_t)$8; Poincaré’s recurrence theorem then implies that almost every initial condition returns arbitrarily close to its start infinitely often (Bailey et al., 2019).

The constrained bilinear literature later shifted the emphasis from bounded cycles to ergodic convergence. For compact convex strategy sets, including matrix games on simplices, AltGDA with constant step-size achieves an $(x_t,y_t)$9 ergodic convergence rate when there is an interior Nash equilibrium, with

$y$0

provided

$y$1

(Nan et al., 4 Oct 2025). Without an interior equilibrium, the same work proves an $y$2 local convergence rate in a neighborhood $y$3 under the constant-step condition $y$4 and a refined energy argument (Nan et al., 4 Oct 2025).

These two strands are complementary rather than contradictory. In unconstrained bilinear dynamics, AltGDA generically produces bounded recurrent orbits rather than convergence to equilibrium (Bailey et al., 2019). In compact constrained bilinear games, averaged iterates can nevertheless enjoy $y$5 ergodic gap decay (Nan et al., 4 Oct 2025). A common misconception is that “alternation guarantees convergence” in bilinear games; the record is narrower. The unconstrained theory emphasizes boundedness and recurrence, whereas the constrained theory emphasizes averaged duality-gap convergence.

3. Why alternation differs from simultaneous GDA

Several papers isolate the mechanism by which alternation changes the dynamics. In the bilinear setting, simultaneous GDA

$y$6

is described as the naive “Euler” integrator of the continuous Hamiltonian system $y$7, $y$8. Its drawbacks are explicit: it does not conserve energy, iterates diverge from equilibrium, and the map is volume-expanding, leading to formally chaotic behavior and linear or unbounded regret. AltGDA is instead interpreted as a symplectic integrator of Verlet/leapfrog type: it exactly conserves a perturbed energy, preserves volume, and thereby avoids systematic drift away from equilibrium (Bailey et al., 2019).

In strongly convex–strongly concave problems, the distinction appears through the Jacobian of the iteration map near the Nash equilibrium. For SCSC objectives with twice continuously differentiable $y$9 satisfying

$x_{t+1}$0

and Hessian block norms bounded by $x_{t+1}$1, Alt-GDA with $x_{t+1}$2 satisfies

$x_{t+1}$3

where $x_{t+1}$4, $x_{t+1}$5, and $x_{t+1}$6 (Zhang et al., 2021). Standard linearization then yields local linear convergence with iteration complexity $x_{t+1}$7. The simultaneous counterpart admits a much slower worst-case spectral radius

$x_{t+1}$8

corresponding to $x_{t+1}$9 iterations (Zhang et al., 2021).

A more global comparison is provided for twice-differentiable SCSC functions with Lipschitz gradients, where Alt-GDA uses step-sizes satisfying

$y_t$0

Under these conditions, the iteration complexity is

$y_t$1

where $y_t$2, $y_t$3, and $y_t$4 (Lee et al., 2024). The corresponding simultaneous bound is

$y_t$5

and the paper states that the upper bound for Alt-GDA is strictly smaller than the lower bound for Sim-GDA (Lee et al., 2024).

The same work proposes Alternating-Extrapolation GDA (Alex-GDA), with extrapolation parameters $y_t$6, subsuming Sim-GDA and Alt-GDA as special cases $y_t$7 and $y_t$8 (Lee et al., 2024). Alex-GDA attains

$y_t$9

and is reported to match the extragradient lower bound while requiring two gradient evaluations per iteration instead of four (Lee et al., 2024). This suggests that alternation can be viewed as an intermediate point in a broader design space of sequential and extrapolated schemes.

4. Nonconvex minimax theory: concavity, PL structure, and proximal variants

Outside convex–concave settings, AltGDA is primarily analyzed through structure imposed on the maximization block. One influential line considers nonconvex–concave or nonconvex–PL objectives, where the value function

$$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$0

or its composite analog is smooth or weakly convex enough to support first-order stationarity analysis.

For nonconvex–PL minimax problems with Lipschitz gradient constant $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$1 and PL constant $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$2 in $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$3, deterministic AGDA uses

$$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$4

Choosing

$$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$5

yields

$$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$6

and hence $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$7 to reach $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$8 (Yang et al., 2021). The stochastic single-sample version attains sample complexity $$\min_{x\in\mathbb{R}^n}\max_{y\in\mathbb{R}^m} f(x,y)$$9 under bounded variance (Yang et al., 2021). The same paper emphasizes that AGDA matches the best $$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$0 complexity under the weaker PL assumption instead of strong concavity.

A related but stronger condition is the two-sided PL inequality. For unconstrained nonconvex-nonconcave $$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$1, if both blocks satisfy PL-type inequalities, AGDA with

$$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$2

converges globally at a linear rate in the potential

$$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$3

with contraction factor

$$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$4

where $$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$5 (Yang et al., 2020). This extends linear convergence beyond strong convexity/concavity to a class of nonconvex-nonconcave minimax problems.

For nonconvex–concave composite problems, the proximal single-loop AltGDA of Boţ and Böhm studies

$$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$6

under $$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$7-weak convexity in $$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$8, concavity and $$x_{t+1} = x_t - \eta \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta \nabla_y f(x_{t+1},y_t),$$9-smoothness in $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$0, bounded $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$1, and lower boundedness of $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$2 (Boţ et al., 2020). With

$$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$3

the deterministic algorithm finds an $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$4-stationary point of the Moreau envelope $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$5 after

$$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$6

iterations (Boţ et al., 2020). In the stochastic case, with batch-size $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$7, one obtains $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$8 for $$x_{t+1} = x_t - \eta_x \nabla_x f(x_t,y_t), \qquad y_{t+1} = y_t + \eta_y \nabla_y f(x_{t+1},y_t).$$9 (Boţ et al., 2020). In the nonconvex–strongly-concave regime, the same paper gives single-loop rates $f(x_1,x_2)=x_1^\top A x_2$0 and $f(x_1,x_2)=x_1^\top A x_2$1 for deterministic and stochastic settings (Boţ et al., 2020).

A later accelerated proximal version, Prox-AltGDAm, adds heavy-ball momentum in $f(x_1,x_2)=x_1^\top A x_2$2 and Nesterov momentum in $f(x_1,x_2)=x_1^\top A x_2$3 to the regularized nonconvex–strongly-concave problem

$f(x_1,x_2)=x_1^\top A x_2$4

where $f(x_1,x_2)=x_1^\top A x_2$5 is $f(x_1,x_2)=x_1^\top A x_2$6-smooth and $f(x_1,x_2)=x_1^\top A x_2$7-strongly concave in $f(x_1,x_2)=x_1^\top A x_2$8 (Chen et al., 2021). With $f(x_1,x_2)=x_1^\top A x_2$9, $x$00, and $x$01, the method reaches

$x$02

in

$x$03

iterations (Chen et al., 2021). This improves the stated single-loop dependence on $x$04 relative to prior AltGDA-type methods.

Across these nonconvex results, the main theme is that alternation alone is usually not sufficient; the theory additionally requires concavity, strong concavity, PL, weak convexity plus Moreau smoothing, or analogous regularity conditions in the inner block. This suggests that AltGDA is best seen as a favorable update order whose guarantees depend critically on problem geometry.

5. Variants: adaptive, stochastic, zeroth-order, and regularized AltGDA

A substantial part of the AltGDA literature concerns practical modifications that preserve the alternating structure while weakening oracle assumptions or reducing hyperparameter sensitivity.

For double-regularized nonconvex–strongly-concave problems

$x$05

AGDA+ introduces nonmonotone backtracking to choose primal-dual step-sizes without knowing the global Lipschitz constant $x$06 and concavity modulus $x$07 (Zhang et al., 2024). The algorithm alternates the proximal updates

$x$08

and uses local-smoothness tests to accept or reject trial steps. The paper states that AGDA+ achieves the optimal iteration complexity $x$09 and is the first step-size search method for NCSC minimax problems that require only $x$10 calls to $x$11 on average per backtracking iteration (Zhang et al., 2024).

In stochastic minimax problems with decision-dependent distributions, the alternating adaptive method AASGDA updates

$x$12

where each adaptive gradient includes a correction through the learned distribution map $x$13: $x$14

$x$15

(Gao et al., 14 Sep 2025). Under the PL condition in $x$16, with

$x$17

and online least-squares estimation yielding $x$18, the method reaches an $x$19-stationary point within $x$20 stochastic gradient evaluations (Gao et al., 14 Sep 2025).

When gradients are unavailable, zeroth-order alternating methods replace each block gradient by a two-point estimator. For NC-PL minimax problems, ZO-AGDA uses

$x$21

and alternates descent in $x$22 with ascent in $x$23 (Xu et al., 2022). The paper states total function-value query complexity $x$24 for deterministic ZO-AGDA and $x$25 for stochastic variance-reduced ZO-VRAGDA (Xu et al., 2022).

Regularization can also reshape the landscape itself. In two-player zero-sum Markov games, entropy-regularized AltGDA updates

$x$26

for

$x$27

(Zeng et al., 2022). The fixed-$x$28 theory gives geometric decay to the unique regularized Nash equilibrium, while a diminishing schedule

$x$29

yields last-iterate rate $x$30 and therefore $x$31 gradient updates to an $x$32 Nash equilibrium of the original unregularized game (Zeng et al., 2022).

These variants do not alter the core semantics of AltGDA. They preserve the sequential dependence $x$33 while modifying the oracle, geometry, or line-search mechanism.

6. Geometry-aware and manifold generalizations

AltGDA extends naturally beyond Euclidean spaces when one block lies on a manifold and the other lies in a convex set. In fair principal component analysis, the problem is reformulated as

$x$34

where $x$35 is the Stiefel manifold and $x$36 is the simplex (Xu et al., 2022). The ARPGDA method alternates a Riemannian gradient descent step

$x$37

with a projected gradient ascent step

$x$38

where a quadratic regularizer $x$39 is added to stabilize the ascent block (Xu et al., 2022). Under smoothness assumptions, ARPGDA finds an $x$40-stationary point within $x$41 iterations (Xu et al., 2022).

A broader Riemannian alternating descent-ascent framework, RADA, considers

$x$42

with $x$43 a compact Riemannian submanifold and $x$44 proper, closed, convex with compact domain (Xu et al., 2024). The framework defines

$x$45

and alternates a descent step on the smoothed value function $x$46 with the proximal ascent step

$x$47

(Xu et al., 2024). With $x$48, the framework returns an $x$49-Riemannian-game-stationary point and an $x$50-Riemannian-optimization-stationary point within $x$51 iterations (Xu et al., 2024).

These manifold results reveal two general features. First, alternation is compatible with non-Euclidean descent mechanisms such as retractions, tangent-space projections, and Riemannian gradients. Second, in many geometric settings the ascent step is regularized through a proximal or strongly concave surrogate, suggesting that the most successful manifold instantiations of AltGDA use alternation together with value-function smoothing.

7. Applications, empirical behavior, and common interpretive boundaries

The application areas explicitly named across the literature include training of Generative Adversarial Networks, adversarial deep learning, economic duopoly models, fair PCA, sparse PCA, fair PCA on manifolds, sparse spectral clustering, robust nonlinear regression, and two-player zero-sum Markov games (Bailey et al., 2019). Empirical reports consistently associate alternating updates with reduced oscillation or faster practical progress relative to simultaneous updates, but the phenomena vary by regime.

In the original bilinear zero-sum illustrations, AltGDA strategies cycle around $x$52 in a $x$53 game with $x$54, $x$55, and $x$56, while cumulative regret oscillates yet remains bounded (Bailey et al., 2019). In quadratic SCSC experiments with $x$57–$x$58, Alt-GDA is reported to converge linearly with rate approximately $x$59, while Sim-GDA converges much more slowly at approximately $x$60; on CIFAR-10 and SVHN with WGAN-GP, alternating SGD and AMSGrad reach lower FID in many fewer iterations than simultaneous variants (Zhang et al., 2021). Prox-AltGDAm on the Wasserstein Robustness Model for MNIST reports approximately $x$61 speedup in convergence of $x$62 versus no-momentum variants and approximately $x$63–$x$64 higher robust accuracy within the same number of epochs (Chen et al., 2021). In FPCA, ARPGDA is described as roughly one-fifth the time of RSG on synthetic Gaussian data and about $x$65 faster than SG/RSG on the USGS dataset (Xu et al., 2022). RADA-RGD and RADA-PGD are reported as $x$66–$x$67 faster than nested-loop ManPG and augmented-Lagrangian methods on SPCA, FPCA, and SSC (Xu et al., 2024). AGDA+ is reported to require markedly fewer gradient calls than fixed-step GDA, TiAda, and SGDA-B on quadratic WCSC tests (Zhang et al., 2024).

Several boundaries are important for interpretation. First, AltGDA is not uniformly convergent in all minimax settings. In unconstrained bilinear games, it can cycle indefinitely even while enjoying finite regret and bounded iterates (Bailey et al., 2019). Second, superior performance over simultaneous GDA is well established in several structured regimes, especially SCSC and constrained bilinear settings (Zhang et al., 2021, Lee et al., 2024, Nan et al., 4 Oct 2025), but the exact advantage depends on the problem class and performance metric. Third, many nonconvex guarantees concern stationarity of the value function, Moreau envelope, or proximal-gradient mapping rather than direct convergence to a global minimax solution (Boţ et al., 2020). Fourth, stronger guarantees often rely on additional ingredients beyond plain alternation: proximal regularization (Boţ et al., 2020), momentum (Chen et al., 2021), entropy regularization (Zeng et al., 2022), extrapolation (Lee et al., 2024), or value-function smoothing on manifolds (Xu et al., 2024).

Taken together, the AltGDA literature portrays alternation as a structurally meaningful algorithmic choice rather than a cosmetic implementation detail. In some regimes it induces conserved quantities, telescoping potentials, or improved Jacobian spectra; in others it serves as the backbone for proximal, adaptive, or geometric methods. A plausible implication is that the main value of AltGDA lies not merely in “updating in turn,” but in enabling analyses and algorithmic designs that exploit the freshly updated descent block in the ascent step.