---
title: Anchored Gradient Descent Ascent
url: https://www.emergentmind.com/topics/anchored-gradient-descent-ascent
type: topic
---

# Anchored Gradient Descent Ascent

Anchored Gradient Descent Ascent (AGDA) is a stabilization of simultaneous gradient descent-ascent for minimax and saddle-point problems in which each iterate is pulled toward a fixed reference point, typically the initialization. In the smooth unconstrained convex-concave setting, with saddle operator \(G(z)=(\nabla_x L(z),-\nabla_y L(z))\) and \(z=(x,y)\), the method takes the form
\[
z_{t+1}=z_t-\alpha_t G(z_t)+\beta_t(z_0-z_t),
\]
or, coordinatewise,
\[
x_{t+1}=x_t-\alpha_t\nabla_x L(x_t,y_t)+\beta_t(x_0-x_t),\qquad
y_{t+1}=y_t+\alpha_t\nabla_y L(x_t,y_t)+\beta_t(y_0-y_t).
\]
The anchor \(z_0\) is fixed, the anchoring weight \(\beta_t\) vanishes over time, and the method remains a single-gradient-call modification of simultaneous GDA. Modern analyses place AGDA within monotone operator theory, connect it to Halpern-type regularization, and establish last-iterate guarantees that were unavailable for vanilla GDA on classical oscillatory examples such as bilinear games [1905.10899], [2604.03782], [2604.12235].

## 1. Core formulation and problem classes

AGDA is studied primarily for convex-concave saddle-point problems
\[
\min_{x\in \mathbb{R}^n}\max_{y\in \mathbb{R}^m} L(x,y),
\]
where \(L(\cdot,y)\) is convex for every \(y\), \(L(x,\cdot)\) is concave for every \(x\), and the associated saddle operator
\[
G(z)=\bigl(\nabla_x L(z),-\nabla_y L(z)\bigr)
\]
is monotone and Lipschitz. In this setting, anchoring adds the restoring term \(\beta_t(z_0-z_t)\) to simultaneous GDA, so the primal variable is pulled toward \(x_0\) and the dual variable toward \(y_0\). The 2019 formulation explicitly identifies the anchor as the fixed initial iterate and relates the mechanism to Halpern’s method and James–Stein shrinkage [1905.10899].

A broader operator-theoretic formulation treats AGDA as a method for the monotone inclusion
\[
0\in F(z)+A(z),
\]
where \(F:\mathbb{R}^d\to\mathbb{R}^d\) is single-valued, monotone, and \(L\)-Lipschitz, \(A:\mathbb{R}^d\rightrightarrows\mathbb{R}^d\) is maximally monotone, and the solution set \(S=\operatorname{zer}(F+A)\) is nonempty. This encompasses monotone variational inequalities and convex-concave saddle-point problems with constraints or regularization. The proximal anchored iteration is
\[
z_{t+1}=J_{\alpha_tA}\Bigl((1-\beta_t)z_t+\beta_tz_0-\alpha_tF(z_t)\Bigr),
\qquad
J_{\alpha A}=(I+\alpha A)^{-1},
\]
which reduces to standard AGDA when \(A=0\) [2604.12235].

For composite saddle problems
\[
\min_x\max_y\; \Phi(x,y)+g(x)-h(y),
\]
with \(\Phi\) continuously differentiable, convex in \(x\), concave in \(y\), and \(g,h\) proper closed convex, one sets
\[
F(x,y)=\bigl(\nabla_x\Phi(x,y),-\nabla_y\Phi(x,y)\bigr),\qquad
A(x,y)=\partial g(x)\times \partial h(y).
\]
The resolvent then separates, yielding proximal anchored gradient descent-ascent:
\[
x_{t+1}=\operatorname{Prox}_{\alpha_t g}\Bigl((1-\beta_t)x_t+\beta_t x_0-\alpha_t\nabla_x\Phi(x_t,y_t)\Bigr),
\]
\[
y_{t+1}=\operatorname{Prox}_{\alpha_t h}\Bigl((1-\beta_t)y_t+\beta_t y_0+\alpha_t\nabla_y\Phi(x_t,y_t)\Bigr).
\]
If \(g=I_X\) and \(h=I_Y\), these become projected updates onto closed convex sets \(X\) and \(Y\) [2604.12235].

## 2. Early theory: ODE motivation, deterministic rates, and stochastic convergence

The first AGDA analysis in this corpus appears as “anchored simultaneous gradient descent” in a 2019 ODE-guided treatment of convex-concave minimax dynamics. In the deterministic smooth setting the update is
\[
z_{k+1}=z_k-\frac{1-p}{(k+1)^p}G(z_k)+\frac{(1-p)\gamma}{k+1}(z_0-z_k),
\qquad p\in(1/2,1),
\]
and in the stochastic subgradient setting the anchored term is
\[
\frac{(1-p)\gamma}{(k+1)^{1-\varepsilon}}(z_0-z_k),
\qquad \varepsilon>0.
\]
The continuous-time analogue is
\[
\dot z(t)=-g(t)+\frac{\gamma}{t}(z_0-z(t)),\qquad g(t)=G(z(t)),
\]
with Lyapunov-type quantity
\[
V(t)=t^2\|g(t)\|^2+2\gamma t\langle g(t),z(t)-z_0\rangle+\gamma(\gamma-1)\|z(t)-z_0\|^2.
\]
The paper derives the continuous-time decay
\[
\|g(t)\|^2\le \frac{4\gamma^2}{t^2}\|z_0-z_\star\|^2,
\]
and proves in discrete time that, under convex-concavity and smoothness, AGDA satisfies
\[
\|G(z_k)\|^2\le \mathcal O\!\left(k^{-(2-2p)}\right)
\]
for \(p\in(1/2,1)\), \(\gamma\ge 2\). Its stochastic anchored theorem is stronger in a different sense: under convex-concavity and a second-moment/Lipschitz-type oracle condition, but without strict convexity/concavity and without differentiability, the iterates converge in \(L^2\) to
\[
P_{\operatorname{Zer}(G)}(z_0),
\]
the projection of the anchor onto the saddle-point set [1905.10899].

This early theory established two points that remained central in later work. First, anchoring regularizes the last iterate rather than only ergodic averages. Second, the anchor does not merely damp motion; it also selects a solution when the saddle set is nonunique, because the fixed point \(z_0\) remains present in the dynamics throughout the run [1905.10899].

## 3. Exact last-iterate rates in smooth convex-concave problems

A 2026 analysis sharpened the AGDA theory for smooth convex-concave minimax problems by resolving an open question left by the 2019 rate. It studies
\[
\min_{x\in \mathbb{R}^n}\max_{y\in \mathbb{R}^m} L(x,y),
\qquad
G(z)=\bigl(\nabla_x L(z),-\nabla_y L(z)\bigr),
\]
under monotonicity of \(G\), \(K\)-Lipschitz continuity, and existence of a saddle point \(z^\star\) with \(G(z^\star)=0\). The update remains
\[
z_{t+1}=z_t-\alpha_t G(z_t)+\beta_t(z_0-z_t),
\]
but the schedule changes to
\[
\alpha_t=\frac{1}{K\sqrt{t+\gamma}},\qquad
\beta_t=\frac{\gamma}{t+\gamma},\qquad
\gamma\ge 2.
\]
Under this schedule the paper proves the exact last-iterate bound
\[
\|G(z_t)\|^2\le \frac{C}{t},
\qquad
C=K^2(E+\gamma D)^2,
\qquad
D=(\sqrt{12}+1)\|z_0-z^\star\|.
\]
Equivalently,
\[
\|G(z_t)\|\le \frac{K(E+\gamma D)}{\sqrt{t+\gamma}}.
\]
This closes the gap left by the earlier \(\mathcal O(1/t^{2-2p})\) analysis and shows that the borderline \(1/t\) decay for the squared gradient norm is achievable in the actual iterate, not merely in an average [2604.03782].

The proof is purely discrete-time. Its structure has three parts: a boundedness argument giving
\[
\|z_t-z^\star\|^2\le 12\|z_0-z^\star\|^2,
\]
a recurrence for consecutive differences \(d_t=z_{t+1}-z_t\) implying
\[
\|z_{t+1}-z_t\|\le \frac{E}{t+\gamma},
\]
and the reconstruction identity
\[
\alpha_t G(z_t)=(z_t-z_{t+1})+\beta_t(z_0-z_t),
\]
which converts iterate stability into gradient decay. The result is notable not only for its rate but also for its formal provenance: the paper reports that it was discovered autonomously by an AI system capable of writing formal proofs in Lean, and the natural-language proof is presented as an informalization of the formal development [2604.03782].

A common misconception is that AGDA only supports suboptimal last-iterate rates. That statement is accurate for the general composite monotone inclusion analysis discussed below, where the primary theorem is an \(\mathcal O(1/\sqrt{T})\) bound on a residual norm, but it is not accurate for the smooth unconstrained convex-concave case: there the 2026 result gives the exact \(\mathcal O}(1/t)\) rate for \(\|G(z_t)\|^2\) [2604.03782].

## 4. Composite, constrained, and operator-theoretic AGDA

The operator-theoretic extension of 2026 studies AGDA in its most general monotone inclusion form and derives a concrete anchored gradient descent-ascent method for constrained and regularized min-max problems. The standing assumptions are that \(F\) is monotone and \(L\)-Lipschitz, \(A\) is maximally monotone, and \(S=\operatorname{zer}(F+A)\neq\varnothing\). The proposed schedule is
\[
\alpha_t=\frac{1}{L\sqrt{t+\gamma}},\qquad
\beta_t=\frac{\gamma}{t+\gamma},\qquad
\gamma\ge 2.
\]
The convergence metric is the tangent residual
\[
r_{F,A}(z):=\min_{c\in A(z)}\|F(z)+c\|
      =\operatorname{Distance}(0,F(z)+A(z)),
\]
which reduces to \(\|F(z)\|\) when \(A\equiv 0\). The paper also recalls the natural residual
\[
r^{\mathrm{nat}}_{F,A}(z):=\|z-J_A(z-F(z))\|,
\]
and proves
\[
r^{\mathrm{nat}}_{F,A}(z)\le r^{\mathrm{tan}}_{F,A}(z)
\]
for every \(z\). Consequently, a tangent-residual bound immediately implies a natural-residual bound [2604.12235].

Its main theorem states that for the iterates generated by
\[
z_{t+1}=J_{\alpha_tA}\Bigl((1-\beta_t)z_t+\beta_tz_0-\alpha_tF(z_t)\Bigr),
\]
one has
\[
r_{F,A}(z_T)\le \|F(z_T)+c_T\|
\le \frac{L(2E+\gamma D)}{\sqrt{T-1+\gamma}},
\]
where \(c_T\in A(z_T)\) is induced by the resolvent step,
\[
D=(\sqrt{12}+1)\|z_0-z^\star\|,
\qquad
E=\max\!\bigl\{\gamma\|z_1-z_0\|,\;12\gamma D\bigr\}.
\]
Using \(\|z_1-z_0\|\le D\), the explicit version becomes
\[
r_{F,A}(z_T)\le
\frac{25(\sqrt{12}+1)L\gamma\|z_0-z^\star\|}{\sqrt{T-1+\gamma}}
=O\!\left(\frac{L\|z_0-z^\star\|}{\sqrt{T}}\right).
\]
This is a last-iterate guarantee for \(z_T\) itself, and by residual domination it also yields an \(\mathcal O(1/\sqrt{T})\) last-iterate bound for the natural residual [2604.12235].

The proof introduces auxiliary vectors
\[
c_{t+1}:=
\frac{(1-\beta_t)z_t+\beta_tz_0-\alpha_tF(z_t)-z_{t+1}}{\alpha_t},
\qquad c_{t+1}\in A(z_{t+1}),
\]
so that the proximal iteration can be rewritten as
\[
z_{t+1}=z_t-\alpha_t\bigl(F(z_t)+c_{t+1}\bigr)+\beta_t(z_0-z_t).
\]
This “\(c\)-vector trick” makes the composite recursion resemble the unconstrained anchored recursion while preserving the nonsmooth or constrained contribution of \(A\) [2604.12235].

In saddle-point language, the operator formulation shows that AGDA is not limited to smooth unconstrained min-max. The same fixed-anchor mechanism extends directly to proximal and projected updates, with the resolvent enforcing nonsmooth regularization or feasibility while the anchor continues to regularize both players’ motion [2604.12235].

## 5. Position within the stabilized GDA landscape

The modern literature places AGDA alongside several other mechanisms for suppressing the rotational behavior of simultaneous GDA. The comparison is explicit in the 2026 composite monotone-inclusion paper:

| Method family | Stabilization mechanism | Stated last-iterate behavior |
|---|---|---|
| Vanilla GDA / forward-backward | None | Can diverge or oscillate even on simple bilinear problems |
| AGDA | Fixed anchor \(z_0\) | \(\mathcal O(1/\sqrt{T})\) tangent residual in the composite setting; \(\mathcal O(1/t)\) for \(\|G(z_t)\|^2\) in the smooth unconstrained setting |
| EG / OGDA | Optimism or lookahead | \(\mathcal O(1/\sqrt{T})\) last-iterate residual |
| EAG / FEG and composite variants | Optimism plus anchoring | \(\mathcal O(1/T)\) last-iterate, stated as optimal |

This comparison shows that anchoring alone is sufficient for last-iterate convergence, but not for the fastest rates currently stated in the same generality; the optimal \(\mathcal O(1/T)\) last-iterate rate is attributed to methods that combine anchoring with optimism or extragradient structure [2604.12235].

Several nearby methods are anchor-like without being standard AGDA. “Dissipative Gradient Descent Ascent” augments the state with auxiliary variables \((\hat x_k,\hat y_k)\) and pulls the iterate toward a dynamically updated filtered reference:
\[
x_{k+1}=x_k-\eta\nabla_x f(x_k,y_k)-\rho(x_k-\hat x_k),\qquad
\hat x_{k+1}=(1-\rho)\hat x_k+\rho x_k,
\]
and similarly for \(y\). The paper is explicit that this is not literal anchoring to a fixed center; the reference evolves as an exponential moving average, and the method is best described as dynamically anchored and dissipative rather than standard AGDA. Its theory gives linear convergence in bilinear and strongly convex-strongly concave settings [2403.09090].

A different competitor dispenses with anchors entirely. “Slingshot stepsize schedules” make plain GDA converge by using time-varying, asymmetric, and periodically negative stepsizes. That work presents anchoring as prior art among stabilization devices, but its claim is that convergence can be recovered solely by changing the stepsize schedule, without adding anchoring, optimism, or extragradient corrections [2505.01423].

The continuous-time Newtonian dissipation analysis of gradient descent-ascent is also adjacent rather than direct AGDA theory. It contains no anchoring term, but it explains how antisymmetric coupling can generate rotational behavior and how dissipative structure can restore convergence. This supplies conceptual background for why anchor terms may act as damping-like regularizers even though the paper itself does not analyze AGDA [1903.02536].

## 6. Variants, scope, and limitations

Anchoring has also been adapted beyond the standard smooth convex-concave fixed-anchor setting. A notable example is the “semi-anchored multi-step gradient descent-ascent method” for structured nonconvex-nonconcave composite minimax problems,
\[
\min_u\max_v \{f(u)+\phi(u,v)-g(v)\},
\]
whose saddle-subdifferential operator satisfies a weak Minty variational inequality. SA-MGDA is derived from a Bregman proximal point method and introduces anchoring only on the dual side. The exact dual subproblem is centered at
\[
v_k-\tau\nabla_v\phi(u_k,v_k),
\]
so the method is explicitly asymmetric: the ascent variable is tethered to a previous-iterate-based anchor while the descent variable is updated by a proximal gradient step. Under weak MVI it obtains a nonergodic \(O(1/k)\) stationarity guarantee, and under strong MVI it obtains linear convergence in Bregman distance [2105.15042].

These developments delimit the meaning of AGDA. Fixed-anchor AGDA refers to a method that pulls iterates toward a constant reference point \(z_0\). Dynamic-anchor methods such as DGDA, dual-only semi-anchored methods such as SA-MGDA, and dissipation-based continuous-time analyses are structurally related, but they are not the same algorithmic object [2403.09090], [2105.15042], [1903.02536].

The limitations of standard AGDA are also explicit in the literature. In the general monotone inclusion setting, the main guarantee is for residual convergence—specifically tangent residual, and hence natural residual—rather than a direct primal-dual objective-gap theorem [2604.12235]. The composite note does not claim that \(\mathcal O(1/\sqrt{T})\) is optimal; on the contrary, it states that stronger \(\mathcal O(1/T)\) rates are known when anchoring is combined with optimism [2604.12235]. The 2019 stochastic anchored theorem requires a small \(\varepsilon>0\) in the anchor exponent for the proof, while also stating that this is likely a proof artifact [1905.10899]. The 2026 exact \(\mathcal O(1/t)\) result applies to smooth unconstrained convex-concave problems, not to the full composite inclusion setting [2604.03782].

Taken together, these results support a precise characterization. AGDA is a single-call, last-iterate-oriented regularization of simultaneous GDA in which a vanishing pull toward a fixed anchor suppresses oscillation without introducing lookahead. In smooth unconstrained convex-concave problems it now has an exact \(\mathcal O(1/t)\) last-iterate rate for the squared saddle-gradient norm [2604.03782]. In the general composite monotone inclusion framework it admits a proximal-resolvent generalization with an \(\mathcal O(1/\sqrt{T})\) last-iterate residual bound [2604.12235]. Its broader significance lies in showing that anchoring alone, without optimism, is already sufficient to stabilize a large class of minimax dynamics that vanilla simultaneous GDA does not control [1905.10899], [2604.12235].

Source: https://www.emergentmind.com/topics/anchored-gradient-descent-ascent