Extragradient Method (EG) Explained

• Extragradient Method (EG) is a two-step first-order method that uses an extrapolation step followed by a correction to solve saddle-point, variational inequality, and monotone problems.
• It approximates the proximal point method by employing a midpoint evaluation, which enhances stability and convergence compared to standard gradient descent-ascent.
• EG offers strong convergence guarantees including last-iterate and linear rates in specific regimes, with extensions to stochastic and composite optimization settings.

The extragradient method (EG) is a two-step first-order method for solving saddle-point problems, variational inequalities, monotone equations, and related inclusions. Its defining feature is an extrapolation or “look-ahead” step followed by a correction step evaluated at the extrapolated point. In the unconstrained saddle formulation

$$\min_{x\in \mathbb{R}^m}\max_{y\in \mathbb{R}^n} f(x,y),$$

with

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$

EG takes the form

$$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$

or, in coordinates,

$$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$

$$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$

The method is classical in monotone VI theory and has become central in modern min-max optimization because it remains stable in regimes where plain gradient descent-ascent or forward methods can oscillate or diverge (Mokhtari et al., 2019, Gorbunov et al., 2021).

1. Canonical problem classes and formulations

EG appears in several mathematically equivalent guises. In unconstrained variational inequalities one seeks $x^\star\in\mathbb{R}^d$ such that $F(x^\star)=0$, under monotonicity

$(F(x)-F(x'),x-x')\ge 0$

and $L$-Lipschitz continuity

$\|F(x)-F(x')\|\le L\|x-x'\|.$

This formulation covers many saddle-point models, including convex-concave minimax problems and Nash equilibrium computation (Gorbunov et al., 2021).

For constrained VIs over a compact convex set $$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$0, the projected form is

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$1

which is the standard deterministic extragradient iteration in variational-inequality language (Nan et al., 2023). In generalized inclusion form, EG extends to problems

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$2

with resolvent steps replacing projections; this places EG alongside forward-backward-forward and related operator-splitting schemes (Tran-Dinh, 2023, Tran-Dinh et al., 2024).

In differentiable games, EG is equally expressible through the game vector field $$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$3 as

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$4

which makes clear that the method is a two-stage explicit dynamical system rather than a single forward step (Azizian et al., 2019). This unification across saddle problems, monotone equations, and inclusions is one reason EG has remained a foundational method.

2. Two-step geometry and the proximal-point interpretation

The main structural interpretation of EG is as an explicit approximation to the proximal point method (PP). For saddle-point problems, the PP update is defined implicitly by

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$5

whose optimality conditions are

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$6

PP is robust but implicit, and therefore typically computationally impractical at scale (Mokhtari et al., 2019).

A central analytical result is that EG is a second-order approximation of PP. If $$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$7 denotes the PP step and $$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$8 the EG step from the same iterate, then

$$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$9

The midpoint correction supplies precisely the Taylor terms needed to match the implicit PP step up to second order (Mokhtari et al., 2019). In the Euclidean setting this viewpoint aligns EG with Nemirovski’s conceptual mirror-prox interpretation, where EG serves as an implementable explicit surrogate for a proximal or mirror-prox step (Mokhtari et al., 2019).

This perspective also clarifies why EG differs qualitatively from vanilla gradient descent-ascent (GDA). GDA evaluates the gradient only at the current iterate and is only a first-order approximation to a stable implicit method, whereas EG uses an additional midpoint evaluation. The extra evaluation increases per-iteration cost, but it is precisely what stabilizes rotational saddle dynamics and monotone VI dynamics that defeat one-step explicit schemes (Mokhtari et al., 2019, Zheng et al., 2024).

The same proximal interpretation extends beyond convex-concave minimax. In smooth nonconvex-nonconcave problems satisfying positive interaction dominance, damped EG is analyzed as a second-order approximation of damped proximal point and inherits linear convergence under corresponding structural assumptions (Hajizadeh et al., 2022).

3. Convergence regimes and representative rates

EG admits several distinct convergence theories, depending on the geometry of the operator or saddle function. Some results are ergodic, others are last-iterate, and in bilinear or strongly monotone regimes the method is linearly convergent.

Setting	Representative guarantee	Source
Smooth convex-concave unconstrained saddle problem	Averaged primal-dual gap and function value converge at $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$0	(Mokhtari et al., 2019)
Monotone, $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$1-Lipschitz VIP	Last-iterate $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$2 for $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$3	(Gorbunov et al., 2021)
Bilinear $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$4 with full-rank square $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$5	Linear contraction $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$6, complexity $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$7	(Mokhtari et al., 2019)
Strongly convex-strongly concave smooth saddle problem	Linear contraction $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$8, complexity $$z_{k+\frac12}=z_k-\eta F(z_k),\qquad z_{k+1}=z_k-\eta F(z_{k+\frac12}),$$9	(Mokhtari et al., 2019)
Anchored EG in Hilbert space	Strong convergence to $$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$0; with $$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$1, $$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$2 if $$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$3	(Boţ et al., 2024)
Biaffine min-max, dynamic stepsizes	Shared-step EG achieves $$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$4; separate extrapolation/update stepsizes achieve $$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$5	(Wu et al., 1 Jun 2026)

In the bilinear model

$$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$6

EG becomes

$$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$7

$$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$8

or equivalently

$$x_{k+\frac12}=x_k-\eta \nabla_x f(x_k,y_k),\qquad y_{k+\frac12}=y_k+\eta \nabla_y f(x_k,y_k),$$9

With

$$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$0

the Lyapunov quantity

$$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$1

contracts linearly (Mokhtari et al., 2019). This is the standard explanation for EG’s success on purely rotational games where GDA fails.

A broader spectral analysis shows that EG interpolates between two mechanisms. Near a stationary point $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$2, if $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$3 ranges over the spectrum of $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$4, then for $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$5-extrapolation methods

$$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$6

For EG, the contraction combines a strong-monotonicity-type contribution through $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$7 and a purely adversarial contribution through $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$8, yielding a single analysis across strongly monotone, bilinear, and intermediate regimes. In this sense, EG can be much faster than GD in games, and the paper proves optimality for a broad class of finite-extrapolation last-iterate methods up to constants (Azizian et al., 2019).

4. Last-iterate theory, cocoercivity, and stepsize refinements

A major modern development is that EG admits last-iterate guarantees under assumptions weaker than previously known. For monotone and $$x_{k+1}=x_k-\eta \nabla_x f(x_{k+\frac12},y_{k+\frac12}),\qquad y_{k+1}=y_k+\eta \nabla_y f(x_{k+\frac12},y_{k+\frac12}).$$9-Lipschitz $x^\star\in\mathbb{R}^d$0, the residual norm decreases monotonically: $x^\star\in\mathbb{R}^d$1 provided $x^\star\in\mathbb{R}^d$2. Together with

$x^\star\in\mathbb{R}^d$3

this yields the first last-iterate $x^\star\in\mathbb{R}^d$4 bound for EG under monotonicity and Lipschitz continuity alone: $x^\star\in\mathbb{R}^d$5 The proof is notable because it does not rely on a Lipschitz Jacobian or on cocoercivity of the EG update operator (Gorbunov et al., 2021).

That distinction is substantive. Writing

$x^\star\in\mathbb{R}^d$6

one may view EG formally as gradient descent on a transformed operator. However, $x^\star\in\mathbb{R}^d$7 is not generally cocoercive: for every $x^\star\in\mathbb{R}^d$8 and every $x^\star\in\mathbb{R}^d$9, there exists an $F(x^\star)=0$0-cocoercive operator $F(x^\star)=0$1 such that $F(x^\star)=0$2 is not $F(x^\star)=0$3-cocoercive for any $F(x^\star)=0$4. In the affine monotone case, by contrast, $F(x^\star)=0$5 is $F(x^\star)=0$6-cocoercive for $F(x^\star)=0$7. EG therefore inherits some proximal-point-like behavior without literally becoming GD on a cocoercive field (Gorbunov et al., 2021).

Recent work also refines stepsize theory. Under the $F(x^\star)=0$8-symmetric $F(x^\star)=0$9-Lipschitz condition

$(F(x)-F(x'),x-x')\ge 0$0

EG with adaptive stepsizes of the form

$(F(x)-F(x'),x-x')\ge 0$1

achieves linear convergence for strongly monotone operators, sublinear convergence for monotone operators, and local guarantees for weak Minty operators (Choudhury et al., 25 Oct 2025). This relaxes the conventional reliance on a global Lipschitz constant.

For biaffine min-max problems, dynamic stepsize schedules produce a different kind of improvement. Fixed-stepsize EG has a $(F(x)-F(x'),x-x')\ge 0$2 last-iterate gradient-norm rate, while deterministic power-law schedules accelerate the shared-stepsize case to

$(F(x)-F(x'),x-x')\ge 0$3

and allowing different extrapolation and update stepsizes further improves the rate to

$(F(x)-F(x'),x-x')\ge 0$4

The same work proves the shared-stepsize rate is tight up to $(F(x)-F(x'),x-x')\ge 0$5, via a matching lower bound (Wu et al., 1 Jun 2026).

5. Stochastic, shuffled, and variance-reduced extragradient methods

The stochastic theory of EG is more delicate than the deterministic one because the midpoint correction must be approximated accurately enough at the epoch or sample level. In finite-sum minimax problems,

$(F(x)-F(x'),x-x')\ge 0$6

same-sample stochastic extragradient (SEG) uses

$(F(x)-F(x'),x-x')\ge 0$7

A negative result is that SEG-US, SEG-RR, and SEG-FF can all diverge in expectation on a convex-concave monotone problem for any positive stepsizes, even with bounded gradient variance (Chae et al., 2024).

The remedy proposed there is flip-flop anchoring, SEG-FFA. Within each epoch it performs forward and reverse SEG passes with half-step extrapolation and then anchors the epoch by

$(F(x)-F(x'),x-x')\ge 0$8

Its key matching property is

$(F(x)-F(x'),x-x')\ge 0$9

whereas SEG-RR and SEG-FF incur only $L$0 epoch matching. Under star-monotonicity, smoothness, and bounded variance,

$L$1

and in the strongly monotone regime

$L$2

This sharpens the corresponding $L$3 lower-bound regime for baseline shuffling methods (Chae et al., 2024).

Variance reduction yields another line of development. For monotone VIs satisfying an error-bound condition,

$L$4

SVRG-EG attains last-iterate linear convergence under assumptions much weaker than strong monotonicity, including settings such as bilinear saddle-point problems. Under weak sharpness it obtains a stronger linear guarantee, and increasing iterate averaging preserves an $L$5 gap rate (Nan et al., 2023).

A different stochastic scaling idea is player sampling in multiplayer games. Doubly-stochastic extragradient (DSEG) replaces the full simultaneous gradient by a mini-batch over players, reducing the gradient work per iteration from $L$6 to $L$7. In non-smooth noisy convex games the refined bound

$L$8

improves the noise term relative to full EG, and the approach is reported to train faster and better GANs and mixtures of GANs, particularly with cyclic sampling heuristics (Enrich et al., 2019).

6. Generalizations, comparisons, and frontier directions

EG has accumulated a large family of generalizations. A unifying template for nonlinear equations is

$L$9

where the search direction $\|F(x)-F(x')\|\le L\|x-x'\|.$0 may be $\|F(x)-F(x')\|\le L\|x-x'\|.$1, $\|F(x)-F(x')\|\le L\|x-x'\|.$2, or an affine combination of current and past evaluations. This template recovers classical EG, EG+, past-extragradient or Popov’s method, optimistic-gradient variants, Tseng’s forward-backward-forward splitting, reflected methods, and golden-ratio methods, and supports $\|F(x)-F(x')\|\le L\|x-x'\|.$3 best-iterate and, in several cases, last-iterate rates in both equations and inclusions (Tran-Dinh et al., 2024, Tran-Dinh, 2023).

Projection accuracy is another active axis. Feasible inexact projection variants replace exact projections by points $\|F(x)-F(x')\|\le L\|x-x'\|.$4 satisfying

$\|F(x)-F(x')\|\le L\|x-x'\|.$5

One constant-step version converges under pseudo-monotonicity and Lipschitz continuity, while a line-search version preserves the two-projection structure of classical EG and removes the Lipschitz assumption altogether (Millán et al., 2023).

The method also extends beyond Hilbert-space Euclidean VI settings. In Banach spaces, generalized metric projection and the duality mapping yield an extragradient scheme converging strongly to a common point of a generalized equilibrium problem, a variational inequality for an $\|F(x)-F(x')\|\le L\|x-x'\|.$6-inverse-strongly monotone operator, and fixed-point sets of relatively nonexpansive mappings (Jouymandi et al., 2015). In composite optimization, an extended extragradient method uses two proximal-gradient steps,

$\|F(x)-F(x')\|\le L\|x-x'\|.$7

and under the Kurdyka-Łojasiewicz assumption the generated sequence converges to a critical point and has finite length; in the convex case it achieves the standard $\|F(x)-F(x')\|\le L\|x-x'\|.$8 rate (Nguyen et al., 2016).

For nonconvex-nonconcave minimax optimization, EG now has two distinct roles. Under positive interaction dominance, damped EG is a first-order explicit surrogate for proximal point with linear convergence to the unique stationary point (Hajizadeh et al., 2022). Under negative comonotonicity, fast anchored and two-time-scale variants such as FEG achieve

$\|F(x)-F(x')\|\le L\|x-x'\|.$9

that is, an $$F(z)=\begin{bmatrix}\nabla_x f(x,y)\ -\nabla_y f(x,y)\end{bmatrix},\qquad z=\begin{bmatrix}x\ y\end{bmatrix},$$00 rate on the squared stationarity measure (Lee et al., 2021).

Comparison with neighboring methods remains central. EG is usually contrasted with GDA because GDA can diverge even in bilinear problems, whereas EG converges linearly there (Mokhtari et al., 2019, Zheng et al., 2024). It is also contrasted with OGDA: both can be viewed as proximal-point approximations, but EG uses a true midpoint evaluation whereas OGDA uses a negative-momentum or optimistic correction (Mokhtari et al., 2019). At the same time, alternative stabilization mechanisms continue to be explored; for example, dissipation-based GDA augments the state and requires only one gradient evaluation per iteration, and is compared theoretically and numerically against EG in bilinear and strongly convex-strongly concave settings (Zheng et al., 2024).

The current picture is therefore not that EG is a single fixed algorithm, but that it is a structural principle: an explicit two-stage correction scheme that approximates an implicit stable dynamics closely enough to recover convergence in monotone, adversarial, and weakly regularized regimes. Recent work on anchoring, adaptive smoothness models, dynamic stepsizes, stochastic epoch matching, and generalized operator frameworks indicates that the method’s central questions have shifted from basic convergence to finer issues of last-iterate behavior, stepsize optimality, stochastic robustness, and structural generality (Gorbunov et al., 2021, Boţ et al., 2024, Wu et al., 1 Jun 2026).