Last-Iterate Convergence of Anchored Gradient Descent

Abstract: We study the monotone inclusion problem $0\in F(z)+A(z)$, where $F$ is monotone and Lipschitz, and $A$ is maximally monotone, a framework that encompasses monotone variational inequalities and convex-concave saddle-point problems with constraints or regularization. It is well known that vanilla gradient descent diverges for this problem, whereas optimism-based methods such as Extragradient and accelerated methods that combine both optimism and anchoring, such as Extra Anchored Gradient, achieve last-iterate convergence. However, the anchoring-only method, anchored gradient descent, has been studied only in the unconstrained setting [RYY19, SST+26]. In this note, we extend the anchored gradient descent method to the monotone inclusion problem and prove a last-iterate convergence rate of $O(1/\sqrt{T})$ in terms of the tangent residual. We build on the recent proof in the unconstrained setting [SST+26] and use techniques from [COZ24] to extend it to the general inclusion setting.

• The paper demonstrates that Proximal Anchored Gradient Descent (P-AGD) achieves an O(1/√T) last-iterate convergence rate for composite monotone inclusion problems.
• It employs a proximal update via the resolvent operator to generalize anchoring techniques from unconstrained settings to those with constraints and regularizers.
• The analysis clarifies convergence landscapes, establishing anchoring as a robust approach compared to extragradient methods in various optimization applications.

Last-Iterate Convergence of Anchored Gradient Descent: Theory and Implications

Problem Formulation and Motivation

The paper addresses the composite monotone inclusion problem, $0 \in F(z) + A(z)$, where $F$ is a monotone, $L$-Lipschitz operator ($F: \mathbb{R}^d \to \mathbb{R}^d$), and $A$ is maximally monotone ($A: \mathbb{R}^d \rightrightarrows \mathbb{R}^d$). This framework encapsulates monotone variational inequalities, convex-concave saddle-point problems, and broad classes of constrained optimization and regularized learning objectives.

Gradient descent (GD), i.e., forward-backward splitting, is known to be divergent for such saddle-point and monotone inclusion problems unless additional correction mechanisms are introduced. Standard mechanisms to guarantee convergence in the last iterate include extragradient-style "optimism" and anchoring. Prior results on anchoring-only methods, specifically anchored gradient descent, were limited to the unconstrained case where $A=0$.

Methodology and Main Algorithm

The central algorithm investigated is Proximal Anchored Gradient Descent (P-AGD), formally defined as:

$$z_{t+1} = J_{\alpha_t A} \bigl(z_t - \alpha_t F(z_t) + \beta_t (z_0 - z_t)\bigr)$$

where $J_{\alpha_t A} = (I + \alpha_t A)^{-1}$ denotes the resolvent operator of $A$. For special cases, $F$0 recovers projection or proximal mappings, depending on whether $F$1 encodes constraint sets or regularizers. The step size/weight schedules are:

$F$2

This extends and generalizes the anchoring-only schedule of [surina2026anchored] for unconstrained monotone operators to fully composite settings.

The optimization progress is measured via the tangent residual:

$F$3

which generalizes stationarity; for $F$4, this reduces to the operator norm $F$5.

Theoretical Results

The primary theoretical contribution is a proof that P-AGD achieves a last-iterate convergence rate of $F$6 in the tangent residual for composite monotone inclusions. Specifically, for the iterates $F$7 generated by P-AGD, for all $F$8:

$F$9

where $L$0 is a moderate, explicitly computable constant (e.g., $L$1).

The proof leverages the resolvent structure and introduces iterates $L$2 via the resolvent identity, allowing the P-AGD update to be rewritten in a recursion amenable to standard stability analysis. By induction and norm inequalities, the authors upper bound both the deviation from the anchor and the inter-iterate differences, circumventing the need for strong convexity, smoothness, or specific structure beyond monotonicity and Lipschitz continuity.

A secondary but conceptually important result is the clarification of the landscape of convergence rates for first-order algorithms for monotone inclusions:

• Vanilla GD: Divergent, even for simple bilinear problems.
• Extragradient (EG) / Optimistic GDA: Last-iterate $L$3 rate via look-ahead correction (optimism).
• Anchored GD (AGD) and P-AGD: Last-iterate $L$4 via anchoring only.
• Optimism + Anchoring (e.g., EAG/FEG): Achieve the optimal $L$5 last-iterate rate.

The composite generalizations to $L$6 for EAG and FEG achieve accelerated rates and are covered in [cai2024accelerated] and [kovalev2022first].

Comparisons and Context

A key distinction is that the previous anchoring-only last-iterate convergence results, such as [ryu2019ode] and [surina2026anchored], were strictly for unconstrained or single-operator monotone problems ($L$7). The extension to the composite/proximal setting required careful adaptation of both the update rule (proximal mapping replacing simple subtraction) and the theoretical analysis (handling the set-valued nature and non-expansivity of the resolvent).

Moreover, the $L$8 rate—though suboptimal compared to $L$9—is significant for applications where projections or proximal operators are inexpensive or where algorithmic simplicity is prioritized. This provides a new baseline for "anchor-only" approaches in the presence of constraints or non-smooth regularization.

Practical and Theoretical Implications

From an algorithmic perspective, this work shows that anchoring is a robust mechanism that, by itself (without optimism or extragradient steps), stabilizes the iterates in the presence of arbitrary constraints and composite objectives. The result unifies analysis for projection-based, proximal, and unconstrained settings under a common framework reliant only on monotonicity and Lipschitz properties.

This convergence guarantee for the last-iterate, as opposed to averages, is critical for problems where feasibility and interpretability of individual iterates are required (e.g., in training GANs, equilibrium computation, or certain online learning protocols).

On the theory side, the work closes a gap in the literature regarding the power and limitations of anchoring. It delineates which acceleration mechanisms are prerequisite to achieve optimal rates and which are sufficient for mere stability.

Future Directions

Future research may focus on:

• Lower bounds for anchored methods in various composite setups.
• Adaptive or parameter-free schedules for $F: \mathbb{R}^d \to \mathbb{R}^d$0 and $F: \mathbb{R}^d \to \mathbb{R}^d$1.
• Extensions to stochastic settings and saddle-point structures beyond monotone inclusions.
• Empirical characterization and tradeoffs in high-dimensional and non-Euclidean geometries.
• Practical acceleration via hybridization with optimism or variance-reduction mechanisms.

Conclusion

The paper rigorously extends anchored gradient descent to the composite monotone inclusion framework, establishing an $F: \mathbb{R}^d \to \mathbb{R}^d$2 last-iterate rate for the tangent residual. This result completes the picture of anchoring-based convergence mechanisms, enabling their application in a wide range of structured optimization, game-theoretic, and machine learning contexts. The techniques serve as a foundation for further advances in monotone operator splitting and algorithmic acceleration for modern large-scale optimization.