Projected Gradient Descent with Momentum

• Projected Gradient Descent with Momentum is a constrained smooth optimization method that augments projected gradient steps with momentum while preserving feasibility.
• The method integrates momentum through feasible displacements and an Armijo line-search, achieving an O(ε⁻²) iteration complexity for approximate stationarity.
• Empirical results show that PGMM outperforms standard spectral projected gradient methods in logistic regression and CUTEst problems with minimal extra cost.

Searching arXiv for the cited paper and closely related projected-gradient-with-momentum work. Projected gradient descent with momentum denotes a class of first-order methods for constrained smooth optimization that augment projected-gradient steps with momentum information while preserving feasibility with respect to a convex constraint set. In the formulation developed in "Projected Gradient Methods with Momentum" (Lapucci et al., 23 Jan 2026), the problem is

$\min f(x)\quad \text{subject to}\quad x\in C,$

where $C\subset \mathbb{R}^n$ is a nonempty closed convex set, $f:\mathbb{R}^n\to\mathbb{R}$ is continuously differentiable and bounded below on $C$, and the Euclidean projection $P_C(\cdot)$ is practically available. The central issue is that momentum is straightforward in unconstrained optimization but nontrivial under constraints: the method therefore integrates momentum through feasible projected displacements and an Armijo line-search, obtaining a scheme with theoretical guarantees, $O(\epsilon^{-2})$ iteration complexity for reaching approximate stationarity, and empirical improvements over the standard spectral projected gradient method on $\ell_1$-ball constrained logistic regression and box-constrained CUTEst problems (Lapucci et al., 23 Jan 2026).

1. Problem setting and stationarity

The analysis is carried out under the assumption that $f$ is $L$-smooth on an open convex set $D\supset C$, meaning

$C\subset \mathbb{R}^n$0

The projection operator is the Euclidean projector

$C\subset \mathbb{R}^n$1

In this setting, first-order stationarity on $C\subset \mathbb{R}^n$2 is expressed by

$C\subset \mathbb{R}^n$3

Equivalently, the paper uses the projected-gradient residual

$C\subset \mathbb{R}^n$4

for any $C\subset \mathbb{R}^n$5, with $C\subset \mathbb{R}^n$6 if and only if $C\subset \mathbb{R}^n$7 is stationary. This residual plays a dual role: it is both the optimality measure used in the complexity statements and the quantity that links feasible descent directions to stationarity in the convergence analysis (Lapucci et al., 23 Jan 2026).

This formulation places projected gradient descent with momentum in the regime of smooth, possibly nonconvex, constrained minimization rather than in the unconstrained heavy-ball setting. The constraint geometry is handled entirely through Euclidean projection and convexity of $C\subset \mathbb{R}^n$8, so feasibility is enforced at the level of direction construction and line-search rather than by a post hoc correction.

2. Feasible incorporation of momentum

In the unconstrained case $C\subset \mathbb{R}^n$9, the classical heavy-ball or Polyak iteration is

$f:\mathbb{R}^n\to\mathbb{R}$0

with step-size $f:\mathbb{R}^n\to\mathbb{R}$1 and momentum $f:\mathbb{R}^n\to\mathbb{R}$2. A direct projection-based extension is

$f:\mathbb{R}^n\to\mathbb{R}$3

The paper states that little is known on the convergence of this naïve extension beyond convex-nonsmooth cases. That observation isolates a common misconception: momentum cannot simply be transferred from the unconstrained heavy-ball method to the constrained setting by inserting a projection at the end of the update (Lapucci et al., 23 Jan 2026).

The proposed alternative constructs two base feasible displacements at iteration $f:\mathbb{R}^n\to\mathbb{R}$4: $f:\mathbb{R}^n\to\mathbb{R}$5 Here $f:\mathbb{R}^n\to\mathbb{R}$6 is chosen, for example, via the BB spectral rule. Both $f:\mathbb{R}^n\to\mathbb{R}$7 and $f:\mathbb{R}^n\to\mathbb{R}$8 lie in the tangent cone to $f:\mathbb{R}^n\to\mathbb{R}$9 at $C$0. The actual search direction is then restricted to the convex combination

$C$1

Because $C$2 is convex, this guarantees $C$3.

Two strategies are proposed for choosing $C$4. The first uses an $C$5-dimensional quadratic model with a positive-definite matrix $C$6, for example $C$7 or a spectrally scaled Hessian-approximation. The second uses a reduced $C$8 model: with $C$9 and $P_C(\cdot)$0, one solves

$P_C(\cdot)$1

where $P_C(\cdot)$2 is symmetric and the solution is obtained in closed form by checking the interior and the edges of the simplex. Practically, $P_C(\cdot)$3 is built by matching the Taylor model of $P_C(\cdot)$4 in the two-dimensional subspace $P_C(\cdot)$5 at three interpolation points.

3. PGMM algorithm

The concrete method developed in detail is PGMM, Projected Gradient with Momentum. It embeds the feasible momentum direction into a classic Armijo line-search framework. Starting from $P_C(\cdot)$6 and setting $P_C(\cdot)$7, the method uses parameters $P_C(\cdot)$8, $P_C(\cdot)$9, $O(\epsilon^{-2})$0, and spectral bounds $O(\epsilon^{-2})$1 (Lapucci et al., 23 Jan 2026).

At iteration $O(\epsilon^{-2})$2, the method computes

$O(\epsilon^{-2})$3

The spectral parameter is then set by

$O(\epsilon^{-2})$4

and $O(\epsilon^{-2})$5 otherwise. Using this $O(\epsilon^{-2})$6, the projected gradient and projected momentum displacements are

$O(\epsilon^{-2})$7

If $O(\epsilon^{-2})$8, the algorithm builds $O(\epsilon^{-2})$9 via 3-point interpolation of $\ell_1$0 on $\ell_1$1, computes $\ell_1$2 as a minimizer of the reduced model over $\ell_1$3, and sets

$\ell_1$4

Safeguards are then enforced: if

$\ell_1$5

for safeguards $\ell_1$6, then $\ell_1$7 is corrected to satisfy positive-definiteness bounds and the reduced subproblem is solved again. If $\ell_1$8, the method falls back to the pure projected-gradient displacement $\ell_1$9.

The step is accepted through a monotone Armijo backtracking procedure: $f$0 The update is $f$1. This design makes momentum conditional and controlled rather than unconditional.

4. Descent conditions, convergence, and complexity

The convergence theory is organized around a general line-search framework with a gradient-related feasible-direction assumption. There exist constants $f$2, $f$3, and $f$4 such that, for all $f$5:

1. $f$6 for all $f$7;
2. $f$8;
3. $f$9.

Under $L$0-smoothness and this assumption, classical descent and Lipschitz arguments show that any Armijo step-length satisfies

$L$1

The resulting worst-case complexity bound is that, to find $L$2 with $L$3, it is sufficient to perform at most

$L$4

iterations. The same bound applies to total unsuccessful steps and to the number of gradient and projection evaluations. If $L$5 is compact, any limit point of $L$6 is stationary (Lapucci et al., 23 Jan 2026).

The paper then establishes that the specific directions produced either by the $L$7 subproblem or by the reduced $L$8 $L$9 subproblem satisfy the required assumptions, provided $D\supset C$0 in the first case or $D\supset C$1 in the reduced model case. For PGMM itself, the stated corollary is that, under the smoothness assumption, the spectral rule $D\supset C$2 with $D\supset C$3, and the algorithmic construction above, the iteration complexity to achieve $D\supset C$4 is $D\supset C$5, and all accumulation points are first-order stationary.

A notable feature is that the complexity statement is given for smooth, possibly nonconvex objectives. This indicates that the method is analyzed in terms of first-order stationarity rather than global optimality.

5. Computational profile and implementation considerations

Per iteration, PGMM requires one gradient evaluation $D\supset C$6, two projections $D\supset C$7, up to three extra function evaluations to build $D\supset C$8 by interpolation, a $D\supset C$9 convex QP solution in closed form, and an Armijo line-search in which each trial costs one function evaluation (Lapucci et al., 23 Jan 2026). The paper states that the dominant costs are therefore gradient and projection operations, exactly as in standard projected-gradient methods, plus a small overhead of a few function evaluations.

The reduced subproblem is operationally important. All elements from the direction-selection step can be carried out in $C\subset \mathbb{R}^n$00 cost beyond the two projections and one gradient evaluation per iteration, since solving the $C\subset \mathbb{R}^n$01 subproblem or adjusting $C\subset \mathbb{R}^n$02 takes negligible time. This is the main reason the momentum mechanism does not alter the asymptotic per-iteration structure of a projected-gradient method.

The practical rationale for momentum is also stated explicitly: PGMM typically outperforms pure projected gradient whenever the extra curvature information from momentum speeds up convergence, for example in ill-conditioned or nearly orthogonal gradient regimes, while the projection cost remains moderate. This suggests that the method is most attractive when projections are not overwhelmingly more expensive than gradient computations and when the local geometry makes pure projected-gradient steps inefficient.

6. Empirical comparison and methodological scope

The reported experiments compare PGMM with the state-of-the-art Spectral Projected Gradient (SPG) solver on two benchmarks: $C\subset \mathbb{R}^n$03-ball constrained logistic-regression, using 100 instances drawn from LIBSVM, and box-constrained CUTEst problems, using 50 test problems (Lapucci et al., 23 Jan 2026). In both settings, the same nonmonotone SPG line-search machinery is used, except that PGMM employs a monotone Armijo rule; the stopping criterion is identical,

$C\subset \mathbb{R}^n$04

and the initializations are identical.

Performance profiles show that PGMM solves nearly all instances faster in CPU time and in fewer iterations than SPG. The overhead in function evaluations due to interpolation is described as minor, and PGMM never exceeds $C\subset \mathbb{R}^n$05 the SPG runtime on any problem. The experiments are presented as evidence that adding a rigorously controlled momentum term yields substantial practical gains in constrained smooth optimization, analogously to the unconstrained setting.

At the same time, the scope of the results is sharply delimited. The theory assumes a nonempty closed convex feasible set, smoothness of the objective on an open convex superset of the feasible region, and practical availability of Euclidean projection. The algorithmic guarantees are for first-order stationarity, not for stronger notions of optimality. The paper also distinguishes its framework from the naïve projected heavy-ball update, whose convergence theory remains limited beyond convex-nonsmooth cases. In that sense, projected gradient descent with momentum is not merely projected gradient plus inertia; in the analyzed formulation, it is a feasibility-preserving, model-based, line-search method in which momentum enters through constrained search-direction design rather than through an unconstrained extrapolation followed by projection.