---
title: Repeated Projected Gradient Descent (RPGD)
url: https://www.emergentmind.com/topics/repeated-projected-gradient-descent-rpgd
type: topic
---

# Repeated Projected Gradient Descent (RPGD)

Searching arXiv for recent and foundational papers on projected gradient descent and repeated projected gradient descent across quantum tomography, constrained optimization, low-rank recovery, and nonconvex stationarity.
“Repeated Projected Gradient Descent” (“RPGD”, *Editor’s term*) denotes the repeated application of a projected-gradient update of the form \(x_{k+1}=\mathcal P(x_k-\alpha_k\nabla f(x_k))\), or domain-specific variants with the same gradient-plus-projection structure, on constrained optimization problems whose feasible sets range from convex sets to nonconvex closed sets, structured-sparse models, low-rank varieties, and quantum-state spaces [2601.16683]. In the surveyed literature, the term itself is often implicit rather than explicit, but the underlying mechanism is stable across applications: a gradient step is computed in an ambient Euclidean or matrix space, and the resulting iterate is projected back onto a set that encodes feasibility, structure, or physical admissibility [2403.02530]. In quantum state tomography, for example, the basic update is written as \(\rho_{k+1}=\mathcal S[\rho_k-\delta\nabla\mathcal C(\rho_k)]\), with \(\mathcal S\) enforcing positivity and unit trace [1612.09531].

## 1. Canonical iteration and problem classes

At its most classical, RPGD addresses constrained smooth optimization problems of the form
\[
\min_{x\in\mathcal S} f(x),
\]
where \(\mathcal S\subset\mathbb R^n\) is a closed, convex set and \(f\) is smooth, possibly nonconvex. The baseline projected-gradient iteration is
\[
x_{k+1}=\mathcal P_{\mathcal S}[x_k-\alpha_k\nabla f(x_k)],
\]
or, in a direction-search formulation,
\[
\hat d_k=\mathcal P_{\mathcal S}[x_k-\eta_k\nabla f(x_k)]-x_k,\qquad x_{k+1}=x_k+\alpha_k\hat d_k.
\]
This setting underlies the general convergence and complexity theory developed for projected gradient methods with momentum [2601.16683].

A broader formulation allows \(\mathcal C\) to be merely nonempty and closed, possibly highly nonconvex, with set-valued metric projection
\[
P_C(x):=\operatorname*{argmin}_{y\in C}\|x-y\|.
\]
In that setting, one projected-gradient step is
\[
x_{k+1}\in P_C(x_k-\alpha_k\nabla f(x_k)),
\]
typically combined with an Armijo-type backtracking line search. The resulting infinite sequence is the object studied in the modern stationarity theory for PGD on arbitrary closed sets [2403.02530].

The same template appears in highly structured inverse problems. In model-based sparsity, the iteration is
\[
\chi^{(i)}=\theta^{(i)}-\eta^{(i)}\nabla f(\theta^{(i)}),\qquad
\theta^{(i+1)}=P_{\mathcal C_k,r}[\chi^{(i)}],
\]
with projection onto a model-sparse ball [1209.1557]. In low-rank matrix estimation, the update is
\[
X^{(t+1)}=P\bigl(X^{(t)}-\eta\nabla f(X^{(t)})\bigr),
\]
where \(P\) is truncated SVD onto rank-\(r\) matrices [2403.02704]. In spectral compressed sensing, projected gradient descent is applied to a non-convex factorized objective after a one-step hard-thresholding initialization [1707.09726]. In generalized low-rank tensor regression, projected gradient descent is repeatedly applied to a potentially non-convex tensor constraint set \(\Theta\) [1611.10349].

These formulations differ in geometry, but they share a common separation of roles: the gradient step optimizes the objective in ambient coordinates, whereas the projection step reimposes feasibility.

## 2. Projection geometry and feasible sets

The projection operator is the defining structural element of RPGD. In convex settings it is the Euclidean projection
\[
\mathcal P_{\mathcal S}[x]=\arg\min_{y\in\mathcal S}\|x-y\|,
\]
which is single-valued and central to the projected-gradient stationarity measure
\[
\phi_\eta(x)=\|\mathcal P_{\mathcal S}[x-\eta\nabla f(x)]-x\|
\]
used in general nonconvex-complexity analysis [2601.16683]. When the feasible set is only closed, the metric projection may be set-valued, and the algorithm may choose any element of \(P_C(x)\) [2403.02530].

Across applications, the projection map is adapted to the target structure.

| Feasible structure | Projection mechanism | Representative papers |
|---|---|---|
| Density matrices \(\rho\succeq 0,\ \mathrm{Tr}(\rho)=1\) | Eigen-decomposition plus simplex projection on eigenvalues using Michelot’s algorithm | [1612.09531] |
| Model-based sparsity \(\supp(\theta)\in \mathcal M(\mathcal C_k),\ \|\theta\|\le r\) | Best support in \(\mathcal C_k\), then scaling to radius \(r\) | [1209.1557] |
| Rank-\(r\) matrices | Best rank-\(r\) approximation by truncated SVD | [2403.02704] |
| Incoherence-constrained low-rank factors | Row-wise clipping onto \(\mathcal C_1\) or \(\mathcal C_2\) | [2401.06592], [1707.09726] |
| Box and balance constraints for graph partitioning | Projection onto \(\mathcal B_\infty\) and balance slabs via alternating projections, Dykstra, or exact projection for \(d\le 2\) | [1902.03522] |

In quantum tomography, the projection \(\mathcal S\) is implemented by diagonalizing a Hermitian matrix \(X=U\Lambda U^\dagger\), projecting the eigenvalue vector onto the probability simplex, and reconstructing \(U\,\mathrm{diag}(\mu)\,U^\dagger\) [1612.09531]. In structured sparsity, the bounded projection is reduced to an unbounded model projection followed by a radial scaling step [1209.1557]. In low-rank optimization, truncated SVD realizes projection onto \(\{X:\operatorname{rank}(X)\le r\}\) and is therefore the analogue of Euclidean projection for the determinantal variety [2403.02704].

For nonconvex sets, projection regularity becomes a central analytical issue. The local least-squares theory of PGD uses differentiability and Lipschitz-continuous differentiability of \(P_{\mathcal C}\) at the solution and at the post-gradient point \(z_\eta^*=x^*-\eta A^\top(Ax^*-b)\), yielding an exact asymptotic linear rate through the linearized map
\[
H=\nabla P_{\mathcal C}(z_\eta^*)(I-\eta A^\top A)\nabla P_{\mathcal C}(x^*)
\]
[2112.11760]. A plausible implication is that, in RPGD, projection design is not a secondary implementation detail; it is often the decisive object controlling both local geometry and attainable rates.

## 3. Algorithmic variants, acceleration, and related projected schemes

The simplest RPGD scheme uses a single fixed or line-searched projected gradient step per iteration, but the literature contains several systematic variants. In quantum tomography, three projected-gradient algorithms are studied: Projected Gradient Descent with Momentum (PGDM), FISTA, and Projected Gradient Descent with Backtracking (PGDB) [1612.09531]. PGDM updates a velocity term,
\[
M_{k+1}=\zeta_k M_k-\gamma_k\nabla\mathcal C(\rho_k),\qquad
\rho_{k+1}=\mathcal S(\rho_k+M_{k+1}),
\]
whereas FISTA uses Nesterov-type extrapolation,
\[
\rho_{k+1}=\mathcal S\!\left[\rho_k+\frac{k-2}{k+1}(\rho_k-\rho_{k-1})-\delta\nabla\mathcal C(\rho_k)\right].
\]
PGDB instead enforces monotonic decrease by backtracking along a projected direction [1612.09531].

A more abstract momentum construction is developed for convex-constrained smooth optimization in the Projected Gradient Method with Momentum (PGMM). Rather than inserting a raw inertial term inside the projection, PGMM builds two projected directions,
\[
\hat d_k=\mathcal P_{\mathcal S}[x_k-\eta_k\nabla f(x_k)]-x_k,\qquad
\hat s_k=\mathcal P_{\mathcal S}[x_k+(x_k-x_{k-1})]-x_k,
\]
and combines them as
\[
d_k=\alpha_k\hat d_k+\beta_k\hat s_k,\qquad \alpha_k,\beta_k\ge 0,\ \alpha_k+\beta_k\le 1,
\]
with \((\alpha_k,\beta_k)\) chosen by a two-dimensional quadratic program and an Armijo line search on \(d_k\) [2601.16683]. The paper explicitly notes that the acronym RPGD is not used there, but conceptually the method is a repeated projected-gradient scheme with momentum [2601.16683].

Preconditioning and scaling form another variant class. In deterministic matrix completion, standard PGD is applied to a lifted balanced objective \(L_1\), while a scaled PGD variant applies gradient steps preconditioned by \((Y^\top Y)^{-1}\) and \((X^\top X)^{-1}\) on the unregularized factorized objective \(L_2\), followed by projection onto an incoherence set [2401.06592]. In low-rank matrix estimation, a perturbed projected-gradient method interleaves ordinary projected steps with tangent-space perturbation phases when progress stalls, in order to escape saddle points and converge to an approximate solution or a second-order local minimizer [2403.02704].

Projected-projected gradient descent (PPGD) is closely related but distinct. In low-rank optimization on the determinantal variety, the method first projects \(-\nabla f(X)\) onto the Bouligand tangent cone, then projects the trial point back to the rank-bounded set. A new “P with rank reduction” algorithm augments this with rank-reduced projections and selects the best candidate decrease at each iteration [2201.03962]. This suggests that, within the broader RPGD family, the gradient direction itself may be projected before the iterate is projected.

## 4. Convergence, stationarity, and complexity

The convergence theory of RPGD is fragmented across geometric regimes. For smooth, convex-constrained optimization, the modern baseline is a worst-case nonconvex complexity of \(O(\epsilon^{-2})\) to reach \(\phi_\eta(x_k)\le\epsilon\), where
\[
\phi_\eta(x)=\|\mathcal P_{\mathcal S}[x-\eta\nabla f(x)]-x\|.
\]
This bound is proved for classical projected gradient directions under Armijo line search, and the same \(O(\epsilon^{-2})\) order is retained by the momentum-enhanced PGMM framework [2601.16683].

For arbitrary nonempty closed sets \(C\), the stationarity picture is sharper but more geometric. Under continuous differentiability, every accumulation point of PGD is Bouligand stationary, and under locally Lipschitz continuous gradient on the ambient space, every accumulation point is proximally stationary [2403.02530]. The same work emphasizes that Bouligand and proximal stationarity are the strongest stationarity properties that can be expected in that setting, and contrasts PGD with another projected method, P\(^2\)GD, that can converge to Mordukhovich-stationary points which are not Bouligand stationary [2403.02530].

On the determinantal variety \(\mathbb R_{\le r}^{m\times n}\), first-order low-rank methods face the specific pathology called an apocalypse: a convergent sequence whose limit is not Bouligand stationary. The rank-reduction-enhanced projected-projected scheme of [2201.03962] is designed precisely to avoid that phenomenon. Its finite termination point is stationary, and every accumulation point of an infinite sequence is Bouligand stationary [2201.03962].

Local linear theory is particularly explicit for constrained least squares. If \(x^*\) is a fixed point of
\[
x^{(k+1)}=P_{\mathcal C}(x^{(k)}-\eta A^\top(Ax^{(k)}-b)),
\]
and the projection is Lipschitz-continuously differentiable at \(x^*\) and at \(z_\eta^*=x^*-\eta A^\top(Ax^*-b)\), then the asymptotic linear rate is the spectral radius \(\rho(H)\) of the linearized map \(H\), and the paper gives an explicit region of convergence and an iteration bound to reach any relative accuracy \(\epsilon\) [2112.11760]. In manifold-like cases, the rate simplifies to the familiar restricted-Hessian expression
\[
\rho(H)=\max\{|1-\eta\lambda_1|,\ |1-\eta\lambda_d|\},
\]
where \(\lambda_1,\lambda_d\) are the largest and smallest eigenvalues of the Hessian restricted to the tangent space [2112.11760].

Several application-specific linear convergence results complement this general theory. In deterministic matrix completion with Ramanujan sampling, PGD on the lifted balanced objective converges linearly, with a rate that depends on the condition number \(\kappa\) of the ground truth, while scaled PGD achieves a linear rate independent of \(\kappa\) under its local curvature and smoothness conditions [2401.06592]. In low-rank matrix estimation under rank-\(2r\) restricted \(L\)-smoothness and \(\mu\)-strong convexity, projected gradient descent has a local convergence rate independent of the effective condition number \(\kappa(X_*)\); when \(L/\mu<3\), it also enjoys global linear convergence for an explicit interval of step sizes, and the constrained landscape has no spurious local minimizers [2403.02704].

In quantum tomography, convergence is studied empirically rather than through a general theorem. PGDB is monotonic in the likelihood cost, whereas PGDM and FISTA are non-monotonic but converge rapidly in practice; all three methods reach the same maximum-likelihood state in the reported experiments [1612.09531].

## 5. Representative application domains

RPGD appears in the literature as a reusable computational pattern rather than as a single domain-specific algorithm.

| Domain | Representative formulation | Representative papers |
|---|---|---|
| Quantum state tomography | Maximum-likelihood estimation over density matrices \(\rho\succeq 0,\ \mathrm{Tr}(\rho)=1\) | [1612.09531] |
| Model-based sparsity and GLMs | \(\arg\min_{\theta\in H} f(\theta)\) subject to \(\supp(\theta)\in\mathcal M(\mathcal C_k)\) | [1209.1557] |
| Generalized low-rank tensor regression | PGD on non-convex tensor constraint sets \(\Theta\) | [1611.10349] |
| Spectral compressed sensing | PGD on a factorized low-rank Hankel objective with incoherence projection | [1707.09726] |
| Low-rank matrix completion and estimation | Rank-constrained PGD, scaled PGD, and perturbed projected gradient methods | [2401.06592], [2403.02704], [2201.03962] |
| Multi-dimensional balanced graph partitioning | Randomized PGD on a non-convex continuous relaxation with box and balance constraints | [1902.03522] |

In quantum tomography, RPGD solves a convex maximum-likelihood problem over density matrices from noisy count data. The Gaussian negative log-likelihood cost
\[
\mathcal C(\rho)=\sum_{i=1}^N\left(\frac{r\,\mathrm{Tr}(\Pi_i\rho)-n_i}{\sqrt{n_i}}\right)^2
\]
is minimized subject to positivity and unit trace, with per-iteration gradient evaluation and Born-rule evaluation costing \(O(Nd^2)\) [1612.09531]. The paper reports that PGD techniques reach \(\rho_{\mathrm{ML}}\) significantly faster than the diluted iterative algorithm and SDPT3 in the vast majority of scenarios [1612.09531].

In structured sparsity, the relevant constraint is a non-convex union of subspaces induced by a family of allowed supports \(\mathcal C_k\). Under a Stable Model-Restricted Hessian condition, the projected gradient algorithm contracts linearly toward a model-consistent reference point, with an explicit decomposition into optimization error and approximation error [1209.1557]. The same paper treats canonical generalized linear models as a principal application [1209.1557].

In generalized low-rank tensor regression, projected gradient descent is analyzed on potentially non-convex low-rank tensor sets through localized Gaussian width. The resulting statistical error scales with
\[
w_G[\Theta(t_0)\cap\mathbb B_F(1)]/\sqrt n,
\]
and the paper argues that the non-convex approach has a superior rate for a number of examples relative to convex regularization [1611.10349]. Low Tucker-rank estimation is the clearest example: the non-convex PGD bound depends on a minimum over matricization dimensions, whereas the convex bound depends on a maximum [1611.10349].

In spectral compressed sensing, the low-rank structure is transferred to the Hankel matrix \(Hx\). After one-step hard thresholding and projection, PGD is applied to a non-convex factorized objective that enforces both Hankel structure and data fidelity. Under a sampling-with-replacement model, \(O(r^2\log n)\) observed entries are sufficient for successful recovery in the noiseless theorem, and the iterates converge linearly to the solution manifold [1707.09726].

In graph partitioning, randomized projected gradient descent is used on the continuous relaxation of multi-dimensional balanced graph partitioning. The iterate update is
\[
y^{(t+1)}=(I+\gamma_t A)x^{(t)},\qquad x^{(t+1)}=\Pi_K(y^{(t+1)}),
\]
with \(K\) the intersection of the box \([-1,1]^n\) and multiple balance slabs. Exact projection for \(d\le 2\) has running time \(O(|E|+|V|\log^{d-1}|V|)\), and experiments on graphs containing up to hundreds of billions of edges show superior performance to the compared approaches [1902.03522].

## 6. Limitations, misconceptions, and recurring trade-offs

A common misconception is that all projected first-order methods have essentially the same limiting behavior. The recent stationarity literature does not support that claim. PGD on arbitrary closed sets accumulates at Bouligand stationary points, and even proximally stationary points under locally Lipschitz gradient, whereas other projected methods can accumulate at weaker Mordukhovich-stationary points [2403.02530]. Likewise, low-rank projected-projected methods can suffer apocalypses unless the algorithm is modified with rank reduction [2201.03962].

Another misconception is that adding momentum automatically preserves the convergence profile of baseline PGD. In the tomography literature, PGDM and FISTA are explicitly non-monotonic, even though they are often faster in practice; only PGDB is monotonic in the cost [1612.09531]. In the abstract momentum framework, additional safeguards, spectral bounds on the local model, and Armijo line search are required to retain the standard projected-gradient complexity guarantees [2601.16683].

Projection cost is the main practical bottleneck in many RPGD instantiations. Exact model projections for general combinatorial sparsity can be difficult, and the structured-sparsity paper points to the need for specialized algorithms or approximate projections [1209.1557]. In graph partitioning, exact Euclidean projection is practical only for small numbers of balancing dimensions; for larger \(d\), alternating projections and Dykstra-type procedures become the scalable alternative [1902.03522]. In low-rank varieties, larger numerical-rank thresholds can force exploration of multiple rank levels per iteration, increasing cost [2201.03962].

Finally, global guarantees are highly problem-dependent. Local linear convergence for constrained least squares depends on projection smoothness and initialization inside an explicit region of convergence [2112.11760]. Global linear convergence in low-rank matrix estimation requires the restricted-smoothness/strong-convexity regime with \(L/\mu<3\) [2403.02704]. In quantum tomography, the main evidence is empirical rather than theorem-level global analysis [1612.09531]. This suggests that “RPGD” names a structural algorithmic pattern, not a single theorem: its behavior is determined jointly by objective regularity, feasible-set geometry, projection exactness, and step-selection strategy.

Source: https://www.emergentmind.com/topics/repeated-projected-gradient-descent-rpgd