---
title: Scaled Gradient Descent (ScaledGD)
url: https://www.emergentmind.com/topics/scaled-gradient-descent-scaledgd
type: topic
---

# Scaled Gradient Descent (ScaledGD)

Searching arXiv for relevant Scaled Gradient Descent papers to ground the article in the provided research context.
Scaled Gradient Descent (ScaledGD) denotes a family of first-order optimization methods in which the raw gradient is multiplied by an explicit scaling or preconditioning operator before the parameter update. Across the literature, the term is used in several related but non-identical senses: diagonal or projected scaled gradients in constrained convex optimization [1507.07302], Gram-preconditioned factor updates for low-rank matrix and tensor estimation [2005.08898], Tucker- and t-SVD-based tensor recovery algorithms [2206.09109], [2501.01696], damped preconditioning for overparameterized matrix sensing [2302.01186], Hessian-aware scalar gradient scalings [2502.03701], and task-specific gradient rescaling schemes for classification fine-tuning [2604.27987]. The unifying principle is to alter the local metric of descent so that the update better matches curvature, factorization symmetry, constraint geometry, or sample-dependent optimization structure. In the low-rank estimation line of work, ScaledGD is especially associated with condition-number-robust linear convergence obtained by preconditioning each factor gradient with inverses of small Gram matrices [2005.08898], [2206.09109].

## 1. Core definition and algorithmic forms

The most common contemporary use of “ScaledGD” arises in nonconvex low-rank factorization. For a rank-\(r\) matrix parameterized as \(X=LR^\top\), Tong, Ma, and Chi define the factorized loss \(L(L,R)=f(LR^\top)\) and update the factors by
\[
L_{t+1}=L_t-\eta_t\nabla_LL(L_t,R_t)(R_t^\top R_t)^{-1},\qquad
R_{t+1}=R_t-\eta_t\nabla_RL(L_t,R_t)(L_t^\top L_t)^{-1},
\]
with
\[
\nabla_LL(L,R)=\bigl(\nabla f(LR^\top)\bigr)R,\qquad
\nabla_RL(L,R)=\bigl(\nabla f(LR^\top)\bigr)^\top L
\]
[2005.08898]. The scaling matrices are thus iteration-varying, low-dimensional, and adapted to the opposing factor.

This same structural idea appears in tensor factorization. In Tucker-form tensor RPCA, one observes
\[
X=X_\star+S_\star,\qquad
X_\star=(U_\star^{(1)},U_\star^{(2)},U_\star^{(3)})\bcdot C_\star,
\]
and minimizes
\[
L(F,S)=\tfrac12\Big\|(U^{(1)},U^{(2)},U^{(3)})\bcdot C+S-X\Big\|_F^2
\]
over \(F=(U^{(1)},U^{(2)},U^{(3)},C)\) and \(S\), with scaled updates for each mode and the core tensor [2206.09109]. In t-SVD/t-product tensor estimation, the factorization \( \mathcal X_\star=U_\star\ast V_\star^H \) leads to updates of the form
\[
U_{t+1}=U_t-\eta\,\nabla_U f(U_t,V_t)\ast G_t^{-1},\qquad
V_{t+1}=V_t-\eta\,\nabla_V f(U_t,V_t)\ast G_t^{-1},
\]
where \(G_t=V_t^H\ast V_t\) is the Gram tensor [2501.01696]. A related t-SVD robust tensor PCA formulation uses
\[
L_{k+1}=L_k-\eta_{k+1}\nabla_L f\,(R_k^\top * R_k)^{-1},\qquad
R_{k+1}=R_k-\eta_{k+1}\nabla_R f\,(L_k^\top * L_k)^{-1}
\]
together with sparse-threshold updates [2501.04565].

Outside low-rank estimation, the term can refer to broader variable-metric methods. In projected scaled gradient methods for convex optimization, the iterate is
\[
x^{k+1}=P_C\!\bigl(x^k-\tau_kD_k\nabla f(x^k)\bigr),
\]
with diagonal positive-definite \(D_k\) and Euclidean projection \(P_C\) onto a convex set \(C\) [1507.07302]. In Hessian-aware “first-ish order” methods, the scaling is a scalar \(s_k>0\) chosen from curvature information so that \(p_k=-s_kg_k\) satisfies a second-order descent condition [2502.03701]. This suggests that ScaledGD is best understood as a design pattern rather than a single algorithm.

## 2. Preconditioning, symmetry, and conditioning

In low-rank factorization, the main motivation for scaling is to neutralize the bad geometry induced by ill-conditioning and factor non-uniqueness. The matrix product \(LR^\top\) is invariant under the reparameterization \((L,R)\mapsto(LM,RM^{-T})\) for any \(M\in\mathrm{GL}(r)\). Standard gradient descent does not naturally respect this geometry, whereas ScaledGD uses the factor Gram matrices \(R_t^\top R_t\) and \(L_t^\top L_t\) as inexpensive local preconditioners [2005.08898], [2310.06159].

The 2020 analysis makes this precise through a distance that both aligns the factors by an invertible matrix \(Q\) and weights errors by \(\Sigma_\star^{1/2}\):
\[
\dist^2(F,F_\star)
:=\inf_{Q\in\GL(r)}
\bigl\|\,(LQ-L_\star)\Sigma_\star^{1/2}\bigr\|_F^2
+\bigl\|\,(RQ^{-\top}-R_\star)\Sigma_\star^{1/2}\bigr\|_F^2
\]
[2005.08898]. Under restricted strong convexity and smoothness over rank-\(2r\) matrices, this metric contracts linearly with a rate independent of the matrix condition number \(\kappa=\sigma_1/\sigma_r\), provided the initialization lies in a suitable local basin.

The condition-number robustness is a defining claim of the low-rank ScaledGD literature. For matrix sensing, robust PCA, and completion, the 2020 results show linear convergence with iteration count \(O(\ln(1/\epsilon))\), independent of \(\kappa\), in contrast to vanilla GD whose convergence depends linearly on \(\kappa\) [2005.08898]. The chapter-length synthesis in 2023 emphasizes the same point and interprets the Gram preconditioners as promoting desirable invariance properties without sacrificing the low per-iteration cost of factor-space GD [2310.06159].

A similar mechanism appears in Tucker and t-SVD tensor settings. In the Tucker RPCA algorithm, the update for each factor uses the inverse of a “preconditioned factor” Gram matrix derived from the current core tensor matricization, while the core update is scaled by inverse factor Grams in each mode [2206.09109]. In simultaneous blind super-resolution and demixing, the authors explicitly state that the factors \((R^HR)^{-1}\) and \((L^HL)^{-1}\) “precondition” the gradient so that the method becomes robust to ill-conditioning and no extra balancing regularizer is needed [2407.09900].

## 3. Spectral initialization and basin-entry mechanisms

A recurring feature of ScaledGD in nonconvex low-rank estimation is the need for a tailored initialization that places the iterate inside a contraction region. The canonical construction is spectral.

For matrix recovery under Gaussian measurements, one computes \(Y=\mathcal A^*(y)\), extracts its top-\(r\) SVD \(Y=\widetilde V\widetilde\Sigma\widetilde W^\top\), and sets
\[
L_0=\widetilde V\widetilde\Sigma^{1/2},\qquad
R_0=\widetilde W\widetilde\Sigma^{1/2}
\]
[2604.00060]. This “warm start” ensures
\[
\|L_0R_0^\top-M^*\|\le c_0\,\sigma_{\min}(M^*)
\]
under suitable sampling conditions [2604.00060].

In Tucker tensor RPCA, the spectral step is preceded by trimming:
\[
\widetilde X_0=T_{\zeta_0}(X),\qquad
(U_0^{(1)},U_0^{(2)},U_0^{(3)},C_0)=\mathrm{HOSVD}_r(\widetilde X_0)
\]
[2206.09109]. Under a corruption fraction
\[
\alpha\le c_0/(\mu^2\,r_1r_2r_3\,\kappa),
\]
the initialization obeys
\[
\dist(F_0,F_\star)\le\epsilon_0\,\sigma_{\min}(X_\star),
\]
thereby entering the basin in which ScaledGD exhibits global linear convergence [2206.09109].

In t-SVD tensor estimation, the initialization is also spectral but specialized to the corruption model. For robust PCA, one thresholds the observation tensor, performs a rank-\(r\) truncated t-SVD, and splits the singular tensor symmetrically into factor tensors. For completion, one forms the scaled zero-filled observation \(\tfrac1p\mathcal P_\Omega(Y)\), takes its t-SVD, and then projects the factors onto a scaled \(\ell_{2,\infty}\)-ball to enforce incoherence [2501.01696].

The importance of initialization extends beyond merely attaining a local basin. A 2024 study on matrix factorization shows that Nyström initialization can qualitatively change the convergence regime of ScaledGD. In the symmetric case, setting \(X_0=A\Omega\) with Gaussian sketch \(\Omega\) places the iterate in the column space of \(A\), and ScaledGD then exhibits a two-phase behavior: a linear phase followed by a quadratic phase [2410.18965]. In the asymmetric exact-parametrized setting, the same work proves one-step convergence with \(\eta=1\):
\[
X_1Y_1^\top=A
\]
under Nyström initialization \(X_0=A\Omega,\;Y_0=0\) [2410.18965]. This suggests that, in some factorizations, initialization is not merely a technical device but a determinant of the attainable local rate.

## 4. Theoretical guarantees across major problem classes

The best-developed theoretical guarantees for ScaledGD concern linear convergence independent of condition number in low-rank estimation.

For general low-rank matrix estimation with restricted strong convexity and smoothness, if
\[
\dist(F_0,F_\star)\le0.1\,\sigma_r(X_\star)/\sqrt{\kappa_f},\qquad
0<\eta\le0.4/L,
\]
then
\[
\dist(F_t,F_\star)\le(1-0.7\,\eta\,\mu)^t\bigl(0.1\,\sigma_r(X_\star)/\sqrt{\kappa_f}\bigr)
\]
[2005.08898]. Problem-specific instantiations include matrix sensing, robust PCA, and matrix completion, all with iteration count \(O(\ln(1/\epsilon))\) independent of the target matrix condition number \(\kappa\) [2005.08898].

For Tucker tensor RPCA, Theorem 1 establishes that under \(\mu\)-incoherence, Tucker rank \((r_k)\), and fractional sparsity
\[
\alpha\le c_0/(\mu^2r_1r_2r_3\kappa),
\]
with \(\eta\in[1/7,1/4]\) and geometrically decaying threshold schedule \(\zeta_{t+1}=\rho\zeta_t,\;\rho=1-0.45\eta\), the iterates satisfy
\[
\|X_t-X_\star\|_F\le0.03\,\rho^t\,\sigma_{\min}(X_\star),
\]
with a rate \(\rho\) independent of \(\kappa\) [2206.09109]. The theorem also gives an \(\ell_\infty\)-error bound and implies exact recovery in \(O(\log(1/\epsilon))\) steps [2206.09109].

For t-SVD robust tensor PCA, under \(\mu\)-incoherence and slice-wise sparsity
\[
\alpha\le \frac{1}{10^{4}\,\mu\,R^{1.5}\,I_3^{1.5}\,\kappa},
\]
Theorem 1 states that
\[
\tau=1-0.6\,\eta
\]
is the convergence rate and is independent of \(\kappa\), together with explicit Frobenius, \(\ell_\infty\), and sparse-support guarantees [2501.04565]. The broader 2025 t-SVD tensor estimation paper proves analogous condition-number-free linear convergence for both tensor RPCA and tensor completion [2501.01696].

For blind super-resolution and demixing formulated as low-rank matrix demixing, the main theorem states that if
\[
\eta_t\le\frac1{20},\qquad
n\ge C\,K^2s^2r^2\kappa^2\mu_0\mu_1\log^2(sn),
\]
then with high probability
\[
\sum_{k=1}^K \|L_{k,t}R_{k,t}^H-Z_{k,\natural}\|_F^2
\le
\Bigl(1-\tfrac{\eta_t}{90}\Bigr)^t\frac{\delta^2\sigma_0^2}{K},
\]
again with convergence rate independent of \(\kappa\) [2407.09900].

A more recent 2026 result sharpens the matrix recovery theory under Gaussian designs. It shows that ScaledGD for general low-rank matrix recovery achieves both sample complexity \(O((n_1+n_2)r)\) and iteration complexity \(O(\log(1/\epsilon))\), whereas earlier ScaledGD analyses had \(O((n_1+n_2)r^2)\) sample complexity [2604.00060]. The global linear convergence theorem gives
\[
\dist\bigl((L_t,R_t),(L^*,R^*)\bigr)\le
8\sqrt r\,\Bigl(1-\tfrac{\mu}{10}\Bigr)^t\,c_0\,\sigma_{\min}(M^*)
\]
for \(\mu\in(0,1/32]\) [2604.00060]. This closes a previously noted gap between fast convergence and optimal sampling efficiency.

## 5. Variants beyond the standard low-rank setting

Several ScaledGD variants modify either the preconditioner or the meaning of “scaling.”

An important extension addresses overparameterization. In PSD matrix sensing with unknown true rank, ScaledGD\((\lambda)\) uses the damped preconditioner
\[
P_t=(X_t^\top X_t+\lambda I_r)^{-1}
\]
inside the update
\[
X_{t+1}=X_t-\eta\,\nabla f(X_t)\,P_t
\]
[2302.01186]. The damping resolves the singularity of \(X_t^\top X_t\) when the factor rank is overspecified. From a small Gaussian initialization \(X_0=\alpha G\), the method converges in
\[
T=O(\log\kappa\cdot\log(\kappa n)+\log(1/\epsilon))
\]
iterations, improving dramatically over the polynomial \(\kappa\)-dependence known for vanilla GD in the same regime [2302.01186]. The 2023 ScaledGD overview presents this as a global convergence mechanism from small random initialization when rank is over-specified [2310.06159].

A stochastic analogue appears in low-rank matrix completion. The 2016 Scaled stochastic gradient descent method constructs minibatch preconditioners
\[
A_U=\alpha c H +(1-\alpha)H_b,\qquad
A_V=\alpha c G +(1-\alpha)G_b,
\]
where \(G=U^\top U,\;H=V^\top V\) are global Grams, \(G_b,H_b\) are batch Grams, \(c=b/\max(n,m)\), and \(\alpha\in[0,1]\) trades off global and local second-order information [1603.04989]. This scaling is explicitly designed to restore scale invariance of the factorization and to improve conditioning. The paper states that in the full-batch case the block-diagonal Hessian approximation reduces the condition number of the local quadratic model to one [1603.04989].

Not all methods labeled ScaledGD rely on factor Grams. In separable logistic regression, “ScaledGD” refers to gradient descent with a risk-dependent stepsize
\[
w_{t+1}=w_t-\alpha_t\nabla L(w_t),\qquad
\alpha_t=\eta\,(-\ell^{-1})'(L(w_t)),
\]
equivalently standard GD on the transformed objective
\[
\phi(w)=-\ell^{-1}(L(w))
\]
[2504.04105]. The key theorem states that after burn-in \(T_0=\lceil1/\gamma^2\rceil\), the averaged iterate achieves risk bounded by \(\exp(-c\eta)\) with \(c=\tfrac14\gamma^2\), and the paper proves a matching minimax lower bound showing \(O(1/\gamma^2)\) is optimal among first-order batch methods [2504.04105]. Here the “scaling” is in the stepsize rather than a matrix preconditioner.

In convex constrained optimization, projected scaled gradient methods use diagonal positive-definite \(D_k\) with projection onto a feasible set, unifying gradient projection, projected Newton, projected Landweber, and EM-type algorithms [1507.07302]. The central result is convergence under bounded perturbations, and under boundedness of \(C\) the operator is bounded perturbation resilient [1507.07302]. This line is historically distinct from the low-rank factorization literature but belongs to the same broader family of scaled-gradient ideas.

## 6. Computational profile, empirical behavior, and practical significance

A central claim of ScaledGD is that it obtains quasi-Newton-like benefits with first-order cost. In matrix factorization, the extra work beyond one gradient evaluation consists only of forming and inverting two \(r\times r\) Gram matrices, an \(O(r^3)\) overhead that is negligible when \(r\ll n\) [2005.08898]. The 2020 paper explicitly contrasts this with alternating minimization, which solves larger least-squares systems, and with projected methods that require more expensive inner operations [2005.08898].

In Tucker tensor RPCA, the per-iteration work is dominated by forming the preconditioned factors \(\breve U_t^{(k)}\), costing \(O(n^2r)\) for third-order tensors, while the scaled-gradient multiplications cost \(O(nr^2)\) per mode; the overall cost is reported as \(O(\sum_k n_k r_k^2)\) [2206.09109]. The paper contrasts this with Riemannian GD, which requires an HOSVD each step with \(O(n^3)\) cost [2206.09109]. This computational comparison is central to the method’s positioning: faster than second-order or manifold methods while more robust than unpreconditioned GD.

Empirical results consistently reflect the theoretical condition-number robustness. In synthetic tensor RPCA phase transitions, ScaledGD shows a sharp transition boundary approximately linear in \(\alpha\approx O(1/r)\) and almost independent of \(\kappa\) [2206.09109]. In shot-noise experiments it matches RiemannianGD in accuracy but runs an order of magnitude faster [2206.09109]. On MNIST digit-sequence denoising, it recovers clean digits and flags outliers more sharply than convex TNN at a fraction of the runtime; on VIRAT video background-foreground separation, freezing two modes yields a \(5\times\) speedup with negligible quality loss [2206.09109].

The matrix recovery literature reports the same qualitative pattern. In the 2026 Gaussian measurement experiments with \(n_1=n_2=100\) and \(r=30\), ScaledGD converges in far fewer iterations than vanilla GD and matches Riemannian GD, while GD runtime grows with \(\kappa\) and ScaledGD remains flat [2604.00060]. In low-rank matrix estimation more broadly, ScaledGD is reported to tolerate ill-conditioning up to \(\kappa=50\), whereas vanilla GD slows by a factor proportional to \(\kappa\) [2005.08898].

This empirical regularity has shaped a common interpretation: the factor-Gram preconditioners “balance the scales” of the factors and effectively whiten the local geometry. That interpretation is explicit in several sources [2005.08898], [2407.09900], though the precise proofs rely on specialized Lyapunov distances, RIP or incoherence arguments, perturbation lemmas, and basin-entry conditions rather than on a direct Hessian equivalence.

## 7. Conceptual distinctions, misconceptions, and evolving directions

A common misconception is that “ScaledGD” refers to a single canonical algorithm. The literature instead contains several distinct families sharing the idea of gradient scaling. The Gram-preconditioned low-rank methods of Tong, Ma, Chi, and collaborators [2005.08898], [2206.09109], [2310.06159] are not the same as projected scaled gradient methods in convex optimization [1507.07302], risk-adaptive stepsize scaling for logistic regression [2504.04105], scaled sign methods for communication-efficient optimization [2109.01806], or Hessian-aware scalar scaling rules [2502.03701]. The shared name reflects a common variable-metric intuition rather than a uniform update formula.

A second misconception is that the scaling is equivalent to full second-order optimization. In the low-rank setting, the preconditioner is deliberately inexpensive and highly structured: only small Gram matrices are inverted [2005.08898]. In the projected convex setting, the scaling matrices are diagonal [1507.07302]. In Hessian-aware scalar ScaledGD, only curvature along the gradient direction is used, together with one Hessian-vector product [2502.03701]. These methods occupy an intermediate space between plain GD and Newton-type methods.

The field is also evolving in two notable directions. One is sharper theory: ScaledGD has progressed from local condition-number-free rates [2005.08898] to overparameterized global guarantees with damping [2302.01186] and then to optimal-sampling matrix recovery results in the general asymmetric case [2604.00060]. Another is architectural hybridization. The learnable self-supervised deep unfolding model RTPCA-LSGD turns t-SVD ScaledGD for robust tensor PCA into a trainable recurrent architecture with learnable \(\zeta_0,\zeta_1,\tau,\eta\), while preserving the original algorithmic structure as the backbone [2501.04565]. This suggests a broader trend in which ScaledGD serves both as an analyzable optimization primitive and as a module within learned iterative systems.

Taken together, the ScaledGD literature presents a coherent methodological theme: use a carefully chosen scaling operator—diagonal, projected, Gram-based, damped, stochastic, or Hessian-aware—to reshape the local descent geometry while preserving the simplicity of first-order updates. In low-rank matrix and tensor estimation, this theme has yielded a particularly strong body of theory and algorithms with linear convergence at rates independent of the condition number of the target object [2005.08898], [2206.09109], [2501.01696], [2604.00060].

Source: https://www.emergentmind.com/topics/scaled-gradient-descent-scaledgd