---
title: Online Scaled Gradient Methods (OSGM)
url: https://www.emergentmind.com/topics/online-scaled-gradient-methods-osgm
type: topic
---

# Online Scaled Gradient Methods (OSGM)

Online Scaled Gradient Methods (OSGM) denote a family of gradient-based optimization methods in which the gradient is multiplied by an online-learned scaling, typically a scalar, diagonal matrix, or full preconditioner, so that step selection is itself treated as a sequential learning problem. In the most explicit recent formulation, OSGM is “a framework that utilizes online learning to adapt stepsizes and provably accelerate first-order methods,” with updates of the form $x_{k+1} = x_k - P_k \nabla f(x_k)$ and convergence guarantees stated relative to the optimal scaling matrix for the realized iteration trajectory rather than only to worst-case smoothness constants [2505.23081], [2411.01803].

## 1. Historical emergence and scope

The contemporary OSGM framework was introduced as a method to “accelerate the convergence of gradient-based methods with online learning,” with the scaling matrix $P_k$ updated by an online learning algorithm at each iteration [2411.01803]. This line was then split into a theoretical treatment, “Gradient Methods with Online Scaling Part I. Theoretical Foundations,” and a practical treatment, “Gradient Methods with Online Scaling Part II. Practical Aspects,” which together define OSGM as a regret-based approach to adaptive first-order optimization [2505.23081], [2509.11007].

The term also sits within a broader lineage of online and scaled gradient procedures. Earlier work developed “online and stochastic universal gradient methods” for convex finite sums with Hölder continuous gradients, explicitly aiming to bridge smooth and nonsmooth first-order methods without prior knowledge of the smoothness degree [1311.3832]. In online learning with linear models, parameter-free scale-invariant algorithms were proposed that are invariant under arbitrary rescaling of the features and match, up to logarithmic factors, the regret of Online Gradient Descent with optimally tuned separate learning rates per dimension [1902.07528]. In stochastic manifold optimization, Stochastic Scaled-Gradient Descent (SSGD) generalized projected stochastic gradient descent by allowing unbiased scaled gradients rather than unbiased gradients [2112.14738]. Other related strands include online linear regression of gradients in statistically relevant directions [1901.11457] and stochastic gradient methods with online diagonal Hessian-based preconditioning [2206.00285].

This suggests that OSGM is best understood in two concentric senses. In the narrow sense, it denotes the 2024–2025 framework that uses online convex optimization to learn gradient scalings [2411.01803], [2505.23081]. In a broader sense, it covers online methods whose essential mechanism is to modulate gradient steps by learned scale, geometry, or curvature information rather than by a fixed learning rate [2112.14738], [2206.00285].

## 2. Core formulation: online learning of the scaling operator

The canonical OSGM update is
\[
x_{k+1} = x_k - P_k \nabla f(x_k),
\]
where $P_k$ belongs to a convex candidate set $\mathcal{P}$ that may contain scalar, diagonal, sparse, or full matrices [2411.01803], [2505.23081]. The distinctive step is that the choice of $P_k$ is not prescribed by a hand-designed rule; it is produced by an online learning algorithm that receives feedback about how well the current scaling contracts a chosen optimality measure.

Several surrogate feedback functions are used. For smooth strongly convex problems, one option is the function value ratio
\[
r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}.
\]
A second option is the hypergradient surrogate
\[
h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2},
\]
which does not require knowledge of $f(x^*)$ and is the mechanism through which OSGM formalizes hypergradient descent [2505.23081], [2411.01803]. The 2024 formulation also considers a gradient-norm contraction surrogate,
\[
g_x(P) := \frac{\|\nabla f(x^+)\|}{\|\nabla f(x)\|},
\]
with $x^+ = x - P\nabla f(x)$ [2411.01803].

The scaling update is then performed by an online convex optimization rule, typically projected Online Gradient Descent:
\[
P_{k+1} = \Pi_{\mathcal{P}}\!\left[ P_k - \eta \nabla \ell_{x_k}(P_k) \right],
\]
where $\ell_{x_k}$ is one of the surrogate losses and $\Pi_{\mathcal{P}}$ is projection onto $\mathcal{P}$ [2505.23081], [2411.01803]. In the theoretical treatment, the regret statement has the form
\[
\sum_{k=1}^K l_{x_k}(P_k) \leq \min_{P \in \mathcal{P}} \sum_{k=1}^K l_{x_k}(P) + o(K),
\]
which makes the average surrogate loss asymptotically no worse than that of the best fixed comparator in hindsight [2505.23081].

OSGM is not only a rule for $P_k$; it also includes a “landscape action” that determines whether and how the proposal $x_{k+1/2}=x_k-P_k\nabla f(x_k)$ becomes the next iterate. The framework explicitly lists vanilla, monotone, lookahead, and monotone lookahead actions [2505.23081]. In the 2024 exposition, monotone comparison oracles and modified ratio surrogates $r_x^z(P)$ are also used when only a lower bound $z<f(x^*)$ is available [2411.01803].

## 3. Convergence theory and complexity guarantees

The defining theoretical claim of OSGM is that online adaptation can yield convergence rates “asymptotically no worse than the optimal stepsize” or optimal scaling matrix on the realized trajectory [2505.23081], [2411.01803]. For smooth strongly convex optimization, the 2024 paper proves an $O(\kappa^\star \log(1/\varepsilon))$ complexity result, where $\kappa^\star$ is “the condition number achievable by the optimal preconditioner,” improving on the previous $O(\sqrt{n}\kappa^\star \log(1/\varepsilon))$ result [2411.01803]. The theoretical Part I emphasizes the same theme as an “improved complexity result on smooth strongly convex problems” [2505.23081].

A central feature is that the guarantee is trajectory-dependent. Rather than comparing only to a globally optimal fixed preconditioner defined over the whole domain, OSGM competes with the best scaling for the actual sequence of iterates. The 2024 paper states this through bounds of the form
\[
f(x^{K+1}) - f(x^\star) \leq (f(x^1) - f(x^\star))
\left(\theta_K^\star + \frac{\rho_K}{K}\right)^K,
\]
where $\theta_K^\star$ is the best average contraction obtainable by a comparator in $\mathcal{P}$ and $\rho_K$ is the online learning regret [2411.01803]. The 2025 theoretical paper connects this to dynamic regret, allowing competition not only with a fixed scaling matrix but also with varying sequences of scalings [2505.23081].

OSGM also admits superlinear phenomena that are unusual for first-order methods. The theoretical Part I states that OSGM “constitutes a new family of first-order methods with non-asymptotic superlinear convergence, joining the celebrated quasi-Newton methods” [2505.23081]. The 2024 paper makes this concrete for convex quadratics: a variant achieves superlinear convergence, and when the preconditioner class contains the inverse Hessian of the quadratic, the function gap satisfies a bound of the form
\[
f(x^{K+1})-f(x^\star) \leq (f(x^1)-f(x^\star))\left(\frac{C}{K}\right)^K
\]
for a constant $C$ [2411.01803].

For smooth convex objectives, the significance of the hypergradient surrogate is that it yields the first provable acceleration result for the widely used hypergradient descent heuristic. The 2024 paper states that “for smooth convex optimization, we show for the first time that the widely-used hypergradient descent heuristic improves on the convergence of gradient descent,” while Part I states that OSGM explains the empirical success of hypergradient descent and gives it rigorous non-asymptotic analysis [2411.01803], [2505.23081].

## 4. Hypergradients, momentum, and practical variants

The practical development of OSGM centers on how the online scheduler is stabilized and how momentum is incorporated. A direct hypergradient version, OSGM-H, uses the per-step loss
\[
h_x(P) = \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2}
\]
and an online gradient step on $P$ [2509.11007]. In practice, however, “pure OSGM-H can exhibit periodic ‘spikes’ in objective value unless monotonicity is enforced,” especially on ill-conditioned problems [2509.11007]. The monotone or lookahead landscape action is therefore not a technical ornament but a practical stabilizer.

The main practical method is OSGM-Best, described as a heavy-ball-based variant that uses a potential
\[
Q_\omega(x,x_{-}) = f(x) - f^* + \omega \|x - x_{-}\|^2
\]
and a feedback
\[
h_z(P,\beta)=\frac{Q_\omega(z^+(P,\beta)) - Q_\omega(z)}
{\|\nabla f(x)\|^2 + \omega \|x-x_{-}\|^2},
\]
together with online updates for both the scaling $P_k$ and momentum parameter $\beta_k$ [2509.11007]. The practical paper states that OSGM-Best “matches the performance of quasi-Newton variants while requiring less memory and cheaper iterations,” and more specifically that it matches or closely tracks L-BFGS-M5 or L-BFGS-M10 while frequently outperforming Adam or AdaGrad on convex and nonconvex benchmarks [2509.11007].

The same paper extends OSGM to nonconvex optimization by regularizing the feedback in stepsize-space:
\[
\tilde{h}_x(P) = h_x(P) + \lambda \|P - \alpha I\|^2.
\]
The role of the added term is to make the scheduler’s loss convex in $P$ under suitable smoothness assumptions, so that online learning machinery remains valid [2509.11007]. In this regime, the method guarantees sublinear convergence of the gradient norm for smooth nonconvex problems, and for objectives satisfying the Polyak-Łojasiewicz condition it “recovers known fast rates” [2509.11007].

## 5. Related families frequently associated with online scaling

A large surrounding literature studies online or stochastic methods that are not identical to the 2024–2025 OSGM meta-framework but share its defining mechanism: the update uses a scaled, geometry-aware, or curvature-aware gradient computed or adapted online.

| Family | Representative mechanism | Notable guarantee |
|---|---|---|
| Scale-invariant online linear learning | Per-coordinate parameter-free updates using $G_{t,i}$, $S_{t,i}$, and $M_{t,i}$ | Regret matches oracle OGD up to logarithmic factors; $O(d)$ per trial [1902.07528] |
| Stochastic scaled-gradient descent on manifolds | $x_t=\Pi_C[x_{t-1}-\eta \Gamma(x_{t-1};\xi_t)]$, with $\mathbb{E}\Gamma=D(x)\nabla F(x)$ | $\sqrt{1/T}$ finite-sample rate on the sphere, minimax optimal up to polylog factors; asymptotic normality [2112.14738] |
| Preconditioned stochastic finite-sum methods | Online diagonal Hessian estimate $D_t$ via Hutchinson, used in Scaled SARAH and Scaled L-SVRG | Linear convergence under smoothness and PL; improved behavior on badly scaled problems [2206.00285] |
| Scaled sign methods | $x_{k+1}=x_k-\alpha \|g_k\|_1 \operatorname{sign}(g_k)$ | Linear convergence under strong convexity or PL; stochastic and distributed variants [2109.01806] |
| Universal online/stochastic Hölder methods | Adaptive Bregman-based updates without prior knowledge of $v$ or $M_v$ | Regret and convergence guarantees across $C^{1,v}$; linear convergence for strongly convex regularized finite sums [1311.3832] |

These families clarify what is specific to OSGM and what is generic to online scaling. OSGM, in the strict sense, makes the scaling matrix itself the online learning variable and builds convergence analysis from regret minimization [2505.23081], [2411.01803]. The surrounding methods instead emphasize particular geometries—feature rescaling, manifold constraints, diagonal curvature surrogates, sign-based communication compression, or Hölder-smooth surrogate models [1902.07528], [2112.14738], [2206.00285], [2109.01806], [1311.3832].

Two additional related lines sharpen this distinction. One paper models second-order behavior through online linear regression of noisy gradients in a statistically relevant subspace selected by online PCA- or QR-inspired updates, while performing gradient descent outside that modeled subspace [1901.11457]. Another paper shows that multiscale ill-conditioned objectives can be attacked by a recursive “Big-Step-Little-Step” schedule that exploits multiple scales through interleaved step sizes rather than through an explicit online preconditioner [2111.03137].

## 6. Domain-specific reinterpretations, misconceptions, and boundaries

Several recent application papers instantiate the central idea of online scaling in highly specialized forms. In continual learning, Scaled Gradient Projection (SGP) combines orthogonal gradient projections with “scaled gradient steps along the important gradient spaces for the past tasks,” where the scaling depends on the importance of bases extracted by singular value decomposition of layer activations [2302.01386]. In that setting, the purpose of scaling is to balance catastrophic forgetting against new-task plasticity rather than to accelerate convex optimization. In chaotic-flow optimization, the Online Gradient Flow method propagates an online finite-difference estimate of the gradient of steady-state statistics while simultaneously updating parameters, and explicitly uses EWMA smoothing, minibatching, and RMSprop- or Adam-style normalization to control noise and avoid divergence of adjoint gradients [2507.05149]. In SAR time-series change detection, the online estimator for Kronecker-structured scaled Gaussian models is a recursive natural Riemannian gradient descent whose per-update cost is constant and whose asymptotic efficiency reaches the Intrinsic Cramér-Rao bound [2312.02807].

These examples motivate an important clarification. OSGM is not synonymous with every adaptive optimizer, every preconditioned method, or every online gradient method. The formal 2024–2025 framework is defined by online convex optimization over the scaling operator $P_k$ and by feedback functions such as $r_x(P)$ or $h_x(P)$ [2411.01803], [2505.23081]. By contrast, some neighboring methods scale the gradient through per-coordinate invariance [1902.07528], unbiased scaled manifold gradients [2112.14738], diagonal Hutchinson preconditioners [2206.00285], or domain-specific geometry and memory constraints [2312.02807], [2302.01386]. This suggests that the phrase “online scaled gradient methods” is both a precise recent theory and a broader organizing concept.

A second misconception is that online scaling necessarily means heuristic tuning without theory. The formal OSGM papers were written precisely to rebut that view: they derive trajectory-dependent global convergence, improved strongly convex complexity, and local superlinear convergence, and they provide a rigorous interpretation of hypergradient descent [2505.23081], [2411.01803]. A third misconception is that richer scaling classes are always preferable. The theoretical papers note a trade-off: scalar, diagonal, and full-matrix candidate sets offer different balances between regret constants, computational cost, and attainable local curvature matching [2505.23081], [2411.01803]. The practical paper reaches the same conclusion empirically by positioning OSGM-Best near quasi-Newton performance but with lower memory and cheaper iterations rather than with full Newton-like curvature reconstruction [2509.11007].

Within optimization theory, the significance of OSGM is therefore twofold. First, it reframes step-size and preconditioner selection as an online learning problem with regret guarantees. Second, it supplies a unifying language for a wide range of online scaling procedures that had previously appeared under separate rubrics—hypergradients, scale-invariant online prediction, scaled stochastic manifold optimization, diagonal Hessian preconditioning, and domain-specific geometry-aware updates [2505.23081], [1902.07528], [2112.14738], [2206.00285].

Source: https://www.emergentmind.com/topics/online-scaled-gradient-methods-osgm