Online Scaled Gradient Methods (OSGM)

• OSGM is a family of gradient-based optimization methods that uses online learning to dynamically update scaling matrices for improved convergence.
• The framework leverages surrogate feedback functions, such as the hypergradient and gradient-norm contraction, to adjust the scaling for each iteration.
• OSGM offers trajectory-dependent convergence guarantees and non-asymptotic superlinear behavior, outperforming traditional first-order methods.

Online Scaled Gradient Methods (OSGM) denote a family of gradient-based optimization methods in which the gradient is multiplied by an online-learned scaling, typically a scalar, diagonal matrix, or full preconditioner, so that step selection is itself treated as a sequential learning problem. In the most explicit recent formulation, OSGM is “a framework that utilizes online learning to adapt stepsizes and provably accelerate first-order methods,” with updates of the form $x_{k+1} = x_k - P_k \nabla f(x_k)$ and convergence guarantees stated relative to the optimal scaling matrix for the realized iteration trajectory rather than only to worst-case smoothness constants (Gao et al., 29 May 2025, Gao et al., 2024).

1. Historical emergence and scope

The contemporary OSGM framework was introduced as a method to “accelerate the convergence of gradient-based methods with online learning,” with the scaling matrix $P_k$ updated by an online learning algorithm at each iteration (Gao et al., 2024). This line was then split into a theoretical treatment, “Gradient Methods with Online Scaling Part I. Theoretical Foundations,” and a practical treatment, “Gradient Methods with Online Scaling Part II. Practical Aspects,” which together define OSGM as a regret-based approach to adaptive first-order optimization (Gao et al., 29 May 2025, Chu et al., 13 Sep 2025).

The term also sits within a broader lineage of online and scaled gradient procedures. Earlier work developed “online and stochastic universal gradient methods” for convex finite sums with Hölder continuous gradients, explicitly aiming to bridge smooth and nonsmooth first-order methods without prior knowledge of the smoothness degree (Shi et al., 2013). In online learning with linear models, parameter-free scale-invariant algorithms were proposed that are invariant under arbitrary rescaling of the features and match, up to logarithmic factors, the regret of Online Gradient Descent with optimally tuned separate learning rates per dimension (Kempka et al., 2019). In stochastic manifold optimization, Stochastic Scaled-Gradient Descent (SSGD) generalized projected stochastic gradient descent by allowing unbiased scaled gradients rather than unbiased gradients (Li et al., 2021). Other related strands include online linear regression of gradients in statistically relevant directions (Duda, 2019) and stochastic gradient methods with online diagonal Hessian-based preconditioning (Sadiev et al., 2022).

This suggests that OSGM is best understood in two concentric senses. In the narrow sense, it denotes the 2024–2025 framework that uses online convex optimization to learn gradient scalings (Gao et al., 2024, Gao et al., 29 May 2025). In a broader sense, it covers online methods whose essential mechanism is to modulate gradient steps by learned scale, geometry, or curvature information rather than by a fixed learning rate (Li et al., 2021, Sadiev et al., 2022).

2. Core formulation: online learning of the scaling operator

The canonical OSGM update is

$x_{k+1} = x_k - P_k \nabla f(x_k),$

where $P_k$ belongs to a convex candidate set $\mathcal{P}$ that may contain scalar, diagonal, sparse, or full matrices (Gao et al., 2024, Gao et al., 29 May 2025). The distinctive step is that the choice of $P_k$ is not prescribed by a hand-designed rule; it is produced by an online learning algorithm that receives feedback about how well the current scaling contracts a chosen optimality measure.

Several surrogate feedback functions are used. For smooth strongly convex problems, one option is the function value ratio

$$r_x(P) := \frac{f(x - P \nabla f(x)) - f(x^*)}{f(x) - f(x^*)}.$$

A second option is the hypergradient surrogate

$$h_x(P) := \frac{f(x - P \nabla f(x)) - f(x)}{\|\nabla f(x)\|^2},$$

which does not require knowledge of $f(x^*)$ and is the mechanism through which OSGM formalizes hypergradient descent (Gao et al., 29 May 2025, Gao et al., 2024). The 2024 formulation also considers a gradient-norm contraction surrogate,

$$g_x(P) := \frac{\|\nabla f(x^+)\|}{\|\nabla f(x)\|},$$

with $P_k$0 (Gao et al., 2024).

The scaling update is then performed by an online convex optimization rule, typically projected Online Gradient Descent: $P_k$1 where $P_k$2 is one of the surrogate losses and $P_k$3 is projection onto $P_k$4 (Gao et al., 29 May 2025, Gao et al., 2024). In the theoretical treatment, the regret statement has the form

$P_k$5

which makes the average surrogate loss asymptotically no worse than that of the best fixed comparator in hindsight (Gao et al., 29 May 2025).

OSGM is not only a rule for $P_k$6; it also includes a “landscape action” that determines whether and how the proposal $P_k$7 becomes the next iterate. The framework explicitly lists vanilla, monotone, lookahead, and monotone lookahead actions (Gao et al., 29 May 2025). In the 2024 exposition, monotone comparison oracles and modified ratio surrogates $P_k$8 are also used when only a lower bound $P_k$9 is available (Gao et al., 2024).

3. Convergence theory and complexity guarantees

The defining theoretical claim of OSGM is that online adaptation can yield convergence rates “asymptotically no worse than the optimal stepsize” or optimal scaling matrix on the realized trajectory (Gao et al., 29 May 2025, Gao et al., 2024). For smooth strongly convex optimization, the 2024 paper proves an $x_{k+1} = x_k - P_k \nabla f(x_k),$0 complexity result, where $x_{k+1} = x_k - P_k \nabla f(x_k),$1 is “the condition number achievable by the optimal preconditioner,” improving on the previous $x_{k+1} = x_k - P_k \nabla f(x_k),$2 result (Gao et al., 2024). The theoretical Part I emphasizes the same theme as an “improved complexity result on smooth strongly convex problems” (Gao et al., 29 May 2025).

A central feature is that the guarantee is trajectory-dependent. Rather than comparing only to a globally optimal fixed preconditioner defined over the whole domain, OSGM competes with the best scaling for the actual sequence of iterates. The 2024 paper states this through bounds of the form

$x_{k+1} = x_k - P_k \nabla f(x_k),$3

where $x_{k+1} = x_k - P_k \nabla f(x_k),$4 is the best average contraction obtainable by a comparator in $x_{k+1} = x_k - P_k \nabla f(x_k),$5 and $x_{k+1} = x_k - P_k \nabla f(x_k),$6 is the online learning regret (Gao et al., 2024). The 2025 theoretical paper connects this to dynamic regret, allowing competition not only with a fixed scaling matrix but also with varying sequences of scalings (Gao et al., 29 May 2025).

OSGM also admits superlinear phenomena that are unusual for first-order methods. The theoretical Part I states that OSGM “constitutes a new family of first-order methods with non-asymptotic superlinear convergence, joining the celebrated quasi-Newton methods” (Gao et al., 29 May 2025). The 2024 paper makes this concrete for convex quadratics: a variant achieves superlinear convergence, and when the preconditioner class contains the inverse Hessian of the quadratic, the function gap satisfies a bound of the form

$x_{k+1} = x_k - P_k \nabla f(x_k),$7

for a constant $x_{k+1} = x_k - P_k \nabla f(x_k),$8 (Gao et al., 2024).

For smooth convex objectives, the significance of the hypergradient surrogate is that it yields the first provable acceleration result for the widely used hypergradient descent heuristic. The 2024 paper states that “for smooth convex optimization, we show for the first time that the widely-used hypergradient descent heuristic improves on the convergence of gradient descent,” while Part I states that OSGM explains the empirical success of hypergradient descent and gives it rigorous non-asymptotic analysis (Gao et al., 2024, Gao et al., 29 May 2025).

4. Hypergradients, momentum, and practical variants

The practical development of OSGM centers on how the online scheduler is stabilized and how momentum is incorporated. A direct hypergradient version, OSGM-H, uses the per-step loss

$x_{k+1} = x_k - P_k \nabla f(x_k),$9

and an online gradient step on $P_k$0 (Chu et al., 13 Sep 2025). In practice, however, “pure OSGM-H can exhibit periodic ‘spikes’ in objective value unless monotonicity is enforced,” especially on ill-conditioned problems (Chu et al., 13 Sep 2025). The monotone or lookahead landscape action is therefore not a technical ornament but a practical stabilizer.

The main practical method is OSGM-Best, described as a heavy-ball-based variant that uses a potential

$P_k$1

and a feedback

$P_k$2

together with online updates for both the scaling $P_k$3 and momentum parameter $P_k$4 (Chu et al., 13 Sep 2025). The practical paper states that OSGM-Best “matches the performance of quasi-Newton variants while requiring less memory and cheaper iterations,” and more specifically that it matches or closely tracks L-BFGS-M5 or L-BFGS-M10 while frequently outperforming Adam or AdaGrad on convex and nonconvex benchmarks (Chu et al., 13 Sep 2025).

The same paper extends OSGM to nonconvex optimization by regularizing the feedback in stepsize-space: $P_k$5 The role of the added term is to make the scheduler’s loss convex in $P_k$6 under suitable smoothness assumptions, so that online learning machinery remains valid (Chu et al., 13 Sep 2025). In this regime, the method guarantees sublinear convergence of the gradient norm for smooth nonconvex problems, and for objectives satisfying the Polyak-Łojasiewicz condition it “recovers known fast rates” (Chu et al., 13 Sep 2025).

5. Related families frequently associated with online scaling

A large surrounding literature studies online or stochastic methods that are not identical to the 2024–2025 OSGM meta-framework but share its defining mechanism: the update uses a scaled, geometry-aware, or curvature-aware gradient computed or adapted online.

Family	Representative mechanism	Notable guarantee
Scale-invariant online linear learning	Per-coordinate parameter-free updates using $P_k$7, $P_k$8, and $P_k$9	Regret matches oracle OGD up to logarithmic factors; $\mathcal{P}$0 per trial (Kempka et al., 2019)
Stochastic scaled-gradient descent on manifolds	$\mathcal{P}$1, with $\mathcal{P}$2	$\mathcal{P}$3 finite-sample rate on the sphere, minimax optimal up to polylog factors; asymptotic normality (Li et al., 2021)
Preconditioned stochastic finite-sum methods	Online diagonal Hessian estimate $\mathcal{P}$4 via Hutchinson, used in Scaled SARAH and Scaled L-SVRG	Linear convergence under smoothness and PL; improved behavior on badly scaled problems (Sadiev et al., 2022)
Scaled sign methods	$\mathcal{P}$5	Linear convergence under strong convexity or PL; stochastic and distributed variants (Li et al., 2021)
Universal online/stochastic Hölder methods	Adaptive Bregman-based updates without prior knowledge of $\mathcal{P}$6 or $\mathcal{P}$7	Regret and convergence guarantees across $\mathcal{P}$8; linear convergence for strongly convex regularized finite sums (Shi et al., 2013)

These families clarify what is specific to OSGM and what is generic to online scaling. OSGM, in the strict sense, makes the scaling matrix itself the online learning variable and builds convergence analysis from regret minimization (Gao et al., 29 May 2025, Gao et al., 2024). The surrounding methods instead emphasize particular geometries—feature rescaling, manifold constraints, diagonal curvature surrogates, sign-based communication compression, or Hölder-smooth surrogate models (Kempka et al., 2019, Li et al., 2021, Sadiev et al., 2022, Li et al., 2021, Shi et al., 2013).

Two additional related lines sharpen this distinction. One paper models second-order behavior through online linear regression of noisy gradients in a statistically relevant subspace selected by online PCA- or QR-inspired updates, while performing gradient descent outside that modeled subspace (Duda, 2019). Another paper shows that multiscale ill-conditioned objectives can be attacked by a recursive “Big-Step-Little-Step” schedule that exploits multiple scales through interleaved step sizes rather than through an explicit online preconditioner (Kelner et al., 2021).

6. Domain-specific reinterpretations, misconceptions, and boundaries

Several recent application papers instantiate the central idea of online scaling in highly specialized forms. In continual learning, Scaled Gradient Projection (SGP) combines orthogonal gradient projections with “scaled gradient steps along the important gradient spaces for the past tasks,” where the scaling depends on the importance of bases extracted by singular value decomposition of layer activations (Saha et al., 2023). In that setting, the purpose of scaling is to balance catastrophic forgetting against new-task plasticity rather than to accelerate convex optimization. In chaotic-flow optimization, the Online Gradient Flow method propagates an online finite-difference estimate of the gradient of steady-state statistics while simultaneously updating parameters, and explicitly uses EWMA smoothing, minibatching, and RMSprop- or Adam-style normalization to control noise and avoid divergence of adjoint gradients (Hickling et al., 7 Jul 2025). In SAR time-series change detection, the online estimator for Kronecker-structured scaled Gaussian models is a recursive natural Riemannian gradient descent whose per-update cost is constant and whose asymptotic efficiency reaches the Intrinsic Cramér-Rao bound (Mian et al., 2023).

These examples motivate an important clarification. OSGM is not synonymous with every adaptive optimizer, every preconditioned method, or every online gradient method. The formal 2024–2025 framework is defined by online convex optimization over the scaling operator $\mathcal{P}$9 and by feedback functions such as $P_k$0 or $P_k$1 (Gao et al., 2024, Gao et al., 29 May 2025). By contrast, some neighboring methods scale the gradient through per-coordinate invariance (Kempka et al., 2019), unbiased scaled manifold gradients (Li et al., 2021), diagonal Hutchinson preconditioners (Sadiev et al., 2022), or domain-specific geometry and memory constraints (Mian et al., 2023, Saha et al., 2023). This suggests that the phrase “online scaled gradient methods” is both a precise recent theory and a broader organizing concept.

A second misconception is that online scaling necessarily means heuristic tuning without theory. The formal OSGM papers were written precisely to rebut that view: they derive trajectory-dependent global convergence, improved strongly convex complexity, and local superlinear convergence, and they provide a rigorous interpretation of hypergradient descent (Gao et al., 29 May 2025, Gao et al., 2024). A third misconception is that richer scaling classes are always preferable. The theoretical papers note a trade-off: scalar, diagonal, and full-matrix candidate sets offer different balances between regret constants, computational cost, and attainable local curvature matching (Gao et al., 29 May 2025, Gao et al., 2024). The practical paper reaches the same conclusion empirically by positioning OSGM-Best near quasi-Newton performance but with lower memory and cheaper iterations rather than with full Newton-like curvature reconstruction (Chu et al., 13 Sep 2025).

Within optimization theory, the significance of OSGM is therefore twofold. First, it reframes step-size and preconditioner selection as an online learning problem with regret guarantees. Second, it supplies a unifying language for a wide range of online scaling procedures that had previously appeared under separate rubrics—hypergradients, scale-invariant online prediction, scaled stochastic manifold optimization, diagonal Hessian preconditioning, and domain-specific geometry-aware updates (Gao et al., 29 May 2025, Kempka et al., 2019, Li et al., 2021, Sadiev et al., 2022).