---
title: Stochastic Quasi-Newton Method
url: https://www.emergentmind.com/topics/stochastic-quasi-newton-method
type: topic
---

# Stochastic Quasi-Newton Method

A stochastic quasi-Newton method is a class of optimization algorithms that adapt quasi-Newton methodology—approximating or exploiting curvature (Hessian) information—in settings where only noisy first-order information is available, typical of large-scale or online learning tasks. These methods generalize classic quasi-Newton schemes (e.g. BFGS, L-BFGS) for stochastic, high-dimensional, and often nonconvex or nonsmooth objectives, enabling curvature exploitation under stringent computational and oracle constraints. Across contemporary literature, stochastic quasi-Newton methods encompass variants for composite optimization, nonconvex and finite-sum minimization, variance-reduced schemes, line-search-regularized approaches, coordinate and block-structured updates, proximal extensions, and robust stochastic implementations.

## 1. Formalization and Foundational Principles

Stochastic quasi-Newton methods solve optimization problems of the form
\[
\min_{x\in\mathbb R^n}\, \mathbb E_\xi[f(x;\xi)] + R(x)
\]
where $f(x;\xi)$ is (weighted) smooth and possibly nonconvex, and $R(x)$ is a convex regularizer or constraint penalty (possibly nonsmooth). The method accesses only a stochastic first-order oracle delivering unbiased gradient estimates $g(x,\xi)$ with controlled variance. The key idea is to enhance vanilla stochastic (proximal) gradient steps by applying a state-dependent positive-definite preconditioner $H_k \approx [\nabla^2f(x_k)]^{-1}$, constructed using a quasi-Newton (most commonly L-BFGS) update based on empirical quasi-Newton pairs $(s_k,y_k)$ formed from stochastic gradients and iterates [1412.1196, 1607.01231, 1401.7020, 1602.00223, 1910.09373, 1912.04456, 1503.08316].

The necessity of statistical safeguards (damping, regularization, coordinate selection) is widely recognized, with most modern schemes requiring that $H_k$ maintain uniform spectral bounds, often through Powell/Byrd damping or explicit regularization [1607.01231, 1912.04456, 1805.02338, 1909.01238]. Integration of variance-reduction is pivotal to achieving optimal or near-optimal oracle complexity and linear convergence in strongly convex regimes [1503.08316, 2004.06479, 2409.16971, 2403.15244]. 


## 2. Algorithmic Frameworks and Methodological Advances

A representative stochastic quasi-Newton iteration is given by
\[
x_{k+1} = x_k - \alpha_k H_k g_k
\]
where $H_k$ is constructed via L-BFGS (using, e.g., the last $m$ pairs $(s_j,y_j)$), and $g_k$ is a mini-batch stochastic gradient. Variations exist for composite settings, such as the stochastic extra-step quasi-Newton (SEQN) method [1910.09373], which handles objectives of the form $f(x) + g(x)$, using:
\[
d^k = -W_k F^{\Lambda_k}_{v^k}(x^k),\quad z^k = x^k+\beta_k d^k,\quad x^{k+1} = \operatorname{prox}_{g}^{\Lambda_{k,+}}(x^k+\alpha_k d^k - \Lambda_{k,+}^{-1} v^k_+)
\]
with $F^\Lambda_v(x)$ encoding a proximal fixed-point condition and $W_k$ the quasi-Newton preconditioner.

For general unconstrained nonconvex optimization, the stochastic damped L-BFGS (SdLBFGS) updates the inverse Hessian estimate via a “damped” secant correction that ensures $s_k^T y_k \gg 0$ in expectation, even under high noise [1607.01231, 1805.02338]. Variable sample-size methods extend these ideas to state-dependent or nonstationary oracle variance [1804.05368]. Methods with adaptive step lengths use accept/reject Markov proposals and stochastic line search, as in [1802.04310, 1909.01238], combining L-BFGS secant fits with selection over receding windows of iterates and gradients, often with backtracking and adaptive step-size schedules.

Structured stochastic quasi-Newton methods exploit block, low-rank, or Kronecker product structure in the Hessian—for example, deep learning settings—to accelerate computation and exploit scale [2006.09606], and coordinate/active-set selection further reduces per-iteration costs [1910.09373].

Variance-reduced quasi-Newton methods, such as Vite [1503.08316] and SpiderSQN [2004.06479], integrate SVRG- or SPIDER-style gradient estimators with limited-memory BFGS to achieve optimal sample complexity. Single-loop and asynchronous variants have been developed for better parallel efficiency and practical scalability [2011.00667, 2409.16971].

### Table: Typical Components in State-of-the-Art S-QN Methods

| Component            | Description                                    | Example Papers             |
|----------------------|------------------------------------------------|----------------------------|
| Quasi-Newton Update  | Damped/regularized L-BFGS with spectral bounds | [1607.01231, 1805.02338]   |
| Variance Reduction   | SVRG, SPIDER, SAGA gradient estimators         | [1503.08316, 2004.06479]   |
| Proximal Handling    | Scaled/semismooth Newton-Prox subroutines      | [1910.09373, 2409.16971]   |
| Step-Size Policy     | Fixed, diminishing, adaptive, or clipped       | [1802.04310, 2403.15244]   |
| Structured Curvature | Block/coordinate/low-rank Hessian correction   | [2006.09606, 1910.09373]   |

## 3. Theoretical Guarantees and Complexity

For strongly convex objectives with bounded Hessians and stochastic gradients, classical stochastic quasi-Newton methods achieve an $O(1/k)$ convergence rate in expected objective gap with properly decaying step sizes, matching SGD, but with typically smaller constants due to improved conditioning [1401.7020]. Variance-reduced stochastic quasi-Newton methods—with suitably designed variance-reduction (VR)—achieve global linear (geometric) rates and optimal oracle complexities $O((n + n^{1/2}\epsilon^{-2})$ for finite-sum problems [1503.08316, 2004.06479]. Recent momentum-accelerated quasi-Newton variants further match the best lower bounds on first-order stationarity [2004.06479].

General-purpose nonconvex variants guarantee that $\liminf_{k\to\infty} \mathbb E \|\nabla f(x_k)\|^2 = 0$ almost surely, with $O(\epsilon^{-2})$ sample complexity for returning an $\epsilon$-stationary point [1607.01231, 1412.1196]. More advanced methods, such as those for $(L_0, L_1)$-smoothness, achieve the best-known $O(\epsilon^{-3})$ sample complexity while leveraging curvature and gradient clipping [2403.15244].

Composite and nonsmooth methods based on stochastic quasi-Newton support proximal steps with per-iteration $O(d)$ complexity using compact representations and semismooth Newton updates [2409.16971]. In large-scale linear systems, stochastic quasi-Newton converges almost surely to the true least-squares solution under mild moment assumptions on sketched gradients [1702.07367].

## 4. Practical Techniques, Variants, and Computational Aspects

Implementations universally rely on the two-loop recursion for L-BFGS, with typical memory $m=5$–20, yielding $O(md)$ per-iteration cost, competitive with SGD for moderate $m$ and dominating the cost profile in high-dimensional regimes [1805.02338, 1401.7020]. Curvature pairs are updated at intervals to amortize Hessian-vector products. Spectral regularization, damping, and normalization are standard to avoid divergence or stagnation [1607.01231, 1805.02338, 1912.04456]. Coordinate selection or blockwise updating can trade per-iteration cost for some accuracy loss, but is effective in ultra-high-dimensional settings [1910.09373, 2006.09606].

Adaptive and line-search policies, including stochastic line search and step rejection, provide robustness in deep learning and ill-conditioned problems [1802.04310, 1909.01238]. Semismooth Newton proximal methods with efficient inner solvers make S-QN competitive for composite and $\ell_1$-regularized objectives [1910.09373, 2409.16971]. Methods have been implemented in production frameworks (e.g. TensorFlow, PyTorch), showing convergence stability and performance competitive with, or superior to, first-order methods on deep networks and classical ML benchmarks [1805.02338, 1909.03621].

Parallel and asynchronous designs offer significant wall-clock speedups and scalability, provided the memory and curvature update synchronization is managed properly [2011.00667, 2409.16971]. Absence of common random numbers (CRNs) in sampling oracles requires explicit eigenvalue control in L-BFGS updates and careful adaptive step size routines [2302.09128].

## 5. Empirical Performance and Comparative Evaluation

Stochastic quasi-Newton algorithms consistently outperform SGD and basic first-order variance-reduced methods in both convergence speed and ultimate objective gap, especially for ill-conditioned, sparse, or large-scale problems [1503.08316, 1401.7020, 1910.09373]. Variance-reduced quasi-Newton methods such as Vite and SpiderSQN reach lower objective in fewer epochs and significantly less wall-clock time on convex and nonconvex losses, including deep learning settings [1503.08316, 2004.06479]. In highly ill-conditioned or large logistic regression tasks, coordinate/structured methods yield 2–10× faster convergence than classical alternatives [1910.09373, 2006.09606]. In both linear least-squares and learning tasks, S-QN achieves robustness (convergence under broad sketching/sampling distributions), algorithmic stability (avoidance of divergence or numerical instability with moderate regularization), and scalability in memory and compute [1702.07367, 1802.04310, 2409.16971]. Adaptive and robust step-size schemes further enhance stability in unconstrained stochastic environments [1802.04310, 1909.01238]. Hybrid methods incorporating Nesterov acceleration or momentum exhibit further improvements, especially in deep learning [1909.03621, 2004.06479].

## 6. Notable Extensions, Limitations, and Future Directions

Recent progress includes adaptation to non-uniform smoothness via $(L_0, L_1)$-smoothness and gradient-clipped quasi-Newton steps [2403.15244], guaranteed linear convergence in nonsmooth and composite settings via single-loop and semismooth Newton solvers [2409.16971], and advanced block/coordinate structures for deep learning [2006.09606, 1910.09373]. Analytical frameworks now cover settings without common random numbers [2302.09128] and heavy-tailed Hessian approximations [2502.21078], using high-probability and stopping time analyses.

Ongoing research directions include (i) further scalability for distributed and asynchronous settings, (ii) theory and design for high-noise and limited memory regimes, (iii) efficient integration of curvature exploitation in federated and decentralized learning, and (iv) principled handling of interplay between stochasticity, nonsmoothness, and curvature, especially for modern high-dimensional nonconvex deep architectures [2011.00667, 2502.21078, 2006.09606, 2409.16971]. Open theoretical challenges remain in establishing universally optimal rates for general nonconvex stochastic composite problems and practical schemes under minimal spectral regularity assumptions.

Source: https://www.emergentmind.com/topics/stochastic-quasi-newton-method