--- title: Stochastic BFGS Methods url: https://www.emergentmind.com/topics/stochastic-bfgs-s-bfgs type: topic --- # Stochastic BFGS Methods Stochastic BFGS (S-BFGS) denotes a family of stochastic quasi-Newton methods that adapt the Broyden–Fletcher–Goldfarb–Shanno update to optimization problems in which only noisy gradient information is available. Across the literature, the common objective is to retain the inverse-Hessian preconditioning effect of BFGS while avoiding the instability caused by stochastic gradients, indefinite curvature, large-scale parameter spaces, and the absence of exact line search. The resulting methods range from regularized full-memory recursions for convex stochastic optimization and support vector machines to online limited-memory variants, damped schemes for nonconvex problems, block and sketched updates, Kronecker-factored layerwise methods for deep neural networks, and Bayesian formulations that treat the inverse Hessian as a random object inferred from noisy curvature observations [1401.7625, 1402.4861, 1412.1196, 1409.2045, 2006.08877, 2507.07729]. ## 1. Conceptual scope and relation to deterministic BFGS Deterministic BFGS maintains an inverse-Hessian approximation \(H_k\) through the secant relation and the rank-two inverse update \[ H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k}, \] where \(s_k=x_{k+1}-x_k\) and \(y_k=\nabla f(x_{k+1})-\nabla f(x_k)\). In stochastic optimization, the central difficulty is that \(y_k\) is no longer an exact curvature vector but a noisy quantity derived from sampled gradients, so enforcing the deterministic secant equation too literally can destabilize the metric estimate [1412.1196, 2010.01275]. The stochastic formulations therefore replace exact gradients by mini-batch or sample-average estimators and embed BFGS-type curvature learning inside a stochastic approximation step. Typical iterations have the form \[ x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k \] or, in support-vector-machine and regularized convex formulations, \[ w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t, \] where \(G_k\) or \(g_t\) is an unbiased stochastic gradient estimator, \(B_k\) approximates the Hessian, and the additional diagonal term is a safeguard against poor curvature estimates [1412.1196, 1402.4861]. The literature does not treat S-BFGS as a single algorithm. Rather, it comprises a class of methods that differ in how they form curvature pairs, preserve positive definiteness, limit memory, and control stochastic noise. This suggests that “S-BFGS” is best understood as a methodological lineage rather than a unique recursion. That interpretation is reinforced by work on online L-BFGS, stochastic damped-BFGS, regularized stochastic BFGS, stochastic block BFGS, Kronecker-factored quasi-Newton updates, and Bayesian stochastic BFGS variants [1409.2045, 1412.1196, 1603.09649, 2006.08877, 2507.07729]. ## 2. Stochastic curvature pairs and secant information A defining design choice in S-BFGS is how the curvature pair is constructed. Early convex formulations emphasize that the variable displacement is standard, \[ s_t=w_{t+1}-w_t, \] but the gradient difference must be computed on the same mini-batch at both iterates: \[ y_t=g_{t+1}-g_t, \] with \(g_t\) and \(g_{t+1}\) evaluated using the same sampled data. This reuse is presented as essential for keeping the stochastic secant relation consistent; using two independent mini-batches is explicitly identified as insufficient for convergence guarantees in online limited-memory BFGS [1402.4861, 1409.2045]. In regularized convex S-BFGS, the curvature vector is corrected to \[ r_t:=y_t-\delta s_t, \] which shifts the secant information away from singularity and yields the update \[ B_{t+1}=B_t+\frac{r_t r_t^\top}{r_t^\top s_t}-\frac{B_t s_t s_t^\top B_t}{s_t^\top B_t s_t}. \] In nonconvex stochastic optimization, the analogous construction uses \[ \hat y_k=\bar G_{k+1}-G_k-\delta s_k, \] again computed from the same batch, followed by damping before insertion into a BFGS formula [1402.4861, 1412.1196]. Later work generalizes the secant idea in two different directions. One direction treats inverse-Hessian learning as a stochastic matrix inversion problem. In that view, estimating \(A^{-1}\) is posed as solving \(AX=I\) under symmetry, and the update enforces a sketched condition \[ S_k^\top A X=S_k^\top \] on a random subspace rather than on a single deterministic direction. Deterministic BFGS becomes a special case of this projection viewpoint [1802.04079]. A second direction uses block curvature information: stochastic block BFGS replaces a single secant relation by the matrix-valued block secant equation \[ H_t \nabla^2 f_{T_t}(x_t)D_t=D_t, \] where \(D_t\) contains multiple sketch directions and \(Y_t=\nabla^2 f_{T_t}(x_t)D_t\) supplies several curvature constraints at once [1603.09649]. A different reinterpretation appears in Bayesian S-BFGS. There, the inverse Hessian is treated as a random matrix with posterior \[ \pi(H\mid s_k,y_k)\propto p(y_k\mid H,s_k)\,\pi(H), \] and the update is the maximum a posteriori solution of a quadratic program balancing a prior centered at \(H_k\) against the noisy residual \(s_k-Hy_k\). This directly models the unreliability of stochastic curvature information instead of enforcing an exact secant equation [2507.07729]. ## 3. Positive definiteness, damping, and regularization The central stability problem in S-BFGS is that stochastic or nonconvex curvature pairs need not satisfy \(s_k^\top y_k>0\). Much of the literature can be read as a sequence of mechanisms for restoring positive definiteness and spectral control. In regularized stochastic BFGS for convex stochastic optimization, the corrected pair \(r_t=y_t-\delta s_t\) ensures \[ r_t^\top s_t=(y_t-\delta s_t)^\top s_t\ge (m-\delta)\|s_t\|^2>0 \] under a lower Hessian bound and \(\delta -\frac{1}{\beta_k}. \] This method is not presented as a classical S-BFGS algorithm, but it is explicitly situated alongside stochastic and noisy-gradient quasi-Newton methods and addresses the same fragility of overwriting curvature information under noise [2010.01275]. ## 4. Memory limitation, sketching, and structural approximations Scalability is a defining fault line in the S-BFGS literature. Full dense updates become impractical in high dimension, so many influential developments are structured or limited-memory variants. | Variant | Core mechanism | Representative setting | |---|---|---| | RES / regularized S-BFGS | corrected pair \(r_t=y_t-\delta s_t\), progress term \(\Gamma I\) | convex stochastic optimization, SVMs [1401.7625] | | oLBFGS | fixed window of \(\tau\) curvature pairs, two-loop recursion | large-scale machine learning [1409.2045] | | Stochastic block BFGS | block secant \(H_t\nabla^2 f_{T_t}(x_t)D_t=D_t\) | variance-reduced ERM [1603.09649] | | K-BFGS / K-BFGS(L) | layerwise Kronecker factors \(A_l\otimes G_l\) | deep feed-forward neural networks [2006.08877] | | Bayesian L-S-BFGS | posterior-weighted limited-memory update with pair precision \(p_k\) | stochastic optimization with small batch sizes [2507.07729] | Online limited-memory BFGS is the canonical storage-reduction strategy. It stores only the most recent \(\tau\) pairs \(\{(s_u,y_u)\}\), computes search directions through the standard two-loop recursion, and reduces memory and compute from \(O(n^2)\) for dense stochastic BFGS to \(O(\tau n)\). The method emphasizes that using a common sample set at consecutive iterates is necessary for its convergence proof and that a scaled identity initialization \[ H_{t,0}^{-1}=\hat\gamma_t I, \qquad \hat\gamma_t=\frac{s_{t-1}^\top y_{t-1}}{\|y_{t-1}\|^2}, \] is part of the stability argument [1409.2045]. Stochastic block BFGS increases the amount of curvature extracted per update rather than reducing it. The update is defined as a projection onto multiple curvature constraints, \[ H_t=\arg\min_H \|H-H_{t-1}\|_t^2 \quad\text{s.t.}\quad H\nabla^2 f_{T_t}(x_t)D_t=D_t,\; H=H^\top, \] with closed form \[ H_t=D_t\Delta_t D_t^\top+\bigl(I-D_t\Delta_t Y_t^\top\bigr)H_{t-1}\bigl(I-Y_t\Delta_t D_t^\top\bigr), \qquad \Delta_t=(D_t^\top Y_t)^{-1}. \] The paper also develops limited-memory and factored forms \(H_t=L_tL_t^\top\), together with Gaussian, previous-direction, and self-conditioning sketches [1603.09649]. Deep-network S-BFGS variants use structure rather than memory truncation alone. For each layer \(l\), the gradient and Hessian block satisfy \[ \nabla f_l(i)=g_l(i)\,a_{l-1}(i)^\top, \qquad \nabla^2 f_l(i)=\bigl(a_{l-1}(i)a_{l-1}(i)^\top\bigr)\otimes G_l(i), \] which motivates the approximation \[ \mathbb E_i[\nabla^2 f_l(i)]\approx A_l\otimes G_l. \] The inverse-Hessian factor for a layer is taken as \[ H^l=H_a^l\otimes H_g^l, \] and the parameter update becomes \[ p_l=H_g^l\,\widehat{\nabla f_l}\,H_a^l, \qquad W_l\leftarrow W_l-\alpha p_l. \] The algorithm performs two forward-backward passes on the same minibatch and updates the \(G_l\) factor with BFGS or L-BFGS, while \(A_l\) is handled by Hessian-action BFGS with Levenberg–Marquardt regularization [2006.08877]. Variable sample-size stochastic quasi-Newton methods extend the limited-memory idea into regimes with smoothing and regularization. In that framework, \[ x_{k+1}=x_k-\gamma_k H_k \left(\frac{1}{N_k}\sum_{j=1}^{N_k}u_k(x_k,\omega_{j,k})\right), \] with sample size \(N_k\) increasing over time, and the rsL-BFGS update is designed to preserve secant structure even when the objective itself is being altered by smoothing and regularization [1804.05368]. ## 5. Convergence guarantees and complexity statements For strongly convex stochastic optimization, several S-BFGS variants obtain almost sure convergence together with an expected \(O(1/t)\) decay of the objective gap. Regularized stochastic BFGS proves \[ \liminf_{t\to\infty}\|x_t-x^*\|^2=0 \quad \text{a.s.} \] under bounded sample Hessians, bounded second moments of stochastic gradients, \(\delta