Stochastic BFGS Methods

• Stochastic BFGS is a family of quasi-Newton methods that adapts the BFGS update to work with noisy gradient information while preserving inverse-Hessian preconditioning.
• The approach employs same-batch gradient differences, damping, and regularization techniques to stabilize curvature estimates and maintain positive definiteness.
• Variants include limited-memory, block, and Kronecker-factored methods, enabling scalable applications in convex optimization and deep learning.

Stochastic BFGS (S-BFGS) denotes a family of stochastic quasi-Newton methods that adapt the Broyden–Fletcher–Goldfarb–Shanno update to optimization problems in which only noisy gradient information is available. Across the literature, the common objective is to retain the inverse-Hessian preconditioning effect of BFGS while avoiding the instability caused by stochastic gradients, indefinite curvature, large-scale parameter spaces, and the absence of exact line search. The resulting methods range from regularized full-memory recursions for convex stochastic optimization and support vector machines to online limited-memory variants, damped schemes for nonconvex problems, block and sketched updates, Kronecker-factored layerwise methods for deep neural networks, and Bayesian formulations that treat the inverse Hessian as a random object inferred from noisy curvature observations (Mokhtari et al., 2014, Mokhtari et al., 2014, Wang et al., 2014, Mokhtari et al., 2014, Goldfarb et al., 2020, Carlon et al., 10 Jul 2025).

1. Conceptual scope and relation to deterministic BFGS

Deterministic BFGS maintains an inverse-Hessian approximation $H_k$ through the secant relation and the rank-two inverse update

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$

where $s_k=x_{k+1}-x_k$ and $y_k=\nabla f(x_{k+1})-\nabla f(x_k)$. In stochastic optimization, the central difficulty is that $y_k$ is no longer an exact curvature vector but a noisy quantity derived from sampled gradients, so enforcing the deterministic secant equation too literally can destabilize the metric estimate (Wang et al., 2014, Irwin et al., 2020).

The stochastic formulations therefore replace exact gradients by mini-batch or sample-average estimators and embed BFGS-type curvature learning inside a stochastic approximation step. Typical iterations have the form

$x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$

or, in support-vector-machine and regularized convex formulations,

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$

where $G_k$ or $g_t$ is an unbiased stochastic gradient estimator, $B_k$ approximates the Hessian, and the additional diagonal term is a safeguard against poor curvature estimates (Wang et al., 2014, Mokhtari et al., 2014).

The literature does not treat S-BFGS as a single algorithm. Rather, it comprises a class of methods that differ in how they form curvature pairs, preserve positive definiteness, limit memory, and control stochastic noise. This suggests that “S-BFGS” is best understood as a methodological lineage rather than a unique recursion. That interpretation is reinforced by work on online L-BFGS, stochastic damped-BFGS, regularized stochastic BFGS, stochastic block BFGS, Kronecker-factored quasi-Newton updates, and Bayesian stochastic BFGS variants (Mokhtari et al., 2014, Wang et al., 2014, Gower et al., 2016, Goldfarb et al., 2020, Carlon et al., 10 Jul 2025).

2. Stochastic curvature pairs and secant information

A defining design choice in S-BFGS is how the curvature pair is constructed. Early convex formulations emphasize that the variable displacement is standard,

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$0

but the gradient difference must be computed on the same mini-batch at both iterates: $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$1 with $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$2 and $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$3 evaluated using the same sampled data. This reuse is presented as essential for keeping the stochastic secant relation consistent; using two independent mini-batches is explicitly identified as insufficient for convergence guarantees in online limited-memory BFGS (Mokhtari et al., 2014, Mokhtari et al., 2014).

In regularized convex S-BFGS, the curvature vector is corrected to

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$4

which shifts the secant information away from singularity and yields the update

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$5

In nonconvex stochastic optimization, the analogous construction uses

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$6

again computed from the same batch, followed by damping before insertion into a BFGS formula (Mokhtari et al., 2014, Wang et al., 2014).

Later work generalizes the secant idea in two different directions. One direction treats inverse-Hessian learning as a stochastic matrix inversion problem. In that view, estimating $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$7 is posed as solving $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$8 under symmetry, and the update enforces a sketched condition

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$9

on a random subspace rather than on a single deterministic direction. Deterministic BFGS becomes a special case of this projection viewpoint (Gower et al., 2018). A second direction uses block curvature information: stochastic block BFGS replaces a single secant relation by the matrix-valued block secant equation

$s_k=x_{k+1}-x_k$0

where $s_k=x_{k+1}-x_k$1 contains multiple sketch directions and $s_k=x_{k+1}-x_k$2 supplies several curvature constraints at once (Gower et al., 2016).

A different reinterpretation appears in Bayesian S-BFGS. There, the inverse Hessian is treated as a random matrix with posterior

$s_k=x_{k+1}-x_k$3

and the update is the maximum a posteriori solution of a quadratic program balancing a prior centered at $s_k=x_{k+1}-x_k$4 against the noisy residual $s_k=x_{k+1}-x_k$5. This directly models the unreliability of stochastic curvature information instead of enforcing an exact secant equation (Carlon et al., 10 Jul 2025).

3. Positive definiteness, damping, and regularization

The central stability problem in S-BFGS is that stochastic or nonconvex curvature pairs need not satisfy $s_k=x_{k+1}-x_k$6. Much of the literature can be read as a sequence of mechanisms for restoring positive definiteness and spectral control.

In regularized stochastic BFGS for convex stochastic optimization, the corrected pair $s_k=x_{k+1}-x_k$7 ensures

$s_k=x_{k+1}-x_k$8

under a lower Hessian bound and $s_k=x_{k+1}-x_k$9. This yields $y_k=\nabla f(x_{k+1})-\nabla f(x_k)$0, hence $y_k=\nabla f(x_{k+1})-\nabla f(x_k)$1, while the extra term $y_k=\nabla f(x_{k+1})-\nabla f(x_k)$2 ensures

$y_k=\nabla f(x_{k+1})-\nabla f(x_k)$3

The method therefore falls back toward a regularized stochastic gradient step when curvature information is unreliable (Mokhtari et al., 2014, Mokhtari et al., 2014).

For nonconvex stochastic optimization, stochastic damped-BFGS introduces a Powell-style damping coefficient

$y_k=\nabla f(x_{k+1})-\nabla f(x_k)$4

and forms

$y_k=\nabla f(x_{k+1})-\nabla f(x_k)$5

The ensuing update

$y_k=\nabla f(x_{k+1})-\nabla f(x_k)$6

guarantees $y_k=\nabla f(x_{k+1})-\nabla f(x_k)$7 if $y_k=\nabla f(x_{k+1})-\nabla f(x_k)$8 (Wang et al., 2014).

Large-scale nonconvex optimization motivated a further combination of damping and regularization. In Sd-REG-LBFGS, the paper argues that simply composing standard damping with standard regularization can fail, and instead defines

$y_k=\nabla f(x_{k+1})-\nabla f(x_k)$9

with $y_k$0 chosen so that, if $y_k$1,

$y_k$2

The resulting update preserves positive definiteness and imposes a lower eigenvalue bound $y_k$3 (Chen et al., 2019).

Deep-network quasi-Newton methods introduce an additional issue: the Hessian is both indefinite and highly variable. The Kronecker-factored BFGS and L-BFGS methods in deep learning therefore use a double damping strategy. First, a Powell-style damping mixes $y_k$4 with $y_k$5 to guarantee a lower bound. Second, a Levenberg–Marquardt-like stabilization using $y_k$6 is applied to control the size of both Hessian and inverse-Hessian approximations. The paper derives bounds of the form

$y_k$7

with the practical intent of controlling both lower and upper spectral growth (Goldfarb et al., 2020).

A related but distinct response to noise is secant penalization. SP-BFGS replaces the hard secant condition by a penalty term and produces an update that interpolates smoothly between full BFGS and no update, with the relaxed positive-definiteness condition

$y_k$8

This method is not presented as a classical S-BFGS algorithm, but it is explicitly situated alongside stochastic and noisy-gradient quasi-Newton methods and addresses the same fragility of overwriting curvature information under noise (Irwin et al., 2020).

4. Memory limitation, sketching, and structural approximations

Scalability is a defining fault line in the S-BFGS literature. Full dense updates become impractical in high dimension, so many influential developments are structured or limited-memory variants.

Variant	Core mechanism	Representative setting
RES / regularized S-BFGS	corrected pair $y_k$9, progress term $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$0	convex stochastic optimization, SVMs (Mokhtari et al., 2014)
oLBFGS	fixed window of $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$1 curvature pairs, two-loop recursion	large-scale machine learning (Mokhtari et al., 2014)
Stochastic block BFGS	block secant $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$2	variance-reduced ERM (Gower et al., 2016)
K-BFGS / K-BFGS(L)	layerwise Kronecker factors $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$3	deep feed-forward neural networks (Goldfarb et al., 2020)
Bayesian L-S-BFGS	posterior-weighted limited-memory update with pair precision $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$4	stochastic optimization with small batch sizes (Carlon et al., 10 Jul 2025)

Online limited-memory BFGS is the canonical storage-reduction strategy. It stores only the most recent $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$5 pairs $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$6, computes search directions through the standard two-loop recursion, and reduces memory and compute from $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$7 for dense stochastic BFGS to $x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$8. The method emphasizes that using a common sample set at consecutive iterates is necessary for its convergence proof and that a scaled identity initialization

$x_{k+1}=x_k-\alpha_k(B_k^{-1}+\zeta_k I)G_k$9

is part of the stability argument (Mokhtari et al., 2014).

Stochastic block BFGS increases the amount of curvature extracted per update rather than reducing it. The update is defined as a projection onto multiple curvature constraints,

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$0

with closed form

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$1

The paper also develops limited-memory and factored forms $w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$2, together with Gaussian, previous-direction, and self-conditioning sketches (Gower et al., 2016).

Deep-network S-BFGS variants use structure rather than memory truncation alone. For each layer $w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$3, the gradient and Hessian block satisfy

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$4

which motivates the approximation

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$5

The inverse-Hessian factor for a layer is taken as

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$6

and the parameter update becomes

$w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$7

The algorithm performs two forward-backward passes on the same minibatch and updates the $w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$8 factor with BFGS or L-BFGS, while $w_{t+1}=w_t-\epsilon_t(B_t^{-1}+\Gamma I)g_t,$9 is handled by Hessian-action BFGS with Levenberg–Marquardt regularization (Goldfarb et al., 2020).

Variable sample-size stochastic quasi-Newton methods extend the limited-memory idea into regimes with smoothing and regularization. In that framework,

$G_k$0

with sample size $G_k$1 increasing over time, and the rsL-BFGS update is designed to preserve secant structure even when the objective itself is being altered by smoothing and regularization (Jalilzadeh et al., 2018).

5. Convergence guarantees and complexity statements

For strongly convex stochastic optimization, several S-BFGS variants obtain almost sure convergence together with an expected $G_k$2 decay of the objective gap. Regularized stochastic BFGS proves

$G_k$3

under bounded sample Hessians, bounded second moments of stochastic gradients, $G_k$4, and Robbins–Monro stepsizes, and with the harmonic rule $G_k$5 derives

$G_k$6

The support-vector-machine adaptation proves the same qualitative conclusions in classifier training, formulated as almost sure convergence to the optimal classifier and “linear in expectation” in the stochastic-approximation sense (Mokhtari et al., 2014, Mokhtari et al., 2014).

Online limited-memory BFGS establishes a similar rate under strong convexity and bounded sample Hessians. It proves uniform spectral bounds

$G_k$7

through trace and determinant control, obtains the one-step inequality

$G_k$8

and then shows almost sure convergence and

$G_k$9

for harmonic stepsizes (Mokhtari et al., 2014).

For nonconvex stochastic optimization, the general SQN framework assumes only stochastic first-order oracle access and requires uniform bounds

$g_t$0

Under Lipschitz continuity of $g_t$1, lower boundedness, unbiased bounded-variance stochastic gradients, and square-summable but non-summable stepsizes, the method satisfies

$g_t$2

Its randomized counterpart RSQN yields an $g_t$3 worst-case SFO complexity for obtaining an $g_t$4-solution with $g_t$5 (Wang et al., 2014).

The deep-learning K-BFGS(L) analysis is more conservative. The proof applies to a variant with damping/skipping for $g_t$6, exact inversion for $g_t$7, and decreasing stepsizes $g_t$8 with $g_t$9. Under smoothness, bounded-below objective, unbiased stochastic gradients with bounded variance, and bounded activations, the paper shows

$B_k$0

and then derives

$B_k$1

iterations (Goldfarb et al., 2020).

Sd-REG-LBFGS targets large-scale nonconvex stochastic optimization and proves uniform matrix bounds

$B_k$2

together with the average gradient-norm estimate

$B_k$3

under bounded Hessians of the random function and bounded stochastic-gradient variance. The stated iteration complexity for reaching an $B_k$4-stationarity criterion is

$B_k$5

(Chen et al., 2019).

Two additional strands sharpen the theoretical landscape. Accelerated stochastic matrix inversion proves a linear convergence rate

$B_k$6

for the inversion primitive, improving over the non-accelerated sketch-and-project rate $B_k$7, and then transfers this acceleration to BFGS-like matrix updates (Gower et al., 2018). Variable sample-size SQN proves linear mean convergence and oracle complexity $B_k$8 in strongly convex smooth regimes, retains linear convergence under Moreau smoothing for smoothable nonsmooth problems, and obtains sublinear rates such as $B_k$9 and $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$00 in merely convex settings (Jalilzadeh et al., 2018).

6. Empirical behavior, distinctions within the literature, and recurrent misconceptions

Empirically, the earliest convex and SVM-focused papers report substantial gains over plain SGD. In the support-vector-machine study with squared hinge loss, for $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$01 the stochastic BFGS method reaches $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$02 after about $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$03 processed samples, whereas SGD needs about $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$04. For $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$05, after processing $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$06 samples, S-BFGS attains $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$07, whereas SGD is still at about $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$08. Using a test set of $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$09 points and repeating the experiment $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$10 times, the mean test accuracy of S-BFGS is about $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$11, while SGD often tops out near $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$12 (Mokhtari et al., 2014).

Large-scale online limited-memory BFGS shows that the curvature benefit can persist in extremely high dimension. On a search engine advertising logistic-regression problem with

$$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$13

the paper reports that oLBFGS reaches a low log-likelihood value after processing only about $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$14 feature vectors, while SGD is still far from that level after $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$15. The conclusion stated is that oLBFGS can train the regressor using less than $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$16 of the data required by SGD to obtain similar classification accuracy (Mokhtari et al., 2014).

Deep-learning experiments broaden the application range but also show the importance of structure. Kronecker-factored K-BFGS and especially K-BFGS(L) are tested on MNIST, FACES, and CURVES with 9- and 13-layer ReLU autoencoders. The reported finding is that these methods are consistently better than first-order methods in training loss and often substantially better in test error, while matching or nearly matching KFAC; they also often achieve comparable performance to KFAC with less computation per iteration because they avoid the expensive matrix inversions required by KFAC (Goldfarb et al., 2020).

Variance-reduced block/sketched quasi-Newton methods and Bayesian limited-memory methods extend the empirical record further. Stochastic block BFGS combined with SVRG is tested on large-scale logistic regression using LIBSVM datasets and is reported as more robust and substantially outperforming current state-of-the-art methods, with the previous-direction sketch often best overall (Gower et al., 2016). Bayesian L-S-BFGS is evaluated on multinomial logistic regression with $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$17-regularization on madelon, mushrooms, MNIST, and CIFAR-10, with dimensionality up to $$H_{k+1} = \left(I-\rho_k s_k y_k^\top\right)H_k\left(I-\rho_k y_k s_k^\top\right)+\rho_k s_k s_k^\top, \qquad \rho_k=\frac{1}{s_k^\top y_k},$$18. The reported result is that L-S-BFGS is consistently faster and more robust than SdLBFGS and oLBFGS across all datasets, and that it tolerates larger step sizes than the competing limited-memory stochastic quasi-Newton methods (Carlon et al., 10 Jul 2025).

Several misconceptions recur in this literature. A first misconception is that S-BFGS simply means “BFGS with minibatch gradients.” The repeated use of same-batch gradient differences, damping, regularization, pair rejection, structured factorizations, sketching, and limited memory indicates otherwise (Wang et al., 2014, Goldfarb et al., 2020). A second misconception is that all stochastic quasi-Newton papers study the same object. One stochastic L-BFGS analysis explicitly notes that it is not about classical full-memory S-BFGS; full-memory BFGS appears there only as background or as a proof device for spectral bounds (Zhao et al., 2017). A third misconception is that more aggressive inverse-Hessian learning is always better. The noisy-gradient literature repeatedly documents the opposite phenomenon: naive or lightly modified BFGS can diverge, oscillate, or amplify noise, whereas conservative curvature assimilation, damping, or penalized secant enforcement can improve robustness even when it moves the metric toward the true inverse more slowly (Irwin et al., 2020, Carlon et al., 10 Jul 2025).