---
title: Limited-Memory BFGS for Large-Scale Optimization
url: https://www.emergentmind.com/topics/limited-memory-broyden-fletcher-goldfarb-shanno-l-bfgs-algorithm
type: topic
---

# Limited-Memory BFGS for Large-Scale Optimization

The Limited-Memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS) Algorithm is a large-scale unconstrained optimization method in the quasi-Newton family, designed to approximate the Newton or BFGS step efficiently in high-dimensional settings. By storing only a limited number of curvature (iterate and gradient displacement) pairs, L-BFGS achieves $O(mn)$ per-iteration complexity and memory, with $m\ll n$, rendering it practical for applications such as machine learning, PDE-constrained optimization, and electronic structure computation. Recent research details its algorithmic structure, convergence properties under various regularity conditions, extensions for noisy or nonconvex problems, memory management, and practical enhancements [1409.2045, 2401.03805, 2301.05447, 1903.03471, 1811.02693, 2101.04413, 2508.10211].

## 1. Algorithmic Structure and Update Mechanism

L-BFGS is a variant of the BFGS quasi-Newton method, which approximates the inverse Hessian of the objective function using only first-order gradient information. At iteration $k$, it maintains the most recent $m$ pairs of iterates and gradient differences:
- $s_k = x_{k+1} - x_k$
- $y_k = \nabla f(x_{k+1}) - \nabla f(x_k)$

The full-memory BFGS update for the inverse Hessian $H_{k+1}$ is
$$
H_{k+1} = (I - \rho_k s_k y_k^T) H_k (I - \rho_k y_k s_k^T) + \rho_k s_k s_k^T
$$
where $\rho_k = 1/(y_k^Ts_k)$. In the limited-memory variant, only the most recent $m$ $(s, y)$ pairs are retained, and $H_k$ is never explicitly formed.

The search direction $p_k = -H_k \nabla f(x_k)$ is computed efficiently via the two-loop recursion. Given a "seed" matrix $H_0 = \gamma_k I$ (with $\gamma_k$ typically updated using curvature information), this recursion requires $O(mn)$ operations per iteration.

For stochastic or online settings, as in large-scale learning, L-BFGS adapts by using stochastic gradients and corresponding curvature pairs; in this regime, it is sometimes referred to as oL-BFGS [1409.2045].

## 2. Memory, Computational Complexity, and Variants

The primary advantage of L-BFGS is its per-iteration scalability:
- Memory: $O(mn)$ (store $m$ $n$-dimensional vectors each for $s$ and $y$).
- Computational cost: $O(mn)$ for the two-loop recursion and $O(n)$ for each gradient evaluation (or $O(bn)$ for batch size $b$).

This is in stark contrast to full-memory BFGS, which requires $O(n^2)$ memory and computation per step.

Several enhancements and practical variants exist:
- Adaptive memory strategies, where memory parameter $m$ is increased during training as the objective becomes more quadratic [2012.07434].
- Regularized L-BFGS (RL-BFGS), which modifies the curvature pairs by a regularization parameter to improve robustness and allow for trust-region-like acceptance rules [2101.04413].
- Dense subspace initialization, splitting the space between subspace covered by curvature vectors and its orthogonal complement for better trust-region geometry [1710.02396].
- Displacement aggregation: detecting and aggregating redundant steps to further restrict memory requirements while preserving superlinear convergence [2301.05447, 1903.03471].

## 3. Convergence Theory

L-BFGS retains most convergence properties of full-memory BFGS on smooth objectives, with some caveats for nonconvex or nonsmooth functions.

- **Strong Convexity and Smoothness**: If each $f_i$ satisfies uniform Hessian bounds $\lambda I \preceq \nabla^2 f_i(x) \preceq \Lambda I$, then the L-BFGS or oL-BFGS method converges almost surely to the minimum for stochastic objectives under step-size decay ($\sum_k \epsilon_k = \infty$, $\sum_k \epsilon_k^2 < \infty$) and enjoys $O(1/k)$ expected convergence rate [1409.2045].
- **Nonconvex Objectives**: Globalization strategies, such as cautious updating and adaptive selection of which curvature pairs to retain, allow for global convergence to stationary points under weaker assumptions—requiring only continuity of the gradient (rather than Lipschitz) and Armijo or Wolfe-Powell line searches [2401.03805].
- **Superlinear Convergence**: With displacement aggregation and under standard Dennis–Moré-type regularity, local superlinear convergence is guaranteed if the memory window covers a basis for $\mathbb R^n$ (or, in practice, when the local behavior is sufficiently well captured by $m$-recent pairs) [2301.05447, 1903.03471].
- **Nonsmooth Functions**: On specific nonsmooth convex functions, scaled memoryless BFGS (and more generally scaled L-BFGS) can stall at non-optimal points even when the objective is unbounded below. This failure is more frequent for small memory, larger scaling, and Armijo–Wolfe line searches [1810.00292].

## 4. Enhancements for Nonconvexity, Robustness, and Practicalities

Recent work has focused on extending L-BFGS to settings in which classical assumptions fail or in which practical performance requires further robustness:
- **Globalized L-BFGS**: Adaptive, so-called "cautious updating," where only stored pairs satisfying $q(s, y) \ge \omega_k$ are used, guaranteeing globally bounded inverse Hessian approximations and stationary-point limits of cluster points under weak smoothness [2401.03805].
- **Displacement Aggregation**: By aggregating pairs when one step is in the span of others, the method retains the theoretical update equivalence to full-memory BFGS—crucial for achieving superlinear rates—while capping storage [2301.05447, 1903.03471].
- **Operator-Based Improvements**: Image and projection operator modifications of L-BFGS allow for accelerated quadratic termination even without exact line search, reducing iteration count at the cost of minor preprocessing or additional local solves [2508.10211].
- **Structured L-BFGS**: When partial second-derivative structure is available, "structured" compact representations yield further efficiency and improved iteration counts in large-scale PDE and imaging applications [2208.00057].
- **Adaptive Memory**: Memory can be increased during training to reflect the increased trust in curvature information near minimizers, improving solution quality in deep learning [2012.07434].
- **Learning-Based Enhancements**: Neural network-based step-size policies for L-BFGS eliminate repeated line searches, yielding improved or competitive convergence rates in practice [2010.01311].

## 5. Applications in Machine Learning and Scientific Computing

L-BFGS and its variants are foundational for a range of large-scale optimization problems:
- **Supervised Learning**: Empirical risk minimization (e.g., SVMs, logistic regression) with extremely high-dimensional feature vectors. L-BFGS achieves convergence in $1\%$ of the samples and runtime required by stochastic gradient descent in sparse large-scale classification problems [1409.2045].
- **Neural Network Training**: L-BFGS (including online and mini-batch variants) is employed in deep learning, where it achieves faster and more stable convergence compared to first-order methods, especially in the vicinity of optima [1909.01994, 1811.02693, 2012.07434].
- **Reinforcement Learning**: L-BFGS, with or without line search, attains robust convergence and test performance in deep RL (e.g., DQN and policy gradient tasks) using significantly fewer environment steps than classic DQN or TRPO [1811.02693, 1909.01994].
- **PDE-Constrained Optimization and Imaging Inverse Problems**: Structured and trust-region variants of L-BFGS yield iteration reductions in large-scale systems where partial Hessian information is available [2208.00057].
- **Electronic Structure and Quantum Chemistry**: Low-rank L-BFGS updates and trust-region integration enable efficient, robust mean-field orbital optimization, outperforming DIIS and augmented-Hessian methods in challenging SCF settings [2307.00560].

## 6. Limitations and Advanced Topics

- **Nonsmooth Optimization**: L-BFGS with conventional scaling is fragile on nonsmooth functions, with theoretical analysis showing that it may stall even when the gradient method succeeds. This highlights the need for careful scaling/safeguards or alternative strategies when applying L-BFGS to nonsmooth settings [1810.00292].
- **Hyperparameter Selection**: Memory window ($m$), step-size, and initial Hessian scaling all impact efficiency and robustness. Practical heuristics suggest $m=20\sim 40$ for high-dimensional problems and periodic adaptation based on validation loss for deep learning tasks [2012.07434, 1909.01994].
- **Numerical Stability**: To ensure positive definiteness, updates with $y_k^T s_k \leq 0$ are typically skipped. Dense subspace initializations and regularization further improve numerical properties in trust-region frameworks [1710.02396, 2101.04413].

## 7. Numerical and Empirical Performance

Extensive empirical studies confirm that L-BFGS delivers significant reductions in wall-clock time, sample complexity, and iteration counts compared to both stochastic/first-order and full-memory second-order methods across a wide variety of domains [1409.2045, 2301.05447, 1909.01994, 2307.00560].

Illustrative results:

| Application                    | L-BFGS Convergence  | Reference        |
|---------------------------------|---------------------|------------------|
| Large-scale SVMs (n=10³, 10⁵)   | $\lesssim$20,000 samples for O($10^{-5}$) objective; $\ll$SGD | [1409.2045]     |
| Deep RL (ATARI, DQN)            | Robust learning; faster than TRPO; fewer steps than DQN | [1811.02693], [1909.01994] |
| Large-scale imaging (LIBSVM)     | Fewer iterations and time than classical L-BFGS | [2208.00057]    |
| SCF in quantum chemistry         | 0 failures, robust convergence, fewer gradient builds than DIIS or augmented-Hessian | [2307.00560]    |

The method's empirical performance is enhanced when domain-specific structure, displacement aggregation, adaptive memory management, or learned step-size policies are incorporated.

---

**References**: [1409.2045], [2401.03805], [2301.05447], [1903.03471], [1811.02693], [2101.04413], [1710.02396], [2508.10211], [2208.00057], [2012.07434], [1810.00292], [2307.00560], [1909.01994], [2010.01311]

Source: https://www.emergentmind.com/topics/limited-memory-broyden-fletcher-goldfarb-shanno-l-bfgs-algorithm