---
title: Variable-Metric Proximal Gradient
url: https://www.emergentmind.com/topics/variable-metric-proximal-gradient
type: topic
---

# Variable-Metric Proximal Gradient

Variable-metric proximal gradient (VM-PG) methods form a broad class of algorithms for solving composite optimization problems of the form
\[
\min_{x}~F(x) := f(x) + g(x),
\]
where \(f\) is typically smooth (not necessarily convex) and \(g\) is convex, possibly nonsmooth. The key distinguishing feature of VM-PG is the adaptation of a metric—represented by a positive-definite symmetric operator/matrix, which may vary per iteration—to the problem’s local geometry. This generalizes the classical proximal gradient paradigm by improving curvature adaptation, preconditioning, and exploitability of second-order or quasi-Newton information.

## 1. Mathematical Foundations and Basic Iteration

Variable-metric proximal gradient methods replace the standard Euclidean proximal step
\[
\mathrm{prox}_{g}(z) := \arg\min_{y} \left\{ g(y) + \frac{1}{2\alpha}\|y-z\|^2 \right\}
\]
with a variable-metric version:
\[
\mathrm{prox}_{g}^{H}(z) := \arg\min_{y} \left\{ g(y) + \frac{1}{2}(y-z)^\top H (y-z) \right\},
\]
where \(H\succ0\) is a symmetric positive-definite metric matrix that can depend on the current iterate.

A general iteration of VM-PG takes the form
\[
x_{k+1} = \mathrm{prox}_{\gamma_k g}^{H_k}(x_k - \gamma_k H_k^{-1} \nabla f(x_k)),
\]
where \(H_k\) may be chosen as the scaled identity (recovering the standard method), a diagonal/scaled matrix (for coordinate preconditioning), a quasi-Newton/BFGS matrix, or even the Hessian of \(f\) (proximal-Newton variant) [1910.07056][1506.00385][2301.00631].

This generalization enables matching the iteration to the local curvature or structure of the composite objective, enhancing convergence in ill-conditioned or high-dimensional settings.

## 2. Algorithmic Strategies and Metric Construction

Variable-metric choices include:
- **Diagonal and Barzilai–Borwein metrics:** Efficient per-iteration (O(n) cost), allowing cheap curvature adaptation via diagonal scaling derived from secant conditions [1910.07056][2010.00817].
- **Quasi-Newton/BFGS/limited-memory:** Capture curvature much beyond coordinatewise scaling but with moderate cost. Hessian approximations are updated by secant formulas, sometimes restricted to support subspaces in sparsity problems [1610.00960][2507.18363][2108.03365].
- **Hessian-based (proximal Newton):** Highest curvature fidelity, used when second derivatives are available and subproblem prox computations remain affordable [1812.05243][2308.13776].
- **Bregman metrics:** Generalize distance beyond quadratic forms, using a kernel-generating function \(\phi\) and its Hessian [2510.06615][2109.05690][2008.13627].

Metric selection is often coupled with line search, safeguarding (bounding \(H_k\) from above/below), or inexactness tolerance if the model/prox step can only be solved approximately.

## 3. Theoretical Guarantees: Convergence and Complexity

A range of global and local convergence results accompany VM-PG schemes:

- **Global subsequential convergence:** For merely convex, nonsmooth \(g\) and smooth \(f\), under suitable conditions (e.g., coercivity, metric bounds), every cluster point is stationary [1506.00385][1910.07056][2507.18363][2308.13776].
- **O(1/k) sublinear rates:** Assuming L-smoothness or relative smoothness, sequence averages of objective values or normed gradient mappings decay at this rate [1506.00385][2109.05690].
- **Local linear and superlinear rates:** If \(f\) is strongly convex (or satisfies quadratic growth/KŁ/KL property) and the metric sequence stabilizes or matches local curvature (e.g., Hessian), then local linear (or superlinear) rates are achieved [2507.18363][1910.07056][2108.03365].
- **KL-type global convergence:** Under the Kurdyka–Łojasiewicz property or similar error-bounds, global convergence (not just subsequence) follows, with rates dictated by the KL exponent [2308.13776][2504.11023][2510.06615][2008.13627].
- **Stochastic, variance-reduced, and nonconvex settings:** Recent frameworks extend VM-PG to stochastic/finite-sum settings, combining variance-reduction with variable-metric proximity for guaranteed expected stationarity within \(O(\sqrt{n}\epsilon^{-2})\) complexity in the finite-sum case [2301.00631][2010.00817].

The strength of the convergence result depends delicately on properties of the metric sequence, regularity of \(f\) and \(g\), the inexactness tolerance in subproblem solves, and descent monitoring.

## 4. Extensions: Inexactness, Line-Search, and Bregman/Composite Settings

Practical instantiations often require:

- **Inexact proximal solves:** Subproblem solutions may not be exact; convergence is preserved under absolute or relative error criteria (e.g., absolute function gap versus minimum, or sufficient contraction), provided errors decay or sum [1506.00385][2109.05690][2308.13776][2504.11023].
- **Line-search/backtracking:** Sufficient decrease enforced via Armijo or Armijo–Wolfe conditions, often tailored to the variable metric, preserves monotonicity and enables superlinear rates [1506.00385][2510.06615][2507.18363][2308.13776]. Stepsizes may be chosen via nonmonotone search or fixed in certain settings.
- **Bregman distances:** Replacing the Euclidean norm with a Bregman divergence \(D_\phi(x, y)\) based on a convex kernel \(\phi\), enabling adaptation to manifold settings, entropy geometry, or structured non-quadratic curvature [2510.06615][2109.05690][2008.13627]. The resulting method generalizes VM-PG and connects to mirror-descent and other geometric optimization strategies.
- **Composite/nonsmooth/nonconvex settings:** Advanced VM-PG methods address nonsmooth, nonconvex, fractional, or non-Euclidean objectives with model-based steps, o-minimal definability, and KL machinery [2507.18363][2308.13776][2008.13627][2504.11023].

## 5. Specialized Schemes and Paradigm Integration

Numerous specialized frameworks exist under the VM-PG umbrella:

- **VMEPIHT for ℓ₀ minimization:** Alternates a hard-thresholded proximal-IHT with a restricted quasi-Newton update on the active support, achieving global convergence, eventual support stabilization, and, for least-squares problems, local superlinear rates [2108.03365].
- **QNing:** Treats the (squared) Moreau envelope with a variable metric (often L-BFGS) for outer acceleration, allows for inexact subproblem solutions, and is compatible with sparsity-induced regularization [1610.00960].
- **Homotopy methods:** Introduce a parameterized optimality condition to pursue solution trajectory continuation (e.g., by decreasing regularization weight), with metric adaptation yielding global/even finite iteration complexity under strong convexity or self-concordance [1812.05243].
- **Stochastic and residual-variance-reduced methods:** 3P-SPIDER, VM-mSRGBB, and related approaches unite stochastic recursive gradient estimators with variable-metric preconditioning, obtaining optimal or near-optimal sample complexities in large-scale learning [2301.00631][2010.00817].
- **Fractional and model-based extensions:** Fractional objectives and sophisticated local model functions with inexactness operate within variable-metric frameworks, accommodating extremely broad problem classes while inheriting the foundational convergence guarantees [2507.18363][2504.11023].

## 6. Empirical Performance and Applications

Empirical studies routinely illustrate the practical advantages of VM-PG methods:
- **Ill-conditioned problems:** Diagonal or quasi-Newton metrics yield considerable iteration reduction and improved objective value for quadratic programming, logistic regression, and high-dimensional linear models [1910.07056][2010.00817].
- **Imaging and inverse tasks:** In CT, image deblurring, and compressive sensing, VM-PG produces reduced iteration counts, higher PSNRs, and improved reconstruction accuracy, especially using Hessian-informed or structured metrics [2108.03365][2308.13776][1506.00385].
- **Large-scale composite learning:** Stochastic variable-metric methods outperform scalar step-size counterparts in both convergence speed and robustness to problem scaling, as measured on standard datasets [2010.00817][2301.00631].
- **Robustness and adaptivity:** Variable metric approaches consistently enhance local convergence rates, enable non-Euclidean geometry, and handle composite/nonconvex/nonsmooth objectives, often with negligible per-iteration computational overhead relative to fixed-metric methods [2507.18363][1812.05243].

## 7. Summary Table of Key VM-PG Variants

| Method/Framework         | Metric Adaptation                     | Core Application Domains                 |
|-------------------------|---------------------------------------|------------------------------------------|
| Diagonal BB/VM-PG       | Diagonal via secant/BB (O(n))         | Large-scale ML, ill-conditioned QP       |
| BFGS/QNing              | Quasi-Newton update, limited-memory   | Sparse learning, high-dim. composite     |
| VMEPIHT                 | Subspace quasi-Newton + PIHT          | ℓ₀ minimization, sparse recovery         |
| 3P-SPIDER/VM-mSRGBB     | Stochastic with BB/Hessian or diag    | Stochastic composite, variance-reduced   |
| Bregman/ABPG/VBPG       | Variable Bregman/Hessian metric       | Non-Euclidean, entropy, manifold models  |
| Model-based PQN         | Continuous generator (not uniformly bounded) | Nonsmooth nonconvex composite    |

The direction of the field is toward further integration with stochastic, fractional, and nonconvex optimization; extending variable-metric methods to Bregman and manifold contexts; and developing principled metric updating strategies with theoretical and empirical guarantees across broad function classes [1910.07056][2507.18363][2301.00631][2108.03365][1812.05243][2109.05690][2108.03365].

Source: https://www.emergentmind.com/topics/variable-metric-proximal-gradient