Variable-Metric Proximal Gradient

• Variable-metric proximal gradient is a class of methods that adapts a positive-definite metric in the proximal step to capture local curvature in composite optimization.
• These methods employ strategies like diagonal scaling, quasi-Newton updates, and Hessian-based metrics to improve convergence, especially in ill-conditioned or high-dimensional problems.
• They provide rigorous convergence guarantees, including global subsequential convergence and local linear or superlinear rates under suitable smoothness and convexity conditions.

Variable-metric proximal gradient (VM-PG) methods form a broad class of algorithms for solving composite optimization problems of the form

$\min_{x}~F(x) := f(x) + g(x),$

where $f$ is typically smooth (not necessarily convex) and $g$ is convex, possibly nonsmooth. The key distinguishing feature of VM-PG is the adaptation of a metric—represented by a positive-definite symmetric operator/matrix, which may vary per iteration—to the problem’s local geometry. This generalizes the classical proximal gradient paradigm by improving curvature adaptation, preconditioning, and exploitability of second-order or quasi-Newton information.

1. Mathematical Foundations and Basic Iteration

Variable-metric proximal gradient methods replace the standard Euclidean proximal step

$$\mathrm{prox}_{g}(z) := \arg\min_{y} \left\{ g(y) + \frac{1}{2\alpha}\|y-z\|^2 \right\}$$

with a variable-metric version: $$\mathrm{prox}_{g}^{H}(z) := \arg\min_{y} \left\{ g(y) + \frac{1}{2}(y-z)^\top H (y-z) \right\},$$ where $H\succ0$ is a symmetric positive-definite metric matrix that can depend on the current iterate.

A general iteration of VM-PG takes the form

$$x_{k+1} = \mathrm{prox}_{\gamma_k g}^{H_k}(x_k - \gamma_k H_k^{-1} \nabla f(x_k)),$$

where $H_k$ may be chosen as the scaled identity (recovering the standard method), a diagonal/scaled matrix (for coordinate preconditioning), a quasi-Newton/BFGS matrix, or even the Hessian of $f$ (proximal-Newton variant) (Park et al., 2019, Bonettini et al., 2015, Fort et al., 2023).

This generalization enables matching the iteration to the local curvature or structure of the composite objective, enhancing convergence in ill-conditioned or high-dimensional settings.

2. Algorithmic Strategies and Metric Construction

Variable-metric choices include:

• Diagonal and Barzilai–Borwein metrics: Efficient per-iteration (O(n) cost), allowing cheap curvature adaptation via diagonal scaling derived from secant conditions (Park et al., 2019, Yu et al., 2020).
• Quasi-Newton/BFGS/limited-memory: Capture curvature much beyond coordinatewise scaling but with moderate cost. Hessian approximations are updated by secant formulas, sometimes restricted to support subspaces in sparsity problems (Lin et al., 2016, Jia et al., 24 Jul 2025, Zhang et al., 2021).
• Hessian-based (proximal Newton): Highest curvature fidelity, used when second derivatives are available and subproblem prox computations remain affordable (Tran-Dinh et al., 2018, Zhang et al., 2023).
• Bregman metrics: Generalize distance beyond quadratic forms, using a kernel-generating function $\phi$ and its Hessian (Fujiki et al., 8 Oct 2025, Yang et al., 2021, Zhu et al., 2020).

Metric selection is often coupled with line search, safeguarding (bounding $H_k$ from above/below), or inexactness tolerance if the model/prox step can only be solved approximately.

3. Theoretical Guarantees: Convergence and Complexity

A range of global and local convergence results accompany VM-PG schemes:

• Global subsequential convergence: For merely convex, nonsmooth $g$ and smooth $f$, under suitable conditions (e.g., coercivity, metric bounds), every cluster point is stationary (Bonettini et al., 2015, Park et al., 2019, Jia et al., 24 Jul 2025, Zhang et al., 2023).
• O(1/k) sublinear rates: Assuming L-smoothness or relative smoothness, sequence averages of objective values or normed gradient mappings decay at this rate (Bonettini et al., 2015, Yang et al., 2021).
• Local linear and superlinear rates: If $f$ is strongly convex (or satisfies quadratic growth/KŁ/KL property) and the metric sequence stabilizes or matches local curvature (e.g., Hessian), then local linear (or superlinear) rates are achieved (Jia et al., 24 Jul 2025, Park et al., 2019, Zhang et al., 2021).
• KL-type global convergence: Under the Kurdyka–Łojasiewicz property or similar error-bounds, global convergence (not just subsequence) follows, with rates dictated by the KL exponent (Zhang et al., 2023, Yang et al., 15 Apr 2025, Fujiki et al., 8 Oct 2025, Zhu et al., 2020).
• Stochastic, variance-reduced, and nonconvex settings: Recent frameworks extend VM-PG to stochastic/finite-sum settings, combining variance-reduction with variable-metric proximity for guaranteed expected stationarity within $O(\sqrt{n}\epsilon^{-2})$ complexity in the finite-sum case (Fort et al., 2023, Yu et al., 2020).

The strength of the convergence result depends delicately on properties of the metric sequence, regularity of $f$ and $g$, the inexactness tolerance in subproblem solves, and descent monitoring.

4. Extensions: Inexactness, Line-Search, and Bregman/Composite Settings

Practical instantiations often require:

• Inexact proximal solves: Subproblem solutions may not be exact; convergence is preserved under absolute or relative error criteria (e.g., absolute function gap versus minimum, or sufficient contraction), provided errors decay or sum (Bonettini et al., 2015, Yang et al., 2021, Zhang et al., 2023, Yang et al., 15 Apr 2025).
• Line-search/backtracking: Sufficient decrease enforced via Armijo or Armijo–Wolfe conditions, often tailored to the variable metric, preserves monotonicity and enables superlinear rates (Bonettini et al., 2015, Fujiki et al., 8 Oct 2025, Jia et al., 24 Jul 2025, Zhang et al., 2023). Stepsizes may be chosen via nonmonotone search or fixed in certain settings.
• Bregman distances: Replacing the Euclidean norm with a Bregman divergence $D_\phi(x, y)$ based on a convex kernel $\phi$, enabling adaptation to manifold settings, entropy geometry, or structured non-quadratic curvature (Fujiki et al., 8 Oct 2025, Yang et al., 2021, Zhu et al., 2020). The resulting method generalizes VM-PG and connects to mirror-descent and other geometric optimization strategies.
• Composite/nonsmooth/nonconvex settings: Advanced VM-PG methods address nonsmooth, nonconvex, fractional, or non-Euclidean objectives with model-based steps, o-minimal definability, and KL machinery (Jia et al., 24 Jul 2025, Zhang et al., 2023, Zhu et al., 2020, Yang et al., 15 Apr 2025).

5. Specialized Schemes and Paradigm Integration

Numerous specialized frameworks exist under the VM-PG umbrella:

• VMEPIHT for ℓ₀ minimization: Alternates a hard-thresholded proximal-IHT with a restricted quasi-Newton update on the active support, achieving global convergence, eventual support stabilization, and, for least-squares problems, local superlinear rates (Zhang et al., 2021).
• QNing: Treats the (squared) Moreau envelope with a variable metric (often L-BFGS) for outer acceleration, allows for inexact subproblem solutions, and is compatible with sparsity-induced regularization (Lin et al., 2016).
• Homotopy methods: Introduce a parameterized optimality condition to pursue solution trajectory continuation (e.g., by decreasing regularization weight), with metric adaptation yielding global/even finite iteration complexity under strong convexity or self-concordance (Tran-Dinh et al., 2018).
• Stochastic and residual-variance-reduced methods: 3P-SPIDER, VM-mSRGBB, and related approaches unite stochastic recursive gradient estimators with variable-metric preconditioning, obtaining optimal or near-optimal sample complexities in large-scale learning (Fort et al., 2023, Yu et al., 2020).
• Fractional and model-based extensions: Fractional objectives and sophisticated local model functions with inexactness operate within variable-metric frameworks, accommodating extremely broad problem classes while inheriting the foundational convergence guarantees (Jia et al., 24 Jul 2025, Yang et al., 15 Apr 2025).

6. Empirical Performance and Applications

Empirical studies routinely illustrate the practical advantages of VM-PG methods:

• Ill-conditioned problems: Diagonal or quasi-Newton metrics yield considerable iteration reduction and improved objective value for quadratic programming, logistic regression, and high-dimensional linear models (Park et al., 2019, Yu et al., 2020).
• Imaging and inverse tasks: In CT, image deblurring, and compressive sensing, VM-PG produces reduced iteration counts, higher PSNRs, and improved reconstruction accuracy, especially using Hessian-informed or structured metrics (Zhang et al., 2021, Zhang et al., 2023, Bonettini et al., 2015).
• Large-scale composite learning: Stochastic variable-metric methods outperform scalar step-size counterparts in both convergence speed and robustness to problem scaling, as measured on standard datasets (Yu et al., 2020, Fort et al., 2023).
• Robustness and adaptivity: Variable metric approaches consistently enhance local convergence rates, enable non-Euclidean geometry, and handle composite/nonconvex/nonsmooth objectives, often with negligible per-iteration computational overhead relative to fixed-metric methods (Jia et al., 24 Jul 2025, Tran-Dinh et al., 2018).

7. Summary Table of Key VM-PG Variants

Method/Framework	Metric Adaptation	Core Application Domains
Diagonal BB/VM-PG	Diagonal via secant/BB (O(n))	Large-scale ML, ill-conditioned QP
BFGS/QNing	Quasi-Newton update, limited-memory	Sparse learning, high-dim. composite
VMEPIHT	Subspace quasi-Newton + PIHT	ℓ₀ minimization, sparse recovery
3P-SPIDER/VM-mSRGBB	Stochastic with BB/Hessian or diag	Stochastic composite, variance-reduced
Bregman/ABPG/VBPG	Variable Bregman/Hessian metric	Non-Euclidean, entropy, manifold models
Model-based PQN	Continuous generator (not uniformly bounded)	Nonsmooth nonconvex composite

The direction of the field is toward further integration with stochastic, fractional, and nonconvex optimization; extending variable-metric methods to Bregman and manifold contexts; and developing principled metric updating strategies with theoretical and empirical guarantees across broad function classes (Park et al., 2019, Jia et al., 24 Jul 2025, Fort et al., 2023, Zhang et al., 2021, Tran-Dinh et al., 2018, Yang et al., 2021, Zhang et al., 2021).