Proximal Preconditioned Gradient Methods

Updated 3 July 2026

Proximal preconditioned gradient methods are algorithms for composite optimization that integrate explicit preconditioning to speed convergence in structured and ill-conditioned problems.
They employ diverse strategies—diagonal, operator-aware, and Bregman-based—to adapt to local curvature and efficiently manage nonsmooth or constrained regularization.
These methods deliver rigorous convergence rates and practical improvements across applications like signal processing, machine learning, and computational imaging.

Proximal Preconditioned Gradient Methods comprise a class of algorithms for large-scale composite optimization in which the key innovation is the integration of explicit preconditioning within the proximal gradient (forward-backward) framework. These methods are designed to accelerate convergence—especially in ill-conditioned or structured problems—by exploiting local curvature, problem structure, or operator geometry, while preserving the ability to efficiently handle nonsmooth or constrained regularization via proximal operators. Recent advances extend these methodologies to zeroth-order settings, stochastic optimization, non-Euclidean metrics, multiobjective problems, and primal-dual/saddle-point scenarios, establishing rigorous convergence rates and yielding practical improvements across signal processing, machine learning, and computational imaging.

1. Mathematical Formulation and Algorithmic Structure

At their core, proximal preconditioned gradient methods address minimization problems of composite form: $\min_{x \in \R^n} \; h(x) := f(x) + r(x)$ where $f$ is differentiable (often with Lipschitz-continuous or even twice-differentiable gradients) and $r$ is a convex, lower semicontinuous, possibly nonsmooth regularizer with a tractable proximal operator. The canonical preconditioned step is: $x_{k+1} = \operatorname{prox}_{\gamma_k r}^{P_k} \left(x_k - \gamma_k P_k^{-1} \nabla f(x_k)\right)$ where $P_k \succ 0$ is a positive-definite preconditioner (which may be diagonal, block-diagonal, or of problem-adapted structure), and the metric proximal mapping is: $\operatorname{prox}_{\gamma r}^{P}(v) = \arg\min_{y} \; r(y) + \tfrac{1}{2\gamma} \|y - v\|_{P}^2$ Such preconditioning generalizes classical linear transformations, and recent research, including "Preconditioned Proximal Gradient Methods with Conjugate Momentum" (Chen et al., 17 Mar 2026), "Second-order Methods for Multiobjective Composite Optimization" (Chen et al., 25 Jun 2026), and the stochastic ProxGen framework (Yun et al., 2020), demonstrates the flexibility to integrate blockwise, operator-aware, objective-wise, and non-linear (Bregman/Bregman-like) preconditioners.

In zeroth-order settings, as in "An Inexact Preconditioned Zeroth-order Proximal Method for Composite Optimization" (Liu et al., 2024), both gradient and preconditioner are estimated via finite differences, enabling derivative-free applications.

2. Classes of Preconditioning Strategies

Preconditioning in the proximal gradient context can be categorized by scope and construction:

Diagonal and Block-Diagonal Preconditioning: Efficient for large-scale problems where coordinate-decomposability of the proximal operator is preserved. Diagonal scaling typically matches the local curvature (Hessian diagonals or empirical Fisher information) and is amenable to adaptive or AdaGrad-style schemes (Yun et al., 2020, Lin et al., 2024).
Operator-Aware Preconditioning: For composite terms with a linear operator $A$ (i.e., $g(Ax)$ ), a metric $P$ is chosen so that $AP^{-1}A^\top = I$ on the active row-space, yielding a closed-form or cheaply computable dual proximal (Chen et al., 17 Mar 2026, Chen et al., 25 Jun 2026).
Bregman and Nonlinear Testing Preconditioning: Nonlinear preconditioning replaces the squared norm penalty by a Bregman distance tied to a reference function $f$ 0. This framework, formalized in the "Testing and non-linear preconditioning" methodology (Valkonen, 2017, Laude et al., 2022), allows non-Euclidean updates and closed-form resolvents in mirror and anisotropic spaces, and enables convergence proofs via step-wise inequalities.
Barzilai–Borwein and Curvature-Matching Preconditioners: In multiobjective and ill-conditioned settings, Barzilai–Borwein scaling per objective is combined with a common preconditioner; inexact and subspace-approximate steps reduce computational overhead while maintaining robustness to scaling disparities (Chen et al., 25 Jun 2026).
Primal-Dual and Barrier-Based Preconditioning: For saddle-point or conic constraint problems, primal-dual preconditioners (as in PDHG, ADMM, and Chambolle–Pock methods) can be paired with symmetric-cone barriers in the dual, leading to fast convergence under strong convexity (Valkonen, 2017, Ma et al., 2023).

3. Convergence Theory and Complexity Analysis

The convergence guarantees for these methods are established under a range of assumptions on smoothness, convexity, and preconditioner construction:

Sublinear and Linear Rates: For convex, $f$ 1-smooth $f$ 2 and proper, closed, convex $f$ 3, the typical sublinear rate is $f$ 4 for proximal residuals and, with additional error bounds or PL-type dominance, linear rates $f$ 5 can be obtained. For example, inexact zeroth-order methods achieve an oracle complexity of $f$ 6 for $f$ 7 (Liu et al., 2024), while multiobjective subspace methods attain global convergence and, under error bounds, linear contraction (Chen et al., 25 Jun 2026).
Conditions for Feasibility and Stability: Convergence often requires the preconditioner to be uniformly positive-definite and well-aligned with the smooth part’s curvature. In Primal-Dual PDHG, the sharp condition $f$ 8 is necessary and sufficient within constant metrics for convergence and optimal step-size scaling (Ma et al., 2023).
Preconditioned Stochastic Schemes: The ProxGen family (Yun et al., 2020) demonstrates that arbitrary positive definite preconditioners (even adaptive or stochastic) preserve the $f$ 9 non-asymptotic rate for stationarity in nonconvex stochastic settings, provided that preconditioner eigenvalues are uniformly bounded away from zero. Closed-form variable metric prox mappings (including for $r$ 0 penalties $r$ 1) are established for diagonal scaling.
Accelerated and Momentum-Driven Variants: Generalized Nesterov-type acceleration can be integrated with preconditioners, with rates up to $r$ 2 for function values and $r$ 3 for iterates when using power-type momentum schemes ( $r$ 4) (Lin et al., 2024).

4. Low-Dimensional Subspace and Inexact Models

A notable trend is the restriction of second-order corrections to low-dimensional subspaces (typically two-dimensional) to balance improved conditioning against practical computational cost:

Conjugate Subspace Acceleration: Search directions are constructed in the span of the current preconditioned gradient and a historical or momentum direction, orthogonalized with respect to the local Hessian or preconditioner. Restricted second-order models decompose into tractable one-dimensional subproblems (Chen et al., 17 Mar 2026, Chen et al., 25 Jun 2026).
Inexact Subproblem Solutions: Both primal and dual subproblems (including simplex-constrained duals in the operator-aware setting) are often solved only to relaxed accuracy, using backtracking techniques and approximate Armijo conditions to guarantee monotonic descent and eventual stationarity (Chen et al., 25 Jun 2026).
Practical Impact: Such subspace methods regularly achieve substantial reductions in outer iterations and CPU time for ill-conditioned and high-dimensional problems (e.g., LASSO, simplex-constrained quadratic programs, structured $r$ 5-regularized tasks), often by factors of 2–10 compared to standard methods (Chen et al., 17 Mar 2026, Chen et al., 25 Jun 2026).

5. Extensions: Nonlinear, Stochastic, and Composite Settings

Proximal preconditioning is not limited to deterministic, Euclidean, or single-objective optimization:

Bregman/Anisotropic and Testing-Based Preconditioning: Algorithms incorporating anisotropic or mirror-descent-like updates generalize the definition of preconditioning, replacing quadratic models with Bregman distances adapted to problem geometry (e.g., exponential reference for KL-divergence, logistic regression) (Laude et al., 2022). Dual-space nonlinear preconditioning provides unified frameworks for convergence analysis (Valkonen, 2017).
Stochastic and Deep Learning Applications: In nonconvex deep network training, preconditioned proximal-gradient schemes support per-parameter adaptive learning rates (e.g., AdaGrad, Adam-like, KFAC-style scaling), backstopped by full variable-metric proximal mappings and matching theoretical rates (Yun et al., 2020). This approach enables new exact solutions for nonconvex regularizers and quantization penalties.
Saddle-Point and Barrier-Augmented Primal-Dual Methods: Preconditioner design in primal-dual settings (PDHG, ADMM) is closely tied to decoupling and stabilization of updates, with recent works pushing the boundaries of optimal step size and metric selection. Barrier-based dual preconditioning achieves linear convergence in symmetric-cone problems (Valkonen, 2017).

6. Numerical Performance and Applications

Empirical investigations consistently demonstrate the acceleration and stability of proximal preconditioned gradient methods:

Application Domain	Algorithmic Variant	Performance Gains	Reference
High-dim LASSO, TV	Subspace + Precond.	2–10× faster than FISTA/PDHG	(Chen et al., 17 Mar 2026)
Sparse deep nets	Stoch. prox + precond.	Higher accuracy/sparsity	(Yun et al., 2020)
PET image reconstruction	Accelerated PG + precond.	Fewer iterations/robust acc.	(Lin et al., 2024)
Multiobjective, ill-cond	Subspace Barzilai–Borwein	2–5× fewer iterations	(Chen et al., 25 Jun 2026)
Matrix games/imaging	Primal-dual PrePDHG	20–50% fewer iterations	(Ma et al., 2023)

Key findings include:

Diagonal and operator-aware preconditioning are especially effective for structured and sparse optimization.
Subspace and momentum-augmented variants are robust to ill-conditioning and insensitive to parameter tuning.
Stochastic and adaptive diagonal preconditioning empowers large-scale deep learning tasks with nonconvex and nonsmooth penalties.

7. Theoretical and Practical Significance

Proximal preconditioned gradient methods constitute a mathematically unified and algorithmically versatile set of tools. Their ability to leverage problem structure, capture local curvature, and efficiently accommodate non-Euclidean metrics enables both theoretical advances (global and linear convergence in nontrivial settings, oracle complexity bounds) and practical impact (accelerated convergence and robustness in ill-conditioned and nonsmooth domains).

Major open challenges include constructing adaptive and scalable preconditioners for highly non-Euclidean or nonstationary problems, extending robust inexact subspace and primal-dual implementations, and further unifying the theory across stochastic, multiobjective, and nonlinear Bregman frameworks (Valkonen, 2017, Laude et al., 2022, Chen et al., 25 Jun 2026).