Model-Based Proximal Quasi-Newton

• Model-Based Proximal Quasi-Newton Method is a composite optimization framework that constructs local quadratic models for smooth functions and applies exact proximal steps for nonsmooth terms.
• It adapts variable metrics and low-rank quasi-Newton updates to efficiently handle convex, nonconvex, multiobjective, and manifold optimization problems.
• Empirical studies show that leveraging quasi-Newton curvature approximations accelerates convergence and enhances performance on large-scale and structured optimization tasks.

A model-based proximal quasi-Newton method is a class of composite optimization algorithms in which each iteration minimizes a local model that is quadratic in the smooth component, keeps the nonsmooth component in exact proximal form, and uses a quasi-Newton surrogate for second-order curvature. In current usage, the framework appears in convex and nonconvex Euclidean composite optimization, unconstrained multiobjective optimization, stochastic finite-sum problems, and Riemannian optimization over the Stiefel manifold (Adler et al., 2020, Kanzow et al., 2022, Peng et al., 2021, Wang et al., 16 Jan 2025).

1. Problem classes and mathematical setting

In Euclidean composite optimization, the basic problem is typically written as

$\min_{x\in\mathbb{R}^d} F(x)=f(x)+r(x),$

with $f$ convex and twice differentiable, or continuously differentiable and possibly nonconvex, and $r$ an extended real-valued closed convex function or, in more general nonconvex settings, a proper lower semicontinuous prox-bounded function (Adler et al., 2020, Kanzow et al., 2022, Dahl et al., 12 May 2026). This formulation includes quadratic programming, sparse learning, $\ell_1$-regularized logistic or Poisson regression, and problems in which constraints are encoded through indicator functions.

A multiobjective version considers

$\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$

with each component decomposed as

$F_i(x)=g_i(x)+h_i(x),$

where $g_i$ is twice continuously differentiable and strongly convex, and $h_i$ is proper, convex, and lower semicontinuous, but not necessarily differentiable (Peng et al., 2021). In that setting, the first-order notion is Pareto stationarity, defined by

$$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$

On the Stiefel manifold, the problem becomes

$\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$

where

$f$0

$f$1 is smooth, and $f$2 is convex but possibly nonsmooth and Lipschitz continuous (Wang et al., 2024, Wang et al., 16 Jan 2025). The method is then formulated on the tangent space

$f$3

with retractions and vector transport replacing linear Euclidean updates.

This range of problem classes indicates that the phrase denotes a methodology rather than a single algorithm. A plausible implication is that the unifying feature is not the ambient geometry or convexity class, but the use of a local surrogate model plus a proximal treatment of the nonsmooth term.

2. Local models and the proximal quasi-Newton step

In Euclidean composite problems, a standard local model at iterate $f$4 is

$f$5

where $f$6 is a symmetric approximation of $f$7 (Kanzow et al., 2022). In convex proximal Newton-type methods the same structure appears as

$f$8

with $f$9 equal either to the exact Hessian or to a positive-definite approximation (Adler et al., 2020). The step is therefore a model minimizer rather than a direct minimizer of the true objective.

A widely used stabilization is quadratic regularization. In nonconvex composite optimization one writes

$r$0

and computes

$r$1

so that $r$2 acts as the local metric (Kanzow et al., 2022). In the prox-bounded nonsmooth nonconvex setting, the same idea appears as

$r$3

with the candidate point defined by global minimization of $r$4 (Dahl et al., 12 May 2026).

This step admits a scaled proximal characterization. If $r$5, then

$r$6

so the quasi-Newton curvature is embedded directly into the prox metric rather than appended after a Euclidean prox step (Dahl et al., 12 May 2026).

In multiobjective optimization the local model is scalarized through a pointwise maximum rather than fixed weights. For $r$7, the component model is

$r$8

and the scalar model is

$r$9

Adding a proximal regularizer gives

$\ell_1$0

A point $\ell_1$1 is Pareto stationary if and only if $\ell_1$2 and $\ell_1$3, where $\ell_1$4 is the minimum value of $\ell_1$5 (Peng et al., 2021).

On the Stiefel manifold, the model is transferred to the tangent space. ARPQN uses

$\ell_1$6

with $\ell_1$7, while ManPQN uses

$\ell_1$8

subject to the tangent constraint (Wang et al., 2024, Wang et al., 16 Jan 2025). The iterate is then obtained by a retraction,

$\ell_1$9

Across these formulations, “model-based” means that the algorithmic step is defined by minimizing a local surrogate of the smooth term while the nonsmooth term remains explicit. “Proximal” means that nonsmoothness is handled through a metric prox or a proximal subproblem. “Quasi-Newton” means that curvature enters through $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$0, $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$1, $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$2, or per-objective matrices $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$3, rather than through the exact Hessian at every iteration.

3. Variable metrics, Hessian surrogates, and subproblem computation

The variable metric is the central design object. In convex proximal Newton-type methods, the assumptions are typically that $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$4 is symmetric positive definite and that

$\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$5

which is sufficient for local superlinear convergence (Adler et al., 2020). In nonconvex regularized proximal quasi-Newton methods, $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$6 may be indefinite, and the regularization $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$7 restores positive definiteness of the subproblem metric (Kanzow et al., 2022).

Several quasi-Newton update formulas are used. In multiobjective optimization, each smooth component $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$8 has its own matrix $\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,$9, updated by BFGS,

$F_i(x)=g_i(x)+h_i(x),$0

by self-scaling BFGS, or by Huang BFGS with modified secant vector $F_i(x)=g_i(x)+h_i(x),$1 (Peng et al., 2021). In Euclidean convex problems, L-BFGS is used to construct $F_i(x)=g_i(x)+h_i(x),$2 in experiments, and the Shamanskii structure updates or factors the metric only every $F_i(x)=g_i(x)+h_i(x),$3 iterations (Adler et al., 2020). In nonsmooth nonconvex optimization, limited-memory BFGS and limited-memory SR1 are standard, and a compact limited-memory Kleinmichel formula gives a positive-definite rank-one update under the same curvature condition as BFGS (Dahl et al., 12 May 2026).

Efficient subproblem solution depends on exploiting special metric structure. A foundational result for diagonal plus rank-one and, later, diagonal $F_i(x)=g_i(x)+h_i(x),$4 rank-$F_i(x)=g_i(x)+h_i(x),$5 metrics shows that if

$F_i(x)=g_i(x)+h_i(x),$6

then the proximal mapping in metric $F_i(x)=g_i(x)+h_i(x),$7 reduces to the proximal mapping in the simpler metric $F_i(x)=g_i(x)+h_i(x),$8 plus a low-dimensional root-finding problem for $F_i(x)=g_i(x)+h_i(x),$9 (Becker et al., 2012, Becker et al., 2018). In separable cases this yields exact or highly efficient implementations because the dual function is piecewise linear or piecewise affine.

Large-scale nonconvex composite methods use compact representations

$g_i$0

then decompose the regularized metric into

$g_i$1

The resulting metric prox can be expressed through a simple prox in metric $g_i$2 and the solution of a small strongly monotone nonlinear system by a semismooth Newton method (Kanzow et al., 2022). In that setting, the main operation is repeated evaluation of $g_i$3, and the numerical study reports that only 1–2 semismooth Newton steps suffice to reach residual norms $g_i$4, independent of the memory size (Kanzow et al., 2022).

A related mechanism appears in mini-batch TV image reconstruction. There the aggregate metric has the form

$g_i$5

coming from subsetwise SR1 updates, and the weighted proximal mapping is reduced to a low-dimensional nonlinear system and a diagonal-metric prox inside a dual TV solver (Hong et al., 2023). This suggests that low-rank metric perturbations are not merely a theoretical convenience; they are an implementation principle that recurs across Euclidean, stochastic, and imaging variants.

4. Globalization, stationarity, and convergence theory

Globalization mechanisms fall into three main groups: line search, adaptive regularization, and model-agreement ratios. In convex proximal Newton-type methods, backtracking accepts $g_i$6 when

$g_i$7

and the exact-Hessian Shamanskii regime attains local order at least $g_i$8, while approximate $g_i$9 satisfying $h_i$0 and $h_i$1 yield local superlinear convergence (Adler et al., 2020). Inexact subproblem solves retain linear, superlinear, or higher-order behavior depending on the inexactness exponent $h_i$2 and the forcing sequence $h_i$3 (Adler et al., 2020).

Regularization-based globalization replaces line search by predicted versus actual decrease tests. In the regularized proximal quasi-Newton method, one defines

$h_i$4

and adapts $h_i$5 according to whether the iteration is unsuccessful, successful, or highly successful (Kanzow et al., 2022). Under bounded $h_i$6 and $h_i$7 bounded below, one obtains $h_i$8, and with uniform continuity of $h_i$9 on a set containing the iterates, $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$0, so every accumulation point is stationary (Kanzow et al., 2022). Under a Luo–Tseng type local error bound and bounded positive-definite $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$1, the whole sequence converges to a stationary point and

$$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$2

(Kanzow et al., 2022).

In the prox-bounded nonsmooth nonconvex setting, adaptive regularization again controls globalization through sufficient decrease. Under bounded $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$3, lower bounded $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$4, and local Lipschitz gradient, every accumulation point is stationary, and if $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$5 satisfies the Kurdyka–Łojasiewicz property at a stationary accumulation point, the whole sequence converges to it (Dahl et al., 12 May 2026). For a power desingularizing function $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$6, the successful subsequence has Q-superlinear objective convergence and R-superlinear iterate convergence when $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$7, Q-linear and R-linear convergence when $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$8, and sublinear rates when $$\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.$$9 (Dahl et al., 12 May 2026).

A more general model-function framework proves that boundedness of the variable metric need not be an a priori hypothesis. Under mild first-order assumptions on the model error, every accumulation point of the generated sequence is stationary, and if the objective has the Kurdyka–Łojasiewicz property at the corresponding accumulation point, the whole sequence converges to a stationary point; in that regime the sequence of variable metrics becomes uniformly bounded (Jia et al., 24 Jul 2025). This suggests that metric boundedness can be a consequence of objective regularity rather than an external assumption.

Multiobjective convergence uses Pareto stationarity as the limiting notion. With Armijo-type line search, every accumulation point of the generated sequence is Pareto stationary, and with fixed step $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$0 the same conclusion holds under Lipschitz continuity of the gradients and $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$1 (Peng et al., 2021).

For nonsmooth regularized nonconvex optimization with model Hessians allowed to grow, R2N establishes global convergence of the stationarity measure

$\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$2

to zero without relying on local Lipschitz continuity of $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$3, while permitting model Hessians to be unbounded (Diouane et al., 2024). Under Lipschitz continuity of $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$4, the worst-case complexity is

$\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$5

for $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$6, where $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$7 controls the growth of model Hessians, and

$\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$8

when $\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),$9 (Diouane et al., 2024).

On the Stiefel manifold, ARPQN is globally convergent to stationary points, has local linear convergence under a local strong-convexity-type condition on the pullback Hessian, and reaches an $f$00-stationary point in at most $f$01 outer iterations (Wang et al., 2024). For the more faithful Riemannian proximal Newton model ARPN, exact Hessian or quasi-Newton approximations satisfying the Dennis–Moré condition give local q-superlinear convergence (Wang et al., 2024). ManPQN proves global convergence of accumulation points to stationary points and local linear convergence under analogous manifold assumptions (Wang et al., 16 Jan 2025).

5. Specialized variants and extensions

The multiobjective formulation is distinctive because it avoids fixed scalarization weights. Separate quasi-Newton matrices are constructed for each objective, but the step is single and is obtained from the max-model over componentwise local models (Peng et al., 2021). Constraints can be encoded through indicator functions $f$02, turning the subproblem into a projected model step, and robust multiobjective terms of the form

$f$03

lead, in the polyhedral linear case, to convex quadratic minimization problems with quadratic inequality and linear constraints (Peng et al., 2021).

Stochastic and finite-sum variants adapt the same template to large datasets. A proximal stochastic quasi-Newton algorithm uses the update

$f$04

where $f$05 is an SVRG-type variance-reduced gradient estimator and $f$06 is a low-rank quasi-Newton metric built from subsampled Hessians (Luo et al., 2016). Under strong convexity and suitable step size, the stagewise iterates satisfy

$f$07

so the method converges linearly in expectation (Luo et al., 2016).

A domain-specific imaging variant, the mini-batch quasi-Newton proximal method for constrained total-variation nonlinear image reconstruction, builds per-subset quadratic models, aggregates them into a weighted metric prox subproblem, and exploits the dual representation of TV together with a diagonal-plus-low-rank metric structure (Hong et al., 2023). The weighted projection in the inner loop is handled through the same low-dimensional proximal calculus that appears in generic Euclidean methods.

Riemannian variants preserve the model-based structure while changing the ambient geometry. In ARPQN and ManPQN, the step is computed in the tangent space, the nonsmooth term is retained as $f$08, and the iterate is mapped back by a retraction (Wang et al., 2024, Wang et al., 16 Jan 2025). This suggests that the model-based proximal quasi-Newton paradigm is compatible with changing tangent spaces, provided the curvature operator, vector transport, and retraction are updated coherently.

A broader implication is that the method is modular. The smooth model, the proximal term, the metric update, and the globalization mechanism can be recombined for Euclidean, multiobjective, stochastic, robust, constrained, and manifold settings without altering the basic algorithmic identity.

6. Empirical behavior and implementation practice

Across the cited literature, the empirical pattern is consistent: second-order or quasi-Newton curvature usually reduces outer iteration counts relative to proximal gradient baselines, and low-rank metric structure keeps the extra subproblem cost manageable.

For convex composite optimization, PSOPT uses L-BFGS to construct $f$09, solves subproblems with TFOCS, and on $f$10-regularized logistic regression on gisette both PSOPT and PNOPT converge substantially faster than SpaRSA and FISTA in relative suboptimality versus time, with PSOPT best on gisette (Adler et al., 2020). In regularized proximal quasi-Newton methods for convex and nonconvex composite problems, RPQN with limited-memory BFGS or SR1 is significantly faster than FISTA, SpaRSA, and QGPN on large least-squares and group-sparse instances, and remains robust on nonconvex image restoration (Kanzow et al., 2022). In the 2026 limited-memory nonconvex nonsmooth framework, numerical results show a significant speed up compared to SPG, PANOC+, and R2N (Dahl et al., 12 May 2026).

For multiobjective optimization, the reported robust UMOP experiments compare PGM with PQNM(BFGS), PQNM(SS-BFGS), and PQNM(H-BFGS). For a fixed uncertainty level $f$11, the Pareto frontier produced by PQNM(H-BFGS) is the lowest, followed by PQNM(SS-BFGS), then PQNM(BFGS), with PGM highest (Peng et al., 2021). This suggests that the choice of secant update can affect not only speed but also the quality of the recovered frontier.

In manifold optimization, NLS-ARPQN typically requires fewer line search steps, often fewer ASSN iterations, and lower CPU time than ManPQN and proximal gradient variants, especially when $f$12 and $f$13 are large (Wang et al., 2024). ManPQN itself needs dramatically fewer iterations than ManPG and ManPG-Ada on compressed modes and sparse PCA, and often achieves the lowest CPU time on larger problems despite heavier iterations (Wang et al., 16 Jan 2025).

In computational imaging, BQNPM achieves a weighted proximal mapping at a cost similar to the proximal mapping in ASPMs, while requiring fewer iterations to converge; on three-dimensional inverse-scattering problems, wall-clock reductions of about $f$14–$f$15 are reported relative to ASPM, with similar or better reconstruction quality (Hong et al., 2023).

For computationally intensive $f$16-regularized M-estimators, the proximal quasi-Newton method with aggressive active-set shrinking converges considerably faster than current state-of-the-art on sequence labeling and hierarchical classification, outperforming Prox-GD, OWL-QN, SGD, and BCD in wall-clock time to a small objective gap (Zhong et al., 2014). The implementation emphasis there is not only on quasi-Newton curvature but on shrinking, sparse feature-indexed data structures, and limited numbers of full expensive gradient computations.

Implementation practice in this literature repeatedly favors a small set of recurring techniques: compact limited-memory representations, diagonal $f$17 low-rank metric decompositions, semismooth Newton for tiny dual systems, active-set restriction when sparsity is expected, and predicted-versus-actual reduction tests when the problem is nonconvex or geometrically constrained. These recurring design choices help explain why the method functions as a family of algorithms rather than a single fixed scheme.