---
title: Model-Based Proximal Quasi-Newton
url: https://www.emergentmind.com/topics/model-based-proximal-quasi-newton-method
type: topic
---

# Model-Based Proximal Quasi-Newton

A model-based proximal quasi-Newton method is a class of composite optimization algorithms in which each iteration minimizes a local model that is quadratic in the smooth component, keeps the nonsmooth component in exact proximal form, and uses a quasi-Newton surrogate for second-order curvature. In current usage, the framework appears in convex and nonconvex Euclidean composite optimization, unconstrained multiobjective optimization, stochastic finite-sum problems, and Riemannian optimization over the Stiefel manifold [2007.09525][2210.07644][2108.00125][2501.09516].

## 1. Problem classes and mathematical setting

In Euclidean composite optimization, the basic problem is typically written as
\[
\min_{x\in\mathbb{R}^d} F(x)=f(x)+r(x),
\]
with \(f\) convex and twice differentiable, or continuously differentiable and possibly nonconvex, and \(r\) an extended real-valued closed convex function or, in more general nonconvex settings, a proper lower semicontinuous prox-bounded function [2007.09525][2210.07644][2605.11627]. This formulation includes quadratic programming, sparse learning, \(\ell_1\)-regularized logistic or Poisson regression, and problems in which constraints are encoded through indicator functions.

A multiobjective version considers
\[
\min F(x),\qquad F(x)=(F_1(x),\dots,F_m(x))^\top,
\]
with each component decomposed as
\[
F_i(x)=g_i(x)+h_i(x),
\]
where \(g_i\) is twice continuously differentiable and strongly convex, and \(h_i\) is proper, convex, and lower semicontinuous, but not necessarily differentiable [2108.00125]. In that setting, the first-order notion is Pareto stationarity, defined by
\[
\max_{i=1,\dots,m} F_i'(\bar x;d)\ge 0\quad\text{for all }d\in\mathbb{R}^n.
\]

On the Stiefel manifold, the problem becomes
\[
\min_{X\in\mathrm{St}(n,r)} F(X):=f(X)+h(X),
\]
where
\[
\mathrm{St}(n,r):=\{X\in\mathbb{R}^{n\times r}:X^\top X=I_r\},
\]
\(f\) is smooth, and \(h\) is convex but possibly nonsmooth and Lipschitz continuous [2404.11112][2501.09516]. The method is then formulated on the tangent space
\[
{\rm T}_X\mathrm{St}(n,r)=\{V\in\mathbb{R}^{n\times r}:V^\top X+X^\top V=0\},
\]
with retractions and vector transport replacing linear Euclidean updates.

This range of problem classes indicates that the phrase denotes a methodology rather than a single algorithm. A plausible implication is that the unifying feature is not the ambient geometry or convexity class, but the use of a local surrogate model plus a proximal treatment of the nonsmooth term.

## 2. Local models and the proximal quasi-Newton step

In Euclidean composite problems, a standard local model at iterate \(x^k\) is
\[
q_k(d):=f(x^k)+\nabla f(x^k)^\top d+\frac12 d^\top B_k d+\varphi(x^k+d),
\]
where \(B_k\) is a symmetric approximation of \(\nabla^2 f(x^k)\) [2210.07644]. In convex proximal Newton-type methods the same structure appears as
\[
m_t(x):=(x-x_t)^\top \nabla f(x_t)+\frac12 (x-x_t)^\top H_t(x-x_t)+r(x),
\]
with \(H_t\) equal either to the exact Hessian or to a positive-definite approximation [2007.09525]. The step is therefore a model minimizer rather than a direct minimizer of the true objective.

A widely used stabilization is quadratic regularization. In nonconvex composite optimization one writes
\[
\hat q_k(d)=q_k(d)+\frac12\mu_k\|d\|^2
\]
and computes
\[
d^k\in\arg\min_d \hat q_k(d),
\]
so that \(B_k+\mu_k I\) acts as the local metric [2210.07644]. In the prox-bounded nonsmooth nonconvex setting, the same idea appears as
\[
G_k:=B_k+\mu_k I,\qquad
\hat q_k(x)=f(x^k)+\nabla f(x^k)^\top (x-x^k)+\frac12 (x-x^k)^\top G_k(x-x^k)+\varphi(x),
\]
with the candidate point defined by global minimization of \(\hat q_k\) [2605.11627].

This step admits a scaled proximal characterization. If \(G_k\succ0\), then
\[
\hat x^k\in \operatorname{prox}^{G_k}_{\varphi}\bigl(x^k-G_k^{-1}\nabla f(x^k)\bigr),
\]
so the quasi-Newton curvature is embedded directly into the prox metric rather than appended after a Euclidean prox step [2605.11627].

In multiobjective optimization the local model is scalarized through a pointwise maximum rather than fixed weights. For \(F_i=g_i+h_i\), the component model is
\[
m_i(x;d):=\nabla g_i(x)^\top d+\frac12 d^\top B_i(x)d+h_i(x+d)-h_i(x),
\]
and the scalar model is
\[
\theta_x(d):=\max_{i=1,\dots,m}\left\{\nabla g_i(x)^\top d+\frac12 d^\top B_i(x)d+h_i(x+d)-h_i(x)\right\}.
\]
Adding a proximal regularizer gives
\[
\varphi_{\omega,x}(d):=\theta_x(d)+\frac{\omega}{2}\|d\|^2,\qquad
d_\omega(x)=\operatorname*{argmin}_d \varphi_{\omega,x}(d).
\]
A point \(x\) is Pareto stationary if and only if \(d_\omega(x)=0\) and \(\beta_\omega(x)=0\), where \(\beta_\omega(x)\) is the minimum value of \(\varphi_{\omega,x}\) [2108.00125].

On the Stiefel manifold, the model is transferred to the tangent space. ARPQN uses
\[
\phi_k(V):=\langle\nabla f(X_k),V\rangle+\frac12\langle(\mathcal B_k+\sigma_k I)[V],V\rangle+h(X_k+V),
\]
with \(V\in {\rm T}_{X_k}\mathcal M\), while ManPQN uses
\[
\phi_k(V)=\langle\nabla f(X_k),V\rangle+\frac12\|V\|_{\mathcal B_k}^2+h(X_k+V)
\]
subject to the tangent constraint [2404.11112][2501.09516]. The iterate is then obtained by a retraction,
\[
X_{k+1}=\mathbf R_{X_k}(\alpha_k V_k).
\]

Across these formulations, “model-based” means that the algorithmic step is defined by minimizing a local surrogate of the smooth term while the nonsmooth term remains explicit. “Proximal” means that nonsmoothness is handled through a metric prox or a proximal subproblem. “Quasi-Newton” means that curvature enters through \(B_k\), \(H_k\), \(\mathcal B_k\), or per-objective matrices \(B_i^k\), rather than through the exact Hessian at every iteration.

## 3. Variable metrics, Hessian surrogates, and subproblem computation

The variable metric is the central design object. In convex proximal Newton-type methods, the assumptions are typically that \(H_t\) is symmetric positive definite and that
\[
\|H_t-\nabla^2 f(x_t)\|=o(1),
\]
which is sufficient for local superlinear convergence [2007.09525]. In nonconvex regularized proximal quasi-Newton methods, \(B_k\) may be indefinite, and the regularization \(\mu_k I\) restores positive definiteness of the subproblem metric [2210.07644].

Several quasi-Newton update formulas are used. In multiobjective optimization, each smooth component \(g_i\) has its own matrix \(B_i^k\), updated by BFGS,
\[
B_i^{k+1}=B_i^k-\frac{B_i^k s^k (s^k)^\top B_i^k}{(s^k)^\top B_i^k s^k}+\frac{y_i^k (y_i^k)^\top}{(s^k)^\top y_i^k},
\]
by self-scaling BFGS, or by Huang BFGS with modified secant vector \(\hat y_i^k\) [2108.00125]. In Euclidean convex problems, L-BFGS is used to construct \(H_t\) in experiments, and the Shamanskii structure updates or factors the metric only every \(n\) iterations [2007.09525]. In nonsmooth nonconvex optimization, limited-memory BFGS and limited-memory SR1 are standard, and a compact limited-memory Kleinmichel formula gives a positive-definite rank-one update under the same curvature condition as BFGS [2605.11627].

Efficient subproblem solution depends on exploiting special metric structure. A foundational result for diagonal plus rank-one and, later, diagonal \(\pm\) rank-\(r\) metrics shows that if
\[
V=P\pm Q,\qquad Q=UU^\top,
\]
then the proximal mapping in metric \(V\) reduces to the proximal mapping in the simpler metric \(P\) plus a low-dimensional root-finding problem for \(\alpha\) [1206.1156][1801.08691]. In separable cases this yields exact or highly efficient implementations because the dual function is piecewise linear or piecewise affine.

Large-scale nonconvex composite methods use compact representations
\[
B_k=B_{k,0}+A_k Q_k^{-1}A_k^\top,
\]
then decompose the regularized metric into
\[
H_k=\hat B_0+U_1U_1^\top-U_2U_2^\top.
\]
The resulting metric prox can be expressed through a simple prox in metric \(\hat B_0\) and the solution of a small strongly monotone nonlinear system by a semismooth Newton method [2210.07644]. In that setting, the main operation is repeated evaluation of \(\operatorname{prox}_\varphi^{\hat B_0}\), and the numerical study reports that only 1–2 semismooth Newton steps suffice to reach residual norms \(\|\mathcal L(\alpha)\|\le 10^{-10}\), independent of the memory size [2210.07644].

A related mechanism appears in mini-batch TV image reconstruction. There the aggregate metric has the form
\[
H_k=\Sigma_k+U_kU_k^\top,
\]
coming from subsetwise SR1 updates, and the weighted proximal mapping is reduced to a low-dimensional nonlinear system and a diagonal-metric prox inside a dual TV solver [2307.02043]. This suggests that low-rank metric perturbations are not merely a theoretical convenience; they are an implementation principle that recurs across Euclidean, stochastic, and imaging variants.

## 4. Globalization, stationarity, and convergence theory

Globalization mechanisms fall into three main groups: line search, adaptive regularization, and model-agreement ratios. In convex proximal Newton-type methods, backtracking accepts \(\alpha\in(0,1]\) when
\[
F(x_t+\alpha d_t)-F(x_t)\le \frac{\alpha}{4}\big(d_t^\top \nabla f(x_t)+r(x_{t+1}')-r(x_t)\big),
\]
and the exact-Hessian Shamanskii regime attains local order at least \(\sqrt[n]{n+1}\), while approximate \(H_t\) satisfying \(H_t\succeq mI\) and \(\|H_t-\nabla^2 f(x_t)\|=o(1)\) yield local superlinear convergence [2007.09525]. Inexact subproblem solves retain linear, superlinear, or higher-order behavior depending on the inexactness exponent \(\gamma\) and the forcing sequence \(\eta_t\) [2007.09525].

Regularization-based globalization replaces line search by predicted versus actual decrease tests. In the regularized proximal quasi-Newton method, one defines
\[
pred_k=\psi(x^k)-q_k(d^k),\qquad ared_k=\psi(x^k)-\psi(x^k+d^k),\qquad \rho_k=\frac{ared_k}{pred_k},
\]
and adapts \(\mu_k\) according to whether the iteration is unsuccessful, successful, or highly successful [2210.07644]. Under bounded \(B_k\) and \(\psi\) bounded below, one obtains \(\liminf \|r(x^k)\|=0\), and with uniform continuity of \(\nabla f\) on a set containing the iterates, \(\|r(x^k)\|\to 0\), so every accumulation point is stationary [2210.07644]. Under a Luo–Tseng type local error bound and bounded positive-definite \(H_k\), the whole sequence converges to a stationary point and
\[
\sum_{k=0}^\infty \|x^{k+1}-x^k\|<\infty
\]
[2210.07644].

In the prox-bounded nonsmooth nonconvex setting, adaptive regularization again controls globalization through sufficient decrease. Under bounded \(B_k\), lower bounded \(F\), and local Lipschitz gradient, every accumulation point is stationary, and if \(F\) satisfies the Kurdyka–Łojasiewicz property at a stationary accumulation point, the whole sequence converges to it [2605.11627]. For a power desingularizing function \(\chi(t)=ct^{1-\theta}\), the successful subsequence has Q-superlinear objective convergence and R-superlinear iterate convergence when \(\theta\in(0,\tfrac12)\), Q-linear and R-linear convergence when \(\theta=\tfrac12\), and sublinear rates when \(\theta\in(\tfrac12,1)\) [2605.11627].

A more general model-function framework proves that boundedness of the variable metric need not be an a priori hypothesis. Under mild first-order assumptions on the model error, every accumulation point of the generated sequence is stationary, and if the objective has the Kurdyka–Łojasiewicz property at the corresponding accumulation point, the whole sequence converges to a stationary point; in that regime the sequence of variable metrics becomes uniformly bounded [2507.18363]. This suggests that metric boundedness can be a consequence of objective regularity rather than an external assumption.

Multiobjective convergence uses Pareto stationarity as the limiting notion. With Armijo-type line search, every accumulation point of the generated sequence is Pareto stationary, and with fixed step \(\lambda_k\equiv1\) the same conclusion holds under Lipschitz continuity of the gradients and \(\omega>L/2\) [2108.00125].

For nonsmooth regularized nonconvex optimization with model Hessians allowed to grow, R2N establishes global convergence of the stationarity measure
\[
\chi_k=\nu_k^{-1/2}\xi_{\mathrm{cp}}(x_k,\nu_k^{-1})^{1/2}
\]
to zero without relying on local Lipschitz continuity of \(\nabla f\), while permitting model Hessians to be unbounded [2409.19428]. Under Lipschitz continuity of \(\nabla f\), the worst-case complexity is
\[
O(1/\epsilon^{2/(1-p)})
\]
for \(0\le p<1\), where \(p\) controls the growth of model Hessians, and
\[
O(\exp(c\epsilon^{-2}))
\]
when \(p=1\) [2409.19428].

On the Stiefel manifold, ARPQN is globally convergent to stationary points, has local linear convergence under a local strong-convexity-type condition on the pullback Hessian, and reaches an \(\epsilon\)-stationary point in at most \(O(1/\epsilon^2)\) outer iterations [2404.11112]. For the more faithful Riemannian proximal Newton model ARPN, exact Hessian or quasi-Newton approximations satisfying the Dennis–Moré condition give local q-superlinear convergence [2404.11112]. ManPQN proves global convergence of accumulation points to stationary points and local linear convergence under analogous manifold assumptions [2501.09516].

## 5. Specialized variants and extensions

The multiobjective formulation is distinctive because it avoids fixed scalarization weights. Separate quasi-Newton matrices are constructed for each objective, but the step is single and is obtained from the max-model over componentwise local models [2108.00125]. Constraints can be encoded through indicator functions \(h_i=\iota_S\), turning the subproblem into a projected model step, and robust multiobjective terms of the form
\[
h_i(x)=\max_{u\in U_i}\hat h_i(x,u)
\]
lead, in the polyhedral linear case, to convex quadratic minimization problems with quadratic inequality and linear constraints [2108.00125].

Stochastic and finite-sum variants adapt the same template to large datasets. A proximal stochastic quasi-Newton algorithm uses the update
\[
x_{k+1}=\operatorname{prox}_{\eta R}^{H_r}\bigl(x_k-\eta H_r^{-1}\nu_k\bigr),
\]
where \(\nu_k\) is an SVRG-type variance-reduced gradient estimator and \(H_r\) is a low-rank quasi-Newton metric built from subsampled Hessians [1602.00223]. Under strong convexity and suitable step size, the stagewise iterates satisfy
\[
\mathbb{E}[P(\tilde x_s)-P(x_*)]\le \rho^s [P(\tilde x_0)-P(x_*)],
\]
so the method converges linearly in expectation [1602.00223].

A domain-specific imaging variant, the mini-batch quasi-Newton proximal method for constrained total-variation nonlinear image reconstruction, builds per-subset quadratic models, aggregates them into a weighted metric prox subproblem, and exploits the dual representation of TV together with a diagonal-plus-low-rank metric structure [2307.02043]. The weighted projection in the inner loop is handled through the same low-dimensional proximal calculus that appears in generic Euclidean methods.

Riemannian variants preserve the model-based structure while changing the ambient geometry. In ARPQN and ManPQN, the step is computed in the tangent space, the nonsmooth term is retained as \(h(X_k+V)\), and the iterate is mapped back by a retraction [2404.11112][2501.09516]. This suggests that the model-based proximal quasi-Newton paradigm is compatible with changing tangent spaces, provided the curvature operator, vector transport, and retraction are updated coherently.

A broader implication is that the method is modular. The smooth model, the proximal term, the metric update, and the globalization mechanism can be recombined for Euclidean, multiobjective, stochastic, robust, constrained, and manifold settings without altering the basic algorithmic identity.

## 6. Empirical behavior and implementation practice

Across the cited literature, the empirical pattern is consistent: second-order or quasi-Newton curvature usually reduces outer iteration counts relative to proximal gradient baselines, and low-rank metric structure keeps the extra subproblem cost manageable.

For convex composite optimization, PSOPT uses L-BFGS to construct \(H_t\), solves subproblems with TFOCS, and on \(\ell_1\)-regularized logistic regression on `gisette` both PSOPT and PNOPT converge substantially faster than SpaRSA and FISTA in relative suboptimality versus time, with PSOPT best on `gisette` [2007.09525]. In regularized proximal quasi-Newton methods for convex and nonconvex composite problems, RPQN with limited-memory BFGS or SR1 is significantly faster than FISTA, SpaRSA, and QGPN on large least-squares and group-sparse instances, and remains robust on nonconvex image restoration [2210.07644]. In the 2026 limited-memory nonconvex nonsmooth framework, numerical results show a significant speed up compared to SPG, PANOC+, and R2N [2605.11627].

For multiobjective optimization, the reported robust UMOP experiments compare PGM with PQNM(BFGS), PQNM(SS-BFGS), and PQNM(H-BFGS). For a fixed uncertainty level \(\delta\), the Pareto frontier produced by PQNM(H-BFGS) is the lowest, followed by PQNM(SS-BFGS), then PQNM(BFGS), with PGM highest [2108.00125]. This suggests that the choice of secant update can affect not only speed but also the quality of the recovered frontier.

In manifold optimization, NLS-ARPQN typically requires fewer line search steps, often fewer ASSN iterations, and lower CPU time than ManPQN and proximal gradient variants, especially when \(n\) and \(r\) are large [2404.11112]. ManPQN itself needs dramatically fewer iterations than ManPG and ManPG-Ada on compressed modes and sparse PCA, and often achieves the lowest CPU time on larger problems despite heavier iterations [2501.09516].

In computational imaging, BQNPM achieves a weighted proximal mapping at a cost similar to the proximal mapping in ASPMs, while requiring fewer iterations to converge; on three-dimensional inverse-scattering problems, wall-clock reductions of about \(2\times\)–\(3\times\) are reported relative to ASPM, with similar or better reconstruction quality [2307.02043].

For computationally intensive \(\ell_1\)-regularized M-estimators, the proximal quasi-Newton method with aggressive active-set shrinking converges considerably faster than current state-of-the-art on sequence labeling and hierarchical classification, outperforming Prox-GD, OWL-QN, SGD, and BCD in wall-clock time to a small objective gap [1406.7321]. The implementation emphasis there is not only on quasi-Newton curvature but on shrinking, sparse feature-indexed data structures, and limited numbers of full expensive gradient computations.

Implementation practice in this literature repeatedly favors a small set of recurring techniques: compact limited-memory representations, diagonal \(\pm\) low-rank metric decompositions, semismooth Newton for tiny dual systems, active-set restriction when sparsity is expected, and predicted-versus-actual reduction tests when the problem is nonconvex or geometrically constrained. These recurring design choices help explain why the method functions as a family of algorithms rather than a single fixed scheme.

Source: https://www.emergentmind.com/topics/model-based-proximal-quasi-newton-method