---
title: LMM Subgradient Method for Composite Optimization
url: https://www.emergentmind.com/topics/levenberg-morrison-marquardt-subgradient-method
type: topic
---

# LMM Subgradient Method for Composite Optimization

The Levenberg–Morrison–Marquardt subgradient method is a preconditioned first-order method for composite optimization problems of the form
$$
\min_{x\in\mathbb R^d} f(x):=h(F(x)),
$$
where $F:\mathbb R^d\to\mathbb R^m$ is $C^1$–smooth and $h:\mathbb R^m\to\mathbb R$ is convex and possibly nonsmooth. Its defining feature is an LM-style preconditioner,
$$
(\nabla F(x)^\top \nabla F(x)+\lambda I)^{-1}\nabla F(x)^\top,
$$
inserted between a subgradient of $h$ and the parameter update. In the formulation introduced in "Preconditioned subgradient method for composite optimization: overparameterization and fast convergence," the method is designed to overcome the slow sublinear behavior of the vanilla subgradient method in ill-conditioned or overparameterized composite models, and it is proved to converge linearly under mild regularity conditions at a rate determined solely by the convex function [2509.11486].

## 1. Optimization model and notation

The method is posed for composite minimization
$$
\min_{x\in\mathbb R^d} f(x):=h(F(x)),
$$
with $F:\mathbb R^d\to\mathbb R^m$ of class $C^1$ and $h$ convex, possibly nonsmooth. The convex subdifferential of $h$ at $z$ is denoted $\partial h(z)$. The Jacobian is written $\nabla F(x)\in\mathbb R^{d\times m}$, with adjoint $\nabla F(x)^\top$. For a given point $x$, $\Pi^x:=\mathrm{Proj}_{\mathrm{Im}\,\nabla F(x)}$ denotes the orthogonal projector onto the range of $\nabla F(x)$, and
$$
P(x,\lambda):=\nabla F(x)\big[\nabla F(x)^\top\nabla F(x)+\lambda I\big]^{-1}\nabla F(x)^\top.
$$

The analysis fixes a solution $x^*$ and writes $z^*:=F(x^*)$. The regularity assumptions are organized around three components. Assumption A requires $\nabla F$ to be $L_J$–Lipschitz:
$$
\|\nabla F(x)-\nabla F(y)\|_{op}\le L_J\|x-y\|\quad\text{for all }x,y.
$$
Assumption B concerns the geometry of the image of $F$ near $x^*$. Under strong alignment, the rank of $\nabla F(\cdot)$ is locally constant and equal to $r^*$, so $\mathrm{Im}\,F$ is a $C^1$–manifold and the tangent-manifold error satisfies
$$
\|(I-\Pi^x)[F(x)-z^*]\|\le C\|F(x)-z^*\|^2.
$$
Under weak alignment, appropriate singular directions of $\nabla F(x)$ capture the dominant part of $z-z^*$, while a lower singular-value bound scales with $\|z-z^*\|$.

The convex term $h$ is handled through two alternative hypotheses. Assumption $C_1$ covers nonsmooth sharp objectives: $h$ is convex on $\mathrm{Im}\,F$, has a unique minimizer $z^*$, is $\mu$-sharp,
$$
h(z)-h(z^*)\ge \mu\|z-z^*\|,
$$
and satisfies restricted Lipschitz conditions involving $\Pi^x$ and $I-P(x,\lambda)$. Assumption $C_2$ covers smooth objectives with quadratic growth: $h$ has $\alpha$–quadratic growth, restricted $\beta$–smoothness, and analogous compatibility bounds with the projected and damped geometry [2509.11486].

## 2. Algorithmic update and LM preconditioning

Algorithm 1 in the cited work takes positive sequences $\{\gamma_k\}$ and $\{\lambda_k\}$ and, at each iteration, chooses a subgradient $v_k\in\partial h(F(x_k))$, solves
$$
\big(\nabla F(x_k)^\top\nabla F(x_k)+\lambda_k I\big)s_k=\nabla F(x_k)^\top v_k,
$$
and updates
$$
x_{k+1}=x_k-\gamma_k s_k.
$$
Equivalently,
$$
x_{k+1}=x_k-\gamma_k\big(\nabla F(x_k)^\top\nabla F(x_k)+\lambda_k I\big)^{-1}\nabla F(x_k)^\top v_k.
$$

In the image space, the step is approximately
$$
z_{k+1}\approx z_k-\gamma_k P(x_k,\lambda_k)v_k,
$$
with a second-order Taylor error of order $O(\gamma_k^2/\lambda_k)$. More precisely, the update in $z$-space satisfies
$$
z_{k+1}=z_k-\gamma_k[P_k v_k]+O(\gamma_k^2/\lambda_k),
$$
and the remainder is controlled by
$$
\frac{L_J\gamma_k^2}{8\lambda_k}\|\Pi^x v_k\|^2.
$$

The role of the preconditioner is central. The term $\nabla F(x_k)^\top v_k$ is an ordinary subgradient of $f=h\circ F$. The premultiplier
$$
(\nabla F^\top\nabla F+\lambda I)^{-1}\nabla F^\top
$$
normalizes directions by the local Jacobian of $F$, thereby undoing ill-conditioning or rank-deficiency of $F$. When $\lambda\to 0$ and $\nabla F(x)$ has full column rank, $P(x,0)=\Pi^x$, the projector onto $\mathrm{Im}\,\nabla F(x)$. The eigenvalues of $P(x,\lambda)$ lie in $[0,1]$, and the eigenvalues of $I-P(x,\lambda)$ are at most $\lambda/(\sigma_j(\nabla F)^2+\lambda)$ [2509.11486].

## 3. Alignment assumptions and local geometry

The convergence theory is built on an interaction between the local geometry of $F$ and the growth properties of $h$. Under strong alignment, the image of $F$ behaves locally as a $C^1$–manifold of constant rank. In that regime, the projected error onto the normal space is quadratic in $\|F(x)-z^*\|$, which makes the geometry close to exact parameterization. Under weak alignment, the framework allows overparameterization: even if rank-deficiency or degeneracy is present, there exist singular directions of $\nabla F(x)$ that capture most of the signal in $z-z^*$, while the residual component can be made arbitrarily small relative to $\|z-z^*\|$.

The assumptions on $h$ are correspondingly restricted to the image geometry. In the nonsmooth case, sharpness and restricted Lipschitzness are not imposed on the whole ambient space $\mathbb R^m$ in an unrestricted way; instead, they are formulated through the projected component $\Pi^x v$ and the damped residual $(I-P(x,\lambda))(F(x)-z^*)$. In the smooth case, the analogous hypotheses are quadratic growth and restricted smoothness. This suggests that the method isolates the conditioning of the convex outer function from the possible ill-conditioning of the smooth parametrization map.

The paper emphasizes two regimes. In an exact-rank regime, the method reduces to a Riemannian subgradient on the image-manifold. In overparameterized or ill-conditioned regimes, the damping parameter $\lambda$ ensures invertibility and stability. The resulting theory is explicitly local, centered at a fixed solution $x^*$ and its image $z^*$ [2509.11486].

## 4. Hyperparameter configurations and convergence guarantees

Two hyperparameter configurations are stated. The Polyak-type configuration requires access to $h(z_k)-h^*$ and $\|\Pi^{x_k}v_k\|$ and uses
$$
\gamma_k=\gamma\frac{h(z_k)-h^*}{\|\Pi^{x_k}v_k\|^2},
$$
with $\lambda_k$ between $\mathrm{dist}(z_k,Z^*)$ and a multiple thereof. The geometric-decay configuration avoids knowledge of $h^*$: for nonsmooth $h$, one sets $\gamma_k=\gamma q^k$ and $\lambda_k=\lambda q^k$; for smooth $h$, one sets $\gamma_k=\gamma$ and $\lambda_k=\lambda q^k$, with $q\in(0,1)$.

The principal theorem in the nonsmooth weak-alignment setting states local linear convergence under Assumptions A, B(weak), and $C_1$. If $F(x_0)=z_0$ satisfies
$$
\|z_0-z^*\|\le \delta(\mu/8L),
$$
and Algorithm 1 is run with the Polyak configuration, with $\gamma\le \min\{1,\mu^2/L^2\}$, then for all $k\ge 0$,
$$
\|z_k-z^*\|\le \big(1-\gamma\mu^2/(8L^2)\big)^{k/2}\|z_0-z^*\|.
$$
The proof proceeds by comparing the true iterate with the one-step linearized iterate
$$
\hat z_{k+1}=z_k-\gamma_kP_kv_k.
$$
Taylor’s theorem and the Lipschitzness of $\nabla F$ control the deviation $\|z_{k+1}-\hat z_{k+1}\|$, while restricted sharpness and restricted Lipschitzness yield a contraction for the linearized step. Under strong alignment, the exponent improves from $k/2$ to $k$.

The smooth case is described as analogous, replacing sharpness by quadratic growth and replacing the ratio $L^2/\mu^2$ by the corresponding $\beta/\alpha$ dependence. The central conclusion is that the local linear rate is governed solely by the conditioning of the convex function $h$, rather than by the ill-conditioning or overparameterization of $F$ [2509.11486].

## 5. Relation to classical subgradient methods and nearby LM schemes

The comparison with the classical subgradient method is explicit. A standard update takes the form
$$
x_{k+1}=x_k-\alpha_k g_k,\qquad g_k\in\partial f(x_k).
$$
Even with Polyak stepsizes,
$$
\alpha_k=\frac{f(x_k)-f^*}{\|g_k\|^2},
$$
linear convergence requires $f$ itself to be sharp and Lipschitz on $\mathbb R^d$. For composite objectives $f=h\circ F$, this requirement is restrictive because the composition may be ill-conditioned or rank-deficient near a solution. In that setting, Gauss–Newton subgradient with $\lambda_k\equiv 0$ and the unpreconditioned subgradient can stagnate at sublinear rate or diverge, whereas LMM uses the damped LM preconditioner to recover linear rates controlled by $h$ [2509.11486].

A related but distinct line of work appears in "A Levenberg-Marquardt Method for Nonsmooth Regularized Least Squares," which studies problems of the form
$$
\min_x \Phi(x):=\tfrac12\|F(x)\|_2^2+h(x),
$$
with $h$ proper, lower-semicontinuous, and possibly nonsmooth and nonconvex. There, the LM step is obtained by minimizing a regularized linear least-squares model plus a model of $h$, approximately solved by a first-order method such as proximal gradient, and both a regularization variant and a trust-region variant are shown to have worst-case complexity $O(\epsilon^{-2})$ for reducing a stationarity measure below $\epsilon\in(0,1)$ [2301.02347]. This suggests a methodological neighborhood rather than an identity: both frameworks use LM regularization around Jacobian-linearized models, but the LMM subgradient method is formulated as a preconditioned subgradient method for $h\circ F$, while the nonsmooth regularized least-squares method targets $\tfrac12\|F(x)\|_2^2+h(x)$ and analyzes model-based outer iterations with inner proximal solves.

## 6. Applications and computational behavior

The LMM method is applied to several classes of problems. For squared-variable reformulations of nonnegative least squares, with $F(x)=x\odot x$ and $h=\|A(\cdot)-b\|_2$ or $\ell_1$, weak alignment for $F$ and restricted sharpness or Lipschitzness for $h$ can be verified via $\sigma_{\min}(A)$. In that setting, LMM converges linearly at rate $O(\kappa(A)^2)$ in practice and outperforms both Gauss–Newton-subgradient and the unpreconditioned subgradient.

For low-rank matrix sensing, the method is analyzed for positive-semidefinite factors $F(U)=UU^\top$ and asymmetric factors $F(U,V)=UV^\top$. Under RIP, one obtains strong or weak alignment for $F$ and restricted smoothness or growth for $h(M)=\frac12\|A(M)-b\|_2^2$ or for $\ell_1$ losses. The resulting local linear convergence is described as independent of the condition number of $U^*$ and governed solely by $\delta$–RIP constants. For asymmetric factors, local weak alignment together with balanced spectral initialization again yields linear rates for $\ell_2$ and $\ell_1$ losses, including robustness to gross outliers.

For canonical-polyadic tensor factorization, the paper considers symmetric CP,
$$
F(X)=\sum_k X_k\otimes X_k\otimes X_k,
$$
and asymmetric CP,
$$
F(W,X,Y)=\sum_k W_k\otimes X_k\otimes Y_k.
$$
Local strong alignment is established near a full-rank CP factorization, and $\alpha$–quadratic growth together with Lipschitz properties are verified for $h(T)=\|T-T^*\|_F$. Consequently, LMM enjoys local linear convergence with $\ell_2$ loss. The reported numerical experiments on nonnegative least squares, matrix sensing, and tensor factorization confirm consistent linear convergence to exact solutions, improved performance over unpreconditioned subgradient and Gauss–Newton subgradient in ill-conditioned or overparameterized regimes, and robustness to gross outliers when using the $\ell_1$ loss [2509.11486].

A nearby empirical pattern appears in the nonsmooth regularized least-squares LM literature: for a group-lasso basis-pursuit denoise problem, a nonlinear support vector machine, and a FitzHugh–Nagumo neuron-firing inverse problem, the LM variants required fewer outer iterations than adaptive proximal gradient and a quasi-Newton trust-region method, although at the expense of many more inner proximal-gradient steps [2301.02347]. A plausible implication is that LM-style curvature information can substantially reduce outer-iteration counts, while leaving the efficiency of the inner linearized composite solve as a critical implementation bottleneck.

Source: https://www.emergentmind.com/topics/levenberg-morrison-marquardt-subgradient-method