---
title: PALM Algorithm for Nonconvex Optimization
url: https://www.emergentmind.com/topics/proximal-alternating-linearized-minimization-palm-algorithm
type: topic
---

# PALM Algorithm for Nonconvex Optimization

Proximal Alternating Linearized Minimization (PALM) is a fundamental class of block-coordinate, first-order algorithms for structured, nonsmooth, possibly nonconvex optimization problems. It is designed to handle objectives that decompose into a sum of a smooth (potentially nonconvex) coupling function and several potentially nonsmooth, prox-computable block-separable terms. PALM and its extensions (including stochastic, asynchronous, and inertial enhancements) have established convergence guarantees, oracle complexity bounds, and broad applicability in signal processing, machine learning, imaging, and control.

## 1. Canonical Problem Structure and PALM Updates

PALM is formulated for minimization problems in block variables $(x_1, \ldots, x_m)$:
\[
\min_{x_1, \ldots, x_m} F(x_1,\ldots,x_m) = f(x_1,\ldots,x_m) + \sum_{i=1}^m g_i(x_i)
\]
where:
- $f:\mathbb{R}^d\to \mathbb{R}$ is continuously differentiable (not necessarily convex).
- Each $g_i:\mathbb{R}^{d_i}\to \mathbb{R}\cup\{+\infty\}$ is proper, lower-semicontinuous, and possibly nonconvex, but has a proximal operator computable in closed form or efficiently.

**Block-wise proximal updates (classical PALM)**, for step sizes $\{\alpha_i\}$ satisfying $0<\alpha_i<1/L_i$ (with $L_i$ the block-wise Lipschitz constants of $\nabla_{x_i} f$), are:
\[
x_i^{k+1} = \text{prox}_{\alpha_i g_i} \left( x_i^k - \alpha_i \nabla_{x_i}f(x_1^{k+1},\ldots,x_{i-1}^{k+1},x_i^k,x_{i+1}^k,\ldots,x_m^k) \right),\quad i=1,\ldots,m
\]
with the **proximal operator** defined as:
\[
\text{prox}_{\alpha g}(v) = \arg\min_u \left\{ g(u) + \frac{1}{2\alpha} \|u-v\|^2 \right\}
\]

The method alternates block-wise, using most recent updates for preceding blocks and current values for subsequent ones ("Gauss-Seidel" ordering).

## 2. Theoretical Guarantees and Assumptions

PALM's convergence theory relies on three key structural assumptions:
- **(A1) Block-Lipschitz partial gradients:** For all $i$, $\nabla_{x_i} f$ is $L_i$-Lipschitz in $x_i$ (other blocks fixed).
- **(A2) Prox-computability:** Each $g_i$ is proper, lower-semicontinuous, bounded below, and admits a cheap proximal operator.
- **(A3) Objective bounded below:** $F$ is bounded from below.

Global convergence is achieved under the **Kurdyka–Łojasiewicz (KL) property**: $F$ satisfies the KL inequality at all critical points, which provides a desingularizing function $\varphi$ relating objective gaps and generalized gradient mapping norms.

**Conclusions under these assumptions:**
- The sequence $\{x^k\}$ has finite length: $\sum_k \|x^{k+1}-x^k\|<\infty$.
- $\{x^k\}$ converges to a critical point $x^*$ of $F$.
- The generalized gradient mapping $G$ at a random iterate $\alpha \in \{1, \ldots, K\}$ achieves $E[\|G(x^\alpha)\|^2]=O(1/K)$.
- When $F$ has an error-bound property (KL exponent $\theta=1/2$), **linear convergence** is achieved:
  \[
  F(x^k) - F^* \leq (1-c)^k [F(x^0)-F^*],\quad c\in (0,1)
  \]
  Otherwise, a sublinear rate is achieved.

## 3. Stochastic and Variance-Reduced Extensions: SPRING

For large-scale problems where $f$ is a finite sum $f(x)=\frac{1}{n} \sum_{i=1}^n F_i(x)$, evaluating full gradients is expensive. The **SPRING** algorithm introduces variance-reduced stochastic gradient estimators into the PALM framework [2002.12266].

**SPRING iteration with mini-batch $B_k \subset \{1,\ldots,n\}$:**
- Compute estimators (e.g., for block $x$):
  - *SGD*: $v_x^k = (1/b)\sum_{j\in B_k} \nabla_x F_j(x^k, y^k)$
  - *SAGA*: $v_x^k = (1/b) \sum_{j\in B_k} [\nabla_x F_j(x^k,y^k) - g_{k,j}] + (1/n) \sum_{i=1}^n g_{k,i}$, with $g_{k,i}$ storing last gradients.
  - *SARAH*: $v_x^k=v_x^{k-1} + (1/b) \sum_{j\in B_k} [\nabla_x F_j(x^k, y^k) - \nabla_x F_j(x^{k-1}, y^{k-1})]$ (else full gradient refresh with probability $1/p$).
- Update blocks with stochastic estimators replacing full gradients in classical PALM:
  \[
  x^{k+1} = \text{prox}_{\gamma_x J}(x^k - \gamma_x v_x^k),\quad y^{k+1} = \text{prox}_{\gamma_y R}(y^k - \gamma_y v_y^k)
  \]

### Oracle Complexity Results

SPRING achieves **state-of-the-art oracle complexities** for nonconvex finite-sum problems:
- **SPRING-SAGA with $b=n^{2/3}$:**
  - $O(n^{2/3} L/\epsilon^2)$ SFO evaluations required for $E[\|G(x^\alpha)\|^2]\le\epsilon^2$.
  - Under error-bound: $O((n+L n^{2/3}/\mu)\cdot \log(1/\epsilon))$.
- **SPRING-SARAH with $p\approx n$:**
  - $O(\sqrt{n} L/\epsilon^2)$ SFO calls (matching the lower bound for nonconvex finite sums).
  - Under error-bound: $O((n+L\sqrt{n}/\mu)\cdot \log(1/\epsilon))$.

## 4. Empirical Performance and Practical Implementation

The PALM and SPRING families have been evaluated on large-scale imaging problems, including:
- Sparse nonnegative matrix factorization (sNMF)
- Sparse principal component analysis (sPCA)
- Blind image deconvolution (e.g., Yale, ORL, Kodak datasets)

**Summary of observed behavior:**
- Deterministic PALM converges reliably but has $O(n)$ per-iteration cost.
- Inertial variants (see also [1702.02505], [2005.02204]) accelerate convergence but require full gradients.
- Naive SPRING-SGD exhibits slow convergence due to gradient variance.
- SPRING-SAGA and SPRING-SARAH exhibit low per-iteration complexity ($O(b)$ with $b\ll n$) and fast variance decay, yielding superior objective decrease per unit work.
- In realistic signal and image deconvolution tasks, SPRING-SARAH outperforms full-gradient PALM by an order of magnitude in oracle calls for equivalent accuracy.
- The stochastic PALM framework retains global convergence guarantees of deterministic PALM on nonconvex, nonsmooth problems.

## 5. Algorithmic Enhancements and Variants

**Inertial and Acceleration Schemes:** Inertial PALM (iPALM) incorporates heavy-ball–style extrapolation to speed convergence [1702.02505]. Stochastic inertial PALM (iSPALM) merges inertia with variance-reduced estimation, showing improved wall-clock and epoch efficiency on high-dimensional mixtures and learning settings [2005.02204].

**Asynchronous and Parallel Frameworks:** SAPALM generalizes stochastic block-coordinate PALM to asynchronous parallel execution with stale (delayed) reads and writes, achieving near-linear speedup with $P=O(\sqrt{m})$ workers for $m$ blocks [1606.02338]. Atomic block updates are performed on possibly-outdated parameter vectors, preserving theoretical convergence rates under mild additional assumptions.

## 6. Implementation and Scaling Considerations

When deploying PALM-type algorithms:
- **Step-sizes** must reflect (global or block-wise) Lipschitz constants of the smooth coupling term's partial gradients for convergence. Stochastic variants must adapt step sizes to variance and batch size.
- **Proximal operators** for regularizers $g_i$ must be efficiently computable; in practice, sparsity and cardinality constraints yield closed-form hard-thresholding or $l_0$ projection.
- **Resource scaling:** In stochastic variants, per-iteration cost is $O(b)$ for batch size $b$, with optimal $b$ analytically established for SAGA/SARAH schemes.
- In asynchronous settings, delays must be controlled ($\tau = O(\sqrt{m})$) to guarantee linear speedup.
- The approach is suited to large-scale nonconvex signal and image recovery, dictionary learning, and high-volume data analytics, where variance reduction amortizes over the cost of a single gradient evaluation.

## 7. Comparison and Limitations

| PALM Variant    | Per-Iter Cost | Convergence Rate        | Scalability / Comments                |
|-----------------|--------------|-------------------------|---------------------------------------|
| Deterministic   | $O(n)$       | $O(1/K)$ (sublinear); linear if KL-$1/2$ | Expensive for large $n$               |
| SPRING-SAGA     | $O(b)$       | $O(n^{2/3}L/\epsilon^2)$| Optimal with mini-batch variance reduction |
| SPRING-SARAH    | $O(b)$       | $O(\sqrt{n}L/\epsilon^2)$| Best oracle complexity for finite sums |
| iPALM/iSPALM    | $O(n)$/$O(b)$| Empirically faster; theory for linear under EB | Adds inertia/momentum                 |
| SAPALM          | $O(1)$/core  | $O(1/T)$ in expectation | Near-linear multicore speedup         |

Deterministic PALM is preferred when full gradients are tractable. SPRING and its inertial extensions are established choices for large-scale or distributed settings, with provably optimal complexity for finite-sum, nonconvex, nonsmooth objectives. The primary limitations are sensitivity of deterministic PALM to data size and of stochastic PALM to batch size/variance tuning. Inertial and asynchronous versions require precise parameter management and infrastructure support to realize theoretical rates.

PALM and its modern extensions occupy a central role in scalable first-order optimization for data science, signal processing, and large-scale inverse problems. Their theoretical guarantees are underpinned by the Kurdyka–Łojasiewicz property, and their flexibility is exemplified by a range of practical enhancements and broad empirical validation [2002.12266].

Source: https://www.emergentmind.com/topics/proximal-alternating-linearized-minimization-palm-algorithm