---
title: Proximal Alternating Linearized Minimization Algorithm
url: https://www.emergentmind.com/topics/proximal-alternating-linearized-minimization-algorithm
type: topic
---

# Proximal Alternating Linearized Minimization Algorithm

The Proximal Alternating Linearized Minimization (PALM) algorithm and its advanced variants, such as the Two-step Inertial Bregman PALM (TiBPALM), constitute a robust class of block-coordinate methods for structured optimization. These methods address general nonconvex, nonsmooth, and nonseparable two-block problems of the form
$$
\min_{x \in \mathbb{R}^l, \, y \in \mathbb{R}^m} \; L(x, y) = f(x) + Q(x, y) + g(y)
$$
where $f$ and $g$ are proper, lower-semicontinuous functions (possibly nonconvex and nonsmooth), and $Q$ is $C^1$ with Lipschitz continuous gradient. The PALM framework, particularly in its inertial and Bregman generalized forms, delivers global convergence guarantees under the Kurdyka–Łojasiewicz (KL) property, and allows block-specific generalizations critical for modern large-scale applications [2306.07614].

## 1. Class of Problems and Block Structure

The core setting involves partitioning the optimization variables into two (or, in some extensions, more) blocks, $x$ and $y$, over which the objective $L(x, y) = f(x) + Q(x, y) + g(y)$ is defined. Here,
- $f: \mathbb{R}^l \to (-\infty, +\infty]$ and $g: \mathbb{R}^m \to (-\infty, +\infty]$ are (potentially nonconvex, nonsmooth) proper, lsc functions.
- $Q: \mathbb{R}^l \times \mathbb{R}^m \to \mathbb{R}$ is $C^1$ in $(x, y)$ with $\nabla Q$ Lipschitz continuous on bounded sets.
- No separability of $f + Q + g$ is assumed.

This abstraction is broad enough to encompass constrained and penalized data fitting in signal recovery, machine learning, imaging, and quadratic fractional programming.

## 2. Standard PALM Methodology

The fundamental PALM scheme operates by alternating linearized majorizations of the block-coupled term $Q(x, y)$ and applying blockwise proximal steps:
- At iterate $(x^k, y^k)$, pick step-sizes $\lambda_k > 0$, $\mu_k > 0$. Perform
  $$
  x^{k+1} \in \arg \min_x \left\{
    f(x) + \langle \nabla_x Q(x^k, y^k), x - x^k \rangle + \frac{1}{2\lambda_k} \| x - x^k \|^2
  \right\}
  $$
  $$
  y^{k+1} \in \arg \min_y \left\{
    g(y) + \langle \nabla_y Q(x^{k+1}, y^k), y - y^k \rangle + \frac{1}{2\mu_k} \| y - y^k \|^2
  \right\}
  $$
This translates to block coordinate descent with local quadratic majorants. The PALM iteration admits broad regularity (no convexity of $f$, $g$, or $Q$ needed beyond the outlined smoothness/lsc assumptions).

## 3. Inertial and Bregman Extensions: TiBPALM

To accelerate convergence and handle generic geometry, the TiBPALM algorithm incorporates:
- **Two-step inertia:** Extrapolations of the form $\alpha_{1k}(x_k - x_{k-1}) + \alpha_{2k}(x_{k-1} - x_{k-2})$ allow leverage of momentum from two preceding steps, generalizing standard heavy-ball approaches.
- **Bregman distance regularization:** The squared norm is replaced with a Bregman kernel $D_\phi(u, v) := \phi(u) - \phi(v) - \langle \nabla \phi(v), u - v \rangle$, typically with a strongly convex, smooth $\phi$. This is critical when the geometry of proximal steps is non-Euclidean or a Euclidean prox is not easily solvable.

The TiBPALM update for block $x$ (similarly for $y$) is:
$$
x_{k+1} \in \arg\min_x \Big\{
  f(x) + \langle \nabla_x Q(x_k, y_k), x \rangle + D_{\phi_1}(x, x_k)
  + \alpha_{1k}\langle x, x_{k-1} - x_k \rangle
  + \alpha_{2k} \langle x, x_{k-2} - x_{k-1} \rangle
\Big\}
$$
Momentum and geometry are thus jointly exploited.

## 4. Benefit Function and Convergence Analysis

A distinctive feature of TiBPALM is the introduction of a *benefit function* $H(u,v,w)$ across three consecutive iterates:
$$
H(u, v, w) := L(u) + \frac{\alpha_1 + \alpha_2}{2}\| u - v \|^2 + \frac{\alpha_2}{2}\| v - w \|^2
$$
This legitimizes sufficient decrease arguments accounting for the inertial terms.

Under strong convexity of the Bregman kernels $\phi_i$ (constants $\theta_i$) and appropriate upper bounds on the inertia parameters $\alpha_i$, one establishes the key decrease inequality:
$$
H(z_{k+1}, z_k, z_{k-1}) + a \| z_{k+1} - z_k \|^2 \leq H(z_k, z_{k-1}, z_{k-2})
$$
where $a = (\rho - 2(\alpha_1 + \alpha_2))/2 > 0$ and $\rho = \min\{\theta_1 - L_1^+,\, \theta_2 - L_2^+\}$.

The KL property is invoked to conclude global convergence: the whole sequence $\{z_k\}$ has finite length and converges to a critical point of $L$. This leverages the abstract KL-descent framework for general nonconvex, nonsmooth scenarios [2306.07614].

## 5. Related Inertial and Stochastic Developments

The inertial Bregman PALM framework synthesizes and generalizes several prior advancements:
- **iPALM** (Inertial PALM) incorporates single-step momentum but uses standard prox-terms rather than Bregman distances [1702.02505].
- **Stochastic and variance-reduced schemes** adapt PALM to finite-sum problems using variance-reduced estimators (SAGA, SARAH), with global convergence and accelerated rates established under similar KL-type assumptions [2307.05287, 2002.12266]. The stochastic two-step inertial Bregman PALM (STiBPALM) extends TiBPALM to the stochastic regime.

A comparison is summarized below:

| Method      | Inertia                 | Proximal Geometry | Stochasticity | Reference        |
|-------------|-------------------------|-------------------|---------------|------------------|
| PALM        | None                    | Euclidean         | No            | [2306.07614]     |
| iPALM       | 1-step (heavy-ball)     | Euclidean         | No            | [1702.02505]     |
| TiBPALM     | 2-step                  | Bregman           | No            | [2306.07614]     |
| STiBPALM    | 2-step                  | Bregman           | Yes           | [2307.05287]     |

The advantage of Bregman distances becomes pronounced when prox-subproblems associated with convex indicator or regularizer functions become tractable in non-Euclidean geometry (e.g., Kullback–Leibler distance for nonnegativity, half-shrinkage for $\ell_{1/2}$).

## 6. Empirical Performance and Applications

Numerical studies confirm TiBPALM’s superiority over both noninertial PALM and one-step inertial variants (iPALM, GiPALM) on diverse nonconvex benchmarks:
- **Sparse nonnegative matrix factorization** with $\ell_0$-sparsity and nonnegativity.
- **Sparse signal recovery** under $\ell_{1/2}$ regularization.
- **Quadratic fractional programming** with box constraints.

TiBPALM consistently achieved faster decrease in the objective (fewer iterations and reduced wall-clock time for the same accuracy), attributable to the combination of two-step inertial extrapolation and Bregman geometry, and enabled efficient closed-form inner updates in practical instances [2306.07614].

## 7. Extensions and Theoretical Significance

The PALM paradigm, especially in its inertial and Bregman-extended forms, underpins a rich landscape of modern nonconvex optimization algorithms. It readily admits:
- Block-separable multimodal generalizations (BPALM/A-BPALM) [1908.01402].
- Variable metric and composite proximal variants for composite nonsmooth terms [2209.06799].
- Inexact and infeasible subsolver frameworks (PALM-I) with surrogate sequences restoring descent [2204.06182].
- Unrolled, learned, and interpretable deep optimization networks, where the entire PALM structure is encoded in a parameter-efficient, convergence-guaranteed architecture [2411.06333].

These developments establish PALM and its advanced variants as foundational tools for scalable, structured, and theoretically grounded optimization in modern computational mathematics and machine learning.

Source: https://www.emergentmind.com/topics/proximal-alternating-linearized-minimization-algorithm