Penalty-Based PALM Optimization

• Penalty-Based PALM is a block-coordinate method for nonconvex, nonsmooth optimization that incorporates penalty terms to enforce constraints and coupling.
• It linearizes smooth coupling functions and applies proximal updates to individual blocks, making complex subproblems tractable.
• Variants like asynchronous, inertial, and Bregman PALM extend its applicability from deep network training to matrix and tensor recovery with solid convergence guarantees.

Searching arXiv for recent and foundational PALM-related papers to anchor the article. arxiv_search(query="proximal alternating linearized minimization penalty PALM nonconvex nonsmooth", max_results=10) arxiv_search(query="PALM penalty-based proximal alternating linearized minimization", max_results=10) arxiv_search(query="(Davis, 2016) Asynchronous PALM Algorithm for Nonsmooth Nonconvex Problems", max_results=5) arxiv_search(query="(Yashtini, 2022) Composite Proximal Alternating Linearized Minimization", max_results=5) Penalty-Based Proximal Alternating Linearized Minimization (PALM) denotes a family of block-coordinate methods for nonconvex, nonsmooth composite optimization in which constraints, regularizers, or couplings are represented by indicator functions or explicit penalty terms, and each block is updated by minimizing a proximal model built from a linearization of the smooth coupling term. A canonical setting is

$$\min_{x \in \mathcal{H}} \; f(x_1,\ldots,x_m) + \sum_{j=1}^m r_j(x_j),$$

where $f$ is $C^1$, generally nonconvex, and blockwise Lipschitz-smooth, while each $r_j$ is proper, lower semicontinuous, and may be nonsmooth and nonconvex; many penalty-based formulations fit this template because constraints or couplings can be encoded through indicator functions or penalty functions inside the block terms (Davis, 2016). Across the literature, this pattern is extended to lifted quadratic-penalty formulations for deep network training (Qiao et al., 2021), concave-over-convex composite penalties (Yashtini, 2022), infeasible and inexact block solves (Hu et al., 2022), Bregman and variable-metric geometries (Ahookhosh et al., 2019, Guo et al., 2023), spectral and inertial accelerations (Brandoni et al., 2021, Liu et al., 2020), and learned coupling operators in tensor–matrix recovery (Miranda, 11 Jul 2026).

1. Core formulation and penalty viewpoint

The basic PALM model is a block-structured objective

$$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$

on a finite-dimensional Hilbert product space $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$, with blockwise Lipschitz continuity of the partial gradients of $f$, global Lipschitzness of $\nabla f$ on bounded sets, prox-boundedness of $r$, and a lower bound on $\Psi$ (Davis, 2016). In this setting, the phrase “penalty-based” is not restricted to one specific penalty function. It includes hard constraints modeled by indicators $f$0, soft penalties placed directly in $f$1, and smooth quadratic penalties absorbed into the differentiable part $f$2 or its analogues.

Several later formulations make that viewpoint explicit. In lifted deep-network training, the equality constraints

$f$3

are replaced by a quadratic penalty objective

$f$4

which is then minimized by alternating proximal steps over $f$5, $f$6, and $f$7 (Qiao et al., 2021). In variable-metric CPALM, the nonsmooth term $f$8 itself is a composite penalty

$f$9

where $C^1$0 is strictly increasing, concave, and differentiable, and $C^1$1 is convex, proper, lower semicontinuous, and Lipschitz on its domain; the algorithm majorizes $C^1$2 through the affine upper tangent

$C^1$3

to obtain tractable PALM-type subproblems (Yashtini, 2022). In rank minimization over symmetric PSD matrices, complementarity is moved into the objective through the penalty term $C^1$4, yielding a penalized PALM structure with two semidefinite blocks (Shen et al., 2017).

Formulation	Penalty mechanism	Representative source
$C^1$5	Indicators or block penalties inside $C^1$6	(Davis, 2016)
Lifted deep-network objective $C^1$7	Quadratic penalties for layer consistency	(Qiao et al., 2021)
$C^1$8	Concave-over-convex nonconvex penalty	(Yashtini, 2022)
$C^1$9	Complementarity penalty in rank minimization	(Shen et al., 2017)

This suggests that “penalty-based PALM” is best understood as a structural design principle rather than a single algorithm: penalties can encode feasibility, low complexity, coupling, or stability, provided the resulting block models remain proximable or otherwise tractable.

2. Blockwise proximal-linearized mechanics

In the classical synchronous form, PALM updates one block at a time in cyclic order. For block $r_j$0, the update is

$r_j$1

where $r_j$2 uses the newest available values for blocks $r_j$3 and current values for $r_j$4, and the step size satisfies a block-Lipschitz bound such as

$r_j$5

(Davis, 2016). The linearization acts only on the smooth coupling term; the nonsmooth or constrained part remains in the proximal operator. This is the defining mechanic behind PALM’s ability to treat nonconvex smooth couplings together with nonsmooth penalties.

Subsequent work generalizes the proximal model in several directions. Variable-metric CPALM replaces the Euclidean quadratic by blockwise SPD metrics $r_j$6 and $r_j$7, so the proximal terms become $r_j$8 and $r_j$9, while the nonconvex composite penalty $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$0 is linearized in $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$1 and kept proximal in $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$2 (Yashtini, 2022). Two-step inertial Bregman PALM replaces the Euclidean proximity term by Bregman distances $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$3 and $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$4, and adds two inertial linear terms using both $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$5 and $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$6, as well as the analogous $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$7-terms (Guo et al., 2023). In the multi-block Bregman setting, the local model is

$$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$8

which yields BPALM and A-BPALM as direct Bregman analogues of PALM (Ahookhosh et al., 2019).

The literature also distinguishes between truly linearized block updates and exact block minimizations on penalized objectives. In iPDLAM for lifted deep-network training, the layerwise subproblems are exact minimizations of $$\Psi(x) := f(x) + r(x), \qquad r(x) := \sum_{j=1}^m r_j(x_j),$$9 plus proximal terms around inertially extrapolated points; the paper explicitly notes that these are exact block minimizations, not linearized gradients, and that the “linearization” in the abstract refers to the linear extrapolation of current and last iterates (Qiao et al., 2021). By contrast, the spectral PALM algorithm for dictionary learning remains within the proximal-linearized family but replaces conservative Lipschitz step sizes by Barzilai–Borwein spectral steps safeguarded by Armijo backtracking (Brandoni et al., 2021). PALMNUT introduces uncoupled coordinatewise step sizes derived from coordinatewise Lipschitz-like bounds, again preserving the PALM structure while altering the local quadratic models (Liu et al., 2020).

3. Algorithmic variants

The modern literature presents penalty-based PALM as a broad algorithmic family rather than a single scheme.

Variant	Defining feature	Representative source
Asynchronous PALM	Delayed block updates without synchronization	(Davis, 2016)
SAPALM	Asynchronous PALM with stochastic noise	(Davis et al., 2016)
SPRING	Stochastic PALM with variance reduction	(Driggs et al., 2020)
PALM-I	Infeasible, inexact block solves with residual control	(Hu et al., 2022)
iPDLAM	Inertial proximal alternating minimization on a penalty objective	(Qiao et al., 2021)
CPALM	Variable-metric PALM for composite penalties $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$0	(Yashtini, 2022)
BPALM / A-BPALM	Multi-block Bregman PALM with adaptive backtracking	(Ahookhosh et al., 2019)
Spectral PALM	Barzilai–Borwein and Armijo-enhanced PALM	(Brandoni et al., 2021)
LPAM	Smoothed, safeguarded, learned PALM-type scheme	(Chen et al., 2024)

Asynchronous variants preserve the blockwise proximal-gradient structure but remove synchronization barriers. In Asynchronous PALM, each core reads a possibly inconsistent delayed vector $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$1, computes a stale partial gradient, performs a proximal update on one block, and writes back without a global barrier; the deterministic and stochastic cases both guarantee that cluster points are stationary, and the deterministic case yields global convergence when a KL property holds for an associated Lyapunov function (Davis, 2016). SAPALM adds unbiased noise or stochastic gradient estimates to the asynchronous update and proves rates matching the best known synchronous or asynchronous rates on this problem class, while retaining nonconvex and nonsmooth block-separable regularizers (Davis et al., 2016).

Stochastic synchronous variants pursue finite-sum efficiency. SPRING replaces exact block gradients by variance-reduced estimators such as SAGA and SARAH inside a PALM iteration,

$$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$2

and derives non-asymptotic stationarity guarantees and KL-based convergence behavior analogous to deterministic PALM (Driggs et al., 2020).

Another major extension concerns subproblem solves. PALM-I allows each block subproblem to be solved inexactly and even infeasibly by a primal–dual, penalty, or ADMM-type inner method, provided a KKT-like residual $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$3 is controlled by a tolerance sequence $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$4. The central innovation is a surrogate sequence $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$5 that restores monotonicity despite infeasibility-induced objective oscillations (Hu et al., 2022).

Inertial and learned variants exploit the same block structure for acceleration or model-based network design. iPDLAM augments the lifted deep-network penalty formulation with inertial extrapolation, proximal layerwise minimizations, and a penalty warm-up schedule $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$6 (Qiao et al., 2021). LPAM smooths nonsmooth terms, modifies PALM using a residual-learning architecture, and uses block-coordinate descent iterates as a safeguard when the learned PALM-like step fails sufficient-decrease tests (Chen et al., 2024).

4. Convergence theory

The convergence theory of penalty-based PALM is built around stationarity, descent surrogates, and the Kurdyka–Łojasiewicz framework. In classical and asynchronous PALM settings, cluster points are critical or stationary points, typically expressed as

$$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$7

under block Lipschitz assumptions, prox-boundedness, lower boundedness, and bounded iterates (Davis, 2016). In deterministic asynchronous PALM, the objective $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$8 itself need not decrease because stale gradients introduce errors, so the analysis proceeds through a Lyapunov function $$\mathcal H=\mathcal H_1\times\cdots\times\mathcal H_m$$9 that augments the objective with quadratic terms in recent increments; when $f$0 has the KL property, the whole sequence converges and finite-length estimates follow (Davis, 2016).

Infeasible and inexact settings require a different device. PALM-I constructs a surrogate sequence

$f$1

that absorbs infeasibility errors and is provably nonincreasing, even though the original objective along the inexact iterates need not be monotone. Under square-summable residual tolerances, every accumulation point is stationary; under stronger summability and a KL property, the entire sequence converges to a single stationary point, with asymptotic rates depending jointly on the KL exponent and the decay of $f$2 (Hu et al., 2022).

Bregman and variable-metric PALM retain the same overall logic but replace Euclidean sufficient-decrease arguments by geometry-adapted ones. For BPALM and A-BPALM, one obtains subsequential convergence to critical points under boundedness, and global convergence under the KL inequality; when the objective satisfies the Łojasiewicz gradient inequality with exponent $f$3, the standard trichotomy appears again: finite convergence for $f$4, linear convergence for $f$5, and sublinear polynomial convergence for $f$6 (Ahookhosh et al., 2019). CPALM proves global convergence of every bounded generated sequence to a critical point by combining a sufficient decrease inequality, a subgradient bound, and the KL property for the composite objective (Yashtini, 2022).

Stochastic variants adapt the stationarity measure rather than abandoning it. SPRING defines a generalized gradient map and proves $f$7 decay in the squared stationarity measure under variance-reduced estimators, together with KL-exponent-based convergence rates that mirror deterministic PALM when the objective is semi-algebraic and the iterates are bounded (Driggs et al., 2020). Learned and smoothed variants weaken the target notion of stationarity. LPAM establishes that a subsequence of iterates has accumulation points and that every accumulation point is a Clarke stationary point of the original nonsmooth objective, with complexity governed by the smoothing schedule and the Lipschitz constants of the smoothed model (Chen et al., 2024).

A recurring structural theme is that semi-algebraicity is used as a practical sufficient condition for the KL property. This is stated explicitly in asynchronous PALM, CPALM, BPALM, low-complexity VAR estimation, and coupled tensor–matrix recovery, where the objectives are built from polynomial, indicator, quadratic, and nuclear-norm terms (Davis, 2016, Yashtini, 2022, Ahookhosh et al., 2019, Lin et al., 2016, Miranda, 11 Jul 2026).

5. Representative applications

Penalty-based PALM has been used across a wide range of structured nonconvex inverse and learning problems. In generalized low-rank models, asynchronous PALM applies directly to factor variables with smooth loss terms and potentially nonsmooth or nonconvex regularizers, including $f$8 penalties and nonnegative constraints; many such objectives are semi-algebraic, making them compatible with the deterministic KL-based convergence theory (Davis, 2016).

Deep learning provides a particularly explicit penalty-based instantiation. The lifted deep-network training problem introduces activation and pre-activation variables and enforces layer consistency through quadratic penalties. iPDLAM then alternates over $f$9, $\nabla f$0, and $\nabla f$1, with closed-form proximal-like operators for ReLU and squared loss and a warm-up schedule for the penalty parameter (Qiao et al., 2021). This same design pattern reappears in LPAM-net, where a smoothed nonsmooth penalty $\nabla f$2 is optimized by a PALM-like learned architecture with convergence to Clarke stationary points (Chen et al., 2024).

Matrix and tensor factorization problems are another major domain. SPRING studies sparse nonnegative matrix factorization, sparse PCA, and blind image deconvolution with blockwise proximal maps encoding nonnegativity, simplex-type constraints, $\nabla f$3 penalties, and nonconvex edge-preserving penalties (Driggs et al., 2020). Spectral PALM extends the same blockwise proximal-linearized logic to matrix dictionary learning and tensor-train dictionary learning, where constraints are handled by projections onto unit-norm, orthogonality, or sparsity sets (Brandoni et al., 2021). Multi-block Bregman PALM and A-BPALM are applied to orthogonal nonnegative matrix factorization, using a quartic orthogonality penalty and kernels chosen so that both block updates admit closed forms (Ahookhosh et al., 2019).

Several works emphasize penalty terms that encode domain-specific physics or structural equations. Low-complexity VAR estimation uses a Lyapunov penalty

$\nabla f$4

to incorporate steady-state covariance information, together with cardinality or rank constraints handled by proximal projections; PALM is shown to be globally convergent to a critical point, with monotone decrease of the estimation objective (Lin et al., 2016). A penalty method for rank minimization in symmetric PSD matrices relaxes semidefinite complementarity through $\nabla f$5 and applies PALM or Fast PALM to the penalized problem, combining semismooth Newton for the $\nabla f$6-subproblem with eigenvalue clipping for the $\nabla f$7-subproblem (Shen et al., 2017).

Imaging and MRI motivate several specialized variants. CPALM is demonstrated on parallel MRI reconstruction with log-sum and $\nabla f$8 penalties, treating them as concave-over-convex composite regularizers (Yashtini, 2022). PALMNUT addresses complex image reconstruction with separate magnitude and phase regularization by combining PALM, Nesterov momentum, and coordinatewise step sizes derived from coordinatewise Lipschitz-like bounds (Liu et al., 2020). Coupled tensor–matrix recovery formulates a ridge-stabilized penalized objective with nuclear norms on a tensor unfolding and a matrix, plus a quadratic coupling term

$\nabla f$9

and proves convergence of the implemented PALM scheme to a critical point (Miranda, 11 Jul 2026).

6. Limitations, design choices, and open directions

Penalty-based PALM inherits several recurring assumptions. Many convergence theorems require bounded iterates; a standard sufficient condition is coercivity of the penalized objective or boundedness of its level sets (Davis, 2016). Asynchronous theory additionally requires bounded delays $r$0 and, in deterministic settings, essential cyclicity or analogous block-visit conditions; if delays are too large, the admissible step sizes shrink through explicit terms such as $r$1 or $r$2 (Davis, 2016, Davis et al., 2016).

Another design tension lies between tractability and modeling richness. The strongest PALM results usually rely on exact or easily computable block proximal operators. Once a model combines several nonsmooth penalties on the same block, or requires nonseparable nonsmooth coupling terms, exact prox steps may disappear. PALM-I addresses this by allowing infeasible inexact subsolvers with residual-based stopping rules, but then convergence depends on summability conditions for the residual tolerances and on surrogate-sequence analysis rather than direct objective descent (Hu et al., 2022). LPAM resolves a similar difficulty by smoothing the nonsmooth terms and accepting only those learned PALM-like updates that satisfy explicit sufficient-decrease and gradient-bound conditions; otherwise it falls back to a safeguarded BCD step (Chen et al., 2024).

Penalty choice itself can strongly affect conditioning and empirical behavior. In deep-network training, the penalty parameter should be large enough to approximate the constrained model, but starting with a large penalty can severely slow optimization, which motivates the warm-up strategy $r$3 (Qiao et al., 2021). In coupled tensor–matrix recovery, coupling helps only once the block used to estimate the coupling operator is sufficiently sampled; the paper explicitly reports a low-density regime where coupling does not help and can even hurt because $r$4 is poorly estimated (Miranda, 11 Jul 2026).

Open directions recur across the literature. Asynchronous PALM highlights extensions to nonseparable nonsmooth terms, weakening bounded-delay assumptions, and reducing reliance on fixed Lipschitz constants or conservative step rules (Davis, 2016). Inexact PALM suggests adapting the surrogate-sequence and residual-based methodology to broader penalty-based formulations and richer inner solvers (Hu et al., 2022). Learned PALM-type methods point toward broader function classes than the semi-algebraic or smoothed settings currently analyzed, but the strongest guarantees remain subsequential or Clarke-stationary rather than full-sequence global convergence (Chen et al., 2024). Taken together, these works indicate that penalty-based PALM is not a closed theory but an extensible framework whose success depends on preserving three ingredients simultaneously: blockwise majorization of the smooth coupling, tractable proximal or surrogate block solves, and a descent mechanism strong enough to support KL-type convergence analysis.