TiBPALM: Two-step Inertial Bregman PALM

• The paper introduces a novel two-step inertial scheme combined with Bregman proximal terms to accelerate convergence in nonconvex, nonsmooth optimization problems.
• It integrates advanced momentum extrapolation and stochastic variance-reduced techniques like SAGA and SARAH, yielding significant reductions in iterations and computation time.
• Convergence is rigorously established through the Kurdyka–Łojasiewicz framework and Lyapunov analysis, ensuring stability and robust performance in large-scale applications.

The two-step inertial Bregman proximal alternating linearized minimization (TiBPALM) algorithm is an advanced block-structured framework for nonconvex and nonsmooth optimization. It synthesizes concepts from proximal alternating linearized minimization (PALM), two-step inertial methods, and Bregman distance regularization. TiBPALM is particularly effective for large-scale problems with composite objective structure and has been rigorously analyzed both in deterministic and stochastic, variance-reduced settings (Guo et al., 2023, Guo et al., 2023).

1. Problem Formulation and Algorithmic Structure

TiBPALM targets optimization problems with the general form

$$\min_{x \in \mathbb{R}^\ell,\, y \in \mathbb{R}^m} \ \Phi(x,y) := f(x) + H(x,y) + g(y)$$

where $f$ and $g$ are proper, lower-semicontinuous (lsc), possibly nonconvex, and $H$ admits a finite-sum structure,

$$H(x, y) = \frac{1}{n} \sum_{i=1}^n H_i(x, y), \quad H_i \in C^1,$$

with Lipschitz continuous gradients on bounded sets. The algorithm alternates between block updates for $x$ and $y$, each incorporating two-step inertial extrapolation and Bregman-proximal terms.

At iteration $k$, momentum-modified extrapolations are constructed as:

• $$u_k = x_k + \gamma_{1k}(x_k - x_{k-1}) + \gamma_{2k}(x_{k-1} - x_{k-2})$$
• $$v_k = y_k + \mu_{1k}(y_k - y_{k-1}) + \mu_{2k}(y_{k-1} - y_{k-2})$$

The updates are then:

$f$0

where $f$1 is the Bregman distance generated by a strongly convex kernel $f$2.

2. Bregman Distance and Proximal Structure

Bregman distance, $f$3, generalizes Euclidean proximity to arbitrary convex, differentiable kernels and improves adaptation to specific problem geometry. When $f$4 is $f$5-strongly convex, $f$6. Bregman proximal operators with nonquadratic $f$7 enable the use of Kullback–Leibler, Itakura–Saito, or other divergences, improving modeling flexibility and empirical performance in applications such as sparse recovery and matrix factorization.

3. Two-Step Inertial Extrapolation

Two-step inertia incorporates momentum from the past two iterates, contrasting with one-step inertial schemes (e.g., iPALM). At each iteration, updates depend on four consecutive iterates, leveraging momentum to accelerate convergence, especially in complex nonconvex landscapes. The inertial parameters $f$8 are bounded suitably to guarantee sufficient decrease and stability. Proper inertia scaling is critical: $f$9 with $g$0 determined by the strong convexity of $g$1 and Lipschitz continuity constants.

4. Stochastic and Variance-Reduced Extensions

The stochastic TiBPALM (STiBPALM) framework replaces full gradients with stochastic, variance-reduced estimators, essential for scalability in the finite-sum setting. Supported estimators include:

• SAGA: Utilizes memory of previous gradients and combines current stochastic estimates with average historical gradient information, delivering unbiased variance reduction.
• SARAH: Employs recursive gradient updates interleaved with periodic full-gradient computations, achieving low variance and reduced stochastic noise.

For both methods, the variance reduction properties are formalized by conditional expectation bounds on mean squared error sequences, ensuring their vanishing as the iterates converge (Guo et al., 2023). Parameter tuning for step sizes and curvature is derived from interaction between Bregman strong convexity and gradient estimator variance.

5. Convergence Analysis

The TiBPALM convergence proofs leverage the Kurdyka–Łojasiewicz (KL) property and a tailored Lyapunov (benefit) function combining objective value, recent iterate differences, and, in the stochastic case, variance error terms. The main theoretical results are:

• For KL-exponent $g$2, the sequence has finite length and converges to a critical point;
• For $g$3, linear (geometric) decay of expected norm distance to the limit;
• For $g$4, sublinear rate $g$5;
• For $g$6, finite-time identification.

A summary of convergence criteria:

Condition	Effect on TiBPALM Sequence	Convergence Rate
KL-exponent $g$7	Finite-time identification	Immediate
$g$8	Linear (geometric) convergence	Exponential decay
$g$9	Sublinear convergence	$H$0
Strong convexity, Lipschitz	Sufficient decrease guaranteed	--

The Lyapunov function captures energy decay and is bounded above by previous iterates, leading to summability of $H$1 and almost-sure convergence in expectation.

6. Empirical Performance and Applications

TiBPALM and STiBPALM have been experimentally validated on sparse signal recovery, quadratic fractional programming, sparse nonnegative matrix factorization (NMF), and blind image deblurring (Guo et al., 2023, Guo et al., 2023). In sparse recovery with $H$2 regularization, TiBPALM achieves approximately 30–40% fewer iterations and 30–35% less CPU time than one-step inertial methods, and >50% improvement over no-inertia baselines. In quadratic fractional problems, two-step inertia combined with Bregman kernels regularly outperforms one-step schemes and alternatives across problem dimensions.

Empirically, the combination of two-step inertia, Bregman proximity, and variance reduction yields faster practical convergence than one-step inertial Bregman schemes (iBPALM), PALM, and even advanced non-inertial schemes such as SPRING. Variance-reduced STiBPALM (SAGA, SARAH) consistently outperforms stochastic gradient descent in both speed and robustness.

7. Implementation Considerations and Parameter Tuning

Effective implementation requires storing the last four iterates for inertia, gradient memory for SAGA, and periodic full gradient computation for SARAH. Step sizes are chosen proportional to the inverse curvature of the kernel $H$3. For Euclidean $H$4, the proximal weight is $H$5. Success depends on satisfying theoretical step-size and inertia constraints ensuring sufficient descent and Lyapunov decrease. Bregman kernels should be selected based on target geometry (Euclidean, KL, Itakura–Saito) to maximize empirical efficiency. Stopping criteria typically enforce small combined updates in $H$6 and $H$7 ($H$8 in reported experiments).

• (Guo et al., 2023) "Two-step inertial Bregman proximal alternating linearized minimization algorithm for nonconvex and nonsmooth problems"
• (Guo et al., 2023) "A stochastic two-step inertial Bregman proximal alternating linearized minimization algorithm for nonconvex and nonsmooth problems"