--- title: Stochastic Two-Step Inertial Bregman PALM url: https://www.emergentmind.com/topics/stochastic-two-step-inertial-bregman-palm-stibpalm type: topic --- # Stochastic Two-Step Inertial Bregman PALM The Stochastic Two-step Inertial Bregman Proximal Alternating Linearized Minimization (STiBPALM) algorithm is an optimization method designed for large-scale, composite, nonconvex, and nonsmooth finite-sum problems. STiBPALM integrates two-step inertial acceleration, Bregman distance regularization, proximal alternating minimization, and variance-reduced stochastic gradient estimation within a principled framework that yields strong global convergence guarantees under broadly verifiable conditions, particularly in the presence of nonconvexity and nonsmooth regularizers [2307.05287]. ## 1. Formulation of the Optimization Problem STiBPALM addresses optimization problems of the form: \[ \min_{x\in\R^l,\;y\in\R^m}~\Phi(x, y) = f(x) + \frac{1}{n}\sum_{i=1}^n H_i(x, y) + g(y), \] where \(f:\R^l\to(-\infty,+\infty]\) and \(g:\R^m\to(-\infty,+\infty]\) are proper, lower semicontinuous (possibly nonconvex) regularizers, and each \(H_i:\R^l\times\R^m\to\R\) is continuously differentiable with Lipschitz partial gradients \(\nabla_x H_i\) and \(\nabla_y H_i\) on bounded sets. This composite structure encompasses a variety of practical models, wherein \(f\) and \(g\) may encode constraints such as indicator functions or promote sparsity (e.g., via \(\ell_0\)-constraints). ## 2. Bregman Distance Regularization The algorithm employs Bregman distances to stabilize the updates and handle generalized proximity. For a strongly convex, Gâteaux-differentiable function \(\phi:\R^d\to(-\infty,+\infty]\), the Bregman distance is defined as \[ D_\phi(u, v) = \phi(u) - \phi(v) - \langle \nabla\phi(v), u - v \rangle, \] guaranteeing that \(D_\phi(u,v)\geq \frac{\theta}{2}\|u-v\|^2\) for some \(\theta>0\). Two distinct generators \(\phi_1\) and \(\phi_2\), each with strongly convex and Lipschitz gradient properties, are utilized for the \(x\)- and \(y\)-subproblems, respectively. ## 3. Two-step Inertial Extrapolation The core iterative procedure at each step incorporates two-step inertial extrapolation to potentially accelerate convergence. For iterates \((x_k, y_k)\), the inertial variables are constructed as \[ \begin{aligned} u_k &= x_k + \gamma_{1k}(x_k - x_{k-1}) + \gamma_{2k}(x_{k-1} - x_{k-2}), \\ v_k &= y_k + \mu_{1k}(y_k - y_{k-1}) + \mu_{2k}(y_{k-1} - y_{k-2}), \end{aligned} \] where the inertial parameters \(\gamma_{1k}, \gamma_{2k}, \mu_{1k}, \mu_{2k}\) are nonnegative and bounded above by a constant \(\bar{\gamma}\) to ensure descent properties. This two-step extension generalizes the conventional "heavy-ball" and Nesterov-type inertial extrapolations. ## 4. Proximal Alternating Linearization and Variance Reduction Given the extrapolated points \((u_k, v_k)\), the method alternates two prox-linear Bregman steps for \(x\) and \(y\), replacing the full gradient of the finite-sum nonlinearity with a variance-reduced stochastic estimator: \[ \begin{cases} x_{k+1} \in \arg\min_{x\in\R^l} \left\{ f(x) + \langle \widetilde\nabla_x(u_k, y_k), x \rangle + D_{\phi_1}(x, x_k) + \alpha_{1k}\langle x, x_{k-1} - x_k \rangle + \alpha_{2k}\langle x, x_{k-2} - x_{k-1} \rangle \right\}, \\ y_{k+1} \in \arg\min_{y\in\R^m} \left\{ g(y) + \langle \widetilde\nabla_y(x_{k+1}, v_k), y \rangle + D_{\phi_2}(y, y_k) + \beta_{1k}\langle y, y_{k-1} - y_k \rangle + \beta_{2k}\langle y, y_{k-2} - y_{k-1} \rangle \right\}, \end{cases} \] where \(\alpha_{1k}, \alpha_{2k}, \beta_{1k}, \beta_{2k}\in[0,\bar{\alpha}]\). The stochastic gradients \(\widetilde\nabla_x, \widetilde\nabla_y\) are obtained using variance-reduced techniques, specifically SAGA and SARAH. ### SAGA SAGA utilizes "memory" points per datum for precise variance reduction: \[ \widetilde\nabla_x^{\mathrm{SAGA}}(u_k, y_k) = \frac{1}{b}\sum_{i\in B_k}\left[ \nabla_x H_i(u_k, y_k) - \nabla_x H_i(\varphi_k^i, y_k) \right] + \frac{1}{n}\sum_{j=1}^n \nabla_x H_j(\varphi_k^j, y_k), \] with appropriate updates to \(\varphi_{k+1}^i\) for sampled indices. ### SARAH SARAH combines periodic full-gradient refresh with recursive updates: \[ \widetilde\nabla_x^{\mathrm{SARAH}}(u_k, y_k) = \frac{1}{b}\sum_{i\in B_k}\left[ \nabla_x H_i(u_k, y_k) - \nabla_x H_i(u_{k-1}, y_{k-1}) \right] + \widetilde\nabla_x^{\mathrm{SARAH}}(u_{k-1}, y_{k-1}), \] where a full-gradient is recomputed with a specified probability. Both variants enjoy explicit mean-square error and decay bounds, ensuring the reliability of the stochastic approximations. ## 5. Convergence Properties Under the joint assumptions of Lipschitz continuity (for gradients of \(H_i\)), strong convexity of the Bregman generators (\(\phi_1\), \(\phi_2\)), and the Kurdyka–Łojasiewicz (KL) property for \(\Phi\), STiBPALM guarantees global convergence in expectation. Specifically: - The sequence \(\{z_k\}\) satisfies \(\sum_k \EE \|z_{k+1} - z_k\|^2 < \infty\), and \(\EE \Phi(z_k) \to \Phi^*\). - All limit points are critical, characterized by \(\EE \text{dist}(0, \partial\Phi(z_k)) \to 0\). - If \(\Phi\) has KL exponent \(\theta \in [0,1)\), the following convergence rates hold: - \(\theta = 0\): finite identification in expectation, - \(0 < \theta \leq 1/2\): linear rate, - \(1/2 < \theta < 1\): sublinear rate. Step sizes and inertial parameters must be chosen such that a global-descent constant emerges in the basic inequality governing the iterates [2307.05287]. ## 6. Computational Aspects Each iteration of STiBPALM consists of two main components: - Computing two stochastic gradient estimates, each incurring \(O(b)\) partial-gradient computations (with \(b\) denoting the minibatch size). - Solving proximal–Bregman subproblems for \(x\) and \(y\). These subproblems often admit efficient, sometimes closed-form, solutions in settings with indicator or \(\ell_1\)-regularization. Thus, the dominant per-iteration cost is \(O(b+\operatorname{prox}(f)+\operatorname{prox}(g))\), typically scaling as \(O(b{\cdot}\max\dim H_i)\) in practical scenarios. ## 7. Empirical Evaluation Numerical experiments were conducted on: - Sparse nonnegative matrix factorization (S-NMF) using face datasets (Extended Yale-B, ORL). - Blind image deblurring tasks, encompassing both motion blur and defocus blur on standard “Kodim” images. STiBPALM using variance-reduced gradients (SAGA, SARAH) demonstrated superior performance compared to both deterministic (PALM, iPALM, TiPALM) and prior stochastic methods (SPRING, SiPALM). In all settings, STiBPALM-SAGA and STiBPALM-SARAH achieved greater objective decrease per epoch and wall-clock time, with faster and better convergence of recovered basis images and deblurred kernels [2307.05287]. --- STiBPALM systematically unifies two-step inertial methods, Bregman regularization, and stochastic variance reduction for nonconvex and nonsmooth optimization, offering robust convergence theory and empirically superior performance on challenging large-scale signal processing and machine learning tasks. Source: https://www.emergentmind.com/topics/stochastic-two-step-inertial-bregman-palm-stibpalm