Multi-Block Splitting Alternating Algorithm

• Multi-Block Splitting Alternating Algorithm (MSAA) is a class of iterative methods that partitions high-dimensional variables into blocks and updates them alternately to solve complex problems.
• It employs block decomposition, proximal regularization, and linearization techniques to handle diverse models including convex optimization, nonconvex minimization, and sparse recovery.
• MSAA demonstrates practical benefits with scalable convergence rates, efficient parallel updates, and applicability in areas such as signal processing, machine learning, and deep neural network quantization.

Multi-Block Splitting Alternating Algorithm (MSAA) denotes a class of block-structured iterative methods in which a high-dimensional variable is partitioned into multiple blocks and the resulting subproblems are updated alternately, partially in parallel, or through auxiliary splitting variables. In recent arXiv literature, the label appears in several closely related but not identical senses: as general multi-block alternating minimization, as a multi-splitting alternating algorithm embedded in linearized generalized ADMM, and as a sparse-recovery algorithm for linear systems with concatenated orthogonal matrices. This suggests that MSAA is best understood as a methodological family centered on multi-block decomposition, alternating updates, and structure-exploiting subproblems rather than as a single canonical iteration (Tupitsa et al., 2019, Jian et al., 2022, Zhao et al., 29 Sep 2025).

1. Terminological range

A recurrent source of ambiguity is that the acronym is not used uniformly across the literature. In "Alternating Minimization Methods for Strongly Convex Optimization," MSAA refers to the general multi-block alternating minimization procedure in which, at each iteration, every block is minimized while the other blocks are held fixed. In the multi-block L-GADMM work, the paper itself mostly uses the language of "partially parallel splitting," but the summary explicitly identifies the method as a "MSAA" or "multi-splitting alternating algorithm" because the first $m-1$ subproblems are split and solved in parallel before the last block and multiplier are updated. In "Splitting Alternating Algorithms for Sparse Solutions of Linear Systems with Concatenated Orthogonal Matrices," MSAA is a named algorithm class contrasted with TSAA, the two-block splitting alternating algorithm (Tupitsa et al., 2019, Jian et al., 2022, Zhao et al., 29 Sep 2025).

Despite this variation, the shared structural idea is consistent. The variable $x$ is written as a block tuple, the objective or feasibility model is decomposed into blockwise components, and each iteration combines local minimization, proximal correction, projection, or thresholding steps with a coupling update. The coupling may arise from linear constraints, a smooth cross-term, an augmented-Lagrangian penalty, or a measurement equation.

2. Optimization models and block structure

The model classes associated with MSAA-type methods span separable convex programming, blockwise smooth composite minimization, structured nonconvex optimization, and sparse recovery.

Setting	Representative formulation	Source
Linearly constrained separable convex optimization	$$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$	(Jian et al., 2022, Chen et al., 2016, Lin et al., 2013)
Smooth composite multi-block minimization	$$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$	(Tupitsa et al., 2019)
Structured nonsmooth nonconvex minimization	$$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$	(Ahookhosh et al., 2019)
Sparse linear systems with concatenated orthogonal matrices	$\min \{ \|x\|_0 : y=Ax \}$, with $A=[\Phi_1,\Phi_2,\dots,\Phi_p]$ and each $\Phi_i$ orthogonal	(Zhao et al., 29 Sep 2025)

Across these formulations, block separability serves different purposes. In linearly constrained convex programs, it converts a coupled constrained problem into augmented-Lagrangian or primal-dual subproblems that are either blockwise proximal or fully decoupled. In smooth composite minimization, it supports Gauss-Seidel block minimization under blockwise Lipschitz and PL or strong-convexity assumptions. In the Bregman-proximal nonconvex setting, it enables relative smoothness to be imposed one block at a time. In sparse recovery, the block partition matches the concatenated orthogonal structure of the sensing matrix, so that each subsystem is much smaller than the original system.

3. Iterative mechanisms

The prototypical many-block alternating minimization update has the form

$$x_i^{k+1}=\arg\min_{z\in Q_i} f(x_1^{k+1},\ldots,x_{i-1}^{k+1},z,x_{i+1}^k,\ldots,x_n^k)+g_i(z),$$

executed for $i=1,2,\ldots,n$. This is the exact multi-block alternating minimization scheme analyzed under blockwise smoothness and PL-type conditions, and it reduces to classical block coordinate minimization when $x$0 (Tupitsa et al., 2019).

A second major architecture uses linearization and proximal regularization. In the multi-block L-GADMM construction, the variables are grouped into $x$1 blocks $x$2 and the $x$3-th block $x$4; the first $x$5 blocks are updated in parallel, each subproblem includes a proximal term $x$6, then $x$7 is updated with the newest values, and finally the multiplier is updated. The PDFP extension likewise produces fully decoupled multi-block schemes in which the dual blocks $x$8 can be computed in parallel, while LADMPSAP linearizes each block’s quadratic penalty term so that all block updates become proximal mappings and the penalty parameter is updated adaptively (Jian et al., 2022, Chen et al., 2016, Lin et al., 2013).

Nonconvex MSAA variants replace Euclidean proximalization by Bregman geometry or stochastic splitting. BPALM updates each block by minimizing a linearized model plus a blockwise Bregman distance, with $x$9, and A-BPALM adds blockwise backtracking. STAM, designed for a non-convex minimization model with three blocks, alternates a stochastic $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$0-update, a proximal $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$1-update, a proximal/minimization step in $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$2, and a Douglas-Rachford-style variable update $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$3 (Ahookhosh et al., 2019, Bian et al., 2022).

The sparse-recovery MSAA of the concatenated-orthogonal setting is structurally different but still follows the same alternating logic. Each cycle begins with blockwise hard thresholding

$$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$4

forms a support set

$$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$5

solves a least-squares problem restricted to $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$6, then applies a second support-restricted least-squares step on the $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$7-largest entries. The main computations are matrix-vector products, reduced orthogonal projections, thresholding, and least-squares problems with at most $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$8 and then $$\min \left\{ \sum_{i=1}^m \vartheta_i(x_i)\ \big|\ \sum_{i=1}^m \mathcal{A}_i x_i = b,\ x_i \in \mathcal{X}_i \right\}$$9 variables (Zhao et al., 29 Sep 2025).

4. Convergence, rates, and conditions

Theoretical guarantees for MSAA-type methods are diverse and assumption-sensitive. For the multi-block L-GADMM scheme, the sequence $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$0 generated by Algorithm 1 converges to an optimal solution $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$1 from any starting point under convexity, closedness of $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$2, feasibility, and proper proximal parameters. The same work establishes worst-case $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$3 convergence rates in both ergodic and nonergodic senses. The PDFP extension guarantees convergence under mild and explicit step-size conditions for unconstrained and linearly constrained multi-block problems, and if some subfunctional is strongly convex, PDFP achieves linear convergence. LADMPSAP proves global convergence under conditions on $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$4 and $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$5, derives an ergodic $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$6 rate, and identifies $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$7 as both necessary and sufficient under bounded subgradients. The MLADMM for sparse fused Lasso is stated to converge to an optimal solution without requiring uniqueness and to satisfy a non-ergodic convergence rate of $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$8 for successive iterate differences (Jian et al., 2022, Chen et al., 2016, Lin et al., 2013, Wu et al., 2023).

For exact multi-block alternating minimization, the central condition is blockwise curvature. If each block-gradient is $$\min_{x=(x_1,\ldots,x_n)\in Q_1\times\cdots\times Q_n} f(x_1,\ldots,x_n)+\sum_{i=1}^n g_i(x_i)$$9-Lipschitz and the proximal PL condition holds in every block, then

$$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$0

so the basic MSAA is linearly convergent. The accelerated alternating minimization method attains

$$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$1

which yields $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$2 in the convex case and an accelerated linear rate in the strongly convex case (Tupitsa et al., 2019).

For nonconvex variants, the guarantees are formulated in terms of stationarity and KL geometry. BPALM generates sequences whose cluster points are critical points, and under the KL property the full sequence converges with finite, linear, or sublinear rates according to the Łojasiewicz exponent. STAM obtains an $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$3-stationary point with convergence rate $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$4. In the sparse linear-system setting, the TSAA/MSAA paper proves global convergence to the sparse solution under a coherence-type condition linking the sparsity level to the mutual coherence of the matrix (Ahookhosh et al., 2019, Bian et al., 2022, Zhao et al., 29 Sep 2025).

5. Relation to ADMM and operator-splitting theory

A recurrent misconception is that the two-block ADMM can be extended to many blocks without altering the convergence picture. Several papers state the opposite. For more than two blocks, direct extensions of ADMM are not necessarily convergent; naive multi-block ADMM may diverge; and cyclic ADMM with fixed order is not guaranteed to converge for $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$5 (Chen et al., 2016, Lin et al., 2013, Mihic et al., 2019).

Much of the modern MSAA literature can be read as a response to that difficulty. One response is algorithmic regularization: linearizing quadratic penalty terms, adding proximal terms, using partially parallel or fully parallel block splits, or introducing adaptive penalties. Another response is operator-splitting reinterpretation. The modified 3-block ADMM for convex quadratic semidefinite programming is shown to be a realization of the Davis-Yin three-operator splitting, which explains why the algorithm can skip the minimization over the quadratic block and set $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$6. A later three-operator splitting derived from three-block ADMM proves that when two functions have orthogonal domains, the splitting operator is $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$7-averaged and the method converges for any positive step size, while numerical tests suggest robustness to larger step sizes than Davis-Yin or vanilla three-block ADMM (Chang et al., 2018, Anshika et al., 2024).

Randomization provides a third line of development. Randomly permuted ADMM converges in expectation for linear systems and convex quadratic programs, and RAC-ADMM adds randomness in the assembly of the blocks themselves. The paper gives a criterion for almost sure convergence based on the spectral radius condition

$$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$8

This suggests that, within the broader MSAA landscape, block order and block composition are algorithmic degrees of freedom rather than merely implementation details (Mihic et al., 2019).

6. Applications and empirical behavior

The practical impact of MSAA-type methods is tied to the structure of the underlying problem.

• Calibration of correlation matrices: The multi-block L-GADMM paper applies Algorithm 1 to $$\min_{x=(x_1,\ldots,x_N)} \varphi(x):=f(x)+\sum_{i=1}^N g_i(x_i)$$9, splits the problem into three blocks, and reports that L-GADMM outperforms both TADMM and ADMM-G in iteration count and CPU time, especially for large $\min \{ \|x\|_0 : y=Ax \}$0 (Jian et al., 2022).
• Signal processing and imaging science: The PDFP extension targets separable multi-block minimization arising in signal processing and imaging science, including TV regularized CT reconstruction. The paper states that PDFP converges reliably, that some variants achieve faster convergence and higher PSNR, and that the parallel schemes provide substantial speedup and scalability to larger problems (Chen et al., 2016).
• Machine learning convex programs: LADMPSAP is presented for multi-block separable convex programs in machine learning, including sparse representation, low-rank recovery, matrix completion with non-negativity, and overlapping group lasso. The method is described as highly parallelizable, suitable for parallel or distributed computing, and particularly suitable when the subproblems have closed-form proximal solutions and the sparsity and low-rankness of the iterates can be preserved during the iteration (Lin et al., 2013).
• Orthogonal nonnegative matrix factorization: BPALM and A-BPALM are applied to ONMF through a penalized formulation of $\min \{ \|x\|_0 : y=Ax \}$1 with $\min \{ \|x\|_0 : y=Ax \}$2, $\min \{ \|x\|_0 : y=Ax \}$3, and a penalty on $\min \{ \|x\|_0 : y=Ax \}$4. The paper reports that the related subproblems are solved in closed forms and that A-BPALM outperforms BPALM in terms of function value and constraint satisfaction (Ahookhosh et al., 2019).
• Sparse fused Lasso regression and classification: The MLADMM paper develops a unified multi-block linearized ADMM for regression and classification models with sparse and blocky coefficients. It states that the method has lower time complexity than existing methods, can be extended to various existing fused Lasso models, and has per-iteration cost $\min \{ \|x\|_0 : y=Ax \}$5 after a one-time $\min \{ \|x\|_0 : y=Ax \}$6 preprocessing step (Wu et al., 2023).
• Quantized deep neural networks: STAM is applied to relaxed binary quantization for VGG-11, VGG-16, and ResNet-18 on CIFAR-10 and CIFAR-100. The paper states that the test accuracy indicates the effectiveness of STAM, and the summary reports that for CIFAR-10/VGG-11, STAM ("Our3") achieves test accuracy 90.71%, compared to BC at 88.59% and BR at 89.24% (Bian et al., 2022).
• Sparse linear systems with concatenated orthogonal matrices: The explicit TSAA/MSAA framework only involves matrix-vector products and reduced orthogonal projections, is globally convergent under a coherence-type condition, and numerical experiments indicate that the algorithms can quickly and accurately locate the sparse solution with significantly fewer iterations than several mainstream iterative methods (Zhao et al., 29 Sep 2025).

Taken together, these results portray MSAA not as a single fixed recipe but as a broad design principle for multi-block optimization and recovery. The unifying theme is the replacement of a monolithic problem by a sequence of smaller blockwise tasks whose solvability, parallelism, and convergence can be tailored to convex, nonconvex, stochastic, or structured sparse settings.