---
title: Block-Coordinate MM Optimization
url: https://www.emergentmind.com/topics/block-coordinate-mm
type: topic
---

# Block-Coordinate MM Optimization

Block-Coordinate Majorization–Minimization (Block-Coordinate MM) is a principled optimization framework for decomposing high-dimensional, often nonconvex or nonsmooth problems into subproblems along blocks of variables. Each block is updated by minimizing a majorizing surrogate function, ensuring global descent of the original objective via blockwise upper bounds. This approach unites and generalizes classical coordinate descent, expectation-maximization, proximal gradient, and alternating minimization strategies, and extends efficacy to settings with manifold constraints, overlapping variables, and large-scale structure. Block-Coordinate MM methods are foundational in statistical learning, signal processing, matrix/tensor factorization, and large-scale distributed optimization.

## 1. Core Principles and Algorithmic Structure

Block-Coordinate MM iteratively partitions the optimization variable $x=(x_1,\ldots,x_B)$ into blocks, constructing for each block $i$ a surrogate $M_i(x_i; x_{-i})$ (or $u_i(x_i;z)$) satisfying:

- **Majorization:** $M_i(x_i; x_{-i}) \geq f(x_i, x_{-i})$ for all $x_i$.
- **Tangency:** $M_i(x^k_i; x_{-i}^k) = f(x^k)$.
- **First-order matching:** The directional derivatives of the surrogate and $f$ are aligned at the current iterate.

At each iteration, block(s) are selected—cyclically, in parallel, or randomly—and updated via
$$
x_i^{k+1} = \arg\min_{x_i \in \mathcal{X}_i} M_i(x_i; x_{-i}^k),
$$
with $x_j^{k+1} = x_j^k$ for $j \neq i$ [1511.02746, 2312.10330]. This update guarantees $f(x^{k+1}) \leq f(x^k)$.

Block surrogates can be exact restrictions, local linearizations, quadratic (proximal) upper bounds, or structured composite approximations. The block MM scheme yields embedded descent and decomposability by leveraging problem structure—convexity, separability, or blockwise manifold constraints.

## 2. Majorization Properties and Surrogate Constructions

The surrogate construction is central to both theoretical guarantees and practical efficiency. Required surrogate properties (see [1310.6957, 1511.02746]) include:

- **Tangency (Tightness):** $u_i(x_i^k; x^k) = f(x^k)$.
- **Majorization (Upper-Bound):** $u_i(x_i; x^k) \geq f(x_1^k,\dots,x_{i-1}^k,x_i,x_{i+1}^k,\dots,x_B^k)$ for all $x_i$.
- **First-order consistency:** $\nabla_{x_i} u_i(x_i^k; x^k) = \nabla_{x_i} f(x^k)$.
- **(Optional) Strong convexity and Lipschitz continuity** for quantitative convergence analysis.

Common surrogates include:
- **Exact blockwise minimization:** $u_i(x_i; x^k) = f(x_1^k,\dots,x_{i-1}^k,x_i,x_{i+1}^k,\dots)$. 
- **Quadratic/proximal bounds:** $u_i(x_i; x^k) = f(x^k) + \langle \nabla_{x_i} f(x^k), x_i - x_i^k \rangle + \frac{\gamma}{2} \|x_i - x_i^k\|^2$.
- **Composite surrogates:** For $f=g+h$ with $g$ smooth, $u_i(x_i;z) = g(z) + \langle \nabla_i g(z), x_i - z_i \rangle + \frac{1}{2}(x_i - z_i)^T H_i (x_i - z_i) + h(x_i, z_{-i})$ [1511.02746, 1310.6957].

Manifold-constrained problems require that surrogates respect tangent space structure and retraction maps, as developed for Grassmann and Stiefel blocks [2402.09907, 2312.10330, 1903.00597].

## 3. Convergence Analysis and Complexity

Block-Coordinate MM methods guarantee monotonic decrease of the objective. Under mild and explicit conditions—surrogate quasi-convexity, continuity, unique minimizers in all but one block, and boundedness of initial level sets—every limit point is stationary [1310.6957, 1511.02746, 2312.10330]. Convergence rates are summarized as follows:

| Setting                                 | Rate          | Reference        |
|------------------------------------------|---------------|------------------|
| General convex, strongly convex surrogates| $\mathcal{O}(1/\epsilon)$ | [1511.02746, 1310.6957] |
| Block minimization (no strong convexity)  | $\mathcal{O}(1/r)$       | [1310.6957]        |
| Two-block acceleration                   | $\mathcal{O}(1/r^2)$     | [1310.6957]        |
| Nonconvex (Euclidean, with proximal/trust region) | $\widetilde{O}(1/\epsilon^2)$ | [2012.03503, 2312.10330] |
| Riemannian manifold blocks               | $\widetilde{O}(1/\epsilon^2)$ | [2312.10330]      |

The method in [2012.03503] introduces a trust-region variant (BMM-DR), improving complexity for "flat" surrogates and weak convexity by controlling the step's radius.

For block-separable convex programs with linear constraints, specialized block-coordinate MM strategies such as the Parallel Direction Method of Multipliers (PDMM) exhibit global $O(1/T)$ convergence in ergodic averages under minimal assumptions, even allowing randomized and overlapping block updates [1406.4064].

## 4. Generalizations: Riemannian and Structured Settings

Block MM extends naturally to variables living on Riemannian manifolds (e.g., Grassmann or Stiefel), dropping the requirement for Euclidean convexity [2402.09907, 2312.10330, 1903.00597]. In these settings:

- The block update projects onto manifold subspaces (using exponential or retraction maps).
- Monotonic descent persists; under geodesic $L$-smoothness, compact feasibility, and suitable surrogates, convergence to stationary points is guaranteed.
- The blockwise stationarity condition is formulated in terms of Riemannian gradients in the corresponding product tangent spaces.

Block MM has been successfully applied to Burer–Monteiro formulations of large-scale semidefinite programs, robust PCA on fixed-rank manifolds, and tensor dictionary learning, with rigorous complexity guarantees. The algorithm in [1903.00597] achieves global convergence to first-order critical points for block minimization of Stiefel-constrained variables in SDPs.

## 5. Representative Applications

Block-Coordinate MM encompasses a broad spectrum of classic and modern algorithms:

- **Expectation-Maximization (EM):** Single-block MM via Jensen-based surrogates for the log-likelihood, leading to alternate E- and M-steps [1511.02746, 2012.03503].
- **Convex-Concave Procedure (CCCP):** Decomposing a nonconvex objective as a difference of convex functions, updating via blockwise linearization [1511.02746].
- **Tensor CP Decomposition:** Proximal alternating least squares as block MM often with superior convergence to naive ALS [1511.02746].
- **Nonnegative Matrix/Tensor Factorization:** Lee–Seung multiplicative update algorithms are block MM with structured surrogates and possible proximal stabilization [2012.03503].
- **Overlapping Group Lasso and Composite Penalized Models:** Consensus constraints are accommodated via block copies; block MM is applied to alternating minimization and splitting methods [1406.4064].
- **Robust PCA:** Low-rank manifold block MM yields provable stationary convergence [2312.10330].
- **Large-scale SDPs:** BCM on product Stiefel manifolds provides scalable and provably convergent alternatives to interior-point methods [1903.00597].

## 6. Block-Selection Rules, Parallelism, and Practical Issues

Block MM admits flexibility in block-selection:

- Cyclic, essentially cyclic, or randomized block update schedules [1511.02746, 1310.6957].
- Parallel and asynchronous updates of multiple blocks are allowed, enhancing scalability [1406.4064].
- Non-uniform stochastic block selection ("Lipschitz sampling") accelerates convergence proportional to blockwise curvature [1807.09146].
- Inexact block minimization and variable-metric (e.g., quasi-Newton) subproblem solutions retain convergence under mild conditions [1807.09146].

Practical stabilization mechanisms include step-size tuning, proximal and trust-region regularization, and overlap-aware block updating. Empirical evidence demonstrates block MM's reliability and scalability, especially when communication or computation per block is a bottleneck [1406.4064, 2012.03503].

## 7. Connections to Related Frameworks and Limitations

Block MM subsumes:

- Block Coordinate Descent (BCD), Block Proximal Gradient (BCPG), and Proximal Alternating Linearized Minimization (PALM) as special cases [1511.02746, 1310.6957].
- The Parallel Direction Method of Multipliers (PDMM) as a block-coordinate MM extension of ADMM, supporting randomized and overlapping blocks, with $O(1/T)$ ergodic rates [1406.4064].
- Gauss–Seidel block Alternating Direction Methods (GSADMM) under suitable surrogates [1406.4064].

Limitations occur when surrogates are not tight, convergence rates are sublinear unless strong convexity or error bounds hold, and acceleration beyond $\mathcal{O}(1/r)$ is generally possible only in special two-block settings or under additional regularity [1310.6957]. For nonconvex settings, block MM converges to stationary points rather than global optima, and explicit complexity rates may require stronger smoothness or regularity [2012.03503, 2312.10330].

Block-Coordinate MM provides a unifying, rigorous, and highly adaptable framework for large-scale, structured, and constrained optimization. Its rich theory is matched by significant empirical and algorithmic breadth across statistical, machine learning, and mathematical optimization domains [1511.02746, 2012.03503, 2312.10330].

Source: https://www.emergentmind.com/topics/block-coordinate-mm