---
title: Block-Coordinate MM Optimization
url: https://www.emergentmind.com/topics/block-coordinate-majorization-minimization-block-coordinate-mm
type: topic
---

# Block-Coordinate MM Optimization

Block-Coordinate Majorization–Minimization (Block-Coordinate MM), also known as block MM or block majorization–minimization, is the extension of Majorization–Minimization (MM) to optimization problems whose variables are partitioned into several blocks. At each iteration, one constructs a surrogate for a selected block that upper-bounds the objective with the remaining blocks fixed, is tight at the current iterate, and is then minimized over that block. In convex composite settings this viewpoint is formalized by the Block Successive Upper-bound Minimization (BSUM) framework, which unifies block coordinate minimization (BCM), block coordinate gradient descent (BCGD), and block coordinate proximal gradient (BCPG) [1310.6957]. More generally, block MM combines MM with Block Coordinate Descent (BCD): the rationale is to partition the optimization variables into several independent blocks, obtain a surrogate for each block, and optimize the surrogate of each block cyclically [2402.09907].

## 1. Formal structure and MM interpretation

A standard problem class is
$$
\begin{array}{ll}
\text{minimize} & f(x) := g(x_1,\dots,x_K) + \displaystyle\sum_{k=1}^{K} h_k(x_k) \\
\text{subject to} & x_k \in X_k,\quad k=1,\dots,K,
\end{array}
$$
with \(x=(x_1^T,\dots,x_K^T)^T\), where \(g\) is smooth convex, each \(h_k\) is convex and possibly nonsmooth, and each \(X_k\) is convex [1310.6957]. In classical MM, one replaces the full objective by a surrogate \(U(\cdot;x^r)\) satisfying tightness and majorization, then minimizes that surrogate. Block-Coordinate MM applies the same principle block-wise: for each block \(k\), one builds a surrogate \(u_k(x_k;x)\) for the smooth part and updates only selected blocks.

In BSUM notation, the generic block update is
$$
x_k^{r+1} \in
\arg\min_{x_k\in X_k}\; u_k(x_k; w_k^{r+1}) + h_k(x_k),
$$
for \(k\) in the selected block set, while nonselected blocks remain unchanged [1310.6957]. The MM character is encoded by three conditions on \(u_k\): tightness at the current block value,
$$
u_k(x_k;x)=g(x),
$$
majorization,
$$
u_k(v_k;x)\ge g(v_k,x_{-k}),
$$
and gradient consistency,
$$
\nabla u_k(x_k;x)=\nabla_k g(x).
$$
These are the block-wise analogue of the standard MM requirements that a surrogate touch and upper-bound the target at the current iterate [1310.6957].

This viewpoint immediately clarifies the relationship with coordinate descent. If the surrogate is chosen as the original function restricted to the block, then block MM reduces to BCD or BCM. Conversely, BCPG and BCGD are block-coordinate proximal-MM methods in which \(u_k\) is a quadratic majorizer derived from block Lipschitz continuity of \(\nabla_k g\) [1310.6957]. The same reduction appears in manifold-constrained formulations: if the surrogate is simply the original function restricted to the block, then block MM reduces to BCD [2402.09907].

## 2. Surrogates, update rules, and algorithmic variants

The most common surrogates are exact block restrictions and quadratic majorizers. Exact block minimization yields BCM:
$$
x_k^{r} \in
\arg\min_{x_k\in X_k}\; g(x_1^{r},\dots,x_{k-1}^{r}, x_k, x_{k+1}^{r-1},\dots,x_K^{r-1}) + h_k(x_k),
$$
which is BSUM with \(u_k\equiv g\) [1310.6957]. When exact minimization is difficult, BCPG uses a quadratic surrogate,
$$
x_k^r = \arg\min_{x_k \in X_k}\; \left\langle\nabla_k g(\tilde x^r), x_k - x_k^{r-1}\right\rangle + \frac{L_k}{2}\|x_k - x_k^{r-1}\|^2 + h_k(x_k),
$$
and BCGD is the special case \(h_k\equiv 0\) [1310.6957].

BSUM theory covers several deterministic block-selection rules. Gauss–Seidel updates all blocks cyclically in each iteration. Essentially cyclic rules require that over any window of length \(T\), each block is updated at least once. Gauss–Southwell selects a block with large change, and Maximum Block Improvement (MBI) selects the block giving the largest objective decrease [1310.6957]. These rules matter both algorithmically and analytically, because the constants in the descent inequalities and cost-to-go bounds depend on the selection mechanism.

A distinct issue arises when a block subproblem has multiple minimizers. For convex BCM, the minimizer over a block need not be unique, and the selection rule can affect whether the method stagnates on a low-dimensional face. The “relative interior rule” chooses the next block iterate from the relative interior of the set of block minimizers,
$$
x_{n+1} \in \operatorname{ri} \, M\bigl(X \cap (x_n + I_n), f\bigr),
$$
and is shown to be “not worse, in a certain precise sense, than any other rule” [1910.09488]. The same geometric idea carries over to convex surrogates used in block MM: choosing a point in the relative interior of the surrogate argmin set avoids some pathological stagnation patterns that arbitrary tie-breaking can preserve [1910.09488].

For nonsmooth nonconvex problems, block MM often enriches the surrogate beyond a Euclidean quadratic. In Bregman formulations, the surrogate can take the form of a linearization plus a Bregman divergence, and in block relative smooth settings the kernel itself can depend on the current iterate. This yields block updates that are closer to mirror descent than to Euclidean proximal gradient, but remain MM steps because the Bregman term is precisely the majorizing correction [2107.04395].

## 3. Convergence guarantees and iteration complexity

In the convex Euclidean setting, BSUM provides explicit global rates. Under Assumptions A and B, multi-block nonsmooth convex problems admit global sublinear convergence of order \(O(1/r)\) for BSUM under Gauss–Seidel, essentially cyclic, Gauss–Southwell, and MBI rules; for exact BCM, the same \(O(1/r)\) rate holds without per-block strong convexity; and in a special two-block Gauss–Seidel BSUM, acceleration yields \(O(1/r^2)\) [1310.6957]. The analysis combines sufficient descent, cost-to-go inequalities, and a nonlinear recurrence for the optimality gap \(\Delta^r=f(x^r)-f(x^*)\).

For general constrained nonsmooth nonconvex optimization, a different complexity theory is available. BMM with \(\rho\)-strongly convex and \(L_g\)-smooth surrogates can produce an \(\epsilon\)-approximate first-order optimal point within \(\widetilde{O}((1+L_g+\rho^{-1})\epsilon^{-2})\) iterations and asymptotically converges to the set of first-order optimal points [2012.03503]. If BMM is combined with trust-region methods with diminishing radius, the dependence on \(\rho^{-1}\) disappears and the complexity improves to \(\widetilde{O}((1+L_g)\epsilon^{-2})\), which is especially relevant for “flat” surrogates [2012.03503]. A central analytical device there is a continuous first-order optimality measure
$$
\Phi(x) := \sup_{\theta\in \mathcal{X},\ \|\theta - x\|\le 1} \left\langle -\nabla f(x),\,\theta - x \right\rangle,
$$
used to relate blockwise progress to stationarity [2012.03503].

For nonconvex composite objectives, the Kurdyka–Łojasiewicz inequality provides a global convergence mechanism for MM and naturally extends to block-wise variants when the descent and subgradient bounds survive the block decomposition. In particular, MM sequences for \(F(x)=f(x)+r(x)\) with Lipschitz \(\nabla f\), proper lsc coercive \(F\), and KL geometry have finite length and converge to critical points; rates are finite, linear, or sublinear depending on the KL exponent [1504.07791].

| Setting | Representative framework | Guarantee |
|---|---|---|
| Multi-block nonsmooth convex | BSUM / BCM / BCPG | \(O(1/r)\); special two-block accelerated BSUM gives \(O(1/r^2)\) |
| Constrained nonsmooth nonconvex | BMM with \(\rho\)-strongly convex, \(L_g\)-smooth surrogates | \(\widetilde{O}((1+L_g+\rho^{-1})\epsilon^{-2})\) |
| Diminishing-radius trust-region | BMM-DR | \(\widetilde{O}((1+L_g)\epsilon^{-2})\) |
| Constrained block-Riemannian | RBMM | \(\widetilde{O}(\epsilon^{-2})\) |

A recurrent misconception is that monotonic descent alone implies convergence to a global optimum. The literature is more specific. Convex BSUM yields global sublinear rates under explicit assumptions [1310.6957], whereas general nonconvex block MM typically guarantees convergence to stationary or critical points, possibly with KL-based full-sequence convergence under stronger assumptions [1504.07791, 2012.03503].

## 4. Constrained, coupled, and inexact block MM

Block MM is not confined to unconstrained separable objectives. In multiblock ADMM with nonlinear coupling constraints,
$$
\min_{x,y}\; F(x_1,\ldots,x_m) + h(y)
\quad \text{s.t.} \quad \phi(x) + \mathcal B y = 0,
$$
the primal block updates can be performed by minimizing block surrogates of the augmented Lagrangian rather than the exact subproblems [2201.07657]. The resulting mADMM is a Gauss–Seidel block-coordinate MM method on the augmented Lagrangian, with subsequential convergence to a critical point, global convergence under the KL property, and \(O(1/\epsilon^2)\) iteration complexity for reaching an \(\epsilon\)-stationary point [2201.07657].

A related framework, inertial ADMM, combines ADMM with general minimization-majorization updates for each primal block and adds inertial terms for the primal variables. In that setting, each \(x_i\)-update minimizes a surrogate \(u_i\) for the nonconvex objective together with a majorized penalty term and an inertial proximal correction, while the \(y\)-block is updated by a smooth MM step [2102.05433]. Under standard assumptions, subsequential convergence and global convergence are proved for the generated sequence, again demonstrating that block MM can be embedded inside multiplier methods without losing a rigorous convergence theory [2102.05433].

These constrained formulations also clarify the status of inexact subproblem solves. In the nonconvex BMM theory and in the constrained block-Riemannian theory, the results remain valid when convex subproblems are not solved exactly, provided the optimality gaps are summable [2012.03503, 2312.10330]. This is practically important because exact block minimization is often the wrong computational target; what matters is a controlled decrease mechanism and an error sequence compatible with the global descent argument.

## 5. Bregman, extrapolated, and manifold-constrained formulations

A major recent development is the reinterpretation of block MM in Bregman geometry. For multiconvex problems, BMMe introduces extrapolation and shows that block majorization minimization can be reformulated as a block mirror descent method, with the Bregman divergence adaptively updated at each iteration; this yields subsequential convergence and accelerated multiplicative-update algorithms for \(\beta\)-NMF with \(\beta\in[1,2]\) [2401.06646]. In that formulation, each block step minimizes a majorizer evaluated at an extrapolated point, and the resulting update is simultaneously an MM step and a mirror-descent step.

For nonsmooth nonconvex problems with block relative smoothness, block alternating Bregman MM with extrapolation (BMME) uses block-dependent kernels \(\varphi_i\), block Bregman divergences, and surrogate functions for the nonsmooth terms. It proves subsequential convergence to a first-order stationary point under mild assumptions, and global convergence under stronger conditions [2107.04395]. This is a genuine block-coordinate MM generalization of Euclidean inertial proximal-gradient schemes: the curvature control is encoded in Bregman divergences rather than in \(\|x_i-x_i^k\|^2\), and extrapolation is governed by Bregman inequalities rather than Euclidean momentum bounds [2107.04395].

Classical block MM proofs assume closed convex block domains, but manifold-constrained problems violate this assumption. On the Grassmann manifold, convergence can still be proved when one block belongs to a geodesically convex subset of \(\mathrm{Gr}(N,D)\), the Euclidean block remains in a closed convex set, and the surrogates satisfy tightness, majorization, directional derivative consistency, continuity, and geodesic quasiconvexity [2402.09907]. Under unique block minimizers, compact sublevel sets, regularity, and continuity, the sequence generated by the two-block algorithm converges to a stationary point; if the problem has only one stationary point, the convergence point is the unique global optimum [2402.09907].

A broader block-Riemannian theory covers smooth nonconvex objectives in which each parameter block is constrained within a subset of a Riemannian manifold. In that setting, block majorization-minimization converges asymptotically to the set of stationary points and attains an \(\epsilon\)-stationary point within \(\widetilde{O}(\epsilon^{-2})\) iterations [2312.10330]. An important refinement is that, when the underlying manifold is a product of Euclidean or Stiefel manifolds, the assumptions for the complexity results are completely Euclidean even though the analysis uses Riemannian geometry explicitly [2312.10330].

## 6. Applications, design principles, and recurring pitfalls

Block-Coordinate MM appears across large-scale optimization, machine learning, signal processing, and communications. In convex composite optimization, the BSUM analysis explicitly lists LASSO, IRLS, and MIMO uplink capacity as settings where BCM or BCPG fit the framework and inherit explicit \(O(1/r)\) guarantees [1310.6957]. In matrix factorization, BMMe yields multiplicative updates with extrapolation for \(\beta\)-NMF and regularized KL-NMF [2401.06646]. In constrained signal design, block MM has been used to minimize the Integrated Sidelobe Level for unimodular sequences; there the block surrogates admit FFT/IFFT-based implementations and the resulting algorithm is monotonic [2001.06267]. In low-rank representation, iADMM uses block MM surrogates to update nuclear-norm, group-sparsity, and noise blocks inside a linearly constrained nonconvex model [2102.05433].

Several practical design principles recur throughout the literature. Block partitioning usually follows the natural variable grouping induced by the model: feature groups, users, matrix factors, parameter layers, or manifold-valued subspaces [1310.6957]. Surrogates should be chosen to exploit local structure: exact restriction when block minimization is tractable, quadratic majorizers from Lipschitz constants, Bregman majorizers under relative smoothness, or specialized surrogates for nonsmooth terms that preserve block convexity [1310.6957, 2107.04395]. Update rules can be cyclic, essentially cyclic, or greedy; the choice affects both empirical behavior and the constants in the rate bounds [1310.6957]. When subproblems are nonunique, selection rules matter; relative interior selection is specifically designed to prevent undesirable stagnation on the boundary of minimizer faces [1910.09488].

A second recurring pitfall is to identify block MM with heuristics lacking a genuine majorization relation. The defining feature is not merely blockwise optimization but the existence of valid surrogates satisfying tightness and upper-bound properties on each block [1310.6957, 2402.09907]. A third is to overgeneralize acceleration results: improved \(O(1/r^2)\) rates are established for a special two-block BSUM scheme [1310.6957], and extrapolated Bregman block MM requires careful control of inertial terms [2107.04395, 2401.06646]. The general nonconvex story remains one of stationarity, asymptotic convergence, and \(\widetilde{O}(\epsilon^{-2})\)-type first-order complexity rather than universal acceleration.

Taken together, these results position Block-Coordinate MM as a broad algorithmic family rather than a single update formula. Its unifying idea is stable: replace a difficult block subproblem by a simpler upper bound that is exact at the current iterate, solve the surrogate blockwise, and exploit the resulting descent structure to obtain convergence and complexity guarantees. The specific geometry—Euclidean, Bregman, Grassmann, Stiefel, or general block-Riemannian—changes the surrogate design and the proof technology, but not the central MM logic [1310.6957, 2312.10330].

Source: https://www.emergentmind.com/topics/block-coordinate-majorization-minimization-block-coordinate-mm