Block-Coordinate MM Optimization

• Block-Coordinate MM is an extension of MM that decomposes optimization problems into variable blocks, using surrogate functions that ensure tightness and majorization.
• It unifies approaches like block coordinate descent, BCPG, and BCGD while offering explicit convergence rates for both convex and nonconvex setups.
• Applications span machine learning, signal processing, and matrix factorization, emphasizing tailored surrogate designs and block update rules to address computational challenges.

Block-Coordinate Majorization–Minimization (Block-Coordinate MM), also known as block MM or block majorization–minimization, is the extension of Majorization–Minimization (MM) to optimization problems whose variables are partitioned into several blocks. At each iteration, one constructs a surrogate for a selected block that upper-bounds the objective with the remaining blocks fixed, is tight at the current iterate, and is then minimized over that block. In convex composite settings this viewpoint is formalized by the Block Successive Upper-bound Minimization (BSUM) framework, which unifies block coordinate minimization (BCM), block coordinate gradient descent (BCGD), and block coordinate proximal gradient (BCPG) (Hong et al., 2013). More generally, block MM combines MM with Block Coordinate Descent (BCD): the rationale is to partition the optimization variables into several independent blocks, obtain a surrogate for each block, and optimize the surrogate of each block cyclically (Lopez et al., 2024).

1. Formal structure and MM interpretation

A standard problem class is

$$\begin{array}{ll} \text{minimize} & f(x) := g(x_1,\dots,x_K) + \displaystyle\sum_{k=1}^{K} h_k(x_k) \ \text{subject to} & x_k \in X_k,\quad k=1,\dots,K, \end{array}$$

with $x=(x_1^T,\dots,x_K^T)^T$, where $g$ is smooth convex, each $h_k$ is convex and possibly nonsmooth, and each $X_k$ is convex (Hong et al., 2013). In classical MM, one replaces the full objective by a surrogate $U(\cdot;x^r)$ satisfying tightness and majorization, then minimizes that surrogate. Block-Coordinate MM applies the same principle block-wise: for each block $k$, one builds a surrogate $u_k(x_k;x)$ for the smooth part and updates only selected blocks.

In BSUM notation, the generic block update is

$$x_k^{r+1} \in \arg\min_{x_k\in X_k}\; u_k(x_k; w_k^{r+1}) + h_k(x_k),$$

for $k$ in the selected block set, while nonselected blocks remain unchanged (Hong et al., 2013). The MM character is encoded by three conditions on $x=(x_1^T,\dots,x_K^T)^T$0: tightness at the current block value,

$x=(x_1^T,\dots,x_K^T)^T$1

majorization,

$x=(x_1^T,\dots,x_K^T)^T$2

and gradient consistency,

$x=(x_1^T,\dots,x_K^T)^T$3

These are the block-wise analogue of the standard MM requirements that a surrogate touch and upper-bound the target at the current iterate (Hong et al., 2013).

This viewpoint immediately clarifies the relationship with coordinate descent. If the surrogate is chosen as the original function restricted to the block, then block MM reduces to BCD or BCM. Conversely, BCPG and BCGD are block-coordinate proximal-MM methods in which $x=(x_1^T,\dots,x_K^T)^T$4 is a quadratic majorizer derived from block Lipschitz continuity of $x=(x_1^T,\dots,x_K^T)^T$5 (Hong et al., 2013). The same reduction appears in manifold-constrained formulations: if the surrogate is simply the original function restricted to the block, then block MM reduces to BCD (Lopez et al., 2024).

2. Surrogates, update rules, and algorithmic variants

The most common surrogates are exact block restrictions and quadratic majorizers. Exact block minimization yields BCM:

$x=(x_1^T,\dots,x_K^T)^T$6

which is BSUM with $x=(x_1^T,\dots,x_K^T)^T$7 (Hong et al., 2013). When exact minimization is difficult, BCPG uses a quadratic surrogate,

$x=(x_1^T,\dots,x_K^T)^T$8

and BCGD is the special case $x=(x_1^T,\dots,x_K^T)^T$9 (Hong et al., 2013).

BSUM theory covers several deterministic block-selection rules. Gauss–Seidel updates all blocks cyclically in each iteration. Essentially cyclic rules require that over any window of length $g$0, each block is updated at least once. Gauss–Southwell selects a block with large change, and Maximum Block Improvement (MBI) selects the block giving the largest objective decrease (Hong et al., 2013). These rules matter both algorithmically and analytically, because the constants in the descent inequalities and cost-to-go bounds depend on the selection mechanism.

A distinct issue arises when a block subproblem has multiple minimizers. For convex BCM, the minimizer over a block need not be unique, and the selection rule can affect whether the method stagnates on a low-dimensional face. The “relative interior rule” chooses the next block iterate from the relative interior of the set of block minimizers,

$g$1

and is shown to be “not worse, in a certain precise sense, than any other rule” (Werner et al., 2019). The same geometric idea carries over to convex surrogates used in block MM: choosing a point in the relative interior of the surrogate argmin set avoids some pathological stagnation patterns that arbitrary tie-breaking can preserve (Werner et al., 2019).

For nonsmooth nonconvex problems, block MM often enriches the surrogate beyond a Euclidean quadratic. In Bregman formulations, the surrogate can take the form of a linearization plus a Bregman divergence, and in block relative smooth settings the kernel itself can depend on the current iterate. This yields block updates that are closer to mirror descent than to Euclidean proximal gradient, but remain MM steps because the Bregman term is precisely the majorizing correction (Hien et al., 2021).

3. Convergence guarantees and iteration complexity

In the convex Euclidean setting, BSUM provides explicit global rates. Under Assumptions A and B, multi-block nonsmooth convex problems admit global sublinear convergence of order $g$2 for BSUM under Gauss–Seidel, essentially cyclic, Gauss–Southwell, and MBI rules; for exact BCM, the same $g$3 rate holds without per-block strong convexity; and in a special two-block Gauss–Seidel BSUM, acceleration yields $g$4 (Hong et al., 2013). The analysis combines sufficient descent, cost-to-go inequalities, and a nonlinear recurrence for the optimality gap $g$5.

For general constrained nonsmooth nonconvex optimization, a different complexity theory is available. BMM with $g$6-strongly convex and $g$7-smooth surrogates can produce an $g$8-approximate first-order optimal point within $g$9 iterations and asymptotically converges to the set of first-order optimal points (Lyu et al., 2020). If BMM is combined with trust-region methods with diminishing radius, the dependence on $h_k$0 disappears and the complexity improves to $h_k$1, which is especially relevant for “flat” surrogates (Lyu et al., 2020). A central analytical device there is a continuous first-order optimality measure

$h_k$2

used to relate blockwise progress to stationarity (Lyu et al., 2020).

For nonconvex composite objectives, the Kurdyka–Łojasiewicz inequality provides a global convergence mechanism for MM and naturally extends to block-wise variants when the descent and subgradient bounds survive the block decomposition. In particular, MM sequences for $h_k$3 with Lipschitz $h_k$4, proper lsc coercive $h_k$5, and KL geometry have finite length and converge to critical points; rates are finite, linear, or sublinear depending on the KL exponent (Kang et al., 2015).

Setting	Representative framework	Guarantee
Multi-block nonsmooth convex	BSUM / BCM / BCPG	$h_k$6; special two-block accelerated BSUM gives $h_k$7
Constrained nonsmooth nonconvex	BMM with $h_k$8-strongly convex, $h_k$9-smooth surrogates	$X_k$0
Diminishing-radius trust-region	BMM-DR	$X_k$1
Constrained block-Riemannian	RBMM	$X_k$2

A recurrent misconception is that monotonic descent alone implies convergence to a global optimum. The literature is more specific. Convex BSUM yields global sublinear rates under explicit assumptions (Hong et al., 2013), whereas general nonconvex block MM typically guarantees convergence to stationary or critical points, possibly with KL-based full-sequence convergence under stronger assumptions (Kang et al., 2015, Lyu et al., 2020).

4. Constrained, coupled, and inexact block MM

Block MM is not confined to unconstrained separable objectives. In multiblock ADMM with nonlinear coupling constraints,

$X_k$3

the primal block updates can be performed by minimizing block surrogates of the augmented Lagrangian rather than the exact subproblems (Hien et al., 2022). The resulting mADMM is a Gauss–Seidel block-coordinate MM method on the augmented Lagrangian, with subsequential convergence to a critical point, global convergence under the KL property, and $X_k$4 iteration complexity for reaching an $X_k$5-stationary point (Hien et al., 2022).

A related framework, inertial ADMM, combines ADMM with general minimization-majorization updates for each primal block and adds inertial terms for the primal variables. In that setting, each $X_k$6-update minimizes a surrogate $X_k$7 for the nonconvex objective together with a majorized penalty term and an inertial proximal correction, while the $X_k$8-block is updated by a smooth MM step (Hien et al., 2021). Under standard assumptions, subsequential convergence and global convergence are proved for the generated sequence, again demonstrating that block MM can be embedded inside multiplier methods without losing a rigorous convergence theory (Hien et al., 2021).

These constrained formulations also clarify the status of inexact subproblem solves. In the nonconvex BMM theory and in the constrained block-Riemannian theory, the results remain valid when convex subproblems are not solved exactly, provided the optimality gaps are summable (Lyu et al., 2020, Li et al., 2023). This is practically important because exact block minimization is often the wrong computational target; what matters is a controlled decrease mechanism and an error sequence compatible with the global descent argument.

5. Bregman, extrapolated, and manifold-constrained formulations

A major recent development is the reinterpretation of block MM in Bregman geometry. For multiconvex problems, BMMe introduces extrapolation and shows that block majorization minimization can be reformulated as a block mirror descent method, with the Bregman divergence adaptively updated at each iteration; this yields subsequential convergence and accelerated multiplicative-update algorithms for $X_k$9-NMF with $U(\cdot;x^r)$0 (Hien et al., 2024). In that formulation, each block step minimizes a majorizer evaluated at an extrapolated point, and the resulting update is simultaneously an MM step and a mirror-descent step.

For nonsmooth nonconvex problems with block relative smoothness, block alternating Bregman MM with extrapolation (BMME) uses block-dependent kernels $U(\cdot;x^r)$1, block Bregman divergences, and surrogate functions for the nonsmooth terms. It proves subsequential convergence to a first-order stationary point under mild assumptions, and global convergence under stronger conditions (Hien et al., 2021). This is a genuine block-coordinate MM generalization of Euclidean inertial proximal-gradient schemes: the curvature control is encoded in Bregman divergences rather than in $U(\cdot;x^r)$2, and extrapolation is governed by Bregman inequalities rather than Euclidean momentum bounds (Hien et al., 2021).

Classical block MM proofs assume closed convex block domains, but manifold-constrained problems violate this assumption. On the Grassmann manifold, convergence can still be proved when one block belongs to a geodesically convex subset of $U(\cdot;x^r)$3, the Euclidean block remains in a closed convex set, and the surrogates satisfy tightness, majorization, directional derivative consistency, continuity, and geodesic quasiconvexity (Lopez et al., 2024). Under unique block minimizers, compact sublevel sets, regularity, and continuity, the sequence generated by the two-block algorithm converges to a stationary point; if the problem has only one stationary point, the convergence point is the unique global optimum (Lopez et al., 2024).

A broader block-Riemannian theory covers smooth nonconvex objectives in which each parameter block is constrained within a subset of a Riemannian manifold. In that setting, block majorization-minimization converges asymptotically to the set of stationary points and attains an $U(\cdot;x^r)$4-stationary point within $U(\cdot;x^r)$5 iterations (Li et al., 2023). An important refinement is that, when the underlying manifold is a product of Euclidean or Stiefel manifolds, the assumptions for the complexity results are completely Euclidean even though the analysis uses Riemannian geometry explicitly (Li et al., 2023).

6. Applications, design principles, and recurring pitfalls

Block-Coordinate MM appears across large-scale optimization, machine learning, signal processing, and communications. In convex composite optimization, the BSUM analysis explicitly lists LASSO, IRLS, and MIMO uplink capacity as settings where BCM or BCPG fit the framework and inherit explicit $U(\cdot;x^r)$6 guarantees (Hong et al., 2013). In matrix factorization, BMMe yields multiplicative updates with extrapolation for $U(\cdot;x^r)$7-NMF and regularized KL-NMF (Hien et al., 2024). In constrained signal design, block MM has been used to minimize the Integrated Sidelobe Level for unimodular sequences; there the block surrogates admit FFT/IFFT-based implementations and the resulting algorithm is monotonic (Sankuru et al., 2020). In low-rank representation, iADMM uses block MM surrogates to update nuclear-norm, group-sparsity, and noise blocks inside a linearly constrained nonconvex model (Hien et al., 2021).

Several practical design principles recur throughout the literature. Block partitioning usually follows the natural variable grouping induced by the model: feature groups, users, matrix factors, parameter layers, or manifold-valued subspaces (Hong et al., 2013). Surrogates should be chosen to exploit local structure: exact restriction when block minimization is tractable, quadratic majorizers from Lipschitz constants, Bregman majorizers under relative smoothness, or specialized surrogates for nonsmooth terms that preserve block convexity (Hong et al., 2013, Hien et al., 2021). Update rules can be cyclic, essentially cyclic, or greedy; the choice affects both empirical behavior and the constants in the rate bounds (Hong et al., 2013). When subproblems are nonunique, selection rules matter; relative interior selection is specifically designed to prevent undesirable stagnation on the boundary of minimizer faces (Werner et al., 2019).

A second recurring pitfall is to identify block MM with heuristics lacking a genuine majorization relation. The defining feature is not merely blockwise optimization but the existence of valid surrogates satisfying tightness and upper-bound properties on each block (Hong et al., 2013, Lopez et al., 2024). A third is to overgeneralize acceleration results: improved $U(\cdot;x^r)$8 rates are established for a special two-block BSUM scheme (Hong et al., 2013), and extrapolated Bregman block MM requires careful control of inertial terms (Hien et al., 2021, Hien et al., 2024). The general nonconvex story remains one of stationarity, asymptotic convergence, and $U(\cdot;x^r)$9-type first-order complexity rather than universal acceleration.

Taken together, these results position Block-Coordinate MM as a broad algorithmic family rather than a single update formula. Its unifying idea is stable: replace a difficult block subproblem by a simpler upper bound that is exact at the current iterate, solve the surrogate blockwise, and exploit the resulting descent structure to obtain convergence and complexity guarantees. The specific geometry—Euclidean, Bregman, Grassmann, Stiefel, or general block-Riemannian—changes the surrogate design and the proof technology, but not the central MM logic (Hong et al., 2013, Li et al., 2023).