---
title: Blockwise Minorization–Maximization (Blockwise-MM)
url: https://www.emergentmind.com/topics/blockwise-minorization-maximization-blockwise-mm
type: topic
---

# Blockwise Minorization–Maximization (Blockwise-MM)

Blockwise Minorization-Maximization (blockwise-MM) is a flexible class of iterative optimization algorithms for nonconvex, constrained, and high-dimensional problems. Blockwise-MM generalizes the classic Majorization-Minimization (MM) and Minorization-Maximization principles by partitioning variables into blocks and cyclically updating these using block-specific surrogate functions. This structure allows tailored surrogates per block, geometric flexibility (including Euclidean and manifold constraints), and efficient per-block updates, often vital for large-scale or structured data models.

## 1. Mathematical Formulation and Blockwise Partitioning

Blockwise-MM approaches partition the parameter vector $x = (x_1, \ldots, x_B)$ into $B$ blocks, with each block $x_i$ constrained in a (possibly non-Euclidean) set $M_i$. The central objective is
\[
\min_x \; f(x) \quad \text{s.t.} \quad x \in M_1 \times \cdots \times M_B,
\]
where $f$ may be nonsmooth and nonconvex, and each $M_i$ can be a closed subset of a Riemannian manifold, convex set, or similar structure [2312.10330, 2402.09907].

At each iteration and for each block, a surrogate function $g_i(\cdot \mid x^{(k)})$ (majorant or minorant, depending on minimization or maximization context) approximates the objective restricted to that block with others fixed. The essential properties for each surrogate $g_i$ are:
- **Majorization (or minorization):** $g_i(x_i \mid x^{(k)}) \geq f(x_1^{(k)}, \ldots, x_{i-1}^{(k)}, x_i, x_{i+1}^{(k)}, \ldots, x_B^{(k)})$, with equality at $x_i = x_i^{(k)}$.
- **Blockwise convexity or strong convexity** (for minimization), or quasi-convexity for manifold blocks [2402.09907].
- Smoothness properties used for rate analysis [2012.03503].

## 2. Algorithmic Structure and Blockwise Cyclic Updating

Blockwise-MM algorithms proceed iteratively as follows:
1. For the current iterate $x^{(k)} = (x_1^{(k)}, \ldots, x_B^{(k)})$, sequentially (or in a chosen order) construct a surrogate $g_i(x_i \mid x^{(k)})$ for each block $i$.
2. Update each block by
\[
x_i^{(k+1)} = \operatorname*{arg\,min}_{x_i \in M_i} g_i(x_i \mid x^{(k)}).
\]
3. The cycle continues until convergence to a stationary point or an $\epsilon$-stationary point as measured by a blockwise first-order optimality criterion.

For blockwise-MM on manifolds, updates utilize manifold geometry (e.g., geodesic convexity), and surrogates often include proximal regularization or penalized distance terms [2312.10330, 2402.09907].

## 3. Surrogate Function Design and Geometric Properties

Surrogate functions are block-specific and may exploit structure:
- **Euclidean blocks:** quadratic or prox-linear surrogates, strong convexity, or smoothness constraints [2012.03503].
- **Riemannian/Manifold blocks:** geodesically quasi-convex or geodesically smooth surrogates, ensuring feasibility on nonconvex sets such as Stiefel or Grassmann manifolds [2402.09907, 2312.10330].

Essential surrogate properties per block:
- **Tightness** at the current iterate.
- **Majorization** over the block's feasible set.
- **Directional derivative matching** in the block's tangent space or Euclidean subspace.
- **(Geo-)quasi-convexity**: the surrogate is convex (or quasi-convex) along geodesics for manifold blocks.
- **Continuity** in all arguments [2402.09907].

Surrogates with additional regularization (e.g., proximal terms or trust-region radii) can be used to improve convergence properties and rate bounds [2012.03503, 2201.01652].

## 4. Convergence Theory and Complexity Analysis

Blockwise-MM guarantees monotonic descent of $f$ in the minimization case, and—under appropriate regularity—a sequence of iterates converging asymptotically to stationary points of $f$ [2402.09907, 2312.10330]. Key results include:

- **Global stationarity:** Provided surrogates satisfy the five axioms (tightness, majorization, first-order matching, continuity, geo-quasi-convexity), unique block-wise minimizers exist, and compact sublevel sets, every limit point of the sequence is stationary [2402.09907].
- **Complexity rates:** For surrogates with strong convexity or trust-region radii $r_k$, blockwise-MM produces an $\epsilon$-stationary point within $O(\epsilon^{-2})$ iterations (modulo logarithmic factors), matching optimal rates for first-order methods in nonconvex optimization [2312.10330, 2012.03503]. For diminishing radius or weakly convex/block-convex surrogates, explicit rates such as $O((\log n)^{1+\varepsilon}/n^{1/2})$ for empirical loss or $O((\log n)^{1+\varepsilon}/n^{1/4})$ for expected loss can be achieved (stochastic blockwise-MM setting) [2201.01652].

Proofs rely on surrogate optimality, summable progress gaps, and stationarity measures based on blockwise gradients and feasible directions.

## 5. Practical Implementations and Computational Aspects

Blockwise-MM frameworks offer computational advantages when each block subproblem has an efficient or even closed-form solution:
- **Heteroscedastic regression:** Each block update is reduced to weighted least-squares for regression coefficients, and a minorized quadratic in the dispersion parameters, with closed-form updates, dramatically reducing per-iteration cost compared to Newton's method [1603.04613].
- **Distributed/parallel architectures:** Separable surrogates permit deployment across clusters or workers, with communications limited to reduction of aggregated statistics for each block [1603.04613].
- **High-dimensional/structured models:** The blockwise structure allows updates within blocks (e.g., alternating updates of matrix factors in NMF or tensor decomposition), where joint optimization is intractable [2012.03503, 2201.01652].

Surrogate selection, step sizes (fixed or diminishing), and proximal regularization are tuned to balance per-iteration efficiency and global convergence speed.

## 6. Extensions to Manifold-Constrained and Stochastic Settings

Recent work extends blockwise-MM:
- **Manifold constraints:** Algorithms operate on product manifolds (e.g., Grassmannian, Stiefel, or Hadamard), with surrogates incorporating geodesic properties and first-order stationarity defined via Riemannian gradients [2402.09907, 2312.10330]. Applications include robust PCA, subspace tracking, and CP-dictionary learning.
- **Stochastic blockwise-MM:** Empirical surrogates are dynamically averaged and block-minimized within diminishing radii, permitting weakly convex or multi-convex surrogates. Convergence rates under non-i.i.d. data and dependent processes are established, relevant for online matrix/tensor factorization and online empirical risk minimization [2201.01652].

## 7. Applications and Algorithmic Variants

Blockwise-MM encompasses, generalizes, and subsumes several canonical frameworks:
- **Block coordinate descent (BCD):** As special cases when surrogates are linearizations plus strong convexity [2012.03503].
- **EM and blockwise EM:** When surrogates emerge from expectations over latent distributions [2012.03503].
- **Proximal algorithms:** Surrogates with proximal regularization enable extensions to composite nonsmooth problems and block-projected gradient schemes [2312.10330, 2201.01652].
- **Heteroscedastic regression:** The blockwise MM algorithm provides a scalable alternative to Newton-type methods with guaranteed monotonicity and stationarity [1603.04613].

Typical application domains include matrix/tensor factorizations, structured sparsity models, manifold-constrained subspace estimation, and large-scale regression.

---

Blockwise Minorization-Maximization algorithms furnish a versatile, theoretically grounded set of methods for block-structured nonconvex optimization, accommodating both classical Euclidean and modern manifold constraints, with rigorous monotonicity, convergence, and complexity guarantees [2402.09907, 2312.10330, 2012.03503, 2201.01652, 1603.04613]. Practical deployments leverage blockwise separability, surrogate flexibility, and structural efficiency for scalable optimization in high-dimensional and distributed statistical problems.

Source: https://www.emergentmind.com/topics/blockwise-minorization-maximization-blockwise-mm