---
title: Majorization-Minimization (MM) Theory
url: https://www.emergentmind.com/topics/majorization-minimization-mm-theory
type: topic
---

# Majorization-Minimization (MM) Theory

Majorization-Minimization (MM) Theory

Majorization-Minimization (MM) theory formalizes a class of iterative optimization algorithms fundamental to statistical estimation, signal processing, and modern machine learning. MM algorithms proceed by constructing and sequentially minimizing surrogate upper bounds—majorizers—of the objective function, thereby guaranteeing monotonic improvement and often bypassing difficulties associated with non-convexity, non-differentiability, or high dimensionality. Recent advances encompass universal MM via automatic differentiation, nonconvex and locally majorant surrogates, min–max formulations for saddle structure, and stochastic and incremental adaptations for large-scale machine learning.

## 1. Formal Definition and Core Properties

Given an objective $f : \mathbb{R}^n \to \mathbb{R}$, a majorizer at iteration $x_t$ is a function $M(x_t; z)$ satisfying:

- **Majorization**: $M(x_t; z) \geq f(z)$ for all $z$,
- **Tangency**: $M(x_t; x_t) = f(x_t)$.

An MM algorithm generates a sequence via:
$$
x_{t+1} = \arg\min_z M(x_t; z)
$$
which guarantees monotonic decrease:
$$
f(x_{t+1}) \leq M(x_t; x_{t+1}) \leq f(x_t)
$$
This construction is agnostic to convexity or smoothness, provided suitable surrogates are available. Under regularity, any cluster point of $\{x_t\}$ is stationary [2308.00190][1611.03969][2106.02805].

Quadratic majorizers for $L$-smooth $f$ are immediate:
$$
M(x; z) = f(x) + \nabla f(x)^T(z-x) + \frac{L}{2} \|z-x\|^2
$$
Higher-order or nonconvex surrogates can be constructed by Taylor expansion plus remainder bounds, provided the surplus term enforces global majorization.

## 2. Surrogate Construction and Algorithmic Variants

Surrogate (majorizing) functions in MM are central to algorithmic performance. The choice depends on the structure of $f$:

- **Quadratic/Taylor Bounds**: For $L$-smooth $f$, using Hessian/Lipschitz bounds.
- **Jensen/Convexity Bounds**: Leveraging convexity, e.g., Fenchel duality for non-differentiable $h$.
- **Parameter Splitting/Latent Variable Models**: EM as minorization–maximization, where the $Q$-function is a convex surrogate for the complete data log-likelihood [1611.03969].
- **Bregman Majorization**: For composite models $f(u)=G(\rho(u)) + R(u)$, Bregman divergence $D_h$ enables majorizers that are nonconvex but faithful to structure, leading to superior descent in practice [1802.07072].
- **Locally Majorant/Relaxed MM**: Requiring only local upper-bounds and asymptotic tangency, enabling direct surrogates for non-smooth, non-convex programs [1511.08062].
- **Universal MM via AD**: Automatic interval extension of Taylor's theorem constructs polynomial upper bounds in arbitrary (sub)directions, automated for all $f$ in the AD calculus [2308.00190].
- **Block and Manifold Extensions**: Block-MM partitions large variable sets, constructing surrogates per block; when constraints are manifold-valued (Grassmann, Stiefel), surrogates and convergence analysis are formulated in terms of geodesic convexity and tangent spaces [2402.09907].

## 3. Convergence Theory

The cornerstone of MM is guaranteed monotonic descent of $f(x_t)$. In detail:

- **Monotonicity**: Always $f(x_{t+1}) \leq f(x_t)$.
- **Stationarity**: Under sufficient regularity and proper surrogate tangency, any limit point is stationary: $\nabla f(x^*) = 0$ (or $0$ in the subdifferential for nonsmooth cases).
- **Nonasymptotic Rates**: For $L$-smooth $f$, the sublinear rate
  $$
  \min_{1 \leq t \leq T} \|\nabla f(x_t)\|^2 \leq \frac{2L [f(x_1)-f(x_{T+1})]}{T}
  $$
  holds; for strongly convex $f$, a linear rate in objective gap is established [2106.02805][2308.00190].
- **KL Property and Non-smooth Cases**: With KL-tameness and appropriately convexified surrogates, global convergence to critical points is guaranteed, subsuming nonconvex and non-smooth optimization [2106.02805][1511.08062].
- **Stochastic and Incremental MM**: SAGA/SVRG/SARAH-style variance-reduced MM schemes for finite-sum objectives ($F(x) = \frac{1}{n}\sum f_i(x) + r(x)$), achieve stationarity rates of $O(n^{1/2}/\epsilon^2)$ per gradient, matching current best-known complexities [2305.06848][1402.4419].

## 4. Generalizations and Extensions

The MM concept generalizes along several axes:

- **Generalized MM (G-MM)**: Relaxes the classical requirement that surrogates be tangent at the previous iterate. Progress is tracked via a gap function. The framework allows introduction of problem-specific biases and leads to improved empirical performance on nonconvex and latent-variable models [1506.07613].
- **Min–Max and Primal–Dual MM**: Problems naturally suited to saddle-point or dual formulations benefit from MM operating on both the primal and dual blocks. The MM4MM and PDMM strategies exploit surrogates for both variable blocks, ensuring monotonicity and convergence even on nonconvex landscapes [2411.07561][2110.08600].
- **Universal and Hyperparameter-Free MM**: Advances in automatic differentiation for bounding derivatives enable black-box, hyperparameter-free universal MM solvers with guaranteed descent—without the need for hand-tuned line search or analytic majorizer derivation [2308.00190].
- **Learned Majorizers**: Inverse problems with severely ill-posed geometry motivate learning the majorant, e.g., by constraining a recurrent neural network (RNN) to output spectral or diagonal curvature bounds at every MM step, guaranteeing descent while adapting to nontrivial loss curvature [2602.03855].
- **Nonconvex Majorizers**: For composite nonconvex problems, nonconvex surrogates that are nevertheless globally optimized at each iteration allow escape from shallow stationary points. The tradeoff is solving harder subproblems but with much closer majorizer fidelity [1802.07072].
- **Block MM on Manifolds**: Convergence theory extends from Euclidean to manifold constraint sets, provided surrogates are geodesically quasi-convex and directional derivatives are properly matched in the tangent bundle [2402.09907].

## 5. Applications and Algorithmic Practice

MM algorithms have become foundational in numerous domains:

- **Statistical Estimation and Machine Learning**: Mixture models, regularized regression (including nonconvex penalties: SCAD, MCP), SVM fitting via IRLS, and variational inference all admit MM schemes [1912.11119][1705.04651][1611.03969].
- **Penalized and Sparse Estimation**: MM frameworks offer stable, convergent, and interpretable procedures for nonsmooth and nonconvex regularization, often leveraging reweighted soft-thresholding or surrogate-gradient minimization [1001.4776][1912.11119].
- **Inverse Problems and Signal Processing**: Applications range from phase retrieval and X-ray CT to massive-scale penalized least-squares, where algorithmic design of majorizers (e.g., via duality-based optimization of structured matrix surrogates) is critical for acceleration and computational tractability [1508.02958][2110.08600].
- **Structured and Bayesian Sparsity**: MM is naturally linked to hierarchical Bayesian modeling; alternating MM and Gibbs sampling enables uncertainty quantification and mode exploration in heavily multimodal posteriors (e.g., in MEG/EEG inverse source localization) [1710.08747].
- **Large-scale and Online ML**: Incremental, mini-batch, and variance-reduced MM break the computational bottleneck for large datasets without sacrificing convergence guarantees [1402.4419][2305.06848].

## 6. Limitations and Open Directions

While MM provides a unifying umbrella for optimization under challenging regimes, practical deployment hinges on appropriate surrogate construction:

- **Majorizer Design**: Nontrivial for compositions, nonconvexities, or complex regularizers. Recent structural or data-driven approaches mitigate, but do not eliminate, the challenge [2308.00190][1508.02958][2602.03855].
- **Global vs. Local Minima**: For general nonconvex objectives, MM methods guarantee convergence only to stationary points; global optimality depends on additional problem structure, such as geodesic convexity in the manifold-constrained setting [2402.09907].
- **Computational Load**: When surrogates are expensive to optimize—e.g., nonconvex or high-dimensional subproblems—per-iteration cost can surpass that of first-order approximations, unless global solvers or efficient block updates are available [1802.07072].
- **Hyperparameter Sensitivity**: Universal MM mitigates step size and tuning dependence, but in classical hand-designed fits, majorizer tightness and regularization must be carefully engineered [2308.00190][1511.08062].
- **Exploration vs. Exploitation**: Generalized MM (G-MM) introduces an exploration dimension but open questions remain on balancing progress and diversity; algorithmic tuning and theoretical convergence rate quantification are active fronts [1506.07613][2305.06848].

## 7. Reference Algorithms and Summary Table

A selection of archetypal MM algorithms and their domains is summarized below:

| Algorithm           | Majorizer Class                  | Application Domain          |
|---------------------|----------------------------------|----------------------------|
| EM                  | Latent-variable surrogates       | Mixture models             |
| IRLS MM             | Block diagonal/weighted quad     | SVM/logistic regression    |
| Bregman-prox MM     | Bregman (possibly nonconvex)     | Composite imaging problems |
| PDMM, MM4MM         | Min–max, blockwise               | Poisson, saddle-point      |
| Universal MM (AD)   | Polynomial-AD, hyperparameter-free| Generic ML, deep nets      |
| G-MM                | Relaxed bound, application-biased| Latent SVM, k-means        |
| Incremental/SVR-MM  | Variance-reduced, incremental    | Large-scale ML, ERM        |
| Learned MM networks | Data-driven majorant, RNN        | Ill-posed inverse problems |

All MM variants share the two essential steps: constructing a majorizing surrogate, and exact or approximate minimization thereof at each iteration, underpinned by monotonic descent and stationary convergence [1611.03969][2308.00190][2106.02805].

---

**References:**  
[2308.00190], [2411.07561], [1511.08062], [2602.03855], [1802.07072], [2402.09907], [1611.03969], [1912.11119], [1710.08747], [1001.4776], [1506.07613], [2110.08600], [2106.02805], [1705.04651], [2305.06848], [1402.4419], [1508.02958]

Source: https://www.emergentmind.com/topics/majorization-minimization-mm-theory