---
title: Majorization-Minimization Algorithms
url: https://www.emergentmind.com/topics/majorization-minimization-algorithm
type: topic
---

# Majorization-Minimization Algorithms

Majorization-Minimization (MM) algorithms are a broad family of iterative optimization schemes that replace a difficult objective function with a succession of easier surrogates—majorizers—that tightly bound the objective at the current iterate. By design, each MM step monotonically reduces the objective, and the method is applicable to a wide range of nonconvex, nonsmooth, constrained, or large-scale settings. MM algorithms are foundational to modern optimization for statistics, signal processing, and machine learning, with theoretical guarantees and rich algorithmic variants.

## 1. Core Principles and Formal Definition

The classical MM framework seeks to minimize a given function $f(x)$ by generating a sequence $\{x_k\}$ such that at each iteration:
- A *majorizing surrogate* $U(x; x_k)$ is constructed satisfying:
  1. $U(x; x_k) \ge f(x)$ for all $x$ (global upper bound)
  2. $U(x_k; x_k) = f(x_k)$ (tangency at current iterate)
- The next iterate is obtained by:
  \[
  x_{k+1} = \arg\min_x U(x; x_k)
  \]
- The descent property is immediate:
  \[
  f(x_{k+1}) \le U(x_{k+1}; x_k) \le U(x_k; x_k) = f(x_k)
  \]
Such a setup ensures monotonic decrease of $f(x_k)$ over iterations. The classical MM can be interpreted as a meta-algorithm—turning a hard minimization into a series of simpler subproblems, each easier due to the structure of $U(x; x_k)$.

## 2. Construction and Classes of Surrogates

#### 2.1 Traditional Surrogate Construction

Surrogates are often created using convexity (Jensen's inequality), Taylor expansions with upper bounding of the remainder, EM-style conditional expectations, or quadratic upper bounds exploiting Lipschitz continuity of the Hessian.

Examples include:
- Jensen-based bound for log-sum-exp in EM
- Quadratic majorization of smooth terms: $f(x) \le f(x_k) + \nabla f(x_k)^T (x - x_k) + \frac{L}{2}\|x - x_k\|^2$ for $L$-smooth $f$
- Linearization of nondifferentiable penalties, custom bounding for DC (difference of convex) problems

#### 2.2 Higher-Order and Adaptive Majorization

Recent advances include *higher-order MM* where the surrogate matches the function and its derivatives up to order $p$ at $x_k$, and the error $h(y; x_k) = U(y; x_k) - f(y)$ is $p$-times differentiable with Lipschitz $p$th derivative. Implementation requires constructing $U$ such that $h(y; x_k) \ge 0$ for all $y$, and $h(x_k; x_k) = \nabla h(x_k; x_k) = \dots = \nabla^p h(x_k; x_k) = 0$ [2010.13893], [2103.07984].

Automatic surrogate generation has emerged, e.g., "Universal MM" algorithms leverage Taylor-mode automatic differentiation and interval bounding on Taylor remainders to construct tight surrogate upper bounds programmatically for arbitrary (smooth) $f$ [2308.00190]. This allows "black-box" MM for user-supplied objectives and eliminates the need for hand-designed surrogates.

## 3. Convergence Theory and Rate Results

Assuming surrogates satisfy the classical MM properties and the domain is closed and level sets are compact, the MM sequence:
- Is guaranteed to be non-increasing
- Has limit points that are stationary for nonconvex $f$
- For convex $f$ with strongly convex surrogates, achieves global *linear convergence*:
  \[
    f(x_k) - f^* \le \beta^k (f(x_0)-f^*) \quad \text{for some } \beta \in (0, 1)
  \]
- For $p$th-order MM, global sublinear rates $O(1/k^p)$ in convex settings and local superlinear rates under uniform convexity are established [2010.13893], [2103.07984]
- For objectives with the Kurdyka–Łojasiewicz property, precise local rates ranging from sublinear, linear, to superlinear can be shown, depending on the exponent in the KL-inequality [2010.13893], [2103.07984]

Stochastic variants, including stochastic higher-order MM and stochastic majorization-minimization (SMM), feature sublinear $O(1/\sqrt{n})$ rates for convex problems, $O(1/n)$ for strongly convex, and almost-sure convergence to stationary points for broad nonconvex problems [1306.4650], [2103.07984].

Variance-reduced MM algorithms (incorporating SAGA, SVRG, SARAH estimators) further reduce gradient sample complexity, with optimal rates $\tilde{O}(n^{1/2}/\epsilon^2)$ for nonconvex finite-sum composite problems [2305.06848].

## 4. Algorithmic Variants and Extensions

| Variant                  | Distinguishing Feature            | Typical Application/Advantage                   |
|--------------------------|-----------------------------------|------------------------------------------------|
| Universal MM [2308.00190]| Automatic, derivative-based bounds| Black-box optimization without tuning          |
| Incremental MM (MISO) [1402.4419]| Surrogates updated per sample      | Large-scale sum-structure; linear rates         |
| Stochastic MM [1306.4650]| Surrogate per minibatch or sample | Online/streaming, memory and compute tractable  |
| Variance-reduced MM [2305.06848]| SVRG/SAGA in MM subproblems         | Best-known gradient complexities for finite-sums|
| Higher-order MM [2010.13893],[2103.07984]| $p$th-order Taylor bounding        | Superlinear local convergence under regularity  |
| Bregman MM [2501.07306]  | Adaptive, potentially non-Euclidean surrogates | Accelerated convergence for composite objectives|
| MM for matrix means [1312.4654]| Manifold optimization; closed-form updates | Riemannian means in SPD geometry               |
| Min-max MM (MM4MM) [2411.07561]| Surrogate for min-max reformulations | Nonconvex-constrained signal-processing         |
| Generalized MM (G-MM) [1506.07613]| Relaxes "touching" to "progress"      | Robustness to initialization, application-specific bias |

### Algorithmic building blocks
- SafeRate and SafeCombination (Universal MM): Hyperparameter-free, uses local Taylor majorizers, can adapt step-size over arbitrary smooth $f$ [2308.00190]
- MISO: Incremental update of per-sample surrogates, global surrogate minimized per iteration; achieves accelerated convergence for large T [1402.4419]
- Stochastic Proximal MM: Running surrogate is a weighted sum of per-sample surrogates, updated at every stochastic sample. Step-size schedule critical for rate [1306.4650]
- Variance reduction: MM-SAGA/SVRG/SARAH—MM subproblems solved with variance-reduced first-order estimators [2305.06848]

## 5. Practical Applications and Empirical Results

MM algorithms underpin a wide range of applications, including, but not limited to:
- Gaussian Mixture Regression, Multinomial Logistic Regression, and SVM: MM algorithms can build quadratic or EM-style surrogates yielding closed-form or fast iterative updates [1611.03969],[1705.04651]
- Nonnegative binary matrix factorization: MM with Jensen-type surrogates yields closed-form update rules competitive with logistic-PCA and interpretable factorization models [2204.09741]
- Bilevel hyperparameter optimization: MM on duality-refomulated single-level problems enables efficient conic-program subproblem solutions for otherwise intractable bilevel programs [2403.00314]
- Signal processing min-max problems: MM4MM leverages dual representations and majorized min-max alternation to enable hyperparameter-free, provably monotonic algorithms in phase retrieval, beamforming, sensor placement, etc. [2411.07561]
- High-dimensional regression with nonsmooth/nonconvex penalties: MM with iterated soft-thresholding or semismooth Newton subproblems achieves fast, scalable regression with theoretical guarantees for support recovery and convergence [1001.4776], [1903.11460]
- Deep neural network optimization: Universal MM methods can be applied layerwise to ensure safe, monotonic descent even under extreme overparameterization [2308.00190]
- Dirichlet maximum-likelihood: Variable Bregman MM accelerates parameter estimation compared to Newton-type and fixed-metric methods [2501.07306]

## 6. Limitations, Challenges, and Current Directions

Despite their generality, MM algorithms face important operational and theoretical considerations:
- Surrogate construction is nontrivial for non-smooth, high-dimensional, or non-Euclidean objectives. Recent work on automatic differentiable majorizer construction and variable-metric methods has broadened their applicability [2308.00190],[2501.07306]
- The efficiency of MM steps depends critically on the tractability of the surrogate subproblems. For some models, these may require bespoke solvers or reformulations (e.g., conic programming for bilevel optimization [2403.00314], semismooth Newton for nonconvex regression [1903.11460]).
- MM sequence convergence may be slow under ill-conditioned or tight surrogate settings; acceleration methods such as quasi-Newton acceleration (e.g., SQUAREM), adaptive metric methods, variance-reduction, and higher-order MM address this, but often require problem-specific tuning [1001.4776],[2305.06848].
- Classical MM's requirement that surrogates touch the objective at the current iterate can be unnecessarily restrictive in nonconvex and latent-variable settings; the Generalized MM (G-MM) framework relaxes this via a "progress" requirement, enabling more robust optimization [1506.07613].
- In stochastic and large-scale regimes, memory and communication constraints have led to the development of incremental, online, and variance-reduced MM variants; optimal choices of weights and batch sizes remain an active area of research [1306.4650],[1402.4419],[2305.06848].
- Extensions to non-Euclidean geometries, non-Lipschitz problems, or those with more complex constraint sets are ongoing areas of method development [2501.07306],[2411.07561].

## 7. Summary Table: Key MM Algorithm Variants

| MM Variant / Context                   | Core Innovation                            | Notable Features             |
|----------------------------------------|--------------------------------------------|------------------------------|
| Universal MM [2308.00190]              | Automatic Taylor-remainder surrogates      | Hyperparameter-free, black-box|
| Higher-Order MM [2010.13893],[2103.07984] | $p$th-order surrogates, fast local rates   | Superlinear convergence, adaptive|
| Variance-Reduced MM [2305.06848]       | SAGA/SVRG/SARAH in MM subproblems          | Optimal sample complexity    |
| Bregman MM [2501.07306]                | Variable, adaptive Bregman majorizers      | Accelerated convergence      |
| MISO/Incremental MM [1402.4419]        | Per-sample surrogates, updated incrementally| Linear rates for convex      |
| Generalized MM (G-MM) [1506.07613]     | Progress in place of touching constraint   | Exploratory, less initialization-sensitive|
| MM for bilevel programs [2403.00314]   | Surrogates for dual-based reformulations   | Efficient conic subproblems  |
| MM4MM min-max [2411.07561]             | Surrogates on min-max reformulations       | Monotonic, hyperparameter-free|
| Nonneg. binary NMF [2204.09741]        | Closed-form Jensen surrogates              | Interpretable factors, simple updates |

## References

Key references for further study include:
- [2308.00190] Universal Majorization-Minimization Algorithms
- [2010.13893] A systematic approach to general higher-order MM algorithms
- [2103.07984] Convergence analysis of stochastic higher-order MM algorithms
- [2501.07306] Variable Bregman Majorization-Minimization Algorithm...
- [2305.06848] Stochastic Variance-Reduced Majorization-Minimization Algorithms
- [2403.00314] Lower-level Duality Based Reformulation and Majorization Minimization...
- [1402.4419], [1306.4650] Incremental and stochastic MM for large-scale problems
- [2204.09741], [1705.04651], [1611.03969], [1001.4776] for classical and contemporary applications

The MM paradigm provides a versatile and well-founded approach for tackling the full complexity of contemporary machine learning, optimization, and statistical estimation problems—both theoretically and at scale.

Source: https://www.emergentmind.com/topics/majorization-minimization-algorithm