---
title: Normalized Maximum Likelihood (NML)
url: https://www.emergentmind.com/topics/normalized-maximum-likelihood-nml
type: topic
---

# Normalized Maximum Likelihood (NML)

Normalized Maximum Likelihood (NML) is a universal statistical modeling principle that achieves minimax optimality with respect to regret by normalizing the maximum-likelihood function over all possible data samples. It provides an objective, non-asymptotic, and parameter-free foundation for model selection, coding, prediction, and evidence quantification under the Minimum Description Length (MDL) principle, and serves as the cornerstone of modern universal coding theory. NML’s minimax regret property, its computational and representational challenges in continuous domains, and its principled extensions (e.g., luckiness-weighted NML, quotient-NML, $\alpha$-NML) have fueled a large literature spanning statistics, information theory, machine learning, and computational biology.

## 1. Formal Definition and Minimax Regret

Given a parametric family $\{ p(x^n; \theta) : \theta \in \Theta \}$ for samples $x^n \in \mathcal X^n$, the normalized maximum likelihood density is
\[
p_{\mathrm{NML}}(x^n) = \frac{p(x^n; \hat\theta(x^n))}{\int_{y^n} p(y^n; \hat\theta(y^n))\, dy^n}, \qquad \hat\theta(x^n) = \arg\max_{\theta} p(x^n; \theta).
\]
Here, the denominator (the parametric complexity or normalization constant) aggregates the maximized likelihood over all possible data samples.

The key optimality property, as established by Shtarkov, is that $p_{\mathrm{NML}}$ uniquely achieves minimax pointwise regret:
\[
p_{\mathrm{NML}} = \arg\min_{q(\cdot)} \sup_{x^n} \left[ -\log q(x^n) + \log p(x^n;\, \hat\theta(x^n)) \right].
\]
This yields a universal code—one that matches the ideal (oracle) code for every sample, up to a constant penalty $\log C_n$ independent of $x^n$ [1010.0694, 1401.7116, 2010.07382].

## 2. Properties and Regret Analysis

- **Stochastic Complexity**: The negative log of the NML distribution provides a two-term code length,
  \[
  L_{\mathrm{NML}}(x^n) = -\log p(x^n ; \hat\theta(x^n)) + \log C_n,
  \]
  where $C_n$ is the parametric complexity.

- **Minimax Regret**: For any code $q$, the worst-case regret is lower bounded by $\log C_n$; NML achieves this worst-case bound equally for all $x^n$.

- **Invariance**: The NML criterion is invariant to reparametrization and data-space relabeling (for finite $\mathcal X^n$) [1608.05522, 1010.0694].

- **Asymptotics**: For regular $k$-parameter exponential families,
  \[
  \log C_n = \frac{k}{2} \log \frac{n}{2\pi} + \log \int_\Theta \sqrt{\det I(\theta)} d\theta + o(1),
  \]
  with $I(\theta)$ the Fisher information [1801.03705, 1010.0694, 2508.21466].

- **Finite Sample Effects**: In small-sample regimes, the parametric complexity and stochastic complexity deviate from BIC-type penalizations and can influence practical model selection and hypothesis testing [1608.05522, 1901.00059].

## 3. Computation in Practice and Extensions

NML’s generically intractable normalization—an exponentially large sum or high-dimensional integral—has motivated algorithmic strategies and theoretical extensions:

| Domain/Model                | Key Computation Approach                | Citation              |
|-----------------------------|-----------------------------------------|-----------------------|
| Finite discrete models      | Direct sum, recurrences, asymptotics    | [1608.05522, 1010.0694]|
| Exponential families        | Fourier analysis, saddlepoint, density of MLE | [1801.03705, 1205.3549, 2409.08387] |
| Continuous models           | Reparametrization, coarea formula, luckiness regularization | [2409.08387, 1205.3549, 1708.01861]|
| Riemannian manifold data    | Riemannian volume measures, asymptotic Fisher info | [2508.21466]    |
| High-dimensional settings   | Approximations, re-normalization (e.g., restricted domains for GMMs) | [1709.00925]    |

With continuous and unbounded parameter spaces, the NML normalization may diverge; remedies include restricting the domain, introducing "luckiness" weight functions (LNML), or employing robust alternatives such as $\alpha$-NML [1708.01861, 2409.08387, 2202.12737].

## 4. Luckiness, Generalizations, and Surrogate Criteria

- **Luckiness-weighted NML (LNML)**: Augments the model class with a weight function $w(\theta)$:
  \[
  p_{\mathrm{LNML}}(x^n) = \frac{p(x^n; \hat\theta(x^n))\, w(\hat\theta(x^n))}{\int p(y^n; \hat\theta(y^n))\, w(\hat\theta(y^n))\, dy^n}
  \]
  Uniquely minimax for weighted regret and enables NML-type inference when the ordinary normalization is infinite [1010.0694, 1708.01861, 2206.08757].

- **Quotient-NML (qNML)**: For Bayesian networks, quotient-NML constructs a decomposable, hyperparameter-free, and score-equivalent criterion using ratios of "local" 1D-NMLs [2408.14935].

- **$\alpha$-NML**: Generalizes NML to minimize Rényi-divergence–based regret, interpolating between mixture (Bayesian) and worst-case (NML) predictors, and robust when NML is inapplicable [2202.12737].

- **Weighted NML (NMWL)**: Used for multiple hypothesis testing, incorporating side information or pseudo-data to enable robust discrimination information and control over multiple comparisons [1010.0694].

## 5. NML and Model Selection

NML code length forms the foundation of objective, parameter-free model selection under the MDL principle. By encoding both data fit (via the maximized likelihood) and model complexity (via the parametric complexity term), the NML criterion embodies a rigorous Occam’s razor, penalizing over-flexible models more heavily than BIC/AIC in finite samples [1608.05522, 2010.07382]. It is used to select:

- Model order in PCA: Closed-form NML bounds enable non-asymptotic rank selection [1901.00059].
- Number of clusters in GMMs: NML or re-normalized NML yields higher accuracy and robustness than classical information criteria [1205.3549, 1709.00925].
- Feature sets in maximum-entropy models: NML quantifies both complexity and fit, connecting to the minimax entropy principle [1204.6423].

## 6. Sequential Prediction, Universal Coding, and Bayesian Connections

NML admits a (possibly signed) mixture representation over parameter values, bridging MDL and Bayesian approaches—even though it is not a posterior with respect to any genuine nonnegative prior. This decomposition enables linear-time computation of marginals and predictive distributions in exponential family models [1401.7116]. NML-based predictors and classifiers are minimax regret optimal for universal prediction and coding, delivering finite-sample PAC-type guarantees and automatic regularization, especially in small-sample regimes [2010.07382].

## 7. Theoretical Limitations and Geometry

- **Non-existence**: NML (without regularization) is undefined for many continuous unbounded models, including univariate/multivariate Gaussians, due to normalization divergence [1708.01861, 2409.08387].
- **Geometric Measure Theory**: The rigorous extension of NML normalization to continuous models requires the coarea formula, accounting for the pushforward density of the MLE and avoiding the failure of naive integral decompositions [2409.08387].
- **Riemannian NML**: For data in non-Euclidean spaces (e.g., hyperbolic embeddings), NML must be defined over the intrinsic Riemannian volume, with correspondingly invariant code lengths and Fisher information geometry [2508.21466].

## 8. Applications and Empirical Impact

NML underpins compression-based learning, supervised classification with optimal small-sample risk, robust multiple hypothesis testing, objective model selection for PCA/rank selection, maximum entropy modeling, and structure learning in graphical models. NML and its variants, such as luckiness-NML and $\alpha$-NML, exhibit superior robustness, improved sample efficiency, principled regularization, and minimax guarantees in both theoretical analysis and empirical benchmarks [2206.08757, 2010.07382, 2508.21466, 2408.14935, 1608.05522].

---

In summary, NML furnishes a universal, minimax-regret–optimal, and formidably general framework for model complexity quantification, statistical inference, and predictive modeling across discrete and continuous, parametric and nonparametric, Euclidean and non-Euclidean domains. Its extensions and algorithmic refinements address intractability and non-existence in high-dimensional, unbounded, or geometrically structured settings, cementing its central position in contemporary statistical modeling and information-theoretic inference.

Source: https://www.emergentmind.com/topics/normalized-maximum-likelihood-nml