---
title: Normalized Maximum Likelihood (NML) Code-Length
url: https://www.emergentmind.com/topics/normalized-maximum-likelihood-nml-code-length
type: topic
---

# Normalized Maximum Likelihood (NML) Code-Length

The normalized maximum likelihood (NML) code-length is the canonical criterion in the minimum description length (MDL) framework for universal model selection and data compression. Defined for a parametric model by combining the maximum likelihood fit to the observed data with a universal penalty—the parametric complexity—NML achieves minimax optimality with respect to worst-case code-length regret. Its theoretical definition, exact formulas, integral representations, advanced computation methods, and implications for the tractability and optimality of statistical model selection are central to information-theoretic statistics and the theory of universal coding.

## 1. Formal Definition and Minimax Regret Optimality

Given a statistical model class $\{p(x^n; \theta) : \theta \in \Theta\}$ over sequences $x^n$ in a sample or data space $\mathcal{X}^n$, the normalized maximum likelihood density (or probability mass) is defined by
\[
p_{\mathrm{NML}}(x^n) = \frac{p(x^n; \hat{\theta}(x^n))}{C_n} 
\]
where $\hat{\theta}(x^n)$ is the maximum likelihood estimator (MLE) for data $x^n$, and the normalization $C_n$ (parametric complexity) is
\[
C_n = \int_{y^n \in \mathcal{X}^n} p(y^n; \hat{\theta}(y^n)) \, dy^n
\]
(or $\sum$ over $\mathcal{X}^n$ if discrete). The associated stochastic complexity or code-length is
\[
L_{\mathrm{NML}}(x^n) = -\log p(x^n; \hat{\theta}(x^n)) + \log C_n
\]
This form extends directly to continuous (Lebesgue) or Riemannian volume measures as appropriate.

The NML code-length achieves minimax pointwise regret:
\[
\min_q \max_{x^n} \left\{ -\log q(x^n) + \log p(x^n; \hat{\theta}(x^n)) \right\}
\]
with $p_{\mathrm{NML}}$ being the unique solution [2409.08387], [2311.01431], [1401.7116].

## 2. Discrete and Continuous Model Formulations

### Discrete Models

- $x^n \in \mathcal{X}^n$ with $\mathcal{X}$ finite or countable.
- NML code-length:
  \[
  L_{\mathrm{NML}}(x^n) = -\log P(x^n; \hat{\theta}(x^n)) + \log \sum_{y^n \in \mathcal{X}^n} P(y^n; \hat{\theta}(y^n))
  \]
  Example: Bernoulli, multinomial, and categorical families yield explicit multinomial coefficient representations and enable recurrence-based computation of the normalizer [1608.05522], [2403.06499].

### Continuous Models and the Coarea Theorem

- $x^n \in \mathbb{R}^{n \cdot d}$; $p(x^n; \theta)$ absolutely continuous.
- Naïvely replacing summations by integrals is invalid:
  - The level sets $\{x: \hat{\theta}(x) = \theta\}$ have Lebesgue measure zero.
  - Jacobian correction is required.
- The coarea formula from geometric measure theory resolves this:
  \[
  \int_{\mathbb{R}^{n d}} p(x ; \hat{\theta}(x)) \, dx 
  = \int_{\Theta} p_{\hat{\theta}}(\theta) \, d\theta
  \]
  where $p_{\hat{\theta}}(\theta)$ is the pushforward density of the MLE, integrating $p(x; \theta) J_{\hat{\theta}}(x)^{-1}$ over the corresponding level sets [2409.08387].

#### Practical Impact
- The normalization term can now be computed by a $K$-dimensional integral over parameter space instead of an $n \cdot d$-dimensional data integral.
- This establishes the correctness of the "integral-of-MLE-density-over-parameter-space" method for continuous models under mild regularity (Lipschitzness, nondegenerate Jacobian), not just asymptotically [2409.08387].

## 3. Analytic, Asymptotic, and Algorithmic Techniques

### Asymptotic Expansion

For regular $d$-parameter exponential family models, asymptotic expansion yields
\[
L_{\mathrm{NML}}(x^n) = n H(\hat{\theta}_n) + \frac{d}{2}\log \frac{n}{2\pi} + \log \int_{\Theta} |I(\theta)|^{1/2} d\theta + o(1)
\]
where $H(\theta)$ is the Shannon entropy (or differential entropy for continuous case), and $I(\theta)$ is the Fisher information matrix. The complexity penalty thus refines both BIC and two-part MDL through the explicit Fisher integral [2311.01431], [1205.3549], [1608.05522].

### Computational Methods

**Discrete models:** Binomial and multinomial complexities admit closed forms or recurrences (e.g., Kontkanen–Myllymäki algorithm) [1608.05522], [2403.06499].

**Continuous/exponential families:** Asymptotic Laplace-type approximations, Fourier-analytic integral representations (including non-asymptotic forms for exponential families via characteristic function and partition function techniques) [1801.03705].

**Divergence in Continuous Models:** RNML ("renormalized NML") and LNML ("NML with luckiness") modify the denominator by restricting the domain or by weighting with a prior ('luckiness'), ensuring finite normalization even for unbounded parameter spaces such as multivariate normal models [1708.01861], [1810.08749], [1205.3549].

**Geometric Generalization:** On Riemannian manifolds, the Rm-NML generalizes the concept with intrinsic measure, achieving invariance under coordinate transformations, essential for manifolds such as hyperbolic spaces [2508.21466].

## 4. Exact, Finite-Sample, and Mixture Representations

NML can be represented as a (possibly signed) Bayesian mixture over a sufficient set of parameter points, constructed to exactly reproduce the maximized likelihood and normalizer [1401.7116]. This representation speeds marginalization and sequential prediction, although signed weights are algebraic artifacts.

**Key results:**

- For exponential families, the mixture representation revolves around solving
  \[
  \int \exp(-n D(\hat{\theta}(x) \| \theta)) W_n(d\theta) = 1
  \]
  for all possible MLEs $\hat{\theta}(x)$, where $D(\cdot||\cdot)$ is the Kullback–Leibler divergence.
  
- Fast sequential coding and predictive updating is enabled, reducing computational cost from exponential in data size to linear in the sufficient statistic's range [1401.7116].

## 5. Specialized Applications and Model Selection Implications

### Model Selection via MDL Principle

NML code-length is central to MDL-based model selection. In practice, the model minimizing $L_{\mathrm{NML}}$ is selected. For exponential families, maximum entropy models, PCA rank estimation, clustering (via GMM), and Bayesian network learning, minimization over $L_{\mathrm{NML}}$ directly implements the MDL paradigm [1205.3549], [1204.6423], [1901.00059], [1810.08749].

### Causal Inference

NML-based stochastic complexity is deployed in causal discovery, e.g., selecting between latent confounding and direct causality via model comparison among NML code-lengths, with rigorous guarantees of statistical consistency [2403.06499].

### Statistical Physics

In model comparison between canonical and microcanonical ensembles (e.g., maximum entropy subject to hard vs soft constraints), NML code-length reveals the non-equivalence of ensembles through parametric complexity differences, with consequences for per-node description length in the thermodynamic limit [2307.05645].

### Bayes-NML Connections

- Mixture codes with least-favorable priors (e.g., Jeffreys prior) can asymptotically match NML code-length for regular models.
- NML is Bayes-optimal for uniform priors in microcanonical models and for Jeffreys prior in single-parameter canonical models, but deviates with extensive constraints [2311.01431], [2307.05645].

## 6. Technical Subtleties, Limitations, and Extensions

- In unbounded continuous models, $C_n$ diverges unless renormalization or restriction (domain/prior weighting) is introduced; LNML and RNML are principled solutions, optimizing a tilted minimax regret [1708.01861], [1810.08749].
- Asymptotic approximations (Laplace, Stirling) are valid under regularity but may misestimate parametric complexity in small-sample or boundary cases [2311.01431].
- For non-Euclidean data spaces (e.g., Riemannian, hyperbolic manifolds), the coordinate-invariant Rm-NML formulation incorporates the manifold's metric via the volume element, ensuring the geometric correctness of parametric complexity [2508.21466].
- Fourier-based constructions yield non-asymptotic formulas—where the partition function is analytic—especially for exponential families [1801.03705].

## 7. Summary Table: Key NML Code-Length Formulae

| Context (Model class)                    | General NML Code-Length         | Parametric Complexity Term                                    |
|------------------------------------------|---------------------------------|--------------------------------------------------------------|
| Discrete                                 | $-\log p(x^n; \hat{\theta}(x^n)) + \log \sum_{y^n} p(y^n; \hat{\theta}(y^n))$  | $\sum_{y^n} p(y^n; \hat{\theta}(y^n))$                      |
| Continuous (Euclidean)                   | $-\log p(x^n; \hat{\theta}(x^n)) + \log \int p(y^n; \hat{\theta}(y^n))\,dy^n$ | $\int p(y^n; \hat{\theta}(y^n))\,dy^n$                      |
| Continuous (Coarea-pushforward)          | $-\log p(x^n; \hat{\theta}(x^n)) + \log \int p_{\hat{\theta}}(\theta) d\theta$ | $p_{\hat{\theta}}(\theta)$: MLE pushforward density          |
| Exponential family (asymptotic)          | $n H(\hat{\theta}_n) + \tfrac{d}{2} \log \tfrac{n}{2\pi} + \log \int |I(\theta)|^{1/2}d\theta$         | $\frac{d}{2} \log \frac{n}{2\pi} + \log \int |I(\theta)|^{1/2} d\theta$  |
| Riemannian manifold                      | $-\log p_{\mathrm{vol}}(x^n; \hat{\theta}(x^n)) + \log \int p_{\mathrm{vol}}(y^n; \hat{\theta}(y^n))\,d\mathrm{vol}(y^n)$ | intrinsic, coordinate-invariant parametric complexity        |

All code-length formulas strictly separate model fit (negative log-marginal likelihood at the MLE) from the (typically model-dependent) parametric complexity term, which captures the worst-case excess regret and quantifies Occam’s penalty in model selection.

---

In conclusion, the normalized maximum likelihood code-length provides a unified, information-theoretic, and decision-theoretic basis for model selection and universal lossless compression. Subtleties in the continuous case, now resolved via the coarea formula and MLE pushforward, cement the mathematical foundation of the method for general models [2409.08387]. As a result, NML is central both to the rigorous understanding and to the practical implementation of the minimum description length principle [2311.01431], [1810.08749].

Source: https://www.emergentmind.com/topics/normalized-maximum-likelihood-nml-code-length