---
title: Luckiness-weighted NML
url: https://www.emergentmind.com/topics/luckiness-weighted-nml-lnml
type: topic
---

# Luckiness-weighted NML

Luckiness-weighted Normalized Maximum Likelihood (LNML) is a generalized universal distribution extending the normalized maximum likelihood (NML) to parametric models where NML is ill-defined or divergent, particularly continuous or high-capacity settings. LNML introduces a non-negative "luckiness" or weight function over the parameter space, regularizing the model and ensuring well-posedness of the minimax regret solution. LNML appears in statistical inference, coding theory, model selection, and recent advances in regularized estimation and high-dimensional settings.

## 1. Formal Definition and Core Properties

Given a parametric family $\mathcal{M} = \{p(x;\theta):\theta\in\Theta\}$ and a sample $x^n \in \mathcal{X}^n$, standard NML is defined as
\[
\bar p^{\mathrm{NML}}_n(x^n) = \frac{\max_{\theta\in\Theta} p(x^n;\theta)}{C_n}, \quad C_n = \int_{\mathcal{X}^n}\max_{\theta\in\Theta}p(x^n;\theta)d\mu(x^n).
\]
If $C_n$ diverges (e.g., for Gaussian models), NML is not defined. LNML replaces the maximum likelihood in both numerator and denominator with a luckiness-weighted form using a weight (luckiness) function $\pi(\theta)>0$:
\[
\bar p^{\mathrm{LNML}}_n(x^n) = \frac{\max_{\theta\in\Theta} [p(x^n;\theta)\pi(\theta)]}{C_n(\pi)}, \quad C_n(\pi) = \int_{\mathcal{X}^n} \max_{\theta\in\Theta}[p(x^n;\theta)\pi(\theta)] d\mu(x^n).
\]
With $\pi(\theta)\equiv 1$, LNML reduces to ordinary NML. LNML is the unique pointwise minimax solution to the regret function
\[
R_\pi(q;p,x^n) = \log \frac{p(x^n)\pi(\theta)}{q(x^n)},
\]
so that
\[
\bar p^{\mathrm{LNML}}_n(x^n) = \arg\min_{q:\int q=1} \max_{\theta, x^n} R_\pi(q;p,x^n).
\]
LNML always yields a proper (normalized) distribution assuming integrability of the weighted likelihood.

## 2. Minimax Regret, Asymptotics, and Interpretations

The regret of LNML under the luckiness-weighted regime is defined as
\[
R^{L}(x) = -\log p_{\mathrm{LNML}}(x) + \log p_{\theta^*(x)}(x) = \log C(L) - \log L(\theta^*(x)),
\]
where $\theta^*(x) = \arg\max_{\theta}[L(\theta)p_\theta(x)]$. The worst-case regret is
\[
R^L_{\max} = \sup_{x}\log C(L),
\]
so LNML achieves constant regret determined by the log normalization term. For regular (smooth) parametric families of dimension $d$, the asymptotic expansion of $C(L)$ under the Laplace method is
\[
\log C(L) = \frac{d}{2}\log n + \frac{d}{2}\log(2\pi) + \log \int_\Theta L(\theta)|I(\theta)|^{1/2} d\theta + o(1),
\]
where $I(\theta)$ is the Fisher information. This yields the same leading minimax $\log n$ growth as NML for appropriate $L$.

## 3. Construction and Examples in Key Parametric Families

### 3.1 Multivariate Normal Models

For observations $x_1,\dots,x_n\in\mathbb{R}^m$, with $f(x^n;\mu,\Sigma)$ the standard $(\mu,\Sigma)$-Gaussian likelihood, LNML uses a conjugate-like prior as luckiness:
\[
\pi(\mu,\Sigma;\nu, \mu_0, \Sigma_0, \rho^2) = \frac{1}{(2\pi)^{m\nu/2}|\Sigma|^{\nu/2}} \exp\left\{-\frac{\nu}{2}\mathrm{tr}[\Sigma^{-1}(\Sigma_0 + \rho^2(\mu-\mu_0)(\mu-\mu_0)^\top)]\right\},
\]
with $\nu > m-1$, $\Sigma_0 \succ 0$, $\rho^2 > 0$. This choice ensures convergence of the maximization and the normalization integrals. The resulting LNML has a closed-form:
\[
\bar p^{\mathrm{LNML}}_n(x^n) = \frac{f(x^n; \bar\mu_n, \bar\Sigma_n)\,\pi(\bar\mu_n,\bar\Sigma_n)}{C_n},
\]
with explicit formulas for $\bar\mu_n, \bar\Sigma_n$ as weighted MAP estimators, and $C_n$ involving special functions (multivariate Gamma).

### 3.2 Discrete Memoryless Sources (DMS)

With categorical probabilities $\theta=(\theta_1,\dots,\theta_m)$ and luckiness $L(\theta)=\prod_{i=1}^m \theta_i^{\beta_i}$, the LNML numerator becomes
\[
\prod_{i=1}^m\left(\frac{n_i+\beta_i}{n+B}\right)^{n_i+\beta_i},
\]
for counts $n_i$ ($B = \sum_i \beta_i$), and normalization is via summing over count vectors. For $\beta_i=1/2$ (“Jeffreys” luckiness), leading order regret matches NML, with a different $O(1)$ offset.

### 3.3 Linear Regression with $\ell_2$ Luckiness

For linear regression with Gaussian errors and a ridge-like luckiness, $w(\theta) = \exp(-\lambda\|\theta\|^2/(2\sigma^2))$, LNML in the supervised predictive version (LpNML) yields not only consistent regularization but a predictive distribution that can be computed exactly as a shifted Gaussian, blending in-sample interpolation with conservative extrapolation in under-determined cases [2206.08757].

## 4. Algorithmic and Theoretical Insights

LNML density and its normalization constant typically require inner maximization and outer integration—often intractable in high dimensions. For penalized empirical risk minimization, let $g(\theta, \lambda)$ be a penalty (luckiness), then LNML code length is
\[
L(X|\lambda) = \min_\theta [f_X(\theta) + g(\theta,\lambda)] + \log Z(\lambda),\quad Z(\lambda) = \int_{X\in\mathcal{X}} \max_\theta e^{-f_X(\theta)-g(\theta,\lambda)}d\nu(X).
\]
To address computational challenges, analytic upper bounds ("uLNML") were derived [1804.09904]. Under smoothness and convexity assumptions, $U(X|\lambda)$ is computable in closed-form for $\ell_2$/$\ell_1$ penalties and is uniformly close to the true LNML code length—enabling practical parameter selection (MDL-RS) in high dimensions.

## 5. Role of Luckiness and Incorporation of Side Information

The choice of luckiness function $\pi(\theta)$ (or $w(\theta)$, $L(\theta)$) encodes prior beliefs, regularization, or auxiliary information:

- **Priors or pseudo-priors**: Conjugate-like weights encode beliefs or enforce lower bounds (e.g., for covariance matrices).
- **Side information**: Incidental data or null-hypothesis values can be incorporated by constructing $w(\theta)$ to bias estimators towards plausible regions or to allow finite regret when ordinary NML diverges.
- **Regularization**: $\ell_2$ luckiness directly yields ridge regression behavior, regularizing hypothesis space and controlling complexity in high-capacity settings [2206.08757].
- **Statistical evidence measures**: LNML enables discrimination information $DI(x)$ to assess evidence for model comparison, with asymptotic calibration and robustness to multiplicity [1010.0694].

## 6. Connections to NML, Bayesian Mixtures, and $\alpha$-NML

LNML unifies and interpolates various universal coding/prediction paradigms:

- With $\pi(\theta)\equiv 1$, ordinary NML is recovered.
- LNML is a limiting case of $\alpha$-NML as $\alpha\to\infty$ with a prior $w_\alpha(\theta) \propto [L(\theta)]^\alpha$ [2202.12737]. Mixture/Bayesian codes correspond to $\alpha=1$, LNML to $\alpha\to\infty$.
- LNML (with appropriate luckiness) sits at the edge of the trade-off between mixture predictors and hard minimax NML, providing a uniform constant-regret bound and avoiding divergences present in unconstrained NML.

## 7. Practical Applications and Model Selection

LNML is particularly relevant in:

- **Model selection under MDL**: Explicit, finite-complexity penalty even for non-compact or continuous models, augmenting MDL-based criteria [1708.01861, 1804.09904].
- **High-dimensional penalty selection**: The MDL-RS method leverages analytic uLNML to select regularization parameters efficiently, outperforming cross-validation and BIC/AIC in highly-redundant/high-dimensional regimes [1804.09904].
- **Prediction under distribution shift**: LNML/LpNML provides bounded, calibrated regret and improved robustness over empirical risk minimization, with improved out-of-distribution characteristics [2206.08757].
- **Robust inference and multiple comparisons**: LNML-based discrimination information adapts robustly when integrating diverse side information and controls error rates in high-throughput testing [1010.0694].

---

**References:**
- "Normalized Maximum Likelihood with Luckiness for Multivariate Normal Distributions" [1708.01861]
- "Statistical inference optimized with respect to the observed sample for single or multiple comparisons" [1010.0694]
- "High-dimensional Penalty Selection via Minimum Description Length Principle" [1804.09904]
- "Alpha-NML Universal Predictors" [2202.12737]
- "Beyond Ridge Regression for Distribution-Free Data" [2206.08757]

Source: https://www.emergentmind.com/topics/luckiness-weighted-nml-lnml