Minimum Description Length Principle

• Minimum Description Length Principle is a framework that selects the model minimizing the sum of negative log-likelihood and complexity for optimal data compression.
• It uses a prefix coding scheme to objectively balance model accuracy and complexity, thus avoiding both overfitting and underfitting.
• MDL is applied in time-series forecasting, regression, and reinforcement learning, ensuring robust convergence even in non-i.i.d. or non-stationary settings.

The Minimum Description Length (MDL) Principle is a fundamental methodology in statistical inference and machine learning, rooted in information theory and algorithmic complexity. MDL formalizes Occam’s razor by selecting the model that offers the shortest lossless encoding of both the data and the model itself. It provides an objective and general framework for model selection, prediction, and statistical learning with strong theoretical guarantees and minimal assumptions.

1. Definition and Foundational Framework

The MDL principle arises from the insight that the best statistical hypothesis for a dataset is the one that leads to maximum data compression. Given a countable model class $\mathcal{M} = \{ Q_1, Q_2, \dots \}$ and a sequence of observations $x = x_1:\ell$, each candidate $Q \in \mathcal{M}$ carries an associated complexity $K(Q)$, often interpreted as the length of its shortest prefix code or codeword. Denoting $Q(x)$ as the likelihood of the data under model $Q$, the MDL principle selects the model that minimizes the sum of negative log-likelihood and complexity:

$$\mathrm{MDL}^* = \arg\min_{Q \in \mathcal{M}} \left\{ -\log Q(x) + K(Q) \right\}$$

The predictive distribution for a future sequence $z$ given past data $x$ is then defined as:

$$\mathrm{MDL}^*(z \mid x) = \frac{\mathrm{MDL}^*(x z)}{\mathrm{MDL}^*(x)}$$

This framing is entirely general and does not necessitate the model class to exhibit independence, stationarity, ergodicity, or identifiability.

2. Convergence and Theoretical Guarantees

Quality of prediction under MDL is measured using the total variation distance between the predictive distribution of the selected model and the true generating distribution $P$. For any event $A$ in the appropriate $\sigma$-algebra $\mathcal{F}$, total variation distance is

$$d(P, Q \mid x) = \sup_{A \in \mathcal{F}} \left| P[A \mid x] - Q[A \mid x] \right|$$

The principal result for discrete MDL asserts that if the true distribution $P$ is in $\mathcal{M}$, then

$$d(P, \mathrm{MDL}^* \mid x) \to 0 \quad\text{as}\quad \ell \to \infty \quad (P\text{-almost surely})$$

This strong “merging of opinions” occurs irrespective of independence or mixing conditions, encompassing non-i.i.d., non-stationary, non-ergodic, or non-identifiable settings (0909.4588). Importantly, analogous convergence holds for Bayesian mixture models when the prior weights correspond to codeword probabilities $2^{-K(Q)}$, but the novelty for MDL is the convergence guarantee for maximum a posteriori (MAP) model selection itself, under this extremely weak set of requirements.

3. Practical Applications in Non-i.i.d. and Dependent Data

The generality of the convergence result enables MDL to be deployed in several types of non-i.i.d. and dependent-data problems:

• Time-Series Forecasting: MDL predictions are robust to temporal dependencies, changing distributions, and even adversarial non-ergodic processes, as long as the true data-generating law is within (or closely approximated by) the countable model class $\mathcal{M}$. This provides a strong foundation for time-series forecasting in settings where classical assumptions like stationarity fail.
• Discriminative Learning and Regression: For tasks such as classification and regression, MDL extends naturally by defining the relevant model class in a discriminative fashion. For example, for conditional models $Q(x \mid y)$, a new class $Q'(x) = Q(x \mid y_{1:\infty})$ is constructed, and the full MDL logic remains valid.
• Reinforcement Learning (RL): In sequential decision problems, an agent’s environment can be modeled by a countable set of possible environment measures. The MDL principle then ensures that conditional distributions over observation-action sequences converge (in total variation) to the true environment’s distributions, which in turn guarantees that the estimated value functions converge to their true counterparts:

$$V_{\mathrm{MDL}}(x, y) - V_P(x, y) \to 0 \quad (\text{almost surely})$$

This is achieved without any requirement for Markovian structure, stationarity, or ergodicity.

4. MDL and Model Complexity: Operational and Statistical Implications

The operational core of MDL is the explicit penalization of model complexity and its direct connection with coding theory. The complexity term $K(Q)$ can be interpreted via:

• Coding theory: as the prefix code length required to describe $Q$.
• Algorithmic information theory: as a proxy for Kolmogorov complexity in a model class context.

By minimizing $-\log Q(x) + K(Q)$, MDL realizes an automatic trade-off between overfitting (too low complexity) and underfitting (too high complexity). Notably, the resulting selection is parameter-free from the practitioner’s perspective, provided the coding scheme is fixed.

This insights tie MDL closely to penalized likelihood (AIC, BIC), maximum a posteriori estimation, and Bayesian inference, but with the advantage that the complexity penalty need not take the form of a traditional parameter prior—any prefix code (or more broadly, any “luckiness function”) can be substituted, allowing the user to encode domain knowledge or computational constraints.

5. Statistical Robustness and Broad Applicability

A critical property of MDL is that its primary predictive and estimation guarantees hold under minimal regularity conditions. Classical results for Bayesian or penalized likelihood methods often hinge on independent sampling, identifiability, and structure in the model class. The framework set by MDL (0909.4588) demonstrates that:

• No independence or i.i.d. requirement: The sequence of data points may exhibit arbitrary dependencies.
• No stationarity or ergodicity needed: Data-generating processes may evolve or never settle to steady-state behavior.
• No identifiability condition: The model class may contain multiple equivalent representations of the underlying process.

Consequently, MDL is directly applicable to complex, real-world domains where such classical assumptions break down, yielding consistent prediction and estimation as long as the model class is sufficiently expressive.

6. Implementation and Deployment Considerations

To deploy MDL-based model selection and prediction procedures:

• Model Class Construction: Enumerate a countable set of models $\mathcal{M}$, each with computable likelihoods and assigned codeword lengths $K(Q)$.
• Coding Scheme: Specify a prefix code (e.g., using universal coding or standard code tables) so that $K(Q)$ reflects prior beliefs or computational desirability.
• Sequential Implementation: At each timestep, update with observed data; select the MDL-optimal model and use its conditional for subsequent predictions.
• Scalability: While the theoretical justification is agnostic to computational cost, practical implementations may rely on approximations, pruning, or heuristic search in large or infinite model classes to retain tractability.

Convergence and robustness results apply irrespective of the learning rate or adaptation sequence, provided the updates follow the MDL decision rule.

---

The MDL principle, as formally developed for countable model classes and established in total variation, underpins a large range of modern inferential methods. Its minimal requirements and broad applicability make it foundational to rigorous statistical inference, sequential prediction, and robust learning in non-i.i.d., non-stationary, and even adversarial environments (0909.4588).