Minimum Description Length Objective

• Minimum Description Length (MDL) is an information-theoretic framework that selects models by minimizing the combined encoding length of both the model and the data.
• It unifies model complexity and goodness-of-fit into a single metric, eliminating the need for arbitrary hyperparameter tuning.
• MDL is practically applied in neural network compression, sparse coding, and variable selection to enhance model interpretability and prevent overfitting.

The Minimum Description Length (MDL) objective is an information-theoretic framework for inductive inference, model selection, and learning. MDL posits that the best explanatory model for any data is the one which yields the shortest total description when both the model and the data are encoded optimally. This principle formally unifies model complexity and goodness-of-fit, providing a rigorous, hyperparameter-free approach to model selection, regularization, and representation learning. The MDL objective is realised through either two-part codes—where the model and the data conditioned on the model are encoded sequentially—or more refined universal coding schemes such as Normalized Maximum Likelihood (NML). MDL is foundational in modern statistics, machine learning, neural network compression, pattern mining, symbolic regression, and unsupervised representation learning, including sparse coding, autoencoders, nonnegative matrix factorization, and high-dimensional variable selection.

1. Formal Definition and Two-Part Coding

The MDL objective seeks the hypothesis or model $M$ that minimizes

$L(M, D) = L(M) + L(D|M)$

where $L(M)$ is the codelength required to specify the model (complexity term) and $L(D|M)$ is the codelength required to encode the data given model $M$ (fit term) (Grünwald et al., 2019). For continuous parameters or probabilistic models, the optimal (in expectation) codelength to encode a data point $x$ with probability density $P(x)$ is $h(x) = -\log_2 P(x)$, as dictated by Shannon's coding theorem.

The two-part MDL formulation typically includes:

• Model cost $L(M)$: Encodes the parameters or structure (e.g., neural network weights, dictionary atoms, cluster centers, principal components).
• Data-fit cost $L(D|M)$: Encodes the residuals, errors, or data conditioned on $M$ via likelihood or entropy-based code lengths.

The classical two-part code is connected to penalized likelihood approaches, but in MDL the penalty arises naturally from coding theory (Kolmogorov, Shannon) rather than arbitrary heuristics (Grünwald et al., 2019, Ramírez et al., 2011).

2. Parametric Versus Universal Codes

Universal coding, notably NML coding, refines MDL by defining code-lengths that are minimax optimal with respect to worst-case regret. The normalized maximum likelihood code is: $P_{\rm NML}(x) = \frac{p(x|\hat{\theta}(x))}{C}$ where $\hat{\theta}(x)$ is the maximum likelihood estimator and $C = \sum_{x'} p(x'|\hat{\theta}(x'))$ is a normalising constant (the parametric complexity) (Grünwald et al., 2019). The corresponding code length is: $$L_{\rm NML}(x) = -\log_2 p(x|\hat{\theta}(x)) + \log_2 C$$ This formulation guarantees minimax optimality in coding regret and unifies Bayesian, AIC/BIC, and cross-validation criteria under a single worst-case coding objective.

3. Practical Formulations in Machine Learning

For practical ML objectives, the MDL criterion is instantiated as follows:

Neural Networks and Deep Models

For a neural network $H$ trained on dataset $D$, the objective is (Abudy et al., 19 May 2025, Lan et al., 2021, Wiedemann et al., 2018): $L_{\rm MDL}(H;D) = |H| + |D:H|$ with $|H|$ the total bit-length of the network encoding (using prefix-free codes for units, connections, weights as rationals, activations, biases) and $|D:H|$ the negative log-likelihood (cross-entropy) of the data under $H$ (Abudy et al., 19 May 2025, Lan et al., 2021, Ayonrinde et al., 2024). Compression-based regularizers (entropy of weight distributions, variational coding, etc.) serve as differentiable surrogates for model encoding cost (Wiedemann et al., 2018, Shaw et al., 26 Sep 2025).

Sparse Coding and Dictionary Learning

For data $X$, dictionary $D$, and sparse codes $\alpha$, Ramirez & Sapiro (Ramírez et al., 2011) propose: $$L(X,D,\alpha) = L(E | D, \alpha) + L(\alpha) + L(D)$$ where $E = X - D\alpha$, each term is explicitly coded using universal mixtures for residuals and coefficients, and dictionary atoms are penalized according to universal Laplacian codes on their predictors.

Non-negative Matrix Factorization (NMF)

Squires et al. (Squires et al., 2019) define MDL-NMF as: $$L(W, H) = -\sum \log_2 P_W(W_{ij}) - \sum \log_2 P_H(H_{ij}) - \sum \log_2 P_E(E_{ij})$$ using Gamma distributions for factor matrices and Gaussian for the residual error. This replaces ad hoc regularization with a principled bits-vs-bits trade-off.

4. Model Selection, Regularization, and Hyperparameter Tuning

MDL is foundational for model selection tasks such as principal component cardinality (Tavory, 2018), network structure choice (Brugere et al., 2017), variable selection in high-dimensional regression (Wei et al., 2022), pattern mining (Galbrun, 2020), and symbolic regression (Yu et al., 2024). It automatically applies an Occam's razor penalty, avoiding overfitting without arbitrary regularization weights or grid searches.

Differential Description Length (DDL) (Abolfazli et al., 2019) refines MDL to estimate generalization error directly using differences of codelengths on partitions of training data, and empirically yields superior hyperparameter selection compared to cross-validation and Bayesian evidence.

For deep learning, Abudy et al. (Abudy et al., 19 May 2025, Lan et al., 2024, Lan et al., 2021) demonstrate that MDL-regularized objectives select for perfect, low-complexity symbolic solutions which are unreachable via $L_1$, $L_2$ or standard regularizers.

5. Optimization Strategies and Coding Schemes

Optimization under MDL is generally non-differentiable due to discrete and universal coding; genetic algorithms, simulated annealing, and combinatorial greedy search are routinely used for model search (Abudy et al., 19 May 2025, Lan et al., 2021, Abudy et al., 2023). Continuous relaxations (e.g. entropy penalties, variational Gaussian mixture priors, local reparameterization) allow gradient-based training for compression-aware neural nets (Wiedemann et al., 2018, Shaw et al., 26 Sep 2025).

Specific coding procedures include:

• Prefix-free integer codes (Elias, Li–Vitányi) for model structure.
• Universal mixtures (Gamma, Laplace, Exponential) for unknown scales and priors.
• Enumerative codes for combinatorial models and multinomials (Boullé et al., 2016).
• Adaptive model-specific codes for latent representations (e.g., hierarchical or tree-structured codes for autoencoders) (Ayonrinde et al., 2024).
• Prequential codes for sequential or online data (Grünwald et al., 2019).

6. Applications and Empirical Results

MDL-based model selection and learning has demonstrated improved generalization and interpretability compared with traditional objectives. Key findings include:

• Sparse Autoencoders (MDL-SAE) discover maximally interpretable, independent additive features, outperforming $L_1$ or TopK sparsity by avoiding feature splitting and trivial dictionary expansion (Ayonrinde et al., 2024).
• Compressed Deep Nets via entropy-constrained MDL objectives achieve state-of-the-art quantization and pruning, with theoretical guarantees on bit-cost and compression (Wiedemann et al., 2018).
• Hopfield Networks with MDL automatically determine the number of stored memories and balance memorization against prototype-driven generalization (Abudy et al., 2023).
• Symbolic Regression using MDL-former search avoids error-minimization pitfalls and recovers ground-truth formula structure far ahead of previous approaches (Yu et al., 2024).
• Network Model Selection with MDL-based “efficiency” criteria combine accuracy and parsimony, robust to hidden choice impacts and data perturbations (Brugere et al., 2017).
• High-Dimensional Variable Selection in regression and additive models via MDL are consistent, outperform robust and adaptive lasso, and remain competitive at extreme dimensionality (Wei et al., 2022).
• Pattern Mining with MDL-based codes emphasize succinct, non-redundant pattern sets and connect to Kolmogorov complexity (Galbrun, 2020).
• Formal Language Learning with MDL yields minimal, perfectly generalizing neural architectures where standard regularization fails (Lan et al., 2021, Lan et al., 2024).

7. Theoretical Guarantees, Generalization Bounds, and Future Directions

MDL minimization is theoretically minimax-optimal in worst-case code regret under NML coding (Grünwald et al., 2019, Cubero et al., 2018). Asymptotically optimal description length objectives for Transformers connect MDL to Kolmogorov complexity, proving resource-universal compression bounds and provable generalization guarantees (Shaw et al., 26 Sep 2025). Predictive coding is shown to converge blockwise on MDL objectives, yielding tight high-probability Occam bounds $$R(\theta) \le \hat{R}(\theta) + \frac{L(\theta)}{N}$$ for deep learning (Prada et al., 20 May 2025).

Practical optimization remains a challenge, with ongoing research into quasi-optimal codes, multimodal priors for discrete parameters, and compression-aware variational objectives (Shaw et al., 26 Sep 2025). The scope of MDL as a universal framework for model selection, regularization, and representation learning continues to expand, positioning it as a rigorous alternative to heuristic or unprincipled regularization across statistical and machine learning domains.