Minimum Functional Description Length

• Minimum Functional Description Length is a principle that selects functions based on the total code length needed to describe both the model and its residual uncertainty.
• It is applied in symbolic regression and related areas, integrating exact coding methods, learned estimation, and Fisher information approximations to assess model complexity.
• The approach emphasizes balancing simplicity with accuracy by accounting for the functional form, parameter-free structural penalties, and the geometry of the model space.

Searching arXiv for recent and foundational papers on minimum description length, symbolic regression, and functional model complexity. Minimum Functional Description Length denotes the use of minimum description length (MDL) principles to select functions, formulas, or model classes by the total code length required to describe both a candidate functional hypothesis and the data it explains. In the literature summarized here, the term is most explicit in symbolic regression, where the selected object is a symbolic expression, but closely related formulations also appear in maximum entropy model selection, Fisher-information-based MDL approximations, graph pooling, and deep learning. Across these settings, the unifying idea is that model selection should not be based solely on prediction error or parameter count: it should account for the descriptive cost of the functional form itself, the residual uncertainty left after adopting that form, and, in several formulations, the geometry or expressivity of the induced model family (Desmond, 17 Jul 2025, Yu et al., 2024, Heck et al., 2018).

1. Conceptual basis and scope

Minimum description length is introduced in the cited works as the principle that the preferred model is the one yielding the shortest description of the data. In one general formulation, the total codelength is written as

$$L(D)=\min_{p\in\mathcal M}\bigl[L(p)+L(D\mid p)\bigr],$$

with the data codelength under a model given by

$L_p(D)=-\log p(D)$

(Pandey et al., 2012). In symbolic regression, the same decomposition is written directly in terms of a hypothesis or function: $L(D) = L(H) + L(D|H),$ where $L(H)$ is the description length of the model or function and $L(D|H)$ is the description length of the residuals or information loss given the model (Desmond, 17 Jul 2025).

The qualifier “functional” is used in the supplied literature to distinguish description length associated with a model’s functional form from penalties based only on parameter count. This distinction is explicit in the Fisher information approximation literature, which argues that AIC and BIC fail when models differ in functional form while having the same number of free parameters, whereas MDL-style criteria can incorporate “functional model complexity” through the Fisher information geometry (Heck et al., 2018). A related interpretation appears in the maximum entropy setting, where complexity depends on the chosen functions or moments $\Phi$, and in symbolic regression, where the function tree, operator basis, and constants contribute directly to the code length [(Pandey et al., 2012); (Desmond, 17 Jul 2025)].

The strongest recent formulation in symbolic regression reframes the task itself: instead of minimizing prediction error while searching over formulas, it minimizes description length, on the grounds that formula recovery and numerical fit are not the same objective. The paper “Symbolic regression via MDLformer-guided search: from minimizing prediction error to minimizing description length” (Yu et al., 2024) states that two expressions can fit the same data equally well while having very different symbolic structures, and conversely two formulas with very similar symbolic structure can have different numerical errors at intermediate search steps. This motivates description length as a search signal rather than merely a post hoc regularizer.

2. Formalizations of functional description length

Several distinct but related mathematical instantiations appear in the literature.

In maximum entropy model selection, normalized maximum likelihood (NML) yields a decomposition into an error term and a complexity term: $$\mathrm{NML}(\mathcal M,\mathbf{x}^n) = \mathrm{ERR}(\mathcal M,\mathbf{x}^n)+\mathrm{COMP}(\mathcal M).$$ For maximum entropy models defined by feature subsets $\Phi$, the full stochastic complexity is

$$\mathrm{NML}(\mathcal M_\Phi,\mathbf{x}^n) = nH(p^*_{\mathbf{x}^n}) + \log\int_{\mathbf{y}^n\in\mathcal X^n} \exp\!\bigl(-nH(p^*_{\mathbf{y}^n})\bigr)\,d\mathbf{y}^n,$$

and model selection is performed by minimizing this quantity over candidate feature sets (Pandey et al., 2012). Here, the chosen functions $\Phi$ determine the model structure, so the penalty is a function-class penalty rather than a simple dimensional correction.

In symbolic regression via exhaustive search, the model term is made fully explicit. The final description length is

$L_p(D)=-\log p(D)$0

(Desmond, 17 Jul 2025). The constituents are the negative log-likelihood, a structural penalty $L_p(D)=-\log p(D)$1 for operator assignments over a basis of size $L_p(D)=-\log p(D)$2, a cost for simplification constants $L_p(D)=-\log p(D)$3, and a parameter cost determined by magnitudes and observed Fisher information. This is the most direct use of “functional description length” in the supplied material, because the symbolic function itself is encoded.

In the SR4MDL framework, description length is not computed analytically for each candidate formula; instead, the quantity of interest is the complexity of the simplest symbolic function $L_p(D)=-\log p(D)$4 explaining the observations $L_p(D)=-\log p(D)$5, written as $L_p(D)=-\log p(D)$6, where $L_p(D)=-\log p(D)$7 is “the number of symbols required to express $L_p(D)=-\log p(D)$8” (Yu et al., 2024). A learned estimator $L_p(D)=-\log p(D)$9 is trained with

$L(D) = L(H) + L(D|H),$0

together with an auxiliary symmetric cross-entropy alignment loss between numeric and symbolic latent spaces (Yu et al., 2024). This formulation treats description length as a latent target recoverable from numerical data.

In graph pooling, MDL is instantiated through the map equation, an information-theoretic code length for random walks on graphs. The two-level map equation is

$L(D) = L(H) + L(D|H),$1

and the multilevel version is defined recursively as

$L(D) = L(H) + L(D|H),$2

(Pichowski et al., 2024). Although this is not symbolic regression, it is still a functional-description-length criterion in the sense that the hierarchy is selected by the number of bits needed to describe graph flow under a hierarchical partition.

A broader theoretical interpretation appears in the paper on critical MDL codes, where the optimal MDL code is the NML distribution

$L(D) = L(H) + L(D|H),$3

with coding cost

$L(D) = L(H) + L(D|H),$4

and regret

$L(D) = L(H) + L(D|H),$5

(Cubero et al., 2018). This work interprets MDL codes as functionally aligned with the generative process rather than merely syntactic compression schemes.

3. Symbolic regression as a primary domain

The supplied sources make symbolic regression the clearest domain in which Minimum Functional Description Length acquires a concrete procedural meaning.

The exhaustive symbolic regression framework argues that traditional symbolic regression faces two difficulties: stochastic search may fail to find good functions with unknown probability, and model selection among Pareto-optimal formulas is ambiguous because accuracy and complexity are treated as separate, incommensurate objectives (Desmond, 17 Jul 2025). The proposed remedy is exhaustive enumeration of all functions up to a chosen complexity cap, followed by MDL-based ranking. The procedure is described as generating all tree templates up to a maximum number of nodes, labeling nodes by operator arity, decorating trees with all operator permutations from the basis set, simplifying and deduplicating them, fitting parameters, and broadcasting fitted values to equivalent expressions using Jacobians (Desmond, 17 Jul 2025). This operationalizes minimum functional description length as exhaustive search over function space plus one-dimensional code-length ranking.

The SR4MDL framework addresses a different failure mode. It argues that prediction error is often the wrong search signal if the goal is to recover the underlying formula, because numerical proximity does not imply structural proximity (Yu et al., 2024). The proposed replacement is a search objective based on minimum description length, justified by the claim that MDL is a better proxy for distance to the true formula. The paper explicitly states that, unlike mean squared error, MDL “decreases monotonically” as the candidate formula becomes structurally closer to the target formula (Yu et al., 2024).

This monotonicity claim is central. The paper’s conceptual argument is that if a candidate is literally a subformula of the target, then the remaining transformation needed to reach the target is smaller and its description length is shorter. Empirically, the Feynman I.18.4 case study tracks a chain of transformations $L(D) = L(H) + L(D|H),$6 from $L(D) = L(H) + L(D|H),$7 to $L(D) = L(H) + L(D|H),$8, and reports that the MDLformer’s estimated complexity decreases along this path (Yu et al., 2024). This suggests that description length is being used not only for final model choice but as a search geometry over symbolic expressions.

The search procedure in SR4MDL uses Monte Carlo tree search. Unlike traditional approaches that maintain populations or trees of full formulas and rank them by prediction error, SR4MDL maintains a tree whose nodes represent subformulas of the target (Yu et al., 2024). The root is the input $L(D) = L(H) + L(D|H),$9, and expansion applies unary or binary operators to available subformulas, variables, or constants. Selection is governed by a PUCT-style score

$L(H)$0

with $L(H)$1 (Yu et al., 2024). Candidates estimated to be closer to the final formula receive higher priority through the inverse MDL term.

The same work also introduces a residual transformation view. After enough iterations, the remaining map from the current candidate to the target is assumed to become simple enough to fit by a small parameterized function, often a linear map: $L(H)$2 (Yu et al., 2024). A plausible implication is that minimum functional description length can guide search toward structurally meaningful intermediate forms without requiring the algorithm to emit the exact closed-form target at every step.

4. Learned, exact, and approximate MDL criteria

The literature presents three distinct modes of using functional description length: exact or near-exact coding formulas, learned estimators, and asymptotic approximations.

The exhaustive symbolic regression framework belongs to the first category. It defines a concrete code over function trees, basis operators, constants, and parameters, and combines it with negative log-likelihood into a single scalar objective in nats (Desmond, 17 Jul 2025). Because the structural penalty depends on the chosen operator basis, the same analytic form can receive different code lengths depending on representation. The paper explicitly notes, for example, that $L(H)$3 is “simpler” if $L(H)$4 is a primitive in the basis set than if it must be composed from $L(H)$5, $L(H)$6, and division (Desmond, 17 Jul 2025). This makes the function code language-dependent, a point that is intrinsic to functional description length rather than an implementation detail.

The SR4MDL framework belongs to the second category. Instead of exact coding, it trains MDLformer to infer $L(H)$7 directly from numerical observations (Yu et al., 2024). MDLformer consists of an embedder, a Transformer encoder, attention-based pooling, and an MLP readout head. Numeric inputs $L(H)$8 are padded to a maximum feature dimension, tokenized into sign, mantissa, and exponent components, then passed through an 8-layer Transformer with 8 attention heads and 512 hidden units; positional encoding is removed because the order of data pairs is not meaningful (Yu et al., 2024). The model is trained in two stages, first alignment and then prediction, on about 131 million generated symbolic-numeric pairs obtained by sampling random formulas, simplifying them with SymPy, and sampling inputs from either a Gaussian mixture model or hidden-variable transformations (Yu et al., 2024). The appendix reports 31.9 million trainable parameters, 100k steps for alignment, and 30k steps for prediction (Yu et al., 2024).

The third category is represented by Fisher-information-based approximations. Exact NML is often intractable, so the Fisher information approximation uses

$L(H)$9

with

$L(D|H)$0

(Heck et al., 2018). The second term is described as the measure of functional model complexity, because it captures how the model maps parameters to observable distributions. This permits discrimination between models with equal parameter counts but different functional constraints (Heck et al., 2018).

The same paper, however, identifies a finite-sample pathology: FIA can invert the correct complexity ordering relative to NML (Heck et al., 2018). It therefore proposes a lower-bound sample size $L(D|H)$1 such that, for $L(D|H)$2, the FIA complexity ranking agrees with the NML ranking. This is an important caution for any use of approximate functional description length. The theoretical appeal of a function-sensitive criterion does not remove the need for validity conditions.

5. Relation to Occam’s razor, Kolmogorov complexity, and Bayesian evidence

The supplied works consistently connect functional description length to parsimony, but they do so through technically distinct routes.

The SR4MDL paper explicitly links MDL to Kolmogorov complexity and to the idea that the shortest program or shortest symbolic description is the most parsimonious explanation of the data (Yu et al., 2024). In its framing, MDL is not just a penalty added to an error term; it restores a search geometry aligned with formula recovery. This is stronger than the usual regularization interpretation.

The exhaustive symbolic regression paper provides a Bayesian interpretation. It writes the posterior over functions $L(D|H)$3 as

$L(D|H)$4

and under a Laplace approximation obtains a negative log posterior resembling the MDL score, provided the structural prior on functions satisfies

$L(D|H)$5

(Desmond, 17 Jul 2025). In this sense, functional description length can be viewed as Bayesian evidence combined with a prior on symbolic form.

The maximum entropy literature offers a different bridge. It proves that the minimax entropy principle is a special case of MDL if all candidate models are assumed to have equal complexity (Pandey et al., 2012). If complexities are equal, selection reduces to minimizing $L(D|H)$6; otherwise, MDL can decide among feature sets of different cardinalities because it includes a complexity term (Pandey et al., 2012). This clarifies that minimum functional description length is not synonymous with “choose the lowest entropy fit”; it is specifically the fit-complexity tradeoff expressed in code length.

Graph pooling makes the Occam connection explicit through the map equation, which is said to implement Occam’s razor by balancing model complexity against goodness of fit via MDL (Pichowski et al., 2024). Here, “simplicity” means a hierarchical partition yielding short codelength for a random walker’s trajectory. This broadens the term’s applicability: the “function” being selected need not be an analytic formula; it can be a hierarchical organization of graph flow.

The critical-codes literature adds a more interpretive layer. It argues that NML-optimal codes are “critical,” sit at a second-order phase transition, and generate samples with broad empirical distributions and high relevance (Cubero et al., 2018). In that formulation, the shortest description is functionally aligned with the latent generative mechanism, and attempts to compress beyond the NML optimum lead to degenerate, localized samples (Cubero et al., 2018). This suggests that functional description length can be read as a criterion for preserving inferentially relevant structure rather than simply minimizing syntax length.

6. Empirical behavior, applications, and limitations

The most concrete empirical results in the supplied material come from symbolic regression.

SR4MDL reports recovery of around 50 formulas across the Strogatz and Feynman benchmark datasets comprising 133 problems, outperforming state-of-the-art methods by 43.92% (Yu et al., 2024). In the main table, it attains a $L(D|H)$7 recovery rate on Strogatz and $L(D|H)$8 on Feynman (Yu et al., 2024). On 122 unseen black-box SRbench problems, where exact ground truth is unavailable, evaluation by the Pareto tradeoff between test $L(D|H)$9 and complexity places the method on the top-ranked Pareto front (Yu et al., 2024). MDLformer itself is reported to achieve $\Phi$0, RMSE $\Phi$1, and AUC $\Phi$2, with ablations showing that sequential alignment-then-prediction training is უკეთter than training without alignment, more input samples improve estimation, and ranking remains robust under input noise (Yu et al., 2024).

The exhaustive symbolic regression paper showcases MDL-based model selection on three astrophysical problems: cosmic expansion history, the radial acceleration relation, and inflationary potentials (Desmond, 17 Jul 2025). For cosmic expansion history, the MDL-optimal functions are reported as

$\Phi$3

and

$\Phi$4

preferred over the standard Friedmann equation by 7.12 nats and 4.91 nats respectively (Desmond, 17 Jul 2025). In inflation, the choice of structural prior matters strongly: with the default prior $\Phi$5, the MDL-optimal function is $\Phi$6, whereas a Katz back-off prior trained on successful equations in the domain yields more physically plausible optima such as $\Phi$7 or $\Phi$8 (Desmond, 17 Jul 2025). This is a direct demonstration that functional description length depends on the coding language or prior.

Beyond symbolic regression, the MapEqPool paper evaluates MDL-based adaptive multilevel pooling on six graph classification datasets: MUTAG, PROTEINS, DD, REDDIT-B, REDDIT-5K, and COLLAB (Pichowski et al., 2024). It reports top balanced accuracy on MUTAG at 81.40, PROTEINS at 73.41, and DD at 78.31 (Pichowski et al., 2024). Its significance in the present context is methodological: it shows that MDL can select among hierarchical alternatives with different effective depths via a single global description length, instead of optimizing each level independently.

The limitations described in the sources are equally important. The maximum entropy paper notes that the NML complexity term can be hard to compute in practice, especially for larger data sets, and uses quantization to make computation tractable in gene-selection experiments (Pandey et al., 2012). The FIA paper shows that asymptotic MDL approximations can be misleading in finite samples and proposes the threshold $\Phi$9 as a safeguard, but also stresses that $$\mathrm{NML}(\mathcal M,\mathbf{x}^n) = \mathrm{ERR}(\mathcal M,\mathbf{x}^n)+\mathrm{COMP}(\mathcal M).$$0 guarantees only correct rank order, not good absolute approximation (Heck et al., 2018). The deep-learning description-length literature, while not using the phrase Minimum Functional Description Length directly, reports that variational/Bayesian codelengths are “surprisingly poor” and that simple incremental or prequential codes can yield much tighter compression values (Blier et al., 2018). This suggests that the coding scheme itself is a substantive modeling choice.

7. Interpretation and common points of confusion

A recurring misconception is that description length is merely another name for a generic complexity penalty. The sources do not support that simplification. In symbolic regression, the point of MDL is not only to prefer simpler formulas after fitting; it is to provide a search objective that is better aligned with formula recovery than prediction error (Yu et al., 2024). In maximum entropy model selection, MDL is not equivalent to minimizing entropy alone, because the complexity term distinguishes candidate feature sets of unequal richness (Pandey et al., 2012). In FIA, functional complexity is not reducible to parameter count, since the Fisher information integral distinguishes models with the same dimension but different structural constraints (Heck et al., 2018).

A second misconception is that functional description length is representation-invariant. The symbolic regression literature explicitly shows otherwise. Structural cost depends on the operator basis set, and alternative priors such as the Katz back-off LLM alter which equations are favored (Desmond, 17 Jul 2025). This does not invalidate the principle; rather, it means that the code language is part of the hypothesis space.

A third misconception is that MDL always yields a fully exact and tractable criterion. The literature presents exact NML, asymptotic approximations such as FIA, learned proxies such as MDLformer, and operational codes such as prequential coding (Heck et al., 2018, Yu et al., 2024, Blier et al., 2018). These are related by a shared principle, but they differ substantially in tractability, faithfulness, and finite-sample behavior.

Taken together, the cited works define Minimum Functional Description Length as a family of model-selection and search principles in which the central object is the code length of a functionally specified hypothesis. In symbolic regression, it selects formulas; in maximum entropy settings, feature-defined model classes; in graph pooling, hierarchical graph partitions; in function-sensitive statistical model selection, model manifolds distinguished by Fisher-information geometry. The common thesis is that the shortest adequate functional description provides a more principled criterion than raw prediction error or parameter count alone (Yu et al., 2024, Desmond, 17 Jul 2025, Heck et al., 2018).