Epiplexity: Computational Information Theory

• Epiplexity is a formal framework that quantifies data’s structural content using time-bounded computation, distinguishing learnable patterns from random noise.
• It integrates explicit computation limits into information theory to resolve paradoxes related to deterministic transformations and data order.
• The framework provides actionable insights for data selection and curriculum design, optimizing training processes under fixed compute budgets.

Epiplexity is a formalization of informational content that quantifies what structural knowledge can be extracted from data by computationally bounded learners, distinguishing it from unstructured unpredictability that conventional information theory cannot separate. Unlike Shannon entropy or Kolmogorov complexity, which measure information under the assumption of unbounded computation, epiplexity introduces explicit time constraints, thus aligning information content with the actual capabilities of learning systems. This framework resolves longstanding paradoxes in information theory, guides principled data selection, and provides a rigorous basis for analyzing the relationship between data structure, computational constraints, and learnability (Finzi et al., 6 Jan 2026).

1. Motivation and Conceptual Foundations

Traditional information-theoretic approaches, including Shannon entropy $H(X)$ and Kolmogorov complexity $K(x)$, assume an observer with unbounded computational capacity. This results in three paradoxes in modern learning applications:

• Paradox 1: Information cannot be increased by deterministic transformations: The data-processing inequality ($H(f(X)) \leq H(X)$) and $K(f(x)) \leq K(x) + K(f) + O(1)$ suggest deterministic pipelines cannot introduce new structure, yet practical learning often extracts useful patterns via synthetic procedures, pseudorandom generation, and emergent phenomena in deterministic dynamical systems.
• Paradox 2: Information is independent of data order and factorization: Both entropy and Kolmogorov complexity are symmetric with respect to order, while neural models, cryptographic constructions, and sequential data exhibit direction-sensitive learnability.
• Paradox 3: Likelihood modeling is merely distribution matching: The minimizer of $\min_P \mathbb{E}_{X \sim Q}[ -\log P(X) ]$ is $P=Q$, which treats model learning as trivial matching, inconsistent with the empirical emergence of powerful inductive shortcuts and representations.

All these paradoxes stem from neglecting computational bounds—collapsing all decodable structure as equally trivial, even if adaptation would require infeasible computation.

2. Formal Definition of Epiplexity

Epiplexity explicitly incorporates computation time in measuring information content. Given a universal prefix Turing machine $\mathcal U$ and a time-constructible bound $T(n)$, define $\mathcal P_T$ as the set of all programs $\mathrm P$ that, in at most $T(n)$ steps, can evaluate probabilities and sample outputs for binary strings of length $n$.

The time-bounded two-part code minimizer is

$$\mathrm{P}^* = \arg\min_{\mathrm{P} \in \mathcal{P}_T} \left\{ |\mathrm{P}| + \mathbb{E}_{X}[ -\log \mathrm{P}(X) ] \right\}$$

where $|\mathrm{P}|$ is the length of the description of $\mathrm{P}$.

Define: $$\boxed{ \begin{aligned} \text{Epiplexity:} \quad & S_T(X) := |\mathrm{P}^*|, \ \text{Time-bounded entropy:} \quad & H_T(X) := \mathbb{E}_X[ -\log \mathrm{P}^*(X) ]. \end{aligned} }$$

$S_T(X)$ quantifies the minimal program description length (structural content) a $T$-bounded learner must absorb to model $X$. $H_T(X)$ measures residual unpredictability under this best model. Increasing available computation (raising $T$) can strictly decrease both, as more structure becomes recoverable.

3. Key Properties and Paradox Resolution

Epiplexity exhibits several properties that resolve the paradoxes noted in conventional theory:

• Nonnegativity and boundedness: $$0 \leq S_T(X), H_T(X),\ S_T(X) + H_T(X) \leq n + O(1)$$.
• Monotonicity in compute: If $T' \geq T$, $S_{T'}(X) + H_{T'}(X) \leq S_T(X) + H_T(X)$.
• Deterministic transformations may increase epiplexity: For a cryptographically secure PRG $G$ mapping $k$ to $n$ bits:

$$H_{\text{poly}}(G(U_k)) - H_{\text{poly}}(U_k) \approx n - k, \quad S_{\text{poly}}(G(U_k)) = O(1)$$

Thus, PRG output appears random to any poly-time observer, with increased time-bounded entropy but no increase in structural content.

• Order-dependence: For one-way permutations $f$, modeling $(X, f(X))$ and $(f(X), X)$ yields very different $S_T$ and $H_T$ values. Predicting chess boards from moves is easy, but inverting (moves from board) inflates $S_T$, aligning with model performance.
• Computational structure creation: Models trained via maximum likelihood under finite compute can invent algorithms and inductive shortcuts not required by the true data generator.

4. Illustrative Examples

Multiple synthetic domains demonstrate epiplexity’s discriminative power:

Setting	$S_T$	$H_T$
ECA Rule 15 (periodic)	$O(1)$	$O(1)$
ECA Rule 30 (chaotic)	$O(1)$	$n$
ECA Rule 54 (emergent structure)	$\Omega(n^\gamma)$ ($\gamma > 0$)	$O(n)$
Game of Life, one-step evolution	$O(1)$	-
Game of Life, multi-step ($k \gg n$)	$S_T \gg O(1)$, depending on structures	-
Masked Markov chain (easy induction)	$S_T$ peaks for $0 < h < 8$	-
Masked Rule 30 (hard induction)	$\Omega(h)$	Converges to $h$

Periodic and trivial evolutions yield low epiplexity, while chaotic or unpredictable processes are noise, with structureless randomness and high $H_T$. Emergent or “inductive” domains require programmatic structure to model efficiently—reflected in growing $S_T$.

5. Practical Estimation Schemes

Direct optimization over all $T$-bounded programs is infeasible. In practice, $S_T(X)$ and $H_T(X)$ are estimated via parametric families under compute budgets:

• Prequential Coding (AUC heuristic): Sequentially train a model on data $Z_1, \ldots, Z_M$, track per-step log-loss $\ell_i$, then compute

$$\widehat{S}_{\text{preq}} = \sum_{i=1}^M (\ell_i - \ell_M), \quad \widehat{H}_{\text{preq}} = M \ell_M$$

Optimizing $(N, D)$ (model size, tokens) under a time constraint traces out the compute-optimal two-part code.

• Requential Coding (Teacher–Student KL): Maintain a sequence of “teacher” models $P^{\text{t}}_i$; train “student” $P^{\text{s}}_i$ on synthetic teacher samples. For each token, code cost is $$\mathrm{KL}(P^{\text{t}}_i \Vert P^{\text{s}}_i) + O(1)$$. Summing yields $$|\mathrm{P}_{\text{req}}| \approx \sum_{i=1}^M \mathrm{KL}(P^{\text{t}}_i \Vert P^{\text{s}}_i)$$.

Prequential estimates are computationally cheaper, while requential coding provides a tighter upper bound. The compute-optimal tradeoff is found by sweeping $(N, D)$ and considering the lower convex hull in $(\text{code length},\ \text{compute})$ space.

6. Empirical Characterization Across Domains

Empirical results under fixed compute budgets ($\sim 6 \times 10^{18}$ FLOPs, up to 5 billion tokens) reveal:

• OpenWebText (language): $S_T \approx 10^7$ nats, $H_T \approx 10^{10}$ nats.
• Chess PGN: $S_T \approx 10^6$ nats.
• CIFAR-5M (pixels): $S_T \approx 10^4$ nats; almost all content is unpredictable noise.

Scaling to budgets of $10^{25}$ FLOPs and 1 trillion tokens, language retains the greatest structural epiplexity, with visual and video data trailing significantly.

Epiplexity correlates with practical performance. For instance, reordering chess (board-to-moves) results in higher $S_T$ and better zero-shot transfer. Adaptive Data Optimization for LLM pretraining (Jiang et al., 2025) increases prequential epiplexity, yielding superior out-of-distribution generalization on multiple benchmarks.

7. Implications for Data Selection and Learning

Epiplexity inverts the model-centric view typical of Minimum Description Length and related criteria. Rather than minimizing model code for a fixed dataset, it asks which data (under a fixed compute budget) induces the largest reusable structure in a learner:

• Data with higher $S_T$ contains richer, reusable “circuits” (Editor’s term), fostering transfer and generalization.
• Relying solely on in-distribution loss may select data that is merely entropic or redundant.
• Maximizing $S_T$ suggests new strategies for curriculum design, synthetic data generation, or curation, tailored to the concrete computational limits of a learning system.

A plausible implication is that epiplexity quantifies “learning potential” under budget constraints and gives a principled metric for evaluating and selecting training corpora in large-scale machine learning.

---

Epiplexity and its associated time-bounded entropy provide a comprehensive framework for measuring information as a resource relative to computational constraints, resolving longstanding limitations of classical theory and aligning data-centric learning with the realities of modern AI system design (Finzi et al., 6 Jan 2026).