Hierarchies of Novelty & Open-Endedness

• Hierarchies of novelty and open-endedness are defined as evolving multi-level systems that generate unprecedented configurations through recursive combinatorial processes.
• They integrate algorithmic complexity with statistical methods, revealing scaling laws like Zipf’s law that characterize innovation in biological, technological, and artificial systems.
• This framework offers practical insights for designing systems that sustain open-ended evolution by preserving deep generative rules amid increasing entropy.

Hierarchies of novelty and open-endedness arise in evolving systems—biological, technological, and artificial—through mechanisms that allow for the continual introduction of unprecedented configurations, structures, or functions across multiple levels of organization. These hierarchies are characterized by the systematic accumulation of innovative forms, often via recursive processes that expand and reorganize the available state space, and are subject to both statistical and algorithmic descriptions. Recent research rigorously formalizes these phenomena, revealing that open-ended evolution (OEE) not only entails unbounded complexity growth but also a structured cascade of escalating novelty, each level building on or combining earlier ones.

1. Algorithmic Foundations of Open-Endedness

Open-ended evolution is defined formally via the growth of descriptive (algorithmic) complexity over time. Let the system state at time $t$ be encoded as a string $\sigma_t$ and assessed by its Kolmogorov complexity:

$$K(\sigma_t) = \min_p \{\ell(p) : T_U(p) = \sigma_t\},$$

where $T_U$ is a universal Turing machine and $\ell(p)$ is the length of program $p$ generating $\sigma_t$. Two key postulates specify the conditions for open-endedness:

• Non-decreasing normalized complexity:

$$\frac{K(\Sigma(t))}{t} \leq \frac{K(\Sigma(t+1))}{t+1},$$

where $\Sigma(t)$ is the concatenation or record of all historical states up to $t$.

• Unbounded complexity:

$$\forall N \in \mathbb{N}, \; \exists t \text{ s.t. } \frac{K(\Sigma(t))}{t} > N.$$

These formal conditions imply that, for any time horizon, evolution introduces further unpredictability: there is always a possible expansion or combination not previously realized.

A principle of minimal change underlies evolutionary stepwise novelty, ensuring that while each descendant state is as similar as possible to its predecessor (to minimize “change cost”), the system as a whole accumulates nonredundant and unencoded innovations. This is made precise via the minimization of conditional complexity:

$$S(\sigma_t \rightarrow \sigma_{t+1}) = K(\sigma_{t+1}\mid \sigma_t).$$

The cumulative effect is a hierarchical record where each state potentially encodes new, unprecedented, or higher-level combinations of features, reflecting a history-dependent structure.

2. Statistical Regularities and Zipf's Law

The statistical analog of this algorithmic growth leverages Shannon entropy to approximate the complexity of system states:

$$K(\sigma_t) \approx H(X_t),\quad H(X_t) = -\sum_k p_t(k)\log p_t(k),$$

where $X_t$ is the system’s random state variable.

To model incremental novelty, the system expands its state space by adding new categories with minimal entropy increases. The iterative probabilistic update is

$$p_{n+1}(k) = \theta_{n+1} p_n(k),\quad k\leq n;\qquad p_{n+1}(n+1) = 1-\theta_{n+1},$$

for $0<\theta_{n+1}<1$.

The stationary distribution arising from this variational process is a power law:

$p_n(i) \propto \frac{1}{i},$

which is Zipf’s law. This outcome indicates that in OEE-like systems with combinatorial generativity, a small number of components become ubiquitous while the majority remain rare, a feature that holds across domains such as language (words), protein domains, and even technological artifacts. The power-law scaling is thus a statistical signature of the underlying hierarchical combinatorics of novelty production.

3. The Paradox of Information Conservation

An apparent paradox emerges when tracking information through statistical (Shannon) mutual information. The stepwise mutual information between consecutive states,

$$I(X_{n+1}:X_n) = H(X_{n+1}) - H(X_{n+1}\mid X_n) = \theta_{n+1} H(X_n),$$

can be significant. However, over long histories, the mutual information between distant timepoints vanishes:

$$I(X_m:X_n) \leq \frac{C_m}{C_n}H(X_m), \qquad \lim_{n\to\infty}I(X_m:X_n)=0,$$

since the normalization $C_n\to\infty$ as $n\to\infty$ in systems with unbounded complexity.

This “information loss paradox” means that as the system grows in entropy (and thus “information”), the present state becomes statistically uninformative about the distant past. This is resolved by recognizing that Shannon theory only measures ensemble-level (statistical) correlations, which become diffused. In contrast, algorithmic (non-statistical) information is preserved in the generative rules themselves:

$$I(\sigma_N:\sigma_n) = K(\sigma_N) - K(\sigma_N\mid\sigma_n),\qquad \lim_{n\to\infty}I(\sigma_n:\sigma_m) \approx K(\sigma_m).$$

Thus, while statistical observables “forget” the past, the underlying programs or generative mechanisms keep an indelible record—historical memory persists at the algorithmic level, enabling the recursive and hierarchical emergence of further novelty.

4. Hierarchical Structuring of Novelty

Hierarchies of novelty in OEE systems arise as each new state is not merely an extension, but may represent unique compositions or abstractions over prior states. Key aspects:

• Combinatorial generativity: Each novel state may result from unprecedented combinations or modifications of existing components, recursively building ever-richer structures.
• History-dependence: The cumulative sequence of states encodes a path-dependent hierarchy, with higher levels incorporating and reorganizing prior levels' building blocks. Though lower-level details may statistically “wash out,” they remain latent in the higher-level generative procedures.
• Power-law usage: The frequency of use of basic elements inherits a hierarchical structure, with dominant “core” elements at lower levels and rare, idiosyncratic configurations emerging continuously at higher levels.
• Persistence of rules: Generative rules (programs) encode the mechanisms of novelty induction, meaning that even as individual features are replaced or recombined, the system's capacity for further innovation is preserved or increased.

Such hierarchical novelty is observed in biological evolution (where gene duplications, domain shuffling, and exaptations produce new modules), technological systems (where new devices layer upon previous technologies), and linguistic evolution (with new syntactic constructs built from core vocabularies).

5. Theoretical Implications and Limitations of Statistical Descriptions

The findings support the view that open-ended evolutionary dynamics cannot be fully captured by statistical information theory alone. While scaling laws and power-law distributions reflect the outward statistical signature of OEE, only Algorithmic Information Theory adequately accounts for the preservation and elaboration of generative potential and “deep memory” in such systems.

Shannon theory captures how patterns and regularities scale with system growth, but cannot describe how an evolving system retains, internalizes, and leverages past innovations in the face of continual expansion. Algorithmic approaches, by contrast, model the rules, programs, or mechanisms underlying hierarchical novelty production.

The necessity of both viewpoints is clear for real-world complex systems: statistical approaches explain universally observed scaling phenomena (e.g., Zipf's law), while algorithmic approaches explain persistence of evolutionary potential and the capacity to continually build hierarchies of novelty.

6. Summary Table: Algorithmic vs. Statistical Signatures of Open-Endedness

Aspect	Algorithmic Information Theory	Statistical (Shannon) Information Theory
Complexity metric	Kolmogorov complexity $K(\sigma_t)$	Entropy $H(X_t)$
Memory of history	Algorithmic mutual information persists	Mutual information decays over time
Novelty induction	Minimize conditional complexity	Minimize conditional entropy
Signature of hierarchy	Recursive expansion of program size/rules	Zipf’s law in event frequencies
Limitation	Not observable without access to rules	Cannot account for persistent generative rules

7. Generalization and Applications

The presented framework offers a rigorous way to distinguish and quantify open-ended, hierarchically novel processes in any domain—biological, technological, or artificial. It predicts universal scaling laws and explains why persistent innovation must rely on mechanisms that preserve generative history even as the surface-level details become increasingly “randomized” at scale. The integration of algorithmic with statistical perspectives is essential for understanding and designing systems (natural or artificial) that reliably produce, retain, and scaffold hierarchies of novelty across unbounded timeframes (Corominas-Murtra et al., 2016).