Driven by Compression Progress: A Simple Principle Explains Essential Aspects of Subjective Beauty, Novelty, Surprise, Interestingness, Attention, Curiosity, Creativity, Art, Science, Music, Jokes (0812.4360v2)

Abstract: I argue that data becomes temporarily interesting by itself to some self-improving, but computationally limited, subjective observer once he learns to predict or compress the data in a better way, thus making it subjectively simpler and more beautiful. Curiosity is the desire to create or discover more non-random, non-arbitrary, regular data that is novel and surprising not in the traditional sense of Boltzmann and Shannon but in the sense that it allows for compression progress because its regularity was not yet known. This drive maximizes interestingness, the first derivative of subjective beauty or compressibility, that is, the steepness of the learning curve. It motivates exploring infants, pure mathematicians, composers, artists, dancers, comedians, yourself, and (since 1990) artificial systems.

• The paper demonstrates that improvements in data compression serve as measurable intrinsic rewards for both biological and artificial agents.
• It utilizes a reinforcement learning framework where curiosity is quantified through gains in prediction accuracy and data understanding.
• The study’s insights pave the way for autonomous systems that explore and learn effectively in environments with limited external rewards.

An Analysis of "Driven by Compression Progress"

The paper by Jürgen Schmidhuber presents a theoretical framework linking data compressibility to various facets of cognitive processes such as novelty, attention, creativity, and curiosity. The central thesis posits that the ability to compress data efficiently is integral to generating intrinsic motivation in agents, both biological and artificial.

Core Principle

The framework suggests that a self-improving agent finds certain data interesting when it begins to predict or compress that data more efficiently. This process leads to increased subjective simplicity and perceived beauty. The intrinsic motivation or curiosity reward is quantitatively tied to the progress made in data compression.

Theoretical Implementation

Schmidhuber outlines a formal approach where curiosity is measured in terms of the improvement in data compression or prediction accuracy. The paper introduces a reinforcement learning (RL)-based framework where agents seek actions that maximize their understanding of the environment through intrinsic rewards, even in the absence of external stimuli.

The framework employs a predictive or compressive model to interact with the environment. Improvements in model predictions serve as a metric for intrinsic rewards, guiding the agent’s exploration strategy. This is a departure from traditional RL approaches that rely primarily on external rewards.

Algorithmic Design

The methodology includes:

1. History Recording: Storing all interaction data to enable comprehensive learning.
2. Compressor Usage: Applying models to improve data compressibility, thus enhancing predictive capabilities.
3. Curiosity Reward Calculation: Generating intrinsic rewards based on measurable improvements in data compression.
4. Intrinsic Motivation Optimization: Using an RL framework to optimize curiosity-driven exploration.

Implications and Applications

This principle has profound implications for developing intelligent systems capable of self-motivated learning. By embedding curiosity as a fundamental drive, artificial systems can achieve a deeper understanding of their environments without predefined objectives.

Such systems could show efficacy in problem domains where external rewards are sparse or difficult to define, stimulating advancements in areas like autonomous robotics, creative AI, and developmental learning frameworks.

Future Directions

The paper suggests potential enhancements in adaptive compressor designs and more efficient algorithms for measuring learning progress. Exploration into improved RL techniques and their integration into this framework could foster systems that not only mimic human exploratory behavior but also surpass it in many domains.

Concluding Remarks

Schmidhuber’s theory provides a robust paradigm for understanding and developing intrinsic motivators in AI systems. The link between compression progress and curiosity not only deepens our understanding of cognitive processes but also offers a scalable path towards more autonomous and self-sufficient artificial agents. The framework’s capacity to unify concepts across disciplines suggests a promising direction for future AI research and its applications.