Predictive representations: building blocks of intelligence (2402.06590v3)

Abstract: Adaptive behavior often requires predicting future events. The theory of reinforcement learning prescribes what kinds of predictive representations are useful and how to compute them. This paper integrates these theoretical ideas with work on cognition and neuroscience. We pay special attention to the successor representation (SR) and its generalizations, which have been widely applied both as engineering tools and models of brain function. This convergence suggests that particular kinds of predictive representations may function as versatile building blocks of intelligence.

• The paper demonstrates that predictive representations, especially the successor representation, enable efficient learning and quick adaptation to new reward structures.
• It introduces generalizations such as successor features to enhance transfer learning, exploration, and hierarchical reinforcement learning.
• Findings converge across reinforcement learning, cognitive science, and neuroscience, underscoring predictive representations as foundational to intelligence.

Predictive Representations: Building Blocks of Intelligence

This paper explores the key concept that predictive representations are fundamental components of intelligence, integrating insights from reinforcement learning (RL), cognitive science, and neuroscience. The primary focus is on the successor representation (SR) and its generalizations, which serve as tools not only in engineering applications but also as models to understand brain function. The convergence of findings from different disciplines indicates that certain predictive representations might be indispensable for intelligent behavior, both artificial and biological.

Theoretical Foundations

The theory underpinning this work rests on distinguishing between predictive models and predictive representations. A predictive model describes a probability distribution over the dynamics of a system's state, facilitating predictions regarding future trajectories. Such models, though flexible, are computationally expensive when posed with complex queries, especially under time constraints. Predictive representations offer a more efficient alternative by caching the answers to specific queries, thus trading off some flexibility for computational speed.

In the framework of RL, the paper highlights the SR. The SR captures expected future state occupancy given the current state and policy. It is computationally efficient and supports quick adaptation to changes in the reward structure, although it is less flexible than fully predictive models. The SR and its generalizations, like successor features (SFs) and successor models (SMs), enable versatile applications in RL tasks, including exploration, transfer learning, temporal abstraction, and coordination among multiple agents.

Strong Numerical and Conceptual Results

The paper evaluates different classes of RL algorithms—model-based, model-free, and those relying on the SR. Each approach has its strengths and weaknesses:

• Model-based algorithms: Flexible but computationally expensive, as they involve simulating complete trajectories.
• Model-free algorithms: Efficient but less adaptable since they cache summary statistics like the value function.
• Predictive representations: Combine some advantages of both by caching more complex predictive structures like the SR or SFs, facilitating adaptation to new reward structures while remaining computationally efficient.

The SR's primary advantage is illustrated by its BeLLMan equation, which enables TD updates to learn predictive representations efficiently. For instance, once the SR is learned, re-evaluating the value function under new reward functions becomes linear in complexity. This efficiency is critical in practical applications where adaptation speed is paramount.

Applications and Implications

Practical Applications

1. Exploration: The SR supports methods like count-based exploration and balancing exploration with exploitation, thereby enhancing agents' ability to discover and learn about their environments efficiently.
2. Transfer Learning: Using SFs and Generalized Policy Improvement (GPI) methods, the SR facilitates transferring knowledge across tasks with varying reward structures. This transferability is essential for agents operating in dynamic, multi-task environments.
3. Hierarchical RL: The SR's predictive nature makes it suitable for hierarchical RL, where agents can learn and execute temporally extended actions (options) that are crucial for solving complex tasks.
4. Multi-Agent Coordination: Predictive representations allow agents to model and anticipate the actions of other agents, fostering coordination and collaborative behavior in multi-agent systems.

Cognitive and Neuroscientific Insights

1. Revaluation Studies: Experiments using revaluation paradigms indicate that humans and animals employ predictive representations akin to the SR, especially in tasks where reward structures change. Human and rodent navigation studies further support the SR's role in cognitive maps used for spatial reasoning.
2. Hippocampal Function: The hippocampus likely encodes predictive maps similar to the SR. Grid cells and place cells in the hippocampus exhibit firing patterns that align with the SR and its eigenvectors, suggesting a neural basis for predictive representations.
3. Learning Mechanisms: Biologically plausible learning rules, such as those based on spike-timing dependent plasticity (STDP), can realize predictive representations like the SR in neural circuits. Offline learning mechanisms, including replay, may support efficient SR updates by consolidating experiences during sleep or rest periods.
4. Dopamine and Prediction Errors: The involvement of dopamine in signaling prediction errors extends beyond scalar reward predictions to encompass vector-valued errors used for learning SFs. This aligns with observed dopaminergic responses to novel and sensory-predictive features.

Future Directions

Predictive representations will likely remain a central theme in the development of AI and our understanding of intelligence. Future research might focus on:

• Enhancing the scalability and generalization capabilities of predictive representations in more complex, high-dimensional environments.
• Investigating hierarchical and multi-scale predictive representations to better model temporal and spatial abstractions.
• Expanding the integration of neuroscientific insights into the design of biologically inspired RL algorithms.

In conclusion, predictive representations like the SR are pivotal in bridging the gap between flexible, model-based predictions and efficient, model-free computations, offering a powerful framework for both artificial and biological intelligence. This convergence across disciplines underscores their fundamental role in the architecture of intelligent systems.