Unifying Count-Based Exploration and Intrinsic Motivation (1606.01868v2)

Abstract: We consider an agent's uncertainty about its environment and the problem of generalizing this uncertainty across observations. Specifically, we focus on the problem of exploration in non-tabular reinforcement learning. Drawing inspiration from the intrinsic motivation literature, we use density models to measure uncertainty, and propose a novel algorithm for deriving a pseudo-count from an arbitrary density model. This technique enables us to generalize count-based exploration algorithms to the non-tabular case. We apply our ideas to Atari 2600 games, providing sensible pseudo-counts from raw pixels. We transform these pseudo-counts into intrinsic rewards and obtain significantly improved exploration in a number of hard games, including the infamously difficult Montezuma's Revenge.

• The paper presents pseudo‐counts derived from density models to extend count-based exploration to non-tabular, high-dimensional reinforcement learning settings.
• It bridges intrinsic motivation and traditional exploration by linking pseudo‐counts to prediction gain, yielding notable performance gains on Atari benchmarks.
• The proposed framework enhances scalability in deep RL and sparks new research directions for integrating exploration strategies in continuous domains.

Unifying Count-Based Exploration and Intrinsic Motivation

The paper "Unifying Count-Based Exploration and Intrinsic Motivation" by Bellemare et al. addresses a crucial aspect of reinforcement learning (RL): exploration. The authors present a novel approach that bridges the gap between count-based exploration and intrinsic motivation, introducing the concept of pseudo-counts to extend count-based methods to non-tabular settings.

Core Contributions and Methodology

1. Generalizing Count-Based Methods: Traditional RL methods that rely on visit counts for exploration are theoretically grounded but impractical for large, high-dimensional state spaces. Tabular visit counts become ineffective because states are rarely revisited. The authors address this limitation by proposing pseudo-counts derived from density models, thereby generalizing count-based exploration techniques for non-tabular environments.
2. Pseudo-Counts from Density Models: The crux of the paper is the derivation of pseudo-counts from an arbitrary density model. This pseudo-count quantifies an agent's uncertainty about states in a high-dimensional space. The authors relate pseudo-counts to a density model's prediction gain, offering a computationally feasible way to determine exploration bonuses.
3. Theoretical Foundations: The authors connect intrinsic motivation, specifically prediction gain, and count-based exploration. They demonstrate that pseudo-counts can approximate information gain, traditionally used to drive intrinsic motivation. This finding provides a solid theoretical basis for applying pseudo-counts to guide exploration in complex environments.
4. Empirical Validation: The proposed method is empirically validated on the Atari 2600 platform. By implementing pseudo-counts within an existing exploration algorithm (MBIE-EB), the authors demonstrate significant performance improvements on notoriously difficult games, including "Montezuma's Revenge." The presented results show that agents using pseudo-count-based exploration bonuses explore more effectively and achieve higher scores in a fraction of the training time compared to other methods.

Implications and Future Directions

Theoretical and practical implications of this research are far-reaching:

1. Scalability: By extending count-based methods to high-dimensional, non-tabular settings, the paper offers a scalable solution for RL in complex environments. This contribution is critical for the development of algorithms that can effectively explore and learn from high-dimensional inputs such as images.
2. Integration with Intrinsic Motivation: Unifying count-based exploration with intrinsic motivation lays the groundwork for more robust exploration strategies. This integration harnesses the strengths of both approaches, potentially leading to more efficient learning algorithms.
3. Applications to Modern RL Algorithms: The paper's approach can be readily integrated into modern deep RL algorithms, such as DQN and A3C. The authors provide compelling evidence that pseudo-counts enhance exploration efficacy in these settings, prompting further research into their application across various RL frameworks.
4. Future Research: The authors suggest several promising directions, such as examining the induced metrics of density models, designing value functions compatible with density models, and extending the concept of pseudo-counts to continuous spaces. These avenues hold potential for significant advancements in efficient exploration and optimal learning.

Conclusions

Bellemare et al. contribute a novel framework that unifies count-based exploration and intrinsic motivation through pseudo-counts derived from density models. This innovative approach addresses key limitations of traditional count-based methods, enabling their application in high-dimensional, non-tabular settings. The theoretical foundations and empirical validations presented in the paper establish pseudo-counts as a powerful tool for efficient exploration in reinforcement learning. The implications for future research and practical applications in RL are substantial, marking a significant step forward in the development of robust, scalable exploration strategies.