Improving Intrinsic Exploration by Creating Stationary Objectives (2310.18144v4)

Abstract: Exploration bonuses in reinforcement learning guide long-horizon exploration by defining custom intrinsic objectives. Several exploration objectives like count-based bonuses, pseudo-counts, and state-entropy maximization are non-stationary and hence are difficult to optimize for the agent. While this issue is generally known, it is usually omitted and solutions remain under-explored. The key contribution of our work lies in transforming the original non-stationary rewards into stationary rewards through an augmented state representation. For this purpose, we introduce the Stationary Objectives For Exploration (SOFE) framework. SOFE requires identifying sufficient statistics for different exploration bonuses and finding an efficient encoding of these statistics to use as input to a deep network. SOFE is based on proposing state augmentations that expand the state space but hold the promise of simplifying the optimization of the agent's objective. We show that SOFE improves the performance of several exploration objectives, including count-based bonuses, pseudo-counts, and state-entropy maximization. Moreover, SOFE outperforms prior methods that attempt to stabilize the optimization of intrinsic objectives. We demonstrate the efficacy of SOFE in hard-exploration problems, including sparse-reward tasks, pixel-based observations, 3D navigation, and procedurally generated environments.

• The paper introduces the SOFE framework, which transforms non-stationary intrinsic rewards into stationary objectives by augmenting state representations.
• It demonstrates improved exploration and state coverage in sparse-reward tasks across maze, continuous, and high-dimensional environments.
• The findings imply that simple state augmentation can yield robust RL performance without the need for complex policy decoupling.

Improving Intrinsic Exploration by Creating Stationary Objectives

The paper "Improving Intrinsic Exploration by Creating Stationary Objectives" introduces the Stationary Objectives For Exploration (SOFE) framework, which aims to tackle the non-stationarity in intrinsic reward mechanisms used for exploration in reinforcement learning (RL). This non-stationarity emerges due to the dynamic nature of count-based exploration methods and similar approaches that guide agents through sparse-reward and long-horizon tasks.

Problem Statement

Intrinsic objectives are prevalent in RL to enhance exploration, especially in environments with sparse or no rewards. Count-based exploration methods that rely on state-visitation frequencies inherently exhibit non-stationary behavior. As the dynamics of these counts evolve with the agent's interactions, they induce a partially observable Markov decision process (POMDP). Within such a framework, optimal Markovian policies—central to most RL paradigms—are often unattainable, complicating the learning process and undermining policy performance. Existing solutions, including those that decouple exploration and exploitation policies, add complexity without adequately resolving the core issue.

The SOFE Framework

SOFE addresses the problem by transforming non-stationary rewards into stationary ones through augmented state representations. This process involves identifying sufficient statistics for various exploration methods and encoding them efficiently as inputs to a deep network. By expanding the state space with relevant statistics like state-visit frequencies or pseudo-counts, SOFE enables a simpler optimization landscape. The authors hypothesize that RL agents generalize better over these augmented state spaces than under non-stationary reward conditions.

Key components of SOFE include:

• State Augmentation: Encodes sufficient statistics directly into the state representation, thereby converting the intrinsic objectives into stationary ones.
• Simplified Optimization: Facilitates tractable, end-to-end training across intrinsic and extrinsic objectives, eliminating the need for complex separate-policy architectures.

Experimental Evaluation

The paper details a series of evaluations across hard-exploration tasks to validate the proposed approach. SOFE demonstrates marked improvements over baseline methods, including enhanced state coverage and policy performance in tasks characterized by sparse rewards:

1. Count-based Methods: In various maze environments, SOFE improves exploratory behaviors by leveraging state-visit frequency augmentations, resulting in higher state coverage and more optimal navigation policies.
2. Continuous Domains: In a 3D navigation task, SOFE scales effectively, presenting improved map exploration.
3. High-Dimensional Contexts: When integrated with advanced exploration algorithms like E3B, SOFE shows improved performance on procedurally generated environments such as Minihack and Procgen-Maze, where visual complexity poses additional challenges.

Implications and Future Work

The effectiveness of SOFE in stabilizing intrinsic reward optimization holds significant implications for designing robust exploration strategies in RL. It suggests a pathway for designing exploration strategies that maintain a stable learning target even as environmental complexities increase. The framework's simplicity, requiring only state augmentation rather than complex architecture changes, underscores its potential integration ease across diverse RL ecosystems.

Future research may explore broader applications of SOFE, expanding beyond RL or even integrating with model-based RL strategies. Further examination could also assess the framework's adaptability across various POMDPs and its influence on the learned representations, paving the way for deeper theoretical insights into exploration-exploitation dynamics within RL.