Exploration by Random Network Distillation (1810.12894v1)

Abstract: We introduce an exploration bonus for deep reinforcement learning methods that is easy to implement and adds minimal overhead to the computation performed. The bonus is the error of a neural network predicting features of the observations given by a fixed randomly initialized neural network. We also introduce a method to flexibly combine intrinsic and extrinsic rewards. We find that the random network distillation (RND) bonus combined with this increased flexibility enables significant progress on several hard exploration Atari games. In particular we establish state of the art performance on Montezuma's Revenge, a game famously difficult for deep reinforcement learning methods. To the best of our knowledge, this is the first method that achieves better than average human performance on this game without using demonstrations or having access to the underlying state of the game, and occasionally completes the first level.

• The paper introduces Random Network Distillation, a novel approach that uses prediction error from a fixed random network to generate exploration bonuses.
• It seamlessly integrates intrinsic and extrinsic rewards through a dual-head PPO setup, enhancing exploration efficiency in sparse reward environments.
• Benchmark evaluations, especially on Montezuma's Revenge, show that RND achieves state-of-the-art performance and improved scalability in hard exploration tasks.

Exploration by Random Network Distillation

The paper "Exploration by Random Network Distillation" introduces a novel exploration bonus for deep reinforcement learning (DRL) methods that is simple to implement and computationally efficient. This method, termed Random Network Distillation (RND), leverages the error of a neural network in predicting the features of observations from a fixed, randomly initialized neural network to generate the exploration bonus.

Key Contributions

RND provides several groundbreaking methodological contributions:

1. Simplicity and Efficiency: The exploration bonus is computed using a single forward pass of a neural network, ensuring minimal computational overhead.
2. Flexible Reward Integration: The paper introduces a method to seamlessly combine intrinsic (exploration bonus) and extrinsic (environment reward) rewards, addressing the non-episodic nature of intrinsic rewards.
3. Benchmark Performance: RND establishes state-of-the-art performance on the challenging Atari game Montezuma's Revenge without relying on expert demonstrations or underlying game state access, achieving results surpassing average human performance and occasionally completing the first level.

Methodology

Exploration Bonus

The RND bonus is defined as the error in predicting the output of a fixed, randomly initialized network (target network) by a predictor network trained on the agent's observations. This approach relies on the observation that neural networks exhibit lower prediction errors on familiar inputs, thus higher errors indicate novel states.

Combining Rewards

To dynamically balance intrinsic and extrinsic rewards, the authors propose a dual head approach in the Proximal Policy Optimization (PPO) algorithm, where separate value functions are used for the different reward streams. This allows for varied discount rates and the integration of episodic and non-episodic returns.

Experimental Evaluation

The experimental setup involved extensive ablations on Montezuma's Revenge to isolate the effects of RND and to understand the interplay between intrinsic and extrinsic rewards. Moreover, performance was evaluated across six hard exploration Atari games: Freeway, Gravitar, Montezuma's Revenge, Pitfall!, Private Eye, and Solaris.

Notable Results

1. Montezuma's Revenge: RND combined with a dual-head PPO agent consistently explored over half of the rooms, achieving a mean episodic return exceeding most prior works. The best agent discovered 22 out of 24 rooms on the first level, occasionally completing it.
2. Scalability: The method scaled well with the number of parallel environments, showing improved exploration efficiency and stability with increased environments. This was particularly evident with the recurrent policy (RNN) performance.

Comparative Analysis

The RND approach was benchmarked against PPO without intrinsic rewards, and an alternative exploration method based on forward dynamics error. The RND agent demonstrated superior performance in Montezuma's Revenge, Venture, and Private Eye compared to these baselines.

Implications and Future Directions

Practical Implications

RND bridges a crucial gap in exploration for DRL agents, particularly in sparse reward environments. Its simplicity and computational efficiency make it a viable choice for real-world applications where scalability and resource limitations are significant concerns.

Theoretical Implications

This method offers a robust mechanism to address the 'noisy-TV' problem, common with methods that maximize prediction errors in stochastic environments. By using a deterministic target network, RND mitigates the attraction to unpredictable state transitions.

Future Developments

The promising results open avenues for further research in several directions:

1. Global Exploration Strategies: Addressing more sophisticated exploration problems that require long-term planning and coordination over extended time horizons.
2. Cross-Domain Validation: Applying RND to different DRL benchmarks beyond Atari games to validate its generalizability.
3. Combining With Hierarchical Methods: Integrating RND with hierarchical reinforcement learning to tackle tasks with complex sub-goals and layered challenges.

Conclusion

The Random Network Distillation approach represents a substantial advancement in exploration methods for DRL. By ensuring efficient computation and effective exploration in sparse reward settings, RND enhances both the practical applicability and performance of DRL agents, setting a new standard for future research and development in this domain.