Sample Efficient Deep Reinforcement Learning via Local Planning (2301.12579v2)

Abstract: The focus of this work is sample-efficient deep reinforcement learning (RL) with a simulator. One useful property of simulators is that it is typically easy to reset the environment to a previously observed state. We propose an algorithmic framework, named uncertainty-first local planning (UFLP), that takes advantage of this property. Concretely, in each data collection iteration, with some probability, our meta-algorithm resets the environment to an observed state which has high uncertainty, instead of sampling according to the initial-state distribution. The agent-environment interaction then proceeds as in the standard online RL setting. We demonstrate that this simple procedure can dramatically improve the sample cost of several baseline RL algorithms on difficult exploration tasks. Notably, with our framework, we can achieve super-human performance on the notoriously hard Atari game, Montezuma's Revenge, with a simple (distributional) double DQN. Our work can be seen as an efficient approximate implementation of an existing algorithm with theoretical guarantees, which offers an interpretation of the positive empirical results.

• The paper introduces UFLP, a framework that uses state-history buffers and uncertainty-based resets to enhance sample efficiency in simulators.
• It demonstrates super-human performance on tasks like Montezuma’s Revenge by integrating UFLP with a distributional double DQN.
• Empirical results across benchmarks such as Deep Sea and Cartpole Swingup validate UFLP’s capacity to reduce sample costs in sparse-reward environments.

Sample Efficient Deep Reinforcement Learning via Local Planning

The paper presents a framework named Uncertainty-First Local Planning (UFLP) to enhance the sample efficiency of Deep Reinforcement Learning (RL) when used with simulators. The authors highlight the utilitarian aspect of simulators, which can easily reset the environment to previously observed states. By leveraging this property, UFLP significantly optimizes sample usage through a novel approach of state revisitation based on uncertainty.

The UFLP framework modifies the agent-environment interaction by defaulting the data collection process not to commence from a randomly initialized state, but rather from a state-history buffer. With probability 1 - $\epsilon$, UFLP identifies states with high uncertainty to initiate subsequent exploration. This method adopts a strategic starting point, thereby enhancing exploration efficiency and curtailing unnecessary sample consumption, particularly in environments requiring arduous exploration.

A pivotal example of UFLP's efficacy is demonstrated in the challenging Atari game, Montezuma's Revenge. The authors report that the UFLP exhibited super-human performance integrating a simple distributional double DQN. This aligns with the main goal of the research, which is to furnish a methodology translating theoretical RL efficiency such as core-set state revisitation into a pragmatic implementation with tangible success cases.

The theoretical comparison between local and traditional online access in reinforcement learning scenarios is noted. Standard RL algorithms mimic real-world learning protocols, with agents interacting dynamically without direct state revisitation. Conversely, UFLP's local access harnessing historical states reflects a pragmatic adjustment to simulator-based learning paradigms.

Several baseline RL algorithms were enriched using UFLP, showing a notable reduction in sample costs in exploration-persistent environments like the Deep Sea and Cartpole Swingup tasks of bsuite, and other challenging Atari games like PrivateEye and Venture. The paper further details specific implementation aspects encompassing base agents—employing Double DQN, bootstrapped DDQN, and policy iteration—and their respective uncertainty metrics like ensemble predictions, feature covariance, and random network distillation.

Empirical results demonstrate how local access can markedly improve the sample efficiency for various RL agents, as local sampling from high-uncertainty states enhances exploration in sparse-reward environments. The UFLP framework facilitated unprecedented exploration perspectives, driven by the capability to reset environments to previously important checkpoints, proving beneficial for tasks with non-trivial exploration.

The implications of these findings are dual-fold. Practically, the proposal delineates a tangible performance enhancement methodology for existing RL models, whereas theoretically, it extends the analytical boundaries and utility of simulators within the loop of RL research. This introduces prospects for future studies, particularly in refining uncertainty modeling and extending UFLP's principles into partially observable or stochastic domains.

In the landscape of RL research, this work positions simulators not just as testbeds but as strategic enablers for efficient exploration, challenging preconceived boundaries of trial-based learning protocols. The proposed UFLP framework emphasizes an intersection of strategy-driven exploration and algorithmic efficiency, providing a substantive contribution to both the RL theory and its application.