The Effective Horizon Explains Deep RL Performance in Stochastic Environments (2312.08369v2)

Abstract: Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited function classes for representing the policy or value function. Our goal is to explain why deep RL algorithms often perform well in practice, despite using random exploration and much more expressive function classes like neural networks. Our work arrives at an explanation by showing that many stochastic MDPs can be solved by performing only a few steps of value iteration on the random policy's Q function and then acting greedily. When this is true, we find that it is possible to separate the exploration and learning components of RL, making it much easier to analyze. We introduce a new RL algorithm, SQIRL, that iteratively learns a near-optimal policy by exploring randomly to collect rollouts and then performing a limited number of steps of fitted-Q iteration over those rollouts. Any regression algorithm that satisfies basic in-distribution generalization properties can be used in SQIRL to efficiently solve common MDPs. This can explain why deep RL works, since it is empirically established that neural networks generalize well in-distribution. Furthermore, SQIRL explains why random exploration works well in practice. We leverage SQIRL to derive instance-dependent sample complexity bounds for RL that are exponential only in an "effective horizon" of lookahead and on the complexity of the class used for function approximation. Empirically, we also find that SQIRL performance strongly correlates with PPO and DQN performance in a variety of stochastic environments, supporting that our theoretical analysis is predictive of practical performance. Our code and data are available at https://github.com/cassidylaidlaw/effective-horizon.

• The paper introduces SQIRL, which separates exploration and learning to reveal the effective horizon's role in near-optimal deep RL performance.
• It establishes that k-QVI-solvability explains how minimal value iteration after random exploration can yield strong results in stochastic environments.
• Empirical results demonstrate that SQIRL matches prominent algorithms like PPO and DQN, broadening the application of neural network-based function approximators in RL.

Understanding Deep RL in Stochastic Environments

Background

Reinforcement learning (RL) has traditionally been guided by theoretical frameworks that focus on strategic exploration and minimax sample complexity bounds. However, these theories often do not translate into explaining the success of deep RL algorithms in practice, which typically employ random exploration and leverage expressive function approximators, such as neural networks. A critical challenge has been understanding the performance of these algorithms in stochastic environments.

Separating Exploration and Learning

A new paper introduces insights related to this challenge by considering the concept of the "effective horizon." The effective horizon is the number of lookahead steps needed for an algorithm to approximate the optimal decision-making process. The researchers introduce the SQIRL (shallow Q-iteration via reinforcement learning) algorithm. SQIRL separates the exploration and learning stages in RL by using random exploration to collect data and then applying regression and fitted Q-iteration for learning.

SQIRL only requires basic in-distribution generalization from collected samples, making it applicable alongside neural networks. This algorithm helps elucidate why random exploration seems to work well in practice, despite poor theoretical guarantees in worst-case scenarios.

Sample Complexity and Function Approximation

The findings advance understanding by establishing that many stochastic environments are well-explained by a property called k-QVI-solvability. This property indicates that acting greedily based on the Q-function of a random policy, or after minimal value iteration, yields near-optimal behavior. The paper provides instance-dependent sample complexity bounds for RL that depend on a stochastic version of the effective horizon and the function approximation class used.

Empirically, the research shows SQIRL can utilize a variety of function approximators, including least-squares regression on linear functions and neural networks. This flexibility reveals a significant theoretical expansion of environments where deep RL can be expected to perform well.

Empirical Validation

Empirically, the effectiveness of SQIRL is validated in various stochastic environments. The algorithm's performance is comparable to prominent deep RL algorithms like PPO and DQN. Moreover, environments with a lower effective horizon often see stronger results from deep RL methods, aligning with this new theoretical understanding.

Additionally, SQIRL's performance in scenarios like the BRIDGE environments and full-length Atari games demonstrates that the effective horizon is potentially a key factor in diverse settings. A substantial correlation between the performance of deep RL algorithms and SQIRL underpins the proposed theoretical foundations.

Conclusion

This work emphasizes the importance of the effective horizon and introduces the SQIRL algorithm as significant contributions to bridging the gap between deep RL theory and practice. While there are still cases where SQIRL falls short, its performance alignment with PPO and DQN suggests that these factors explain much of deep RL's effectiveness in stochastic environments. The outcomes of this paper present pathways for future research to further refine our understanding and application of deep RL.