What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2006.05990v1)

Abstract: In recent years, on-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. While RL algorithms are often conceptually simple, their state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. Those choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. This makes it hard to attribute progress in RL and slows down overall progress [Engstrom'20]. As a step towards filling that gap, we implement >50 such ``choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study. We train over 250'000 agents in five continuous control environments of different complexity and provide insights and practical recommendations for on-policy training of RL agents.

• The paper introduces a unified framework evaluating over 50 design choices in on-policy RL to pinpoint critical performance determinants.
• The paper shows that policy network initialization and robust advantage estimation (e.g., GAE) significantly accelerate learning speed and stability.
• The paper finds that input normalization and optimizer settings, including using Adam with a 0.0003 learning rate, improve sample efficiency and training robustness.

Insights into On-Policy Reinforcement Learning: An Empirical Examination of Design Choices

The paper "What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study" by Andrychowicz et al. provides a thorough investigation into the myriad of design decisions inherent in implementing on-policy reinforcement learning (RL) algorithms. The paper meticulously analyzes over 50 such decisions in a unified framework, training more than 250,000 agents across diverse continuous control environments, culminating in practical insights for enhancing on-policy RL training.

The central focus of this work is on dissecting the influence of various implementation choices on the performance of RL agents. This is crucial because many state-of-the-art RL implementations rely on complexities not explicitly laid out in the academic literature, thereby obfuscating the true sources of performance improvements.

Key Contributions and Results

1. Unified Implementation Framework: The paper introduces a comprehensive framework, allowing for broad exploration of high- and low-level choices that affect RL performance. This includes architectural choices, hyperparameter tuning, and network initialization schemes, all of which are systematically varied and assessed.
2. Impact of Network Initialization: One of the significant findings of this paper is the impact of network initialization on agent performance. It is noted that the initialization scheme, particularly for the policy network, can substantially influence learning speed and efficacy. The authors recommend initializing the policy such that the initial action distribution has zero mean with a relatively low standard deviation.
3. Advantage Estimation and Policy Losses: The analysis also highlights the importance of using robust advantage estimation strategies, like Generalized Advantage Estimation (GAE), which balances bias and variance effectively. Additionally, Proximal Policy Optimization (PPO) is favored among the policy loss functions due to its resilience to hyperparameter sensitivity and ability to facilitate stable policy updates.
4. Normalization Strategies: Normalization of inputs, particularly observation normalization, is shown to be critical for performance enhancement in most environments. Likewise, normalization of value function targets presents mixed results, aiding in some scenarios while hindering in others, indicating the need for task-specific tuning.
5. Regularization and Clipping: The paper finds limited evidence supporting the regularization of policies except for certain tasks like HalfCheetah, where it might improve learning. Gradient clipping does afford marginal benefits, especially in preventing gradient explosion in deeper networks.
6. Training and Optimizer Settings: The exploration into training setups and optimizer configurations reveals that common choices like using Adam with a learning rate of 0.0003 perform robustly across tasks. Moreover, employing multiple data passes can enhance sample efficiency, while larger batch sizes improve computational efficiency without necessarily impeding learning stability.
7. Minor Influence of Frame Skipping and Steps Handling: Frame skip levels and handling of episode terminations due to step limits generally exert a minor influence on the agent's performance, assuming a sufficiently large step limit within environments.

Implications and Future Directions

This extensive empirical paper has pivotal implications for both the theoretical understanding and practical implementation of on-policy RL algorithms. By systematically identifying what truly influences RL training outcomes, this work provides a concrete foundation for researchers and practitioners to base their algorithmic improvements on more than theoretical constructs alone.

Moving forward, the run-time dynamics of action spaces and the interplay of complex neural architectures could be explored further. As scalability in RL continues to intersect with deployment in real-world scenarios, the insights gained from understanding these lower-level implementation details will be essential in bridging the gap between research environments and practical applications.

Such expansive studies prompt a reevaluation of perceived performance milestones, providing a richer grounding from which future RL frameworks can achieve greater transparency and reliability in attributing algorithm performance to actual innovations.