Deep Reinforcement Learning that Matters (1709.06560v3)

Abstract: In recent years, significant progress has been made in solving challenging problems across various domains using deep reinforcement learning (RL). Reproducing existing work and accurately judging the improvements offered by novel methods is vital to sustaining this progress. Unfortunately, reproducing results for state-of-the-art deep RL methods is seldom straightforward. In particular, non-determinism in standard benchmark environments, combined with variance intrinsic to the methods, can make reported results tough to interpret. Without significance metrics and tighter standardization of experimental reporting, it is difficult to determine whether improvements over the prior state-of-the-art are meaningful. In this paper, we investigate challenges posed by reproducibility, proper experimental techniques, and reporting procedures. We illustrate the variability in reported metrics and results when comparing against common baselines and suggest guidelines to make future results in deep RL more reproducible. We aim to spur discussion about how to ensure continued progress in the field by minimizing wasted effort stemming from results that are non-reproducible and easily misinterpreted.

• The paper demonstrates that hyperparameter tuning and network configurations significantly impact the performance of policy gradient methods in DRL.
• It reveals that random seed variability can lead to substantial differences in results, highlighting the need for multiple trial evaluations.
• The study recommends rigorous reporting and statistical significance testing to improve reproducibility and reliability in DRL research.

An Examination of Reproducibility in Deep Reinforcement Learning

"Deep Reinforcement Learning that Matters" by Peter Henderson et al. critically examines the reproducibility of state-of-the-art deep reinforcement learning (DRL) methods. Given the surge in RL's popularity, as evidenced by its application in robotics, games like Go, and competitive video games, the authors argue the indispensable need for reproducible results to ascertain the validity and improvement of new algorithms.

The paper systematically underscores the challenges in reproducibility from both extrinsic (e.g., hyperparameters, codebases) and intrinsic (e.g., random seeds, environment properties) sources of variability. Through a series of experiments focused primarily on policy gradient (PG) methods within continuous control settings, the authors offer a detailed empirical analysis to highlight the inconsistencies in the reported performance metrics across the literature.

Key Algorithms Explored

The authors center their investigation on several prominent model-free PG algorithms:

• Trust Region Policy Optimization (TRPO)
• Deep Deterministic Policy Gradients (DDPG)
• Proximal Policy Optimization (PPO)
• Actor Critic using Kronecker-Factored Trust Region (ACKTR)

These methods, widely recognized for their effectiveness in continuous control tasks, employ neural network function approximators to optimize policies within the MuJoCo domains.

Hyperparameters and Network Architecture

The paper demonstrates the significant role that hyperparameter tuning and network architecture play in the performance of algorithms. The authors experimentally varied hyperparameters such as network structure, activation functions, and reward scaling, showing that small modifications can lead to substantial differences in results. For instance, the choice of activation function (ReLU, tanh, Leaky ReLU) had inconsistent impacts on performance across algorithms and environments.

Reward Scaling

In DDPG, reward scaling showed profound effects. The analysis revealed that improper reward scaling could lead to misleading results, indicating the necessity for a principled approach to handling reward targets rather than relying on arbitrary rescaling.

Random Seed Variability

The experiment with random seeds highlighted another source of significant variability. By running trials with different random seeds, the authors found that results could be drastically different even with the same hyperparameter settings, underscoring the need for multiple trials to obtain reliable performance estimates.

Environment Characteristics

The choice of environment markedly impacted algorithm performance, with no single algorithm consistently outperforming others across all environments. For example, DDPG excelled in stable environments like HalfCheetah but faltered in more unstable settings such as Hopper. This variability stresses the importance of comprehensive evaluation across diverse environments to ensure the robustness of novel methods.

Codebase Discrepancies

Discrepancies between codebases also emerged as a crucial factor impacting reproducibility. The authors compared multiple implementations of DDPG and TRPO and found significant performance differences. This finding underscores the necessity for transparent reporting and availability of implementation details to facilitate accurate reproduction of results.

Significance Metrics

Henderson et al. advocate for the use of statistical significance metrics to better assess the reliability of reported results. The authors explored bootstrap methods and significance tests such as the 2-sample t-test and Kolmogorov-Smirnov test to evaluate performance gains. These methods provided insights into the confidence intervals and the significance of observed performance improvements.

Recommendations and Future Directions

The paper culminates in recommendations to enhance reproducibility in DRL research:

• Consistent Reporting: Ensure all hyperparameters, implementation details, and experimental setups are thoroughly documented.
• Multiple Trials: Report average performance across multiple trials with different random seeds to mitigate variability.
• Significance Testing: Use statistical methods to evaluate the significance of performance gains.
• Hyperparameter Agnostic Algorithms: Develop algorithms less sensitive to hyperparameters to facilitate fair comparisons.

The authors also propose future directions, including the development of significance metrics tailored specifically for RL and the exploration of hyperparameter agnostic algorithms.

Conclusion

Henderson et al.'s paper offers a comprehensive examination of the reproducibility challenges in DRL. By providing empirical evidence of variabilities and proposing structured guidelines for better reporting and evaluation practices, the paper serves as a pivotal resource for ensuring that advancements in DRL are both meaningful and dependable. This work lays the groundwork for a more rigorous and standardized approach to DRL research, vital for the field's continued growth and impact.