V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (1909.12238v1)

Abstract: Some of the most successful applications of deep reinforcement learning to challenging domains in discrete and continuous control have used policy gradient methods in the on-policy setting. However, policy gradients can suffer from large variance that may limit performance, and in practice require carefully tuned entropy regularization to prevent policy collapse. As an alternative to policy gradient algorithms, we introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that performs policy iteration based on a learned state-value function. We show that V-MPO surpasses previously reported scores for both the Atari-57 and DMLab-30 benchmark suites in the multi-task setting, and does so reliably without importance weighting, entropy regularization, or population-based tuning of hyperparameters. On individual DMLab and Atari levels, the proposed algorithm can achieve scores that are substantially higher than has previously been reported. V-MPO is also applicable to problems with high-dimensional, continuous action spaces, which we demonstrate in the context of learning to control simulated humanoids with 22 degrees of freedom from full state observations and 56 degrees of freedom from pixel observations, as well as example OpenAI Gym tasks where V-MPO achieves substantially higher asymptotic scores than previously reported.

• The paper presents V-MPO as an on-policy algorithm employing a learned state-value function to stabilize policy updates without complex regularization.
• It demonstrates superior performance on multi-task benchmarks like Atari-57 and DMLab-30, eliminating the need for dynamic hyperparameter tuning.
• V-MPO achieves robust results across diverse environments, significantly improving scores in discrete tasks (e.g., Ms. Pacman) and continuous control scenarios.

Overview of V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control

This essay presents an overview of the V-MPO algorithm introduced in the paper, highlighting its significance in the domain of deep reinforcement learning (RL) with applications to both discrete and continuous control settings. The paper describes V-MPO as an on-policy variant of Maximum a Posteriori Policy Optimization (MPO), offering reliable performance enhancements without resorting to traditionally used techniques like importance weighting or entropy regularization.

Method Innovation

The primary contribution of the V-MPO algorithm lies in addressing the high variance and stability challenges typically associated with policy gradient methods in RL. Policymakers often require entropy regularization to avoid policy collapse—a practice that needs careful tuning. V-MPO circumvents these issues by adopting a learned state-value function, diverging from the typical state-action value function. This approach leads to policy updates that depend on a constructed target distribution aiming for policy improvement subject to a KL constraint.

Assessment and Results

The effectiveness of V-MPO is demonstrated through empirical evaluation across multi-task settings, specifically the Atari-57 and DMLab-30 benchmark suites.

• Multi-task Performance: V-MPO surpasses previously reported performance on both benchmark suites. This is achieved without dynamic hyperparameter tuning, which is often criticized for its computational overhead and complex design.
• Individual Task Scoring: On specific DMLab and Atari levels, the algorithm demonstrates substantial score improvements, notably in challenging environments like Ms. Pacman. These results indicate the algorithm’s robustness in discrete action spaces.
• Continuous Control: V-MPO shows significant applicability in high-dimensional continuous action spaces, with superior results compared to existing methodologies in simulated humanoid control tasks.

Implications and Future Work

The implications of V-MPO are twofold: theoretical and practical. Theoretically, it offers new insights into handling policy updates in RL by leveraging nonparametric distributions for policy improvement. Practically, it opens pathways to scalable RL algorithms capable of handling a diverse array of tasks without intricate parameter tuning.

Future research could explore expanding V-MPO’s applicability to more varied environments and complex tasks. Additionally, understanding the integration of V-MPO with other learning architectures and its adaptability to off-policy settings may unlock new potential in reinforcement learning frameworks.

Conclusion

V-MPO emerges as a promising alternative to conventional policy gradient-based methods for reinforcement learning. By focusing on scalable, robust algorithms, it addresses significant limitations inherent to existing approaches, facilitating advancements in both discrete and continuous control domains. Its development reflects a growing trend towards smarter, more efficient policy optimization techniques, justified by substantial empirical success.