Maximum a Posteriori Policy Optimisation (1806.06920v1)

Abstract: We introduce a new algorithm for reinforcement learning called Maximum aposteriori Policy Optimisation (MPO) based on coordinate ascent on a relative entropy objective. We show that several existing methods can directly be related to our derivation. We develop two off-policy algorithms and demonstrate that they are competitive with the state-of-the-art in deep reinforcement learning. In particular, for continuous control, our method outperforms existing methods with respect to sample efficiency, premature convergence and robustness to hyperparameter settings while achieving similar or better final performance.

• The paper introduces a novel RL algorithm that uses an EM-inspired dual optimization approach to enhance sample efficiency and robustness.
• It leverages both parametric and non-parametric variants to optimize a relative-entropy objective for superior off-policy learning in continuous control.
• Empirical results on tasks like 56 DoF humanoid simulations demonstrate MPO's ability to dramatically reduce sample requirements compared to existing methods.

Maximum a Posteriori Policy Optimisation: A Detailed Exploration

The paper "Maximum a Posteriori Policy Optimisation" introduces a novel reinforcement learning (RL) algorithm, referred to as Maximum a Posteriori Policy Optimisation (MPO). This algorithm employs a coordinate ascent approach to optimize a relative-entropy objective, leading to the creation of two off-policy algorithms designed for continuous control challenges. The presented method not only displays competitive performance but also excels in sample efficiency, reducing premature convergence, and enhancing robustness against variations in hyperparameter settings.

Algorithmic Foundation

MPO leverages the inherent duality between control and estimation within RL, drawing on Expectation Maximisation (EM) techniques from probabilistic estimation to reformulate control challenges. Traditionally, RL focuses on maximizing expected rewards through policy optimization. In contrast, MPO reformulates this problem by considering actions most likely to be taken given an assumption of future success in maximizing rewards.

The Reinforcement Learning Landscape

The paper positions MPO among existing RL algorithms, notably differentiating between on-policy methods like TRPO and PPO, which require large batches and impose parameter change constraints, and off-policy techniques such as DDPG and SVG, which optimize data efficiency albeit with challenges in tuning. MPO attempts to merge the strengths of these approaches, promising the robustness of on-policy methods while enhancing sample efficiency akin to off-policy strategies.

Core Contributions

1. Duality Utilization: MPO employs a dual-perspective optimization strategy, adopting sampling-based E-steps to adjust state-action pair weights, followed by parametric M-steps utilizing deep neural networks. This alternating process draws from EM procedures to ensure stable learning progression.
2. Non-parametric and Parametric Variants: MPO features versions involving parametric and non-parametric variational distributions, each demonstrating specific applicability based on the task dimensionality and inherent complexity.
3. Off-policy Learning Efficiency: By optimizing the variational distribution in the E-step and adopting an EM-style update for the M-step, MPO achieves a remarkable balance between data efficiency and stability, extending the applicability of RL algorithms to more complex environments like humanoid simulations.

Empirical Evaluation

The proposed MPO framework is evaluated across a suite of continuous control problems. Notably, for domains such as a 56 DoF humanoid body, MPO showcases significant data efficiencies, often outpacing state-of-the-art algorithms by an order of magnitude.

Implications and Future Directions

The introduction of MPO highlights the potential of leveraging probabilistic inference techniques within the RL context. By focusing on expectation maximization and variational optimisation, future research might further explore:

• Enhancements in hierarchical policy structures.
• Expanded applicability to multi-agent systems.
• Integration with model-based RL techniques for broader adaptability.

MPO stands as a significant step in reconciling the strengths of on-policy and off-policy strategies, offering a robust and efficient model for diverse and complex RL challenges. As AI continues to tackle increasingly intricate tasks, the principles and methodologies outlined in MPO are poised to guide future algorithmic developments and applications.