Proximal Policy Optimization via Enhanced Exploration Efficiency (2011.05525v1)

Abstract: Proximal policy optimization (PPO) algorithm is a deep reinforcement learning algorithm with outstanding performance, especially in continuous control tasks. But the performance of this method is still affected by its exploration ability. For classical reinforcement learning, there are some schemes that make exploration more full and balanced with data exploitation, but they can't be applied in complex environments due to the complexity of algorithm. Based on continuous control tasks with dense reward, this paper analyzes the assumption of the original Gaussian action exploration mechanism in PPO algorithm, and clarifies the influence of exploration ability on performance. Afterward, aiming at the problem of exploration, an exploration enhancement mechanism based on uncertainty estimation is designed in this paper. Then, we apply exploration enhancement theory to PPO algorithm and propose the proximal policy optimization algorithm with intrinsic exploration module (IEM-PPO) which can be used in complex environments. In the experimental parts, we evaluate our method on multiple tasks of MuJoCo physical simulator, and compare IEM-PPO algorithm with curiosity driven exploration algorithm (ICM-PPO) and original algorithm (PPO). The experimental results demonstrate that IEM-PPO algorithm needs longer training time, but performs better in terms of sample efficiency and cumulative reward, and has stability and robustness.

• The paper introduces IEM-PPO to overcome PPO's exploration limitations by leveraging uncertainty estimation.
• It integrates an intrinsic exploration module within PPO to balance comprehensive exploration with data exploitation.
• Experimental results on the MuJoCo simulator show IEM-PPO achieves higher cumulative rewards and improved stability compared to standard methods.

The paper "Proximal Policy Optimization via Enhanced Exploration Efficiency" addresses the exploration challenge in Proximal Policy Optimization (PPO), a prominent deep reinforcement learning algorithm known for its effectiveness in continuous control tasks. Despite its success, PPO's performance is sometimes hindered by inadequate exploration capabilities.

The authors start by analyzing the traditional Gaussian action exploration mechanism inherent in the PPO algorithm, particularly focusing on its assumptions and limitations in continuous control tasks with dense rewards. They argue that the standard exploration strategies in PPO do not fully achieve a balance between comprehensive exploration and data exploitation, which is critical for optimal performance in complex environments.

To tackle this, the researchers introduce an enhanced exploration mechanism grounded in uncertainty estimation. This innovation is aimed at improving the exploration component of PPO, making it more robust and effective in navigating the complexity of diverse environments.

Building on this foundation, the paper proposes the Intrinsic Exploration Module for PPO (IEM-PPO). This novel approach integrates the enhanced exploration theory into the PPO framework, empowering the algorithm to perform better in complex environments. The IEM-PPO algorithm is specifically designed to address the exploration issue by making the agent more curious and capable of dealing with the uncertainties inherent in intricate tasks.

The experimental validation utilizes the MuJoCo physical simulator, a popular benchmarking platform for continuous control tasks. In a series of comprehensive experiments, the authors compare IEM-PPO with both the original PPO and another variant that incorporates curiosity-driven exploration, known as ICM-PPO. The results are compelling: while IEM-PPO requires a longer training period, it demonstrates superior sample efficiency and achieves higher cumulative rewards. Additionally, IEM-PPO exhibits improved stability and robustness across multiple tasks.

In summary, the paper contributes to the field by addressing a significant limitation in the PPO algorithm regarding exploration. By introducing the IEM-PPO, which leverages uncertainty estimation to enhance exploration, the authors advance the performance capabilities of PPO, making it more suitable for complex continuous control tasks.