A dynamical clipping approach with task feedback for Proximal Policy Optimization (2312.07624v3)

Abstract: Proximal Policy Optimization (PPO) has been broadly applied to robotics learning, showcasing stable training performance. However, the fixed clipping bound setting may limit the performance of PPO. Specifically, there is no theoretical proof that the optimal clipping bound remains consistent throughout the entire training process. Meanwhile, previous researches suggest that a fixed clipping bound restricts the policy's ability to explore. Therefore, many past studies have aimed to dynamically adjust the PPO clipping bound to enhance PPO's performance. However, the objective of these approaches are not directly aligned with the objective of reinforcement learning (RL) tasks, which is to maximize the cumulative Return. Unlike previous clipping approaches, we propose a bi-level proximal policy optimization objective that can dynamically adjust the clipping bound to better reflect the preference (maximizing Return) of these RL tasks. Based on this bi-level proximal policy optimization paradigm, we introduce a new algorithm named Preference based Proximal Policy Optimization (Pb-PPO). Pb-PPO utilizes a multi-armed bandit approach to refelect RL preference, recommending the clipping bound for PPO that can maximizes the current Return. Therefore, Pb-PPO results in greater stability and improved performance compared to PPO with a fixed clipping bound. We test Pb-PPO on locomotion benchmarks across multiple environments, including Gym-Mujoco and legged-gym. Additionally, we validate Pb-PPO on customized navigation tasks. Meanwhile, we conducted comparisons with PPO using various fixed clipping bounds and various of clipping approaches. The experimental results indicate that Pb-PPO demonstrates superior training performance compared to PPO and its variants. Our codebase has been released at : https://github.com/stevezhangzA/pb_ppo

• The paper presents an adaptive clipping strategy using task feedback through UCB to adjust PPO's clipping bounds during training.
• It details a methodology where the dynamic update mechanism balances exploration and exploitation to ensure stable policy improvement.
• Experimental results show that Adaptive-PPO enhances sample efficiency and performance on high-dimensional continuous control benchmarks.

Introduction to Proximal Policy Optimization

Reinforcement Learning (RL) is a subset of Machine Learning where an agent learns to make decisions by performing actions in an environment to maximize some notion of cumulative reward. Two common paradigms in RL involve, firstly, learning Q-networks and updating the policy network, and secondly, using policy gradients to directly update the policy. Proximal Policy Optimization (PPO) is a technique under the second paradigm that aims to improve policy sample efficiency and ease of deployment compared to its predecessor, Trust Region Policy Optimization (TRPO). PPO accomplishes this by keeping policy updates within a predefined surrogate trust region, using something called a clipping bound to prevent the new policy from deviating too much from the old one.

Dynamic Clipping in Policy Updates

However, the fixed setting of the surrogate trust region in PPO may limit its adaptability, as there's no solid evidence that a single optimal clipping bound fits all stages of training. Consequently, there is a push to explore dynamic clipping bounds to enhance PPO's performance. The paper introduces an adaptive method designed to modify the clipping bound dynamically throughout the training process. This approach leverages a bandit-based strategy, utilizing the Upper Confidence Bound (UCB) during online training to balance exploration and exploitation of candidate clip bounds.

Reinforcement Learning Framework

RL is formulated as a Markov Decision Process (MDP) and operates under the paradigms of on-policy and off-policy learning. Importance sampling is introduced to approximate on-policy algorithms as off-policy, which helps to maximize the use of previous data in training current policy. The paper also explains how trust region optimization works, emphasizing the importance of balancing the policy update's step size using importance sampling and KL divergence in TRPO, to ensure policy improvement without the new policy deviating excessively.

Adaptive PPO: Theory and Methodology

The paper proposes the Adaptive-PPO method and provides a detailed theoretical underpinning of its approach. It uses UCB to decide on the best clipping bounds to use during different training stages. This adaptability allows the algorithm to maintain stable and monotonic policy improvement, which can be especially beneficial in complex environments.

Empirical Results

Finally, the paper presents empirical results demonstrating the sample efficiency and performance of Adaptive-PPO compared to the standard PPO with fixed clipping bounds. The experiments conducted across various standard benchmarks, including high-dimensional continuous control tasks, reveal that Adaptive-PPO shows significant improvements, particularly in complex environments where the benefits of dynamically adjusting the trust region's clipping bound are more pronounced.

Conclusion and Future Work

In conclusion, the paper suggests that dynamic clipping provides significant benefits to the performance of PPO algorithms. While further experimentation and tuning may enhance these results, the current findings suggest that Adaptive-PPO is a promising approach to improving the efficiency and applicability of policy gradient methods in complex reinforcement learning tasks. Future work may include testing the algorithm on a wider array of tasks and further refining its implementation.