Pb-PPO: Adaptive Clipping in Proximal Policy Optimization

• Pb-PPO is a bi-level reinforcement learning method that dynamically optimizes clipping bounds via multi-armed bandit feedback.
• It integrates a multi-armed bandit mechanism with standard PPO to adjust clipping parameters based on actual return feedback for enhanced policy updates.
• Empirical benchmarks demonstrate that Pb-PPO achieves higher final returns, improved sample efficiency, and smoother learning curves compared to fixed-bound PPO variants.

Preference-based Proximal Policy Optimization (Pb-PPO) is a bi-level reinforcement learning framework designed to address limitations inherent in conventional Proximal Policy Optimization (PPO), specifically the reliance on a fixed clipping bound for policy updates. Pb-PPO dynamically optimizes the choice of the clipping bound via a multi-armed bandit mechanism guided by feedback from actual returns, aligning the update process with the true objective of maximizing cumulative return. This approach results in improved sample efficiency, stability, and overall performance in continuous control and navigation tasks, as evidenced by empirical benchmarks (Zhang et al., 2023).

1. Motivation and Background

PPO is widely adopted for its stable training characteristics, largely attributed to the use of a clipped surrogate objective that constrains policy updates. The clipping bound $\epsilon$ in PPO controls the extent of update per iteration, with PPO's performance shown to be sensitive to this hyperparameter. Traditional PPO uses a fixed $\epsilon$, yet there is no theoretical guarantee that a constant bound remains optimal throughout training. Fixed bounds can restrict exploration and adaptability, and prior work on dynamically adjusting $\epsilon$ has not directly aligned clipping adjustment with maximizing true cumulative return.

Pb-PPO introduces a principled framework for dynamically and automatically selecting the clipping bound. Unlike previous methods, the mechanism directly uses task-return feedback to tune the bound, thus better matching reinforcement learning’s core objective.

2. Bi-Level Optimization Paradigm

Pb-PPO formalizes the policy update as a bi-level optimization problem:

• Inner Level: For a given clipping bound $\epsilon$, optimize the policy using the standard PPO-style clipped surrogate objective:

$$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$

where $$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$.

• Outer Level: Select the optimal $\epsilon^*$ from a candidate set $\mathbb{Z} = \{\epsilon_0, \ldots, \epsilon_n\}$ using a multi-armed bandit upper confidence bound (UCB) objective:

$$\epsilon^* \leftarrow \arg\max_{\epsilon_i \in \mathbb{Z}} U^\text{UCB}(\epsilon_i) := U(\epsilon_i) + \lambda \hat{H}(\epsilon_i),$$

where $U(\epsilon_i)$ estimates expected return under $\epsilon$0 and $\epsilon$1 quantifies uncertainty.

The two levels are tightly coupled: the outer bandit selects the clipping parameter to maximize true return, and the inner PPO update uses this choice to update the policy.

3. Multi-Armed Bandit Integration

Each clipping bound candidate $\epsilon$2 is treated as a bandit arm. The integration operates as follows:

• Bandit Reward: After each PPO update under $\epsilon$3, evaluate the updated policy $\epsilon$4 on $\epsilon$5 trajectories and compute the average return $\epsilon$6.
• Statistics Maintenance: For each arm, track the number of visits $\epsilon$7, the current expected reward $\epsilon$8 (updated as $\epsilon$9), and total visits $\epsilon$0.
• Uncertainty Quantification: The bandit UCB uncertainty is defined by $\epsilon$1.
• Selection and Update: At each outer iteration, select $\epsilon$2 as the arm with maximal $\epsilon$3, update statistics, and use $\epsilon$4 in the next inner PPO epoch. Optionally, normalize $\epsilon$5 to obtain an advantage-style signal.

This approach ensures that exploration and exploitation of candidate bounds are balanced and that the policy update is directly steered by return-based preference.

4. Theoretical Properties

Pb-PPO preserves PPO's local monotonic improvement guarantee under the standard conditions of small policy updates and proper clipping. The multi-armed bandit outer loop provides probabilistic control (via the UCB rule) over the selection of suboptimal clipping bounds, visiting them only $\epsilon$6 times, with $\epsilon$7 the total number of iterations. The Hoeffding-based derivation justifies the uncertainty term $\epsilon$8. This two-layered structure ensures $\epsilon$9 converges to the best-performing bound as learning proceeds (Zhang et al., 2023).

5. Algorithmic Realization

The overall algorithm can be described as:

1. Collect on-policy data with the current policy.
2. For each candidate $\epsilon$0, compute $\epsilon$1.
3. Select $\epsilon$2.
4. Perform PPO-style updates using $\epsilon$3 for several epochs.
5. Evaluate the new policy, compute reward statistics, and update bandit estimates.
6. (Optionally) Normalize bandit utilities to improve signal for selection.

Pseudocode is documented in the original reference. Typical hyperparameter settings include a candidate set $\epsilon$4 (10 values uniformly spaced in $\epsilon$5), UCB weight $\epsilon$6, discount $\epsilon$7, and evaluation episodes $\epsilon$8 or $\epsilon$9.

6. Empirical Evaluation and Benchmarking

Experiments with Pb-PPO were conducted on continuous control benchmarks in Gym-Mujoco (Ant-v3, HalfCheetah-v3, Hopper-v3, Walker2d-v3) and pybullet-gym (Dog-run), with comparisons to PPO (fixed $$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$0), TRPO, DDPG, TRGPPO, and PPO-$$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$1.

Evaluation metrics include average episodic return, sample efficiency (AUC), final return after $$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$2 steps, and policy-improvement success rate (fraction of iterations with $$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$3). Pb-PPO achieves:

• Highest average final return (e.g., average 3315 across tasks, surpassing PPO-$$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$4/TRGPPO $$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$5).
• Policy-improvement success of $$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$6 (vs. $$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$7 for fixed-$$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$8 PPO).
• More stable and smooth learning curves (narrower variance) and higher monotonic-improvement ratios.

Summary of Comparative Metrics

Method	Avg. Final Return	Policy-Improvement Success
Pb-PPO	3315	5.0%
PPO-$$J_{\text{inner}}(\pi_{\text{new}}; \epsilon) = \mathbb{E}_{\tau \sim \pi_{\text{old}}} \left[ \min\left( r(\tau)A_{\text{old}}(\tau), \mathrm{clip}(r(\tau), 1-\epsilon, 1+\epsilon)A_{\text{old}}(\tau) \right) \right],$$9, TRGPPO	$$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$03000	3–4%
Fixed-$$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$1 PPO	Lower	Lower

7. Ablation and Sensitivity Analyses

Ablations analyze the effect of the number of clipping arms $$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$2, the presence of normalization in the bandit feedback, and arm selection statistics.

• Increasing $$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$3 from 3 to 10 improves performance, which then plateaus.
• Normalizing $$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$4 provides a slight benefit ("wi-ad" outperforms "wo-ad").
• Pb-PPO consistently demonstrates higher monotonic improvement success than fixed-bound baselines.
• Statistical correlation analyses verify that arms with higher $$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$5 reliably produce higher returns, validating the efficacy of preference-based feedback.

A plausible implication is that further tuning of candidate set cardinality and reward normalization may yield incremental gains, though significant improvements beyond $$r(\tau) = \pi_{\text{new}}(\tau)/\pi_{\text{old}}(\tau)$$6 appear limited (Zhang et al., 2023).

---

Pb-PPO systematically addresses the fundamental challenge of aligning adaptive surrogate objective parameters with reinforcement learning goals by leveraging a bi-level, feedback-driven approach. The empirical and theoretical results demonstrate superiority in adaptability, convergence, and robustness over traditional and previously adaptive PPO variants.