Decoupled Proximal Policy Optimization

• The paper introduces Outer-PPO, a decoupled framework that separates update estimation from application, enabling flexible algorithmic tuning.
• It employs non-unity outer step scaling, momentum, and biased initialization to achieve statistically significant performance gains in continuous and discrete control domains.
• Empirical evaluations across Brax, Jumanji, and MinAtar demonstrate enhanced sample efficiency and stability without altering PPO's core surrogate objectives.

Decoupled Proximal Policy Optimization (PPO), most precisely formalized as "Outer Proximal Policy Optimization (outer-PPO)", is a generalization of the standard PPO algorithm in which the estimation of parameter updates and their application are explicitly separated into inner and outer optimization loops. This decoupling permits flexible adaptation of the outer update step—such as scaling its step size or adding momentum—without affecting the core PPO surrogate objectives or inner loop dynamics. Empirical and algorithmic studies of outer-PPO reveal implicit assumptions in canonical PPO implementations and motivate additional algorithmic tuning knobs that yield statistically significant performance gains in diverse continuous and discrete control domains (Tan et al., 1 Nov 2024).

1. Decomposition of PPO into Inner and Outer Loops

Standard PPO is an on-policy actor–critic method with parameters $\theta = (\theta^\pi, \theta^V)$, combining policy and value function. At iteration $k$, data $D_k$ is collected using policy $\pi_{\theta_k}$. Advantages $\widehat{A}_i$ are estimated with GAE, and the core PPO objectives are:

• Policy loss (clipped surrogate):

$$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$

where $$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$.

• Value loss (clipped, for stability):

$$L^V(\theta^V; D_k) = \mathbb{E}_{s \sim D_k} \left[ \max\big( (V_{\theta^V}(s) - V^{\text{targ}}_i)^2, \; \operatorname{clip}(V_{\theta^V}(s), V_{\theta^V_k}(s)-\epsilon_v, V_{\theta^V_k}(s)+\epsilon_v) - V^{\text{targ}}_i)^2 ) \big) \right]$$

with $$V^{\text{targ}}_i = r_i + \gamma V_{\theta^V_k}(s_{i+1})$$.

Traditionally, the entire PPO update is implemented as a sequence of $N$ inner-loop SGD updates, starting from $\theta_k$ and yielding $\theta^*_k$, considered a locally optimal updated parameter. The canonical PPO outer update is simply:

$\theta_{k+1} \leftarrow \theta^*_k$

or, equivalently,

$$\theta_{k+1} = \theta_k + O_k, \;\;\; O_k := \theta_k^* - \theta_k$$

Here, $O_k$ is the outer gradient or update vector computed by the inner loop.

2. Outer-PPO: Generalizing PPO with Decoupled Update Application

Outer-PPO replaces the fixed outer step (unity learning rate) with a general update rule, flexibly adapting the application of the inner-loop update vector $O_k$ by scaling, momentum, or other transformations. The key design axes are:

• Non-unity learning rate (σ):

$$\theta_{k+1} = \theta_k + \sigma O_k, \quad \sigma \in \mathbb{R}^+$$

Varying $\sigma$ interpolates between conservative and aggressive outer steps, independently of the inner-loop trust region (clipping $\epsilon$).

• Nesterov momentum (μ):

$m_{k} = \mu m_{k-1} + O_{k}$

$$\theta_{k+1} = \theta_k + \sigma \big[ m_k + \mu O_k \big]$$

Momentum smooths outer updates, enabling acceleration or stabilization.

• Biased inner-loop initialization ($\alpha$):

The inner-loop can be "warm-started" with a momentum-informed shift:

$\tilde{\theta}_k = \theta_k + \alpha m_{k-1}$

Inner-loop optimization then initializes at $\tilde{\theta}_k$ instead of $\theta_k$.

In this generalized framework, standard PPO corresponds to $(\sigma=1, \mu=0, \alpha=0)$.

3. Algorithmic Structure and Pseudocode

The following pseudocode clarifies outer-PPO with optional momentum:

initialize theta_0, momentum m_{-1} = 0 for k = 0, 1, ...: D_k ← Collect rollouts with π_{θ_k} Compute advantage estimates ĤA over D_k theta_k_star ← InnerOptimization(theta_k, D_k, ĤA, ...hyperparams...) O_k ← theta_k_star - theta_k m_k ← mu * m_{k-1} + O_k theta_{k+1} ← theta_k + sigma * (m_k + mu * O_k)

Special cases:

• $\mu=0$: no momentum; pure scaled updates.
• $\sigma=1$: standard PPO.
• Nonzero $\alpha$ in initialization: bias for inner loop only.

4. Empirical Evaluation and Performance Analysis

Empirical results were obtained on 14 tasks across Brax (6 tasks, continuous control), Jumanji (4 tasks, discrete control), and MinAtar (4 Atari-like benchmarks). Up to 600 hyperparameter trials and 64 independent seeds per task ensured statistical robustness.

• Non-unity outer-LR ($\sigma$):
  • Optimal $\sigma$ per task: $0.5$ to $2.3$.
  • On Brax and Jumanji, $\sigma \approx 1.6$ yielded 5–10% improvement in mean, median, and IQM returns over a tuned PPO baseline ($p<0.05$).
  • Probability of improvement: $>0.6$ (Brax), $>0.7$ (Jumanji).
  • MinAtar: best at $\sigma=1.0$; no significant gain.
• Outer Nesterov momentum ($\sigma\in[0.1,1.0],\,\mu\in[0.1,0.9]$):
  • Optimal $(\sigma, \mu) \approx (0.7, 0.5)$ for Brax and Jumanji; 3–7% gains ($p<0.05$).
  • No net improvement in MinAtar.
• Biased initialization ($\alpha\in[0.1,1.0],\,\mu\in[0,0.9]$):
  • Modest gains; statistically significant (≈4% improvement, $p<0.05$) only on Jumanji.
  • Optimal $\alpha \approx 0.1$–$0.2$.

Summary of effective hyperparameters:

Variant	Best Hyperparameter(s)	Gain Domains
Outer-LR	$\sigma = [1.5, 2.1]$	Brax, Jumanji
Nesterov	$\sigma \sim 0.7$, $\mu \sim 0.5$	Brax, Jumanji
Bias Init	$\alpha = 0.1$–$0.2$	Jumanji (only)

5. Algorithmic Implications and Insights

Outer-PPO demonstrates that standard PPO contains implicit design choices:

• The outer learning rate is fixed at unity ($\sigma=1$).
• No memory or smoothing exists across outer updates ($\mu=0$).
• The inner optimization always starts at the most recent $\theta_k$ ($\alpha=0$).

By decoupling inner update estimation (constrained by the trust region and advantage estimation) from outer update application (effected by any optimizer on the update vector $O_k$), outer-PPO:

• Separates noise/stability control (via clipping and inner loop epochs) from overall update aggression (via $\sigma$).
• Enables temporal smoothing of parameter updates through momentum ($\mu$), which can reduce variance and improve sample efficiency.
• Allows information transfer across iterations by warm-starting the inner loop ($\alpha$), potentially accelerating adaptation.

Empirically, these modifications deliver consistent, statistically significant gains in large-scale benchmark suites without changes to core PPO surrogate losses or the data-collection pipeline.

6. Practical Applications and Tuning Recommendations

Practitioners adopting outer-PPO may treat $\sigma$ (outer learning rate), $\mu$ (momentum), and $\alpha$ (bias in initialization) as additional, computationally inexpensive hyperparameters.

• For continuous and discrete-control problems (e.g., Brax, Jumanji), moderate increases in $\sigma$ ($1.5$–$2.1$) and momentum ($0.5$) offer robust improvements over default PPO, requiring no change to the surrogate objectives, value estimation mechanisms, or rollout strategy.
• Gains are less pronounced or not present on small-scale or heavily simplified domains, such as MinAtar.

Resource and computational requirements remain comparable to well-tuned PPO, as the modifications involve only bookkeeping and outer loop scheduling and work alongside existing PPO implementations.

7. Theoretical and Methodological Impact

Decoupled PPO formalizes the separation between update estimation (inner, trust-region-constrained) and update application (outer, optimizer-defined), revealing that PPO is a specific instance of a more general two-step RL algorithm. This perspective enables a systematic challenge to legacy hyperparameter choices, exposing opportunities for controlled aggression, stabilization, and cross-iteration coupling with minimal disruption.

The outer-PPO formalism thus represents a platform for exploring richer optimizer-based update schemas for RL that remain consistent with strong prior empirical results, as demonstrated by measurable boosts in both sample efficiency and final return (Tan et al., 1 Nov 2024).