Decoupled Proximal Policy Optimization

• The paper introduces Outer-PPO, a decoupled framework that separates update estimation from application, enabling flexible algorithmic tuning.
• It employs non-unity outer step scaling, momentum, and biased initialization to achieve statistically significant performance gains in continuous and discrete control domains.
• Empirical evaluations across Brax, Jumanji, and MinAtar demonstrate enhanced sample efficiency and stability without altering PPO's core surrogate objectives.

Decoupled Proximal Policy Optimization (PPO), most precisely formalized as "Outer Proximal Policy Optimization (outer-PPO)", is a generalization of the standard PPO algorithm in which the estimation of parameter updates and their application are explicitly separated into inner and outer optimization loops. This decoupling permits flexible adaptation of the outer update step—such as scaling its step size or adding momentum—without affecting the core PPO surrogate objectives or inner loop dynamics. Empirical and algorithmic studies of outer-PPO reveal implicit assumptions in canonical PPO implementations and motivate additional algorithmic tuning knobs that yield statistically significant performance gains in diverse continuous and discrete control domains (Tan et al., 2024).

1. Decomposition of PPO into Inner and Outer Loops

Standard PPO is an on-policy actor–critic method with parameters $\theta = (\theta^\pi, \theta^V)$, combining policy and value function. At iteration $k$, data $D_k$ is collected using policy $\pi_{\theta_k}$. Advantages $\widehat{A}_i$ are estimated with GAE, and the core PPO objectives are:

• Policy loss (clipped surrogate):

$$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$

where $$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$.

• Value loss (clipped, for stability):

$$L^V(\theta^V; D_k) = \mathbb{E}_{s \sim D_k} \left[ \max\big( (V_{\theta^V}(s) - V^{\text{targ}}_i)^2, \; \operatorname{clip}(V_{\theta^V}(s), V_{\theta^V_k}(s)-\epsilon_v, V_{\theta^V_k}(s)+\epsilon_v) - V^{\text{targ}}_i)^2 ) \big) \right]$$

with $$V^{\text{targ}}_i = r_i + \gamma V_{\theta^V_k}(s_{i+1})$$.

Traditionally, the entire PPO update is implemented as a sequence of $N$ inner-loop SGD updates, starting from $k$0 and yielding $k$1, considered a locally optimal updated parameter. The canonical PPO outer update is simply:

$k$2

or, equivalently,

$k$3

Here, $k$4 is the outer gradient or update vector computed by the inner loop.

2. Outer-PPO: Generalizing PPO with Decoupled Update Application

Outer-PPO replaces the fixed outer step (unity learning rate) with a general update rule, flexibly adapting the application of the inner-loop update vector $k$5 by scaling, momentum, or other transformations. The key design axes are:

• Non-unity learning rate (σ):

$k$6

Varying $k$7 interpolates between conservative and aggressive outer steps, independently of the inner-loop trust region (clipping $k$8).

• Nesterov momentum (μ):

$k$9

$D_k$0

Momentum smooths outer updates, enabling acceleration or stabilization.

• Biased inner-loop initialization ($D_k$1):

The inner-loop can be "warm-started" with a momentum-informed shift:

$D_k$2

Inner-loop optimization then initializes at $D_k$3 instead of $D_k$4.

In this generalized framework, standard PPO corresponds to $D_k$5.

3. Algorithmic Structure and Pseudocode

The following pseudocode clarifies outer-PPO with optional momentum:

$$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$5 Special cases:

• $D_k$6: no momentum; pure scaled updates.
• $D_k$7: standard PPO.
• Nonzero $D_k$8 in initialization: bias for inner loop only.

4. Empirical Evaluation and Performance Analysis

Empirical results were obtained on 14 tasks across Brax (6 tasks, continuous control), Jumanji (4 tasks, discrete control), and MinAtar (4 Atari-like benchmarks). Up to 600 hyperparameter trials and 64 independent seeds per task ensured statistical robustness.

• Non-unity outer-LR ($D_k$9):
  • Optimal $\pi_{\theta_k}$0 per task: $\pi_{\theta_k}$1 to $\pi_{\theta_k}$2.
  • On Brax and Jumanji, $\pi_{\theta_k}$3 yielded 5–10% improvement in mean, median, and IQM returns over a tuned PPO baseline ($\pi_{\theta_k}$4).
  • Probability of improvement: $\pi_{\theta_k}$5 (Brax), $\pi_{\theta_k}$6 (Jumanji).
  • MinAtar: best at $\pi_{\theta_k}$7; no significant gain.
• Outer Nesterov momentum ($\pi_{\theta_k}$8):
  • Optimal $\pi_{\theta_k}$9 for Brax and Jumanji; 3–7% gains ($\widehat{A}_i$0).
  • No net improvement in MinAtar.
• Biased initialization ($\widehat{A}_i$1):
  • Modest gains; statistically significant (≈4% improvement, $\widehat{A}_i$2) only on Jumanji.
  • Optimal $\widehat{A}_i$3–$\widehat{A}_i$4.

Summary of effective hyperparameters:

Variant	Best Hyperparameter(s)	Gain Domains
Outer-LR	$\widehat{A}_i$5	Brax, Jumanji
Nesterov	$\widehat{A}_i$6, $\widehat{A}_i$7	Brax, Jumanji
Bias Init	$\widehat{A}_i$8–$\widehat{A}_i$9	Jumanji (only)

5. Algorithmic Implications and Insights

Outer-PPO demonstrates that standard PPO contains implicit design choices:

• The outer learning rate is fixed at unity ($$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$0).
• No memory or smoothing exists across outer updates ($$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$1).
• The inner optimization always starts at the most recent $$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$2 ($$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$3).

By decoupling inner update estimation (constrained by the trust region and advantage estimation) from outer update application (effected by any optimizer on the update vector $$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$4), outer-PPO:

• Separates noise/stability control (via clipping and inner loop epochs) from overall update aggression (via $$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$5).
• Enables temporal smoothing of parameter updates through momentum ($$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$6), which can reduce variance and improve sample efficiency.
• Allows information transfer across iterations by warm-starting the inner loop ($$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$7), potentially accelerating adaptation.

Empirically, these modifications deliver consistent, statistically significant gains in large-scale benchmark suites without changes to core PPO surrogate losses or the data-collection pipeline.

6. Practical Applications and Tuning Recommendations

Practitioners adopting outer-PPO may treat $$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$8 (outer learning rate), $$L^\pi(\theta^\pi; D_k) = \mathbb{E}_{(s,a)\sim D_k} \left[ \min\big( \rho_i(\theta^\pi) \widehat{A}_i,\; \operatorname{clip}(\rho_i(\theta^\pi), 1-\epsilon, 1+\epsilon) \widehat{A}_i \big) \right]$$9 (momentum), and $$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$0 (bias in initialization) as additional, computationally inexpensive hyperparameters.

• For continuous and discrete-control problems (e.g., Brax, Jumanji), moderate increases in $$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$1 ($$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$2–$$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$3) and momentum ($$\rho_i(\theta^\pi) = \pi_{\theta^\pi}(a_i|s_i) / \pi_{\theta_k^\pi}(a_i|s_i)$$4) offer robust improvements over default PPO, requiring no change to the surrogate objectives, value estimation mechanisms, or rollout strategy.
• Gains are less pronounced or not present on small-scale or heavily simplified domains, such as MinAtar.

Resource and computational requirements remain comparable to well-tuned PPO, as the modifications involve only bookkeeping and outer loop scheduling and work alongside existing PPO implementations.

7. Theoretical and Methodological Impact

Decoupled PPO formalizes the separation between update estimation (inner, trust-region-constrained) and update application (outer, optimizer-defined), revealing that PPO is a specific instance of a more general two-step RL algorithm. This perspective enables a systematic challenge to legacy hyperparameter choices, exposing opportunities for controlled aggression, stabilization, and cross-iteration coupling with minimal disruption.

The outer-PPO formalism thus represents a platform for exploring richer optimizer-based update schemas for RL that remain consistent with strong prior empirical results, as demonstrated by measurable boosts in both sample efficiency and final return (Tan et al., 2024).