RPO: Reparameterization Proximal Policy Optimization

• The paper introduces RPO, a reinforcement learning algorithm combining reparameterized policy gradients with PPO-like surrogate clipping to mitigate instability in long-horizon differentiable environments.
• RPO leverages cached action-gradients computed via BPTT and latent noise regeneration to enable multiple stable update epochs with efficient sample reuse.
• Empirical evaluations demonstrate that RPO achieves 2–5× faster sample efficiency and superior performance on benchmarks like Humanoid and Anymal compared to existing methods.

Searching arXiv for the primary paper and closely related methods mentioned in the provided data. Reparameterization Proximal Policy Optimization (RPO) is a reinforcement-learning algorithm introduced as a stable and sample-efficient reparameterization policy gradient (RPG) method for differentiable environments. It is defined by combining a PPO-like clipped surrogate objective with reparameterized gradients computed by backpropagation through time (BPTT), together with Kullback–Leibler (KL) divergence regularization and an entropy bonus. In the formulation given by the originating work, RPO addresses a central limitation of vanilla RPG—training instability caused by high-variance gradients and sensitivity in long-horizon or stiff dynamics—while preserving the low-variance advantages of pathwise differentiation and enabling multiple epochs of stable sample reuse (Zhong et al., 8 Aug 2025).

1. Conceptual setting and motivation

RPO is situated at the intersection of two lines of work: reparameterization-based policy gradients and proximal policy optimization. The motivating premise is that RPG is promising for improving sample efficiency by leveraging differentiable dynamics, but that its practical use is constrained by instability. The originating paper identifies the critical barrier as high-variance gradients that can destabilize learning, especially in settings where BPTT through dynamics becomes numerically fragile (Zhong et al., 8 Aug 2025).

The baseline RPG objective is the standard Markov decision process objective

$$J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}[R(\tau)],$$

with trajectory return

$R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$

Under a reparameterizable Gaussian policy,

$$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$

the gradient can be expressed pathwise as

$$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$

The paper characterizes this estimator as often having much lower variance than the score-function (REINFORCE) estimator, but also as potentially highly unstable in long-horizon or stiff dynamics due to exploding or vanishing Jacobians (Zhong et al., 8 Aug 2025).

The central idea of RPO is therefore not to replace reparameterization, but to constrain how reparameterized updates are reused and accumulated. This suggests an attempt to transfer the stabilizing logic of PPO’s surrogate optimization into the RPG setting, where that connection had, according to the paper, been largely unexplored and non-trivial (Zhong et al., 8 Aug 2025).

2. Derivation of the RPO objective

The starting point is the PPO clipped surrogate

$$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$

where

$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$

In the model-free setting, the source describes this clipping as constraining policy update size and enabling multiple epochs of sample reuse with stability (Zhong et al., 8 Aug 2025).

RPO adapts this logic to RPG by replacing the REINFORCE-style gradient with its reparameterized counterpart and by adding KL and entropy regularizers. The paper defines

$$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$

and introduces asymmetric clipping bounds $[1-c_{\text{low}},\,1+c_{\text{high}}]$. The clipped term is

$$L_{\text{clip}}(\theta) = \mathbb{E}_t\!\left[\operatorname{clip}\!\big(\rho_t(\theta),1-c_{\text{low}},1+c_{\text{high}}\big)\cdot A_t\right],$$

where $A_t$ is estimated by a critic, for example with generalized advantage estimation (GAE). Two additional regularizers are then added:

$R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$0

and

$R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$1

The final policy objective is

$R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$2

The implementation-oriented exposition gives default settings: $R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$3, moderate clipping bounds such as $R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$4, $R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$5, a small entropy coefficient of approximately $R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$6, and $R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$7 epochs per batch (Zhong et al., 8 Aug 2025). Because these are presented as defaults rather than universal prescriptions, they are best understood as a reference configuration for the reported experiments rather than as an invariant part of the method.

3. Gradient computation and sample reuse via BPTT

A distinctive technical feature of RPO is that it reuses each rollout for multiple actor updates by caching action-gradients. The procedure described in the source has two phases. First, trajectories are collected under $R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$8, and a single BPTT pass computes

$R(\tau)=\sum_{t=0}^\infty \gamma^t r(s_t,a_t).$9

for each trajectory $$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$0 and time index $$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$1. Second, for each of the subsequent update epochs, the algorithm reconstructs a differentiable graph for the stored actions under the current parameters by regenerating the latent noise

$$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$2

so that

$$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$3

It then computes the importance ratio $$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$4 and forms clipped policy-gradient contributions (Zhong et al., 8 Aug 2025).

The update rule given in the paper is piecewise:

• if $$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$5, then

$$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$6

• otherwise,

$$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$7

The KL and entropy gradients are then added, the contributions are summed and averaged, and an optimizer step is taken. The paper emphasizes that this design allows a single backward pass per batch to reuse cached $$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$8 across $$a_t=\mu_\theta(s_t)+\sigma_\theta(s_t)\cdot \epsilon_t,\qquad \epsilon_t\sim \mathcal{N}(0,I),$$9 epochs, with the main repeated cost being regeneration of $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$0 and forward/backward passes through the policy network (Zhong et al., 8 Aug 2025).

This mechanism is central to the method’s identity. It is not merely a clipped objective layered on top of a standard actor-critic loop; rather, the sample-reuse strategy depends on a specific reconstruction of the computational graph for already-sampled actions. A plausible implication is that RPO’s practicality hinges on environments and policy parameterizations for which this inverse mapping $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$1 is tractable.

4. Stabilization mechanisms: clipping, KL regularization, and entropy

The paper attributes RPO’s stability to the joint action of ratio clipping and explicit KL regularization. Clipping enforces the bound $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$2 in the symmetric case, or its asymmetric analogue under $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$3 and $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$4, so that no single ratio can dominate the gradient or push the policy too far in one batch. The text explicitly interprets this as mimicking the trust region of TRPO (Zhong et al., 8 Aug 2025).

Clipping alone, however, is reported as insufficient in very stiff or long-horizon tasks. For that reason, the additional KL term

$$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$5

penalizes the average per-state shift in the policy distribution. In practice, the source states that one implements the closed-form KL of two Gaussians before the $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$6 squash and backpropagates its gradient. The paper further states that this explicit trust-region-style penalty stabilizes RPG updates and can allow looser clipping bounds for better sample reuse (Zhong et al., 8 Aug 2025).

An entropy bonus is also incorporated through $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$7, with a small positive coefficient. In the presentation given, entropy is part of the default objective rather than an optional auxiliary term. The combination of clipping, KL regularization, and entropy is described as producing more “monotonic” empirical updates and as avoiding catastrophic drops. The paper also states that the reparameterized surrogate gradient remains unbiased for the clipped expectation and has bounded variance under mild assumptions on $$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$8 (Zhong et al., 8 Aug 2025).

These statements concern the clipped expectation rather than the original, unclipped RPG objective. That distinction is methodologically important: the paper’s stability claims are attached to the surrogate objective actually optimized by RPO.

5. Compatibility with variance reduction and actor-critic structure

RPO is presented as fully compatible with common RPG variance-reduction techniques. The source lists the following classes explicitly: value-function baselines via any actor-critic trained by TD($$\nabla_\theta J(\theta) = \mathbb{E}\!\left[\sum_{t=0}^\infty \left(\frac{\partial a_t}{\partial \theta}\right)\cdot \left(\frac{\partial R(\tau)}{\partial a_t}\right)\right].$$9) or double-critic; GAE for computing $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$0; short-horizon truncation such as SHAC or AHAC, or mixed-length backpropagation such as MB-MIX; entropy control variates as in SAPO or Soft Actor-Critic; and control variates that combine RP and LR estimators, exemplified by Adaptive Gradient Policy Optimization (Zhong et al., 8 Aug 2025).

The paper’s key claim in this regard is structural: in each such case, only the computation of $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$1 or $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$2 changes, while the surrogate optimization in

$$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$3

remains unchanged (Zhong et al., 8 Aug 2025). This positions RPO as a wrapper around the optimization stage of RPG rather than as a replacement for existing baseline estimators or truncation schemes.

The actor-critic update described in the algorithmic summary is correspondingly conventional on the critic side. After policy optimization over $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$4 epochs, the value critic $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$5 is updated by minimizing

$$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$6

The exposition does not claim novelty for the critic objective; its role is to support advantage estimation in the policy update (Zhong et al., 8 Aug 2025).

A plausible implication is that RPO can be inserted into existing differentiable-simulation actor-critic pipelines with relatively localized changes, provided that action-gradient caching and noise regeneration are supported.

6. Empirical evaluation and reported results

The reported evaluation uses a suite of differentiable-simulator control tasks comprising four locomotion benchmarks—Hopper, Ant, Anymal, and Humanoid in DFlex—and one dexterous manipulation benchmark, Hand Reorient in Rewarped. The baselines listed are PPO (model-free), SHAC, SAPO, and GI-PPO (Zhong et al., 8 Aug 2025).

The paper reports three headline empirical outcomes. First, RPO achieves $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$7–$$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$8 faster sample efficiency on all tasks versus SAPO and SHAC, reaching a given return thousands to millions of steps sooner. Second, final performance is described as state-of-the-art or better, with example mean returns of approximately $$L^{PPO}(\theta) = \mathbb{E}_t\!\left[\min\!\big(r_t(\theta)\cdot A_t,\ \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\cdot A_t\big)\right],$$9 on Humanoid versus $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$0 for SAPO, and approximately $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$1 on Anymal versus $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$2. Third, KL-divergence curves remain small and stable for RPO, around $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$3–$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$4 per update, whereas SAPO is reported to spike to $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$5–$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$6, leading to sudden drops (Zhong et al., 8 Aug 2025).

Category	Reported content
Benchmarks	Hopper, Ant, Anymal, Humanoid in DFlex; Hand Reorient in Rewarped
Baselines	PPO, SHAC, SAPO, GI-PPO
Key quantitative claims	$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$7–$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$8 faster sample efficiency; Humanoid $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$9 vs $$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$0 SAPO; Anymal $$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$1 vs $$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$2; KL $$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$3–$$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$4 vs $$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$5–$$\rho_t(\theta)\equiv \frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}$$6 spikes for SAPO

These results are presented as evidence that the combination of cached BPTT action-gradients, importance-weight clipping, and KL regularization yields both robust learning dynamics and high sample efficiency in challenging differentiable environments (Zhong et al., 8 Aug 2025). Because the numerical examples are limited to the stated benchmark suite, any broader generalization to other environment classes would be an inference rather than a direct empirical finding.

7. Interpretation, relation to prior methods, and scope

RPO is framed as combining the low-variance, low-bias RPG estimator with the stability associated with PPO’s surrogate and trust-region ideas (Zhong et al., 8 Aug 2025). The substantive novelty claimed by the paper is the establishment of a connection between PPO’s surrogate objective and RPG, together with the demonstration that the reparameterization gradient of a PPO-like surrogate objective can be computed efficiently using BPTT. The source explicitly characterizes that bridge as previously unexplored and non-trivial (Zhong et al., 8 Aug 2025).

Relative to vanilla RPG, the main distinction is not the use of pathwise gradients themselves, but the addition of a clipped surrogate objective tailored for RPG and stabilized further by KL regularization. Relative to model-free PPO, RPO preserves proximal-style sample reuse while operating in a setting where gradients are propagated through reparameterized actions and differentiable dynamics. Relative to SHAC- or SAPO-style short-horizon RPG methods, RPO is presented as a more stable sample-reuse procedure rather than as a different variance-reduction primitive (Zhong et al., 8 Aug 2025).

A common misconception would be to treat RPO as merely “PPO with reparameterized actions.” The source indicates a narrower and more technical claim: the crucial contribution is an efficient way to compute the reparameterization gradient of a PPO-like surrogate by regenerating latent noise for stored actions and reusing cached action-gradients across multiple epochs. Another possible misconception would be to assume that clipping alone is sufficient for stability. The paper directly rejects that view for very stiff or long-horizon tasks, motivating the explicit KL penalty (Zhong et al., 8 Aug 2025).

Within the scope defined by the source, RPO is best understood as a proximal optimization layer for differentiable-simulator policy gradients: it leaves the underlying actor-critic and variance-reduction ecosystem largely intact, but changes the update geometry and sample-reuse regime in a way intended to control instability without discarding the efficiency advantages of pathwise differentiation (Zhong et al., 8 Aug 2025).