Buffer-Optimized PPO Methods

• Buffer-Optimized PPO is a set of methods that repurpose on-policy buffers for enhanced sample reuse through controlled clipping and update scheduling.
• These techniques modify the PPO loss by dynamically adjusting clipping ranges and incorporating KL-based constraints to maintain stable policy updates even with aggressive reuse.
• Hybrid strategies like FIFO trajectory replay and reuse of parameter checkpoints improve performance by balancing sample efficiency and trust-region stability.

“Buffer-Optimized PPO” (Editor’s term) denotes a family of proximal policy optimization methods in which the primary design question is how trajectory data, replayed rollouts, or policy/update checkpoints are retained and reused, rather than treating PPO solely as a clipped surrogate loss. In the original formulation, PPO already operates on a short-lived on-policy buffer: it collects a batch under a frozen behavior policy, computes advantages, performs multiple epochs of minibatch optimization, and then discards the batch (Schulman et al., 2017). Subsequent work extends this buffering logic along several axes: time-varying clipping over the current batch (Farsang et al., 2021), KL-based objectives that tolerate much heavier optimization on a fixed batch (Xie et al., 2024), FIFO trajectory replay with best-trajectory-aware sampling (Liu et al., 21 Feb 2025), multi-policy replay over the last $M$ policies (Wang et al., 10 Feb 2026), outer-loop buffering of update vectors (Tan et al., 2024), and checkpoint-buffer exploration in parameter space (Zhang et al., 30 Sep 2025).

1. PPO as a short-lived on-policy buffer method

The canonical PPO-Clip objective is

$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$

with

$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$

The algorithm collects a batch of $N\times T$ transitions under $\pi_{\theta_{\text{old}}}$, computes $\hat{A}_t$, and performs $K$ epochs of minibatch SGD on that fixed dataset before discarding it (Schulman et al., 2017).

This structure makes PPO explicitly buffer-based, although the buffer is ephemeral rather than replay-style. The key technical role of clipping is to support repeated reuse of the same samples without allowing the current policy to drift arbitrarily far from the behavior policy that generated them. Within a single outer iteration, clipping or KL control bounds how far the ratio $r_t(\theta)$ can move from $1$, which is why PPO can perform multiple epochs over one batch while remaining approximately on-policy (Schulman et al., 2017).

A common misconception is that PPO is “buffer-free.” More precisely, PPO avoids a long-lived replay buffer of the DQN or SAC type, but it does rely on a finite on-policy buffer that is deliberately reused several times. From a buffer-optimization perspective, the original PPO design problem is already one of choosing batch size, number of epochs, minibatch size, and clip range so that the temporary buffer is exploited efficiently without destabilizing the trust-region approximation (Schulman et al., 2017).

2. On-policy buffer optimization through clipping schedules

A direct way to optimize PPO’s short-lived buffer is to change not the objective form, but the schedule of the clipping parameter. “Decaying Clipping Range in Proximal Policy Optimization” keeps the standard PPO clipped surrogate and replaces the constant $\epsilon$ by a training-time-dependent $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$0 (Farsang et al., 2021): $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$1 The paper studies two schedules: $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$2

Algorithmically, the schedule is applied once per PPO outer iteration, not per step, and remains constant across all epochs and minibatches for the batch collected in that iteration. The rollout, advantage computation, number of epochs, learning rate, GAE-$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$3, discount factor, value loss coefficient, and entropy coefficient are unchanged; only the clipping range is scheduled (Farsang et al., 2021).

The empirical picture is task-dependent. Over the entire training period, linear decay is consistently best in CartPole, Pendulum, and Acrobot. In locomotion, Hopper is similar across methods with linear decay marginally better, whereas exponential decay yields steeper learning curves and notably higher final returns on Walker2D and higher final returns on HalfCheetah. The clipping fraction is high early, declines mid-training, and rises again later; with decaying $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$4, the late rise is more pronounced because the shrinking clip range increases the number of updates that hit the bounds (Farsang et al., 2021).

In buffer terms, this changes the effective value of repeated passes over the same batch. Large early $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$5 permits larger departures from $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$6 while optimizing a fixed batch; small late $$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$7 makes the same buffer more conservative. This suggests that the marginal utility of extra epochs is itself schedule-dependent, even when the data collection scheme is unchanged.

3. KL-controlled objectives and aggressive reuse of fixed batches

A stronger response to PPO’s buffer-reuse limits is to redesign the surrogate so that repeated optimization on a fixed batch remains explicitly KL-controlled. “Simple Policy Optimization” replaces PPO’s ratio clipping with clipping of the per-state KL divergence between the old and new policies (Xie et al., 2024). Let

$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$8

and define the per-sample objective

$$L^{\text{CLIP}}(\theta) = \hat{\mathbb{E}}_t \Big[ \min\big( r_t(\theta)\hat{A}_t,\; \operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat{A}_t \big) \Big],$$9

When $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$0, this reduces exactly to the standard surrogate; outside the trust region, the multiplicative factor penalizes further KL increase (Xie et al., 2024).

The practical consequence is unusually direct for buffer optimization. In the paper’s over-optimization experiment, the number of policy update epochs on the same data is increased from 8 to 1000. For PPO, average KL becomes very large and optimization degrades. For SPO, average KL stays below $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$1 even with 1000 epochs, and the surrogate objective improves smoothly as $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$2 is relaxed. The method is also more robust to deeper networks: KL remains low and stable as network depth increases, whereas PPO’s KL can grow large and some Atari environments show performance collapse or near-complete failure to learn (Xie et al., 2024).

The significance is structural. PPO’s clipping only indirectly constrains KL, so aggressive reuse of a fixed batch can turn the on-policy buffer into a de facto off-policy source within the same iteration. SPO instead makes the reuse budget a first-class trust-region object. For buffer-optimized PPO designs that aim to extract many more gradient steps per collected sample, this KL-centered formulation is a direct answer to the failure mode of “too many epochs on the same data.”

4. Explicit trajectory replay buffers

Beyond optimizing the current batch, another line of work introduces persistent trajectory replay while attempting to preserve PPO-style stability.

Variant	Buffer object	Update rule
PPO	Current $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$3 batch	Multiple epochs, then discard
HP3O	FIFO trajectory replay buffer	Best trajectory plus random sampled trajectories
ExO-PPO	Last $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$4 policies in replay buffer	Off-policy ratios with extended surrogate and KL penalty

HP3O augments PPO with a trajectory replay buffer using a FIFO strategy so as to keep only recent trajectories and attenuate data distribution drift (Liu et al., 21 Feb 2025). Each update batch contains the trajectory with the best return in the buffer, $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$5, plus other trajectories sampled uniformly at random. HP3O remains PPO-style at the objective level, but the data are partially off-policy. HP3O+ further introduces a best-trajectory-induced baseline $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$6, which yields an additional value penalty term in the policy improvement bound and is reported to reduce variance (Liu et al., 21 Feb 2025).

Theoretical analysis in HP3O extends the policy-improvement lower bound to a mixture over prior policies in the buffer, using importance ratios $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$7 and a TV-distance penalty. Empirically, the method is evaluated on continuous control tasks including HalfCheetah, Hopper, Walker2d, Swimmer, InvertedPendulum, LunarLander, CartPole, and Humanoid. On HalfCheetah, PPO achieves $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$8, HP3O $$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\text{old}}}(a_t\mid s_t)}.$$9, and HP3O+ $N\times T$0. On Hopper, PPO achieves $N\times T$1, P3O $N\times T$2, and HP3O+ $N\times T$3. On Swimmer, PPO achieves $N\times T$4, HP3O $N\times T$5, and HP3O+ $N\times T$6 (Liu et al., 21 Feb 2025).

ExO-PPO pushes the replay formulation further by organizing the trajectories generated by the past $N\times T$7 policies in a replay buffer for off-policy training (Wang et al., 10 Feb 2026). The surrogate no longer uses hard clipping alone. Instead, it introduces a segmented exponential “extended ratio” $N\times T$8 that matches PPO inside $N\times T$9 and decays smoothly outside, combined with a KL penalty: $\pi_{\theta_{\text{old}}}$0 The replay buffer stores data from the last $\pi_{\theta_{\text{old}}}$1 policies, along with behavior probabilities and GAE-based advantages computed once at collection time and then reused. The default setting uses $\pi_{\theta_{\text{old}}}$2, $\pi_{\theta_{\text{old}}}$3, and $\pi_{\theta_{\text{old}}}$4 for the extended ratio; the paper reports that moderate $\pi_{\theta_{\text{old}}}$5 values give the best balance, and that ExO-PPO improves faster and reaches higher asymptotic performance than PPO, ESPPO, and P3O-Scopic in most Atari games while maintaining smoother ratio statistics (Wang et al., 10 Feb 2026).

These two approaches represent different buffer doctrines. HP3O optimizes trajectory selection within a recency-constrained FIFO memory and explicitly privileges the best recent rollout. ExO-PPO instead treats the buffer as a finite window over recent behavior policies and redesigns the surrogate so that off-policy replay remains usable rather than being nullified by hard clipping.

5. Buffers over updates and policy checkpoints

Buffer optimization in PPO is not restricted to replaying trajectories. It can also operate on histories of update vectors or policy parameters.

“Beyond the Boundaries of Proximal Policy Optimization” decomposes PPO into an inner loop that estimates an update vector and an outer loop that applies it (Tan et al., 2024). Defining

$\pi_{\theta_{\text{old}}}$6

standard PPO becomes

$\pi_{\theta_{\text{old}}}$7

which is equivalent to gradient ascent with unity learning rate on the outer gradient. Outer-PPO exposes this outer optimizer explicitly: $\pi_{\theta_{\text{old}}}$8 and also studies outer Nesterov momentum and biased initialization of the inner loop using an accumulated momentum vector (Tan et al., 2024).

The paper evaluates these methods against an aggressively tuned PPO baseline on Brax, Jumanji, and MinAtar. Non-unity learning rates and momentum both achieve statistically significant improvement on Brax and Jumanji, given the same hyperparameter tuning budget, while no improvement is reported on MinAtar. In effect, the sequence $\pi_{\theta_{\text{old}}}$9 functions as a reusable update history. This suggests that “buffer-optimized PPO” need not mean trajectory replay; it can also mean temporal reuse of outer gradients or optimizer state across iterations.

ExploRLer uses a different non-trajectory buffer: an anchor set of iteration-end checkpoints $\hat{A}_t$0 (Zhang et al., 30 Sep 2025). PPO or TRPO is run unchanged within an iteration; after each iteration, the final checkpoint is added to $\hat{A}_t$1. Every $\hat{A}_t$2 iterations, ESA explores the parameter-space neighborhood of these anchors, generates candidate policies, evaluates each candidate with 3 episodes, and replaces the next policy by the best candidate. The anchor set is then cleared (Zhang et al., 30 Sep 2025).

The core claim is that standard on-policy updates traverse only a sparse path through parameter space, leaving nearby “empty spaces” unexplored. ExploRLer is therefore buffer-optimized at the parameter level: it stores recent policies rather than transitions, then uses a zero-order search to relocate PPO to a better local region without increasing the number of gradient updates. On MuJoCo, this yields improvements such as Ant $\hat{A}_t$3 for PPO versus $\hat{A}_t$4 for ExploRLer-P, Hopper $\hat{A}_t$5 versus $\hat{A}_t$6, Walker2d $\hat{A}_t$7 versus $\hat{A}_t$8, and Humanoid $\hat{A}_t$9 versus $K$0 (Zhang et al., 30 Sep 2025).

6. Theoretical lenses, misconceptions, and open problems

A recurring criticism of PPO is that ratio clipping is a heuristic proxy for trust-region control rather than a direct solution of a bounded-ratio optimization problem. “Bounded Ratio Reinforcement Learning” formalizes that alternative viewpoint (Ao et al., 20 Apr 2026). BRRL imposes the per-state-action constraint

$K$1

derives an analytic optimal solution,

$K$2

and proves monotonic performance improvement. BPO then minimizes an advantage-weighted divergence between the parameterized policy and this analytic target, providing a new theoretical lens on why PPO’s clipped loss works and where it diverges from a more principled bounded-ratio update (Ao et al., 20 Apr 2026).

This sharpens several misconceptions. First, buffer optimization is not synonymous with off-policy replay. It includes short-lived on-policy batch scheduling, outer-gradient histories, and checkpoint buffers. Second, ratio clipping does not reliably bound KL under aggressive optimization; SPO is explicit that ratio clipping only indirectly and imperfectly controls KL, and BRRL argues that there is a significant disconnect between trust-region foundations and PPO’s heuristic clipped objective (Xie et al., 2024, Ao et al., 20 Apr 2026). Third, more epochs are not universally beneficial. PPO’s original design assumes moderate multi-epoch reuse; SPO shows that extreme reuse becomes tenable only when the trust region is reformulated more directly (Schulman et al., 2017, Xie et al., 2024).

Open directions in the literature are correspondingly varied. One direction is adaptive control of the trust region: decaying clipping proposes fixed linear or exponential schedules and explicitly suggests more sophisticated adaptive clipping based on clip fraction, policy KL divergence, or learning progress (Farsang et al., 2021). Another is dynamic buffer management: HP3O remarks that prioritized replay based on loss is a promising extension, while ExO-PPO shows that moderate replay windows over recent policies can balance sample efficiency and stability (Liu et al., 21 Feb 2025, Wang et al., 10 Feb 2026). A third is auxiliary-model exploitation. POME keeps policy optimization on-policy but learns transition and reward models from the same trajectories, uses the discrepancy between model-free and model-based one-step value targets as an exploration bonus, and explicitly suggests extensions using experience replay buffers, prioritized by prediction error or discrepancy (Pan et al., 2018).

Taken together, the literature does not converge on a single canonical “buffer-optimized PPO.” Instead, it identifies a design space. One axis asks how aggressively a single on-policy batch should be reused. A second asks whether trajectories from recent policies can be replayed with principled importance weighting and trust-region control. A third asks whether the reusable object should be trajectories, update vectors, or policy checkpoints. The most consistent conclusion across these directions is that sample reuse alone is not the difficult part; the difficult part is controlling the behavioral-policy mismatch induced by reuse, whether that mismatch is expressed through clipping fractions, per-state KL, replay-window age, or bounded likelihood ratios.