Efficient RL Training for LLMs with Experience Replay

Abstract: While Experience Replay - the practice of storing rollouts and reusing them multiple times during training - is a foundational technique in general RL, it remains largely unexplored in LLM post-training due to the prevailing belief that fresh, on-policy data is essential for high performance. In this work, we challenge this assumption. We present a systematic study of replay buffers for LLM post-training, formalizing the optimal design as a trade-off between staleness-induced variance, sample diversity and the high computational cost of generation. We show that strict on-policy sampling is suboptimal when generation is expensive. Empirically, we show that a well-designed replay buffer can drastically reduce inference compute without degrading - and in some cases even improving - final model performance, while preserving policy entropy.

• The paper presents a theoretical framework for optimal replay buffer design in RL fine-tuning of LLMs, balancing off-policy bias, sample diversity, and compute cost.
• The paper demonstrates compute savings of up to 40% with improved test accuracy and reduced variance across benchmarks like OpenR1-Math-220k.
• The paper applies refinements such as positive-bias sampling and AsymRE loss to stabilize training and enhance generalization in cost-intensive scenarios.

Efficient Experience Replay in RL Fine-tuning of LLMs

Introduction

The paper "Efficient RL Training for LLMs with Experience Replay" (2604.08706) systematically reframes the use of experience replay (ER) in reinforcement learning (RL) fine-tuning of LLMs, directly contesting the prevailing "generate-then-discard" approach. Whereas ER is foundational in many RL domains, its adoption in LLM post-training has been limited—driven by a belief that high-fidelity on-policy sampling is essential to avoid deleterious off-policy biases and sample staleness. This work provides a comprehensive theoretical and empirical treatment demonstrating that, when rollout generation is computationally intensive, well-designed replay buffers can produce substantial efficiency gains without degrading, and in some cases improving, final model performance.

Theoretical Framework: Bias-Variance Trade-offs and Optimal Buffer Design

A primary contribution is the mathematical formalization of replay buffer design as a three-way trade-off between off-policy bias, sample diversity, and computational cost. The authors provide a non-convex stochastic optimization analysis showing that strict on-policy sampling is not compute-optimal when generation is expensive. They extend classical SGD bias-variance decomposition by accounting for the additional bias introduced by staleness (off-policiness) in replayed samples and the increased gradient estimator variance produced under higher staleness and lower sample diversity.

The main result identifies the optimal replay buffer design via the staleness horizon ($N/R$), replay ratio ($B/R$), and the rollout-to-backward-pass cost factor ($\mu$), demonstrating:

• As $\mu$ (rollout cost relative to backprop) increases, optimal buffer size and average sample reuse also increase.
• When off-policy induced variance and correlation are low, aggressive experience replay becomes strictly optimal.

Figure 1: Theoretical prediction that optimal staleness and replay ratio increase with the relative rollout cost $\mu$.

Additionally, the analysis quantifies how buffer-induced off-policy noise, minibatch sample correlation, and SGD step size interplay to define optimal hyperparameter regimes.

Asynchronous RL Pipeline and Replay Buffer Implementation

The standard industry pipeline for RL fine-tuning of LLMs is asynchronous: a pool of inference workers generates trajectories (on stale weights), while trainers update model parameters with only the freshest samples, discarding trajectories immediately after use. The introduction of a replay buffer decouples inference and training, improving GPU utilization and enabling repeated rollout consumption. The authors define compute cost and efficiency metrics, establishing that buffer-based pipelines with appropriate $W/T$ ratios (workers/trainers) amortize rollout costs and minimize trainer/inference stalling.

Figure 2: Impact of buffer size and trainer/inference worker ratios on off-policiness, replay ratio, and sample reuse statistics.

Empirical Results: Efficiency and Stability Gains

Main Findings

Empirical evaluation was performed using Qwen3-0.6B and Qwen2.5-7B models on the OpenR1-Math-220k and MATH benchmarks. Key results include:

• Compute savings of up to 40% without loss of final test accuracy by replaying trajectories with carefully chosen buffer sizes and replay ratios.
• Stabilized training—buffered runs exhibit significantly reduced variance in final accuracy across seeds and fewer optimization crashes, especially visible in moderate compute regimes.

  Figure 3: Experience replay enables faster and more robust training than the no-buffer baseline at iso-compute, on Qwen2.5-7B.

• Higher pass@$k$: Buffer configurations not only preserve but often improve pass@$k$ for $k > 1$, speaking to increased output diversity and generalization.

Figure 4: Test accuracy and pass@$k$ as a function of buffer size and compute: larger buffers aid generalization and stabilize training.

• Replay ratio and buffer size: Excessive sample reuse does eventually degrade performance. There is an empirically measurable optimal replay ratio, above which local diversity loss outweighs compute savings.
• Wall-time efficiency matches or exceeds predicted compute savings due to buffering-induced smoothing of trainer/inference worker utilization.

  Figure 5: Wall-time test accuracy reflects significant speed-up using replay buffers compared to on-policy training.

Pareto Frontier and Learning Rate Control

A substantial sweep over learning rates and buffer configurations delineated Pareto frontiers in compute vs. final accuracy, confirming that experience replay outperforms all strictly on-policy baselines when hyperparameters are optimally tuned.

Figure 6: Accuracy as a function of compute for buffer and non-buffer configurations across a large hyperparameter sweep, delineating distinct Pareto frontiers.

Buffer Refinements: Positive-bias Sampling and Loss Functions

Further refinements, including positive-bias sampling (retaining more correct samples), and utilizing the AsymRE loss (as opposed to importance-weighted GRPO), yield additional stability and accuracy improvements in the high-off-policy regime.

Figure 7: Positive-bias sampling and alternative losses increase the stability and peak accuracy of experience replay training.

Implications, Limitations, and Future Directions

This work thoroughly counters the prevailing belief in LLM RL fine-tuning that replay-induced off-policy effects irreparably harm performance. Instead, it demonstrates that sample efficiency and computational efficiency can be substantially improved via carefully tuned, theory-guided ER, and that regularization via mild staleness can, counterintuitively, mitigate overfitting and instability in RL fine-tuning. These results also have direct implications for the design of large-scale, cost-sensitive LLM alignment pipelines, enabling significant GPU hour reductions.

Notably, these findings are robust for small to mid-size open-source LLMs. Further validation for extreme-scale models is warranted, as the off-policy tolerance could vary with increased model and buffer dynamics. Sophisticated buffer implementations—non-uniform or prioritized sampling, loss-adaptive buffer management, and off-policy corrections—remain largely untapped for even greater efficiency.

Conclusion

The paper establishes that experience replay, grounded in precise theoretical analysis and thorough empirical validation, stands as a critical lever for compute-efficient RL fine-tuning of LLMs. A well-tuned buffer can yield substantial efficiency gains without loss of accuracy, challenging the long-standing on-policy dogma and broadening the horizons for practical, scalable RL post-training pipelines.