Experience Replay Buffer in RL

• Experience Replay Buffer is a finite memory storage that decouples experience collection from parameter updates by storing transitions (s, a, r, s') for random mini-batch sampling.
• Sampling strategies like uniform, prioritized, and RPE-based methods optimize data reuse and variance reduction, boosting learning efficiency in various RL setups.
• Advanced buffer structures—such as stratified, topological, and distilled buffers—address challenges in continual, multi-agent, and safe reinforcement learning.

An experience replay buffer is a central data structure in reinforcement learning (RL) that enables agents to store and reuse past interactions with the environment. Buffers decouple the temporal sequence of experience collection from that of parameter updates, providing statistical and computational benefits that are foundational to modern deep RL, continual learning, and off-policy methods.

1. Core Structure and Functionality

The experience replay buffer $\mathcal{B}$ is typically a finite-capacity memory that stores tuples $(s_t, a_t, r_t, s_{t+1})$, and, in some variants, additional data such as policy actions, goals, or model outputs. As new experiences arrive, old entries are evicted according to a policy (FIFO is standard, but variants exist). During training, mini-batches are repeatedly sampled for updates. This decoupling of data acquisition and gradient steps enables critical algorithmic properties:

• Breaking temporal correlations: Random sampling reduces the temporal dependencies between transitions, mitigating the bias in stochastic updates and stabilizing learning dynamics.
• Reusing rare or valuable experiences: Especially in sparse/rare event, multi-agent, or continual learning settings, the buffer ensures efficient reuse of informative samples.
• Supporting off-policy algorithms: Methods such as Q-learning and actor-critic frameworks fundamentally depend on a replay buffer to enable learning from trajectories generated by outdated or exploratory policies.

2. Sampling Strategies and Their Optimization

2.1 Uniform and Non-Uniform Sampling

The canonical approach is uniform sampling, where each stored transition has equal probability $p_i = 1/N$ of being drawn for training. However, research demonstrates that uniform sampling is not optimal in various contexts—particularly for non-i.i.d. data, rare events, or when explicit mitigation of catastrophic forgetting is desired (Krutsylo, 16 Feb 2025). Empirical and theoretical work shows that, even with randomly chosen, fixed weights $\{w_i\}$, non-uniform sampling distributions ($p_i = w_i/\sum_j w_j$) can robustly outperform the uniform baseline across continual learning scenarios, buffer sizes, and models.

2.2 Prioritized Strategies

The most influential non-uniform sampling regime is Prioritized Experience Replay (PER), which assigns each transition a sampling priority, often proportional to the magnitude of the temporal-difference (TD) error:

$$p_i = |\delta_i|^\alpha, \quad P(i) = \frac{p_i}{\sum_j p_j}$$

PER has demonstrated improved convergence and data efficiency in off-policy deep RL, particularly in discrete control domains (Lahire et al., 2021). However, it is sensitive to hyperparameters, and priorities can become stale due to infrequent updates.

2.3 Importance Sampling and Optimal Policy

From a stochastic optimization perspective, the ideal sampling distribution minimizes the variance of the stochastic gradient (Lahire et al., 2021). The optimal per-sample probability is proportional to the norm of the gradient of the loss for each sample:

$$p_i^* \propto \|\nabla_\theta \ell(Q_\theta(x_i), y_i)\|_2$$

As this is often intractable, practical methods such as LaBER (Large Batch Experience Replay) estimate up-to-date surrogate priorities (e.g., TD error) on large buffer batches and sample importance-weighted sub-batches, yielding robust performance improvements and variance reduction beyond uniform or PER (Lahire et al., 2021).

2.4 Biologically-Inspired and Alternative Signals

Recent work explores reward prediction error (RPE)-based prioritization as a more effective informativeness signal in continuous control than TD error. In RPE-PER (Yamani et al., 30 Jan 2025), priority is set by the discrepancy between a reward-predicting critic and actual rewards:

$p_i = |R_\theta(s_i, a_i) - r_i|^\alpha + \epsilon$

This approach leverages explicit reward modeling and is empirically validated to accelerate and stabilize learning over standard PER in challenging tasks.

3. Buffer Organization Beyond the Flat List

3.1 Structured Replay Buffers

Extensions to buffer structure increase sampling and propagation efficiency by partitioning or organizing transitions:

• Event Tables and Stratified Sampling: SSET partitions the buffer into event tables based on user- or system-defined event conditions and trajectory histories, allowing explicit over-sampling of rare or crucial transitions; rigorous correction terms ensure unbiased updates (Kompella et al., 2022).
• Graph-Based Topological Buffers: TER encodes experiences as a transition graph, enabling backward (topological) value backups to efficiently propagate Q-values from terminal to starting states. This approach achieves faster convergence and superior performance in both tabular and high-dimensional tasks (Hong et al., 2022).
• Compressed and Distilled Buffers: Continual learning settings motivate buffer distillation, reducing memory needs by synthesizing a small set of maximally informative, possibly synthetic samples (as few as one per class), which can retain competitive task performance (Rosasco et al., 2021).

3.2 Refreshing and Evolving Buffer Content

Whereas standard buffers are static after insertion, frameworks like LiDER periodically revisit stored states using the current policy to "refresh" experiences, storing only improved (higher-return) rollouts in a parallel buffer (Du et al., 2020). This targets the issue of "stale" memories predominantly generated by obsolete policies.

4. Hyperparameterization, Computational Trade-Offs, and Stability

4.1 Buffer Size Trade-Offs

Empirical studies reveal a non-monotonic dependency between replay buffer size and learning performance. Too-small buffers encourage overfitting to recent transitions and underrepresent global state coverage, while overly large buffers introduce "stale" samples generated under outdated policies, slowing learning and harming convergence (Zhang et al., 2017, Fedus et al., 2020). The effect is algorithm and environment-dependent:

• Agents employing uncorrected $n$-step returns (multi-step targets) benefit strongly from larger replay capacity, with gains in stability and performance (Fedus et al., 2020).
• Hybrid sampling—drawing a fraction of samples from a "recent" sub-buffer—can recover or improve stability at $O(1)$ cost for large buffers (Zhang et al., 2017).

4.2 Replay Ratio and Bandwidth

The replay ratio (number of learning updates per environment interaction) critically influences data utilization. Increasing this ratio typically improves performance, with care needed to avoid learning from excessively outdated samples (Fedus et al., 2020). Distributed frameworks such as Reverb (Cassirer et al., 2021) provide rate-limited, sharded buffer implementations that maintain target sample:insert ratios across thousands of concurrent clients, facilitating scalable distributed RL.

4.3 Variance Reduction and Theoretical Guarantees

Experience replay is shown, via $U$-/ $V$-statistic modeling, to provably reduce the estimator variance of function approximation and kernel methods, in both policy evaluation and supervised settings (Han et al., 1 Feb 2025). Replay-based estimators outperform standard plug-in estimators in both RMSE and variance, particularly in data-scarce or non-i.i.d. environments.

Parameter	Effect on Learning	Trade-off
Buffer size	Diversity, decorrelation vs. staleness	Memory, stability
Replay ratio	Data utilization, faster learning	Staleness, computation
Sampling scheme	Focus, sample efficiency, variance reduction	Complexity, bias

5. Applications and Extended Roles

5.1 Continual Learning

Experience replay is a principal mechanism for mitigating catastrophic forgetting by interleaving seen data with new task data. While most approaches assume uniform sampling, evidence indicates that non-uniform (possibly adaptive) policies can further improve retention across buffer sizes and domains (Krutsylo, 16 Feb 2025). Recent strategies enforce prediction consistency not only on replayed (buffered) samples but also between current and previous model states over new data, countering overfitting when memory is scarce (Zhuo et al., 2023).

5.2 Safe and Policy-Shaped Learning

Replay buffer sampling can be designed to bias towards safety or other desirable policy traits. For example, upweighting high-variance or negative-reward transitions during replay leads to safer, risk-averse policies; convergence is guaranteed as long as the replay probability function stabilizes (Szlak et al., 2021).

5.3 Hierarchical and Multi-Agent RL

In multi-agent settings with sparse rewards, replay buffers can serve as a substrate for curriculum generation. Agents can be assigned subgoals extracted from stored experiences via utility-based selection, and intrinsic rewards shaped via Q-function-based actionable distances, boosting coordinated exploration and credit assignment (Jeon et al., 2022).

6. Open Problems and Future Directions

Despite substantial progress, several dimensions of experience replay remain active areas of research:

• Principled, adaptive sampling mechanisms: Empirical findings consistently indicate that uniform sampling is suboptimal, but the question of task- and agent-optimal, computationally feasible non-uniform policies is largely open (Krutsylo, 16 Feb 2025, Zha et al., 2019).
• Buffer compression, synthetic memories: Advances in continual learning call for buffer-efficient distillation, selection, or even synthesis strategies, with the potential for privacy-preserving or ultra-long-horizon learning (Rosasco et al., 2021).
• Theoretically grounded replay scheduling: Recent frameworks provide finite-time error or variance reduction guarantees (Han et al., 1 Feb 2025, Lim et al., 2023), but more work is needed to bridge theory, practice, and complex neural architectures.
• Replay as a policy or property-shaping tool: Replay buffer design can intentionally bias not just sample efficiency but also the learned policy's safety, fairness, and compositionality, offering a flexible axis of algorithmic control (Szlak et al., 2021).

7. Summary Table: Sampling Methodologies and Their Effects

Method	Sampling Distribution	Informative Signal	Key Benefits
Uniform	$p_i = 1/N$	None	Simplicity, easy decorrelation
PER	$p_i \propto |\delta_i|^\alpha$	TD error	Sample efficiency, focus on learning
RPE-PER	$p_i \propto \text{RPE}_i^\alpha$	RPE	Superior in continuous domains
Stratified (SSET)	User-defined	Events, trajectories	Overweights rare/crucial subsequences
Topological (TER)	Graph order	Transition dependencies	Accelerated Q-value propagation
Distilled Replay	N/A (synthetic)	Optimized representations	Ultra-compact buffer, limited forgetting

Experience replay buffers, and the methods by which they are managed and sampled, are a cornerstone of scalable, stable, and sample-efficient reinforcement learning. Ongoing research continues to expand their roles, theoretical footing, and the scope of their application.