Adaptive Replay Buffer (ARB)

Updated 18 December 2025

Adaptive Replay Buffer (ARB) is a dynamic memory management mechanism that prioritizes experience sampling using signals like TD-error, policy consistency, and entropy measures.
It employs adaptive sampling and update strategies to improve sample efficiency, balance learning stability against plasticity, and mitigate catastrophic forgetting in non-stationary environments.
ARB is applied in various domains including offline-to-online RL, off-policy model-free RL, and continual learning, showing significant performance gains over static replay methods.

An Adaptive Replay Buffer (ARB) is a memory management mechanism in reinforcement learning and continual learning that adaptively prioritizes the sampling or retention of past experiences based on their estimated relevance to current learning objectives. Unlike standard replay buffers that employ uniform or fixed-ratio sampling, an ARB dynamically adjusts the sampling distribution or buffer content using behavior-aware, performance-driven, or information-theoretic signals. The ARB paradigm addresses critical issues of sample efficiency, stability versus plasticity, and catastrophic forgetting in high-dimensional, non-stationary environments.

1. Motivation and Conceptual Foundations

Conventional experience replay relies on uniform or hand-tuned sampling from a fixed buffer of past transitions. Such schemes are suboptimal in several regimes: (1) offline-to-online RL, where the transition from a fixed offline dataset to online data introduces a trade-off between conservatism and adaptability; (2) off-policy RL and model-based RL, where data diversity and on-policyness are critical; (3) continual learning and lifelong learning, where catastrophic forgetting and memory imbalance are prevalent.

Adaptive Replay Buffers are designed to overcome these limitations by introducing dynamically varying data selection or retention criteria. These criteria can be based on:

On-policyness or policy consistency
TD-error, prediction uncertainty, or learning progress
Forgetting or interference metrics in continual learning
Task or class coverage for memory balance

The ARB framework thus subsumes and extends traditional mechanisms such as Prioritized Experience Replay, bandit-based experience selection, and contrastive prototype memory, placing emphasis on the closed-loop adaptivity to agent state and learning dynamics (Song et al., 11 Dec 2025, Smith et al., 18 Apr 2024, Rezaei et al., 9 Oct 2024, Li et al., 2023, Zhang et al., 2 Feb 2024).

2. Formal Definitions and Computational Mechanisms

The unifying element of ARB designs is a two-part mechanism:

Adaptive Sampling: Selection of transitions/trajectories from the buffer according to dynamically computed importance weights.
Adaptive Update/Retention: Buffer update strategy that retains, discards, or augments samples based on informativeness, diversity, or risk of forgetting.

Several representative computational strategies include:

On-Policyness-Based Weighting (O2O RL):

$\widetilde{\Ocal}(s,a;\pi_\theta) = \pi_\theta(a|s)$

Weights are clipped and normalized within the buffer, and, for variance reduction, aggregated at the trajectory level via geometric mean:

$\omega(\tau) = \exp\left(\frac{1}{\lambda T} \sum_{t=0}^{T-1} \log \widetilde{\Ocal}(s_t,a_t;\pi_\theta)\right)$

(Song et al., 11 Dec 2025)

Policy-Learned Replay Selection:

The replay policy $\phi$ assigns sampling probabilities $\lambda_i$ to each experience, updated by REINFORCE using a replay-reward signal calculated as the improvement in main agent return (Zha et al., 2019).

Bandit-Based Cluster Sampling (Continual Learning):

Arms $A_1,\dots,A_K$ correspond to data clusters. Each time-step, a Boltzmann policy samples clusters based on an exponentially-weighted moving average of cluster-specific forgetting:

$p_i^{(j)} = \frac{\exp(\mu_i^{(j)}/t)}{\sum_k \exp(\mu_k^{(j)}/t)}$

(Smith et al., 18 Apr 2024)

Class/Task-Balanced Buffering:

Examples are retained to maintain perfect balance across past classes/tasks and chosen based on informativeness, e.g., confidence variance or boundary proximity (Rezaei et al., 9 Oct 2024, Zhang et al., 2 Feb 2024).

Conflict-Driven Recall and Entropy-Balanced Retention:

The buffer preferentially replays examples exhibiting higher loss increase if the model were updated only on new data, and buffer replacement maximizes class entropy while protecting high-interference memories (Li et al., 2023).

These mechanisms are easily distinguished from prior approaches relying on fixed sampling ratios or global FIFO retention, offering continuous adaptation to agent state, environment shifts, or the curriculum of seen tasks.

3. Integration with RL and CL Architectures

The ARB paradigm functions as a "drop-in" replacement for classical buffer modules in a broad spectrum of frameworks:

Offline-to-Online RL: ARB seamlessly wraps buffer sampling in O2O RL algorithms such as IQL, SAC, CQL, or their derivatives, leaving underlying loss functions and optimizers unchanged (Song et al., 11 Dec 2025).
Off-Policy Model-Free RL: Approaches such as Augmented Replay Memory (Ramicic et al., 2019) and Experience Replay Optimization (Zha et al., 2019) augment classical DDPG/TD3 with adaptive sampling, buffer update, or reward shaping.
Distributed RL: Dynamic Experience Replay (Luo et al., 2020) supports distributed buffers divided into demonstration and agent success zones, with prioritized, adaptive sampling and periodic success-based buffer refresh.
Model-Based RL: Local-forgetting ARB (Rahimi-Kalahroudi et al., 2023) supports deep world model training, preventing local-overfitting by removing outdated transitions within a learned neighborhood of the current state.
Continual Learning: Multiple ARB variants in CL maintain compact, balanced, information-rich rehearsal sets, integrating directly with standard and prototype-based rehearsal/backbone networks (Zhang et al., 2 Feb 2024, Aghasanli et al., 9 Apr 2025, Rezaei et al., 9 Oct 2024).

4. Empirical Evaluation and Comparative Performance

Experiments consistently demonstrate substantial gains of ARB-type buffers over static and uniform sampling baselines:

Setting	Standard Baseline	ARB Variant	Final/Forgetting Improvement
O2O RL (D4RL)	Fixed mixing (e.g., 50/50)	On-policyness ARB (Song et al., 11 Dec 2025)	+5–10 points asymptotic; stable adaptation; AntMaze normalized return +3–13
Continuous Control RL	Uniform replay (DDPG)	AMR, ERO (Ramicic et al., 2019, Zha et al., 2019)	Sample efficiency up to 30–50% higher, final return +10–35%
Continual Learning (Split-CIFAR10, iCaRL/ER)	iCaRL: 32.62%, ER: 29.74%	CORE ARB: 37.95% (Zhang et al., 2 Feb 2024)	$\Delta$ +5–8%; worst-task +6.3%
CL, OOD Generalization	GSS ACC: 31.9%	ACR ACC: 36.1%, OOD +13.41% (Rezaei et al., 9 Oct 2024)	Significant OOD and class/task balance gains
CL, Memory efficiency	ER-AML: 79.63% (CIFAR100)	iSL-LRCP: 83.29% (Aghasanli et al., 9 Apr 2025)	ARB can exceed offline retrain, unsupervised ARB competitive

Early-stage performance dips in O2O RL are mitigated, sample efficiency is improved, memory buffer class/task balance is strengthened, and risk of catastrophic forgetting is reduced. ARBs also yield major improvements in OOD generalization compared to traditional rehearsal-based continual learning baselines.

5. Design Analysis and Ablation Insights

Extensive ablations across ARB papers highlight several central findings:

Aggressiveness of Prioritization: Stronger emphasis on recent/online data (e.g. lower $\lambda$ in on-policyness) enables faster adaptation but increases variance; moderate interpolation yields optimal overall performance (Song et al., 11 Dec 2025).
Trajectory vs Transition Prioritization: Aggregating at trajectory or cluster-levels reduces variance and yields higher final returns compared to noisy per-transition weighting (Song et al., 11 Dec 2025).
Entropy/Balance Constraints: Enforcing class or cluster balance via bandit or entropy constraints avoids overfitting long-tail or majority classes (Li et al., 2023, Smith et al., 18 Apr 2024, Rezaei et al., 9 Oct 2024).
Buffer Refresh and Locality: Local-forgetting mechanisms provide rapid adaptation to non-stationarities without global catastrophic forgetting (Rahimi-Kalahroudi et al., 2023).
Task-Adaptive Slot Allocation: Dynamic, interference/forgetting-driven slot allocation outperforms static rehearsal budgets, especially in highly imbalanced or long task sequences (Zhang et al., 2 Feb 2024).

Hyperparameters such as temperature in Boltzmann sampling (Smith et al., 18 Apr 2024), replay fraction (Li et al., 2023), on-policyness temperature (Song et al., 11 Dec 2025), and support-band width (Aghasanli et al., 9 Apr 2025) require moderate tuning based on architecture and task set.

6. Limitations, Theory, and Future Directions

ARB strategies are predominantly heuristic, with theoretical understanding largely invoking variance-bias tradeoffs and principles from on-policy sampling or memory consolidation. No published convergence or regret guarantees are available for general ARB schemes. Documented limitations include:

Oversampling high-probability suboptimal offline data in on-policyness weighting (Song et al., 11 Dec 2025)
Potential for starvation of rare exploratory transitions
Oscillations or high variance if adaptivity is too aggressive, as with low temperature parameters or rapid updating
Memory/computational footprint for large buffer or per-sample tracking in certain continual learning regimes (Smith et al., 18 Apr 2024)

A plausible implication is that further research into principled, theoretically grounded adaptivity criteria and improved estimation of sample informativeness could yield greater stability and robustness. Hybridization with orthogonal regularization or parameter-isolation methods is also compatible and may enhance performance (Smith et al., 18 Apr 2024).

Adaptive Replay Buffers now represent a core methodological advance supporting scalable, stable, and robust learning in both high-performance reinforcement learning and data-efficient continual learning paradigms.