Dynamic Prioritized Experience Replay in RL

• Dynamic Prioritized Experience Replay is a mechanism in reinforcement learning that dynamically adjusts sampling priorities based on TD errors to focus on informative transitions.
• It employs both proportional and rank-based prioritization schemes with online updates and importance-sampling corrections to mitigate bias from non-uniform replay.
• The approach has been shown to improve learning efficiency in various settings, from Atari games to distributed architectures, while inspiring several advanced variants.

Dynamic Prioritized Experience Replay denotes a family of replay mechanisms in off-policy reinforcement learning in which the replay distribution is allowed to evolve with the agent’s current learning state rather than remaining uniform over stored transitions. In its canonical form, Prioritized Experience Replay (PER) replaces uniform replay with sampling probabilities monotone in transition priority, typically derived from temporal-difference (TD) error magnitude, and pairs this with importance-sampling correction to control the bias introduced by non-uniform replay. The defining dynamic features are that priorities are updated online as new TD errors are computed, newly inserted transitions are typically given maximal priority so that they are replayed at least once, and the importance-sampling exponent is annealed during training to recover less biased updates near convergence (Schaul et al., 2015).

1. Canonical formulation

The original PER framework starts from standard experience replay: transitions are stored in a replay memory and later resampled in order to break temporal correlations, avoid forgetting rare but informative experience, and trade environment interaction for computation and memory. Uniform replay treats all stored transitions equally, whereas PER assumes that transitions with larger expected learning progress should be replayed more often. In the original formulation, the magnitude of the TD error is used as a proxy for that learning progress (Schaul et al., 2015).

For a transition $i = (s_i, a_i, r_i, \gamma_i, s'_i)$, the TD error in DQN is

$$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$

and in Double DQN it is

$$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$

PER defines two stochastic prioritization schemes. In proportional prioritization,

$$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$

where $\epsilon > 0$ prevents starvation and $\alpha \in [0,1]$ controls prioritization strength. In rank-based prioritization, transitions are sorted by descending $|\delta|$, assigned $p_i = 1/\mathrm{rank}(i)$, and sampled from a power-law distribution $P(i) \propto \mathrm{rank}(i)^{-\alpha}$. The rank-based variant is described as robust to outliers because it depends on ranks rather than magnitudes and retains heavy tails, promoting diversity (Schaul et al., 2015).

Because non-uniform replay changes the effective training distribution, PER uses per-sample importance sampling (IS) weights

$$w_i^* = \left(\frac{1}{N \cdot P(i)}\right)^\beta,\qquad w_i = \frac{w_i^*}{\max_j w_j^*},$$

with replay size $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$0 and correction exponent $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$1. The weighted squared TD error becomes

$$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$2

The dynamic character of PER lies in the update cycle: after each learning step, sampled transitions receive refreshed priorities based on their newly computed TD errors, while newly inserted transitions receive maximal priority to guarantee replay. In the Atari Double DQN configuration reported in the original study, proportional prioritization used $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$3 and linearly annealed $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$4 from $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$5 to $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$6, while rank-based prioritization used $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$7 and annealed $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$8 from $$\delta_i = r_i + \gamma_i \max_{a'} Q(s'_i, a'; \theta^-) - Q(s_i, a_i; \theta),$$9 to $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$0 (Schaul et al., 2015).

2. Replay systems, data structures, and scaling

The canonical proportional implementation uses a sum-tree in which leaves store priorities and internal nodes store segment sums. Insert, update, and single-sample operations are $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$1, and minibatch sampling with stratified segments is $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$2. This makes PER practical for $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$3, whereas naive cumulative-sum recomputation would be $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$4 per sample. Rank-based PER can instead use an array-based binary heap to approximate sorted order, together with stratified sampling over equal-probability partitions of the rank-based cumulative distribution (Schaul et al., 2015).

In the Atari configuration of the original paper, the replay buffer size was $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$5, minibatch size $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$6, and replay period $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$7, corresponding to one gradient update per four new transitions and roughly eight replays per transition on average. The learning rate was reduced by $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$8 relative to the DQN baseline, from $$\delta_i = r_i + \gamma_i Q\!\left(s'_i, \arg\max_{a'} Q(s'_i, a'; \theta); \theta^-\right) - Q(s_i, a_i; \theta).$$9 to $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$0, to counter larger gradient magnitudes. Rewards and TD errors were clipped to $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$1 for stability (Schaul et al., 2015).

The distributed Ape-X architecture generalized dynamic PER to a many-actor setting by decoupling acting from learning. Hundreds of CPU actors interact with their own environments using a shared policy, compute initial priorities from local TD errors, and push prioritized transitions into a centralized replay store; a single GPU learner samples from that store, updates the network, recomputes TD errors under current parameters, and writes updated priorities back. In Atari, Ape-X used 360 CPU actors producing about $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$2K raw FPS and a single P100 GPU learner processing about $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$3K transitions per second, with replay capacity soft-limited to about $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$4M transitions, $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$5 multi-step returns, $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$6, $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$7, actor parameter refresh every 400 frames, and target-network updates every 2500 learner batches (Horgan et al., 2018). The actor-side computation of initial priorities was introduced specifically to avoid the strong recency bias of max-priority insertion in a setting where many actors continuously flood replay.

3. Empirical effects and performance regimes

In the original Atari study, DQN with prioritized replay outperformed DQN with uniform replay on 41 of 49 games. For the rank-based variant, the median normalized score increased from $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$8 to $$p_i = |\delta_i| + \epsilon,\qquad P(i) = \frac{p_i^\alpha}{\sum_j p_j^\alpha},$$9. For Double DQN, PER increased the median from $\epsilon > 0$0 to $\epsilon > 0$1 and the mean from $\epsilon > 0$2 to $\epsilon > 0$3, with the mean noted as dominated by Video Pinball. The median performance-equivalence point was reached in about $\epsilon > 0$4 to $\epsilon > 0$5 of training steps depending on the variant, indicating roughly $\epsilon > 0$6 faster aggregate learning (Schaul et al., 2015).

Large-scale distributed replay amplified these gains. Ape-X DQN reported median human-normalized scores of $\epsilon > 0$7 with no-op starts and $\epsilon > 0$8 with human starts after 5 days and 22,800M frames, outperforming Rainbow, C51, Prioritized Dueling DQN, DQN, and Gorila under the reported evaluation setup (Horgan et al., 2018). The associated ablations attributed the performance gain not to recency alone, but to the combination of diverse exploration from many actors and prioritized concentration of learner updates on informative transitions.

The empirical picture is not uniformly positive across domains. In a later study focused on generalization, prediction tasks, and noisy settings, PER still improved value propagation in tabular settings, but behavior changed substantially with neural networks. Certain mitigations—delaying target-network updates and using estimates of expected TD errors—avoided large error spikes, yet prioritized variants generally did not outperform uniform replay in control tasks, and none consistently outperformed uniform replay in the reported neural control experiments (Panahi et al., 2024). This divergence established a persistent distinction between the original Atari success case and later evidence from noisy, bootstrapped, or heavily generalizing function approximators.

4. Bias, instability, and theoretical reinterpretations

The standard failure modes of dynamic PER are well characterized. Oversampling high-$\epsilon > 0$9 items can reduce diversity, increase sensitivity to noise or bootstrapping artifacts, and destabilize learning because both priorities and target values evolve as $\alpha \in [0,1]$0 and $\alpha \in [0,1]$1 change. The original PER paper mitigated these effects through stochastic prioritization with $\alpha \in [0,1]$2, importance sampling with annealed $\alpha \in [0,1]$3, max-normalized IS weights, stratified sampling, and a reduced learning rate (Schaul et al., 2015).

Later analyses made two criticisms sharper. First, raw TD errors can chase stochasticity: in neural settings, bootstrapping and generalization can create large transient TD errors that are then repeatedly replayed, producing error spikes. Second, importance-sampling correction is optimizer-dependent: with plain SGD, PER+ISR was observed to learn more slowly than iid replay in several settings, while with Adam or RMSprop the same correction could remain viable under noise (Panahi et al., 2024). This work proposed Expected Priorities (EPER), which replace instantaneous $\alpha \in [0,1]$4 with an online estimate $\alpha \in [0,1]$5 of expected absolute TD error, and also suggested lagged value functions for priority computation.

A separate theoretical line showed that non-uniform replay can be reinterpreted as a transformed uniformly sampled loss. For a base loss $\alpha \in [0,1]$6, PER with prioritization exponent $\alpha \in [0,1]$7 and IS exponent $\alpha \in [0,1]$8 has a uniformly sampled equivalent

$\alpha \in [0,1]$9

under stop-gradient treatment of the priority term. This analysis explains why PER combined with MSE can bias learning toward outliers when the effective loss power exceeds 2, and it motivates corrected variants such as Loss-Adjusted Prioritized replay (LAP) and the uniformly sampled Prioritized Approximation Loss (PAL) (Fujimoto et al., 2020). In that sense, part of PER’s effect can be understood as loss shaping rather than merely sample selection.

5. Alternative priority signals and adaptation to non-stationarity

The most active post-Atari line of work modifies the prioritization signal itself rather than the replay architecture.

Variant	Priority signal	Dynamic mechanism
EPER (Panahi et al., 2024)	Expected absolute TD error $|\delta|$0	Online smoothing of priorities; mitigates transient spikes
RPE-PER (Yamani et al., 30 Jan 2025)	Reward prediction error $|\delta|$1	Priorities change as the reward head improves
DEER (Duan et al., 18 Sep 2025)	Discrepancy of Environment Dynamics and TD error	Switches weighting after detected environment changes
LFIW replay (Sinha et al., 2020)	Density-ratio weight $|\delta|$2	Continuously reweights critic loss toward the current policy’s stationary distribution

RPE-PER replaces TD-error prioritization with reward prediction error produced by an Enhanced Model Critic Network that predicts Q-values, rewards, and next states. In the reported MuJoCo experiments, it used $|\delta|$3, $|\delta|$4, batch size 256, and 1M environment steps, and it outperformed TD-error PER on most tasks for both TD3 and SAC, with only minor exceptions such as Swimmer under TD3 and Hopper under SAC (Yamani et al., 30 Jan 2025). The rationale is that immediate reward prediction can be less distorted by bootstrapping bias than critic TD error in continuous control.

DEER targets explicitly non-stationary environments. It introduces the Discrepancy of Environment Dynamics,

$|\delta|$5

estimated using critic snapshots at detected environment changes, and combines it with TD error through a change-dependent gate. In the reported MuJoCo benchmarks with non-stationary friction and damping offsets, DEER improved performance by $|\delta|$6 over the best state-of-the-art replay baseline on average, and by about $|\delta|$7 under 200% offsets (Duan et al., 18 Sep 2025). This directly addresses the classical PER problem that post-change high TD errors may reflect obsolete transitions rather than genuinely useful ones.

A different adaptation, “Experience Replay with Likelihood-free Importance Weights,” abandons TD-error prioritization altogether and instead reweights critic updates by an estimate of the current policy’s stationary state-action distribution relative to the replay distribution. The key argument is that the Bellman evaluation operator is a $|\delta|$8-contraction in the weighted $|\delta|$9 norm only when the weighting matches $p_i = 1/\mathrm{rank}(i)$0, and the practical implementation uses a fast/slow buffer density-ratio estimator with self-normalization temperature $p_i = 1/\mathrm{rank}(i)$1 (Sinha et al., 2020).

6. Structural and domain-specific generalizations

Dynamic prioritized replay has also been generalized structurally beyond individual transitions. Prioritized Sequence Experience Replay (PSER) propagates a fraction of a newly observed high priority backward along the preceding episode window using an exponential decay with coefficient $p_i = 1/\mathrm{rank}(i)$2 and window $p_i = 1/\mathrm{rank}(i)$3. In Atari experiments the tuned values were $p_i = 1/\mathrm{rank}(i)$4, $p_i = 1/\mathrm{rank}(i)$5, and $p_i = 1/\mathrm{rank}(i)$6, and the reported human-normalized median score rose from $p_i = 1/\mathrm{rank}(i)$7 under PER to $p_i = 1/\mathrm{rank}(i)$8 under PSER. In Blind Cliffwalk, the paper proved an exponential-vs-linear separation in expected convergence steps between PER and PSER under its assumptions (Brittain et al., 2019).

Sparse-reward goal-conditioned control motivated Hindsight Goal Ranking (HGR), which prioritizes both episodes and hindsight goals by TD error within DDPG+HER. Episode-level priorities are based on the average absolute TD error over all transition/future-goal pairs in an episode, and within a sampled episode, future goals are sampled with probability proportional to their goal-conditioned TD error. On Fetch tasks, HGR reported substantial sample-efficiency gains over vanilla HER, PER extensions, EBP, and MEP, with particularly strong improvements on Push and Slide (Luu et al., 2021).

State-aware replay has been explored in online recommendation through Locality-Sensitive Experience Replay (LSER), which hashes high-dimensional states into buckets using locality-sensitive hashing and then samples within similar-state buckets using reward-driven ranking rather than TD error. In VirtualTB, LSER reached good policies in about 30,000 episodes versus about 50,000 for AER, DER, PER, and SER, while also reducing runtime relative to PER and especially AER (Chen et al., 2021). Robotic manipulation has prompted yet another split design: D-SPEAR uses a shared buffer but decouples critic sampling toward high-TD transitions and actor sampling toward low-TD transitions, with a coefficient-of-variation-driven anchor that adaptively blends uniform and prioritized replay. On robosuite Lift and Door, it outperformed SAC, TD3, and DDPG in both final performance and stability (Zhang et al., 28 Mar 2026).

Taken together, these extensions indicate that “dynamic” in dynamic prioritized experience replay no longer refers only to refreshed TD-error magnitudes. It includes dynamic correction schedules, dynamic choice of prioritization signal, dynamic separation of actor and critic data requirements, dynamic conditioning on current state or current policy, and dynamic adaptation to distribution shift, noise, and environment change. The central design problem remains the same across variants: to focus computation on transitions that are informative under the agent’s present epistemic state without collapsing coverage, amplifying stochasticity, or hard-coding a replay distribution that quickly becomes obsolete.