Double-prioritized State Recycling (DPSR)

• Double-prioritized State Recycling is a reinforcement learning strategy that integrates prioritized sampling and prioritized replacement to maintain high-quality, up-to-date transitions.
• It introduces state recycling to refresh stale transitions, thereby improving sample efficiency and achieving superior performance on Atari and classic control benchmarks.
• The method incorporates importance-sampling corrections and controlled hyperparameter annealing to mitigate bias, ensuring robust learning progress in Deep Q-Networks.

Double-prioritized State Recycling (DPSR) is a reinforcement learning experience replay strategy that augments Deep Q-Networks (DQN) by integrating two axes of prioritization—sampling and replacement—as well as an explicit state-recycling mechanism. DPSR aims to ensure that a replay buffer contains transitions that are both highly informative and up-to-date with respect to the agent’s current policy, addressing limitations in prior experience replay methods such as uniform or prioritized experience replay (PER). Empirical evaluation demonstrates that DPSR significantly improves sample efficiency and ultimate performance on a broad set of Atari games and classic control tasks, outperforming both uniform and PER baselines (Bu et al., 2020).

1. Principle and Motivation

In standard DQN with uniform replay, an agent’s experience buffer $\mathcal{B}$ contains past transitions $(s_i, a_i, r_i, s'_i)$, sampled and replayed uniformly with probability $1/|\mathcal{B}|$. Prioritized Experience Replay (PER) advances this by sampling transitions with probability proportional to their TD error, thereby concentrating updates on more “surprising” or important experiences. However, PER’s FIFO buffer replacement policy ignores the value of old transitions whose learning potential may persist, and overlooks the possibility that transitions initially deemed unimportant could become relevant following policy updates. DPSR addresses both issues by (a) employing double prioritization—prioritized sampling and prioritized replacement—and (b) refreshing “stale” or mis-prioritized transitions via state recycling.

Double prioritization involves:

• Prioritized Sampling: Transitions are sampled for gradient updates according to their TD error, with the sampling exponent $\alpha(t)$ annealed over time.
• Prioritized Replacement: When the buffer reaches capacity, transitions to be evicted are selected with probability favoring low-priority (low TD error) entries, controlled by an exponent $\gamma(t)$.

State recycling periodically rejuvenates a subset of low-priority transitions by simulating from their stored states with the current policy, thereby repopulating the buffer with more informative transitions in regions that may have been stale or under-represented.

2. Mathematical Model

The replay buffer is formalized as $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$, storing each transition's state, action, reward, next state, current priority, and insertion timestep.

2.1. Priority Assignment:

Transition $\mathcal{T}_i$ receives proportional priority: $p_i = |\delta_i| + \varepsilon$ where $$\delta_i = r_i + Q_{\text{target}}(s'_i,\;\arg\max_a Q(s'_i,a)) - Q(s_i,a_i)$$, and $\varepsilon>0$ ensures nonzero probability for all entries.

2.2. Sampling Probability:

At step $(s_i, a_i, r_i, s'_i)$0, the sampling probability for transition $(s_i, a_i, r_i, s'_i)$1 is

$(s_i, a_i, r_i, s'_i)$2

with $(s_i, a_i, r_i, s'_i)$3.

2.3. Replacement Probability:

When $(s_i, a_i, r_i, s'_i)$4 is full, $(s_i, a_i, r_i, s'_i)$5 candidate transitions are drawn for eviction according to

$(s_i, a_i, r_i, s'_i)$6

where $(s_i, a_i, r_i, s'_i)$7 biases selection towards low-error (low-priority) entries; eviction among candidates proceeds via oldest-first.

2.4. State Recycling:

Every $(s_i, a_i, r_i, s'_i)$8 steps, rather than standard replacement, $(s_i, a_i, r_i, s'_i)$9 candidates are drawn via $1/|\mathcal{B}|$0. For each index $1/|\mathcal{B}|$1:

1. Retrieve $1/|\mathcal{B}|$2, discard $1/|\mathcal{B}|$3.
2. Compute current greedy action $1/|\mathcal{B}|$4; if $1/|\mathcal{B}|$5, force a random $1/|\mathcal{B}|$6.
3. Advance $1/|\mathcal{B}|$7 in the environment to observe $1/|\mathcal{B}|$8.
4. Construct transition $1/|\mathcal{B}|$9 and assign $\alpha(t)$0.

Of these $\alpha(t)$1 recycled transitions, the entry with the lowest $\alpha(t)$2 replaces its predecessor.

2.5. New Transitions:

A new transition is assigned $\alpha(t)$3 for immediate eligibility in training.

2.6. Importance-Sampling Correction:

To correct for nonuniform sampling, an importance weight is applied: $\alpha(t)$4 with normalization $\alpha(t)$5 and $\alpha(t)$6 annealed to 1 during training.

3. Algorithmic Realization

The DPSR variant of DQN is structured as follows:

$\mathcal{T}_i$1

Hyperparameters are tuned per-task, with typical values: minibatch size $\alpha(t)$7, learning rate $\alpha(t)$8, replay capacity $\alpha(t)$9, $\gamma(t)$0, $\gamma(t)$1, $\gamma(t)$2, $\gamma(t)$3, $\gamma(t)$4, $\gamma(t)$5, $\gamma(t)$6, $\gamma(t)$7, $\gamma(t)$8 (Bu et al., 2020).

4. Theoretical Properties

No convergence proofs are provided for DPSR. However, the introduction of prioritized sampling ($\gamma(t)$9) and prioritized replacement ($$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$0) alters the empirical data distribution, introducing bias that is mitigated through importance-sampling weight annealing ($$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$1). State recycling reduces estimation variance and bias by refreshing low-priority entries under newer policies, preventing neglect of state–action regions whose value may not be accurately reflected by outdated TD errors. Empirically, these mechanisms collectively yield 1.4–2× improvements in sample efficiency on Atari benchmarks (Bu et al., 2020).

5. Empirical Evaluation

DPSR is evaluated on CartPole-v0 and 24 Atari 2600 games (NoFrameskip-v4). In these benchmarks, DPSR achieves:

• Outperformance versus uniform replay (“Original”) in 23/24 games (“gold”), and 1/24 (“silver”).
• Mean score improvement vs. Original: +161.1%; vs. PER: +137.1%.
• Median improvement vs. Original: +87.0%; vs. PER: +92.1%.

Representative scores:

Game	Original	PER	DPSR
Breakout	87.0	136.4	281.4
Freeway	30.2	29.5	32.2
VideoPinball	7313.0	12025.1	51993.2

These improvements are robust across parameter choices and replay buffer sizes.

6. Practical Ramifications and Limitations

DPSR incurs additional computational and memory overhead. State recycling requires simulator steps for up to $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$2 candidates every $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$3 steps and the buffer must support snapshotting and restoring environment states. In practice, with $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$4 and $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$5, overhead remains modest.

The necessity of storing full state snapshots may be ameliorated via state compression (e.g., latent-feature replay), but this is not explored in (Bu et al., 2020). The bias introduced by double prioritization is manageable if importance-weights ($$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$6) are annealed strongly; otherwise, disabling recycling ($$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$7) can lead to inferior performance compared to PER due to unchecked bias. In highly stochastic or near-deterministic domains, recycling may revive noisy transitions, requiring careful tuning of $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$8, $$\mathcal{B} = \{\mathcal{T}_i = (s_i, a_i, r_i, s'_i, p_i, t_i)\}_{i=1}^N$$9, and $\mathcal{T}_i$0.

DPSR systematically enriches the replay buffer by favoring high-error transitions during sampling, evicting low-error ones in replacement, and rejuvenating stale entries through recycling. This orchestrated approach delivers substantial gains over existing methods on established benchmarks (Bu et al., 2020).