Prioritized Interactive Experience Replay

• Prioritized Interactive Experience Replay (PIER) is a mechanism that integrates non-uniform sampling with interactive elements like current policy, state, and external evaluators to guide experience reuse.
• It extends classical prioritized experience replay by incorporating context-aware signals such as semantic, causal, and policy-conditioned factors, enabling targeted re-exploration.
• Empirical studies show that while PIER variants improve performance in sparse-reward, long-horizon tasks, their benefits depend on the task complexity and associated computational trade-offs.

Prioritized Interactive Experience Replay (PIER) denotes a broad family of replay mechanisms in which the reuse of past experience is guided not only by non-uniform sampling, but also by interaction with the agent’s current policy, current state, environment, or an external evaluator. The phrase is not standardized across the literature, yet the underlying pattern is explicit in several lines of work: classical prioritized experience replay, double-prioritized state recycling, gain×need replay from successor representation, batch-level off-policy-aware selection, causal-pattern replay, semantic replay guided by vision–LLMs, and problem-level prioritization in RL post-training for LLMs (Schaul et al., 2015, Bu et al., 2020, Yuan et al., 2021, Sharony et al., 2 Feb 2026, Fatemi, 6 Jan 2026).

1. From uniform replay to prioritized replay

Experience replay stores transitions and reuses them for learning. In the standard formulation, a replay buffer $\mathcal{B}$ contains transitions such as $\mathcal{T} = (s,a,r,s')$, minibatches are sampled uniformly, and old experience is typically evicted in FIFO order when the buffer is full. This improves sample efficiency and reduces temporal correlation, but it treats all stored experience as equally useful (Schaul et al., 2015).

Prioritized Experience Replay (PER) replaces uniform sampling with a stochastic distribution based on temporal-difference error. In its proportional form, a transition receives priority

$p_i = |\delta_i| + \epsilon,$

and is sampled with probability

$P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$

with importance-sampling correction

$$w_i = \left(\frac{1}{N}\cdot \frac{1}{P(i)}\right)^\beta.$$

Here $\alpha$ controls prioritization strength and $\beta$ controls bias correction. PER was introduced in Deep Q-Networks and achieved a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games (Schaul et al., 2015).

The classical PER formulation already contains the core logic of prioritized replay: transitions with larger apparent learning signal are replayed more often. However, its notion of “importance” is still narrow. The score is usually derived only from current TD error, the replay buffer is still passive, and the sampling distribution does not depend on the agent’s current state, expected future visitation, semantic task progress, or external judgments. These limitations motivate broader PIER-like designs.

2. Modes of interactivity in replay

In the broader literature, replay becomes “interactive” when prioritization is coupled to some online context: the current state and policy, the ability to revisit stored states in the environment, a batch-level estimate of off-policyness, a causal analysis of trajectory motifs, a semantic evaluator, or current rollout statistics (Bu et al., 2020, Yuan et al., 2021, Cicek et al., 2021, Wang et al., 2024, Sharony et al., 2 Feb 2026, Fatemi, 6 Jan 2026).

Mechanism	Priority signal	Interactive component
DPSR	TD error at sampling and replacing	Reset to stored states and generate new transitions
PER-SR	gain $\times$ need	Sampling or update weight depends on current state and policy
KLPER	KL score of candidate batch	Choose batch closest to current behavior policy
CIER / CIPER	causal effect of TSCFs, optionally TD error	Recompute priorities from recent trajectory structure
VLM-guided replay	VLM clip score, optionally times $|\delta|$	External Video-QA model judges sub-trajectories
RL post-training prioritization	$\omega = p(1-p)$	Priorities update from current rollout success statistics

A recurrent misconception is that interactivity requires a human teacher. The literature shows a broader usage. In some methods the interaction is with the environment itself, as when stored states are revisited; in others it is with the current policy, as when successor representation or KL divergence is recomputed online; in still others it is with an automated evaluator such as a VLM or a test suite. Human feedback is only one possible source of interaction.

3. State- and policy-conditioned replay

A concrete PIER-like mechanism is Double-Prioritized State-Recycled Experience Replay (DPSR). DPSR extends PER in two directions. First, it performs double prioritization: prioritized sampling during training and prioritized replacing during storage. Second, it introduces state recycling: before evicting a low-priority experience, the agent resets the environment to the stored state, evaluates the state with the current Q-network, executes a new action, and generates a replacement transition. In the replacing stage, candidate eviction probabilities use a negative exponent,

$\mathcal{T} = (s,a,r,s')$0

so lower-priority transitions are more likely to be selected as eviction candidates. New transitions are initialized with maximal current priority, $\mathcal{T} = (s,a,r,s')$1, which makes them difficult to evict immediately and likely to be replayed early (Bu et al., 2020).

State recycling is the distinctive interactive component. Every $\mathcal{T} = (s,a,r,s')$2 timesteps, when the buffer is full, DPSR samples low-priority experiences, restores their stored states by “saving the full state of the environment,” queries the current Q-network for an action, and, if the new action matches the original one, forces diversity by choosing a different random action. The resulting recycled transition receives a new priority and can replace an older stored transition. Conceptually, the replay buffer is no longer only a dataset; it becomes a control surface for targeted re-exploration (Bu et al., 2020).

A second important line is PER with Successor Representation, which argues that usefulness is not only “gain,” approximated by TD error, but also “need,” defined as the expected discounted future occupancy of a state under the current state and policy:

$\mathcal{T} = (s,a,r,s')$3

In the tabular case this is exactly the successor representation entry $\mathcal{T} = (s,a,r,s')$4. The combined priority becomes state-dependent:

$\mathcal{T} = (s,a,r,s')$5

In deep variants, the same idea is used to reweight update magnitude rather than to resample the entire buffer, because recomputing need for every stored transition is expensive (Yuan et al., 2021).

These methods show two distinct meanings of interactivity. DPSR is interactive because it actively re-engages the environment from stored states. PER-SR is interactive because replay decisions depend on the agent’s current state and policy through successor representation. Both depart from passive replay, but they do so through different control variables.

4. Beyond raw TD error: reliability, causality, batches, and adaptive correction

Several later methods broaden the priority signal itself. KLPER does so at the batch level. Instead of maintaining per-transition priorities, it samples several candidate mini-batches, computes current-policy actions for the batch states, forms an action-difference matrix between current-policy actions and stored actions, fits a Gaussian “Batch Generating Policy” $\mathcal{T} = (s,a,r,s')$6, and selects the batch with minimal

$\mathcal{T} = (s,a,r,s')$7

The chosen batch is therefore the one most likely to have been generated by the current behavior policy, reducing off-policyness without persistent per-transition scores (Cicek et al., 2021).

ReaPER keeps TD error as a central signal but discounts it by target reliability. For transition $\mathcal{T} = (s,a,r,s')$8, reliability is estimated from downstream TD errors in the same trajectory, and priority is defined as

$\mathcal{T} = (s,a,r,s')$9

Terminal transitions have $p_i = |\delta_i| + \epsilon,$0, while earlier transitions receive lower reliability when unresolved future TD error remains large. The paper gives a convergence hierarchy in expected final error,

$p_i = |\delta_i| + \epsilon,$1

and argues that the reliability term approximates inverse target-variance weighting (Pleiss et al., 23 Jun 2025).

CIER replaces TD-error-centric prioritization with causal attribution over trajectory motifs. Episodes are viewed as multivariate time series, segmented into Time Series Causal Factors (TSCFs) by TICC and K-Means, then encoded as binary variables $p_i = |\delta_i| + \epsilon,$2 indicating the presence of each factor. GFCI is used to estimate a Partial Ancestral Graph, and Average Treatment Effects are computed as

$p_i = |\delta_i| + \epsilon,$3

These causal strengths are mapped back to constituent transitions and used as replay weights; in CIPER they are combined with PER-style TD-error priorities. The stated goal is both higher data utilization and a degree of explainability (Wang et al., 2024).

ALAP addresses a different problem: the bias introduced by non-uniform sampling itself. It retains PER-style priorities, but replaces heuristic linear annealing of $p_i = |\delta_i| + \epsilon,$4 with a Self-Attention-based estimator of training progress computed from uniformly sampled state–action pairs in a mirror buffer $p_i = |\delta_i| + \epsilon,$5. A Double-Sampling Mechanism separates the distribution used for learning from the distribution used to estimate $p_i = |\delta_i| + \epsilon,$6, and Huber loss is retained while priority clipping is removed. This suggests that PIER is not only about inventing richer priorities, but also about controlling the distribution shift that richer priorities create (Chen et al., 2023).

5. Semantic and problem-level generalizations

The PIER pattern extends beyond classical transition replay. In VLM-Guided Experience Replay, a frozen pre-trained Video-QA model scores visual clips

$p_i = |\delta_i| + \epsilon,$7

where $p_i = |\delta_i| + \epsilon,$8 is a rendered clip and $p_i = |\delta_i| + \epsilon,$9 is a natural-language prompt asking whether the clip contains clear goal satisfaction. The clip-level score is propagated to all transitions in the clip. Sampling uses a mixture

$P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$0

and in continuous control a boosted variant uses

$P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$1

Across game-playing and robotics, spanning discrete and continuous domains, this mechanism yields 11–52% higher average success rates and 19–45% improved sample efficiency compared to previous approaches (Sharony et al., 2 Feb 2026).

In code generation, the BTP pipeline—beam search sampling, testing, and prioritized experience replay—stores experiences

$P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$2

where $P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$3 is the problem, $P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$4 the test suite, $P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$5 a generated program, $P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$6 its model probability, and $P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$7 its test performance. Priority is defined by

$P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$8

or by a rank-based alternative $P(i) = \frac{p_i^\alpha}{\sum_k p_k^\alpha},$9. Replay is therefore driven by a blend of model confidence and external execution outcomes rather than by TD error (Chen et al., 2024).

In RL post-training for LLMs, prioritization can move to the problem level. For GRPO rollouts with binary rewards, if a problem has empirical success rate

$$w_i = \left(\frac{1}{N}\cdot \frac{1}{P(i)}\right)^\beta.$$0

then the variance of the group advantage is exactly $$w_i = \left(\frac{1}{N}\cdot \frac{1}{P(i)}\right)^\beta.$$1. The paper therefore defines a problem-level priority

$$w_i = \left(\frac{1}{N}\cdot \frac{1}{P(i)}\right)^\beta.$$2

optionally using an EMA-smoothed success rate and solved/unsolved pools with periodic retesting. The induced schedule emphasizes problems that are neither consistently solved nor consistently failed, rather than easier tasks early in training (Fatemi, 6 Jan 2026).

These examples show that PIER is not confined to transition-level value learning. The same structural idea appears whenever replay priority depends on evaluator interaction with clips, programs, or problems, and whenever that interaction is folded back into future data selection.

6. Empirical record, misconceptions, and open issues

Empirical results are consistently positive at the level of broad potential, but heterogeneous across tasks and mechanisms. PER itself improved Atari performance on 41 out of 49 games in its original DQN setting (Schaul et al., 2015). DPSR reported 23/24 gold medals and 1 silver across 24 Atari games, with average and median gains over both original replay and PER (Bu et al., 2020). PER-SR showed faster convergence in the Dyna-Q maze and Blind Cliffwalk and improved performance on most tested Atari games (Yuan et al., 2021). In autonomous navigation, DDQN combined with PER achieved the highest planning success rate and was explicitly credited with breaking through dead zones and improving path quality and safety (Lipeng et al., 2024). VLM-guided replay produced substantial gains in both success rate and sample efficiency in semantically rich, long-horizon tasks (Sharony et al., 2 Feb 2026).

A second misconception is that more prioritization is always better. The literature does not support that claim. In the DPSR Atari evaluation, original uniform replay outperformed PER in 14 games and even won “gold” in SpaceInvaders (Bu et al., 2020). In a finite-environment CartPole study, PER often learned faster in episodes but did not clearly surpass uniform replay in final performance and was computationally more expensive (Perkins et al., 5 Nov 2025). In “Advances in Experience Replay,” PER and several combinations with CER and HER performed worse than baseline on LunarLander-v2 and were not uniformly beneficial even in simpler settings (Wan et al., 2018). Purely prioritized semantic sampling without a uniform component also degraded performance in VLM-guided replay ablations (Sharony et al., 2 Feb 2026).

These outcomes indicate that PIER is best understood as a design space rather than a single method. Some variants privilege state revisitation, some privilege policy-conditioned need, some privilege semantic or causal structure, and some emphasize bias correction or reliability. Across these forms, the recurring technical tensions are the same: concentration versus diversity, speed versus bias, semantic relevance versus evaluator misalignment, and richer prioritization versus computational overhead. The available evidence suggests that the strongest gains arise in sparse-reward, long-horizon, semantically structured, or failure-rich regimes, whereas simpler or denser-reward settings can reduce or even reverse the benefit of prioritization.