Successful Transition Replay (STR)

• STR is a replay strategy that explicitly over-samples successful episodes in off-policy reinforcement learning by storing and replaying entire successful trajectories.
• It employs dedicated replay buffer structures, including demonstration zones and global success pools, with TD-error prioritization to enhance value propagation in sparse reward tasks.
• Empirical results, notably in the DER formulation for robotic assembly, show that STR notably shortens training time and boosts success rates while managing overfitting risks.

Searching arXiv for the cited papers to ground the article. Successful Transition Replay (STR) denotes a replay principle in which successful transitions or episodes are explicitly over-sampled in experience replay. In the provided reinforcement-learning literature, the clearest concrete realization is Dynamic Experience Replay (DER), which augments distributed off-policy learning by reserving replay capacity for demonstrations and for successful episodes generated by the agent during training, then periodically refreshing that pool as training proceeds (Luo et al., 2020). Earlier work on replaying transition sequences provides a sequence-level precursor in which high-reward, high-TD-error trajectories are selected, stored, and recombined for additional off-policy updates (Karimpanal et al., 2017). The acronym STR is context-dependent: in other arXiv works it denotes State Transformer in autonomous driving (Sun et al., 2023) and Simultaneous Transmit and Receive in Wi-Fi 7 MLO (Choorakuzhiyil et al., 7 Jan 2025), not Successful Transition Replay.

1. Definition and terminological scope

In the DER formulation, “Successful Transition Replay (STR) would be a technique that explicitly over-samples successful transitions or episodes in the replay buffer,” and DER is presented as “essentially a concrete realization of this idea, designed for distributed off-policy RL in robotic assembly” (Luo et al., 2020). This suggests that STR is best treated as a descriptive category for replay mechanisms that detect successful trajectories, preserve them under special storage rules, and bias sampling toward them.

DER defines demonstrations as two classes of successful experience: full successful episodes recorded from humans in simulation, and successful episodes generated by RL agents during training (Luo et al., 2020). The method therefore does not restrict “success” to imitation data. Instead, demonstrations become a dynamic subset of replay memory that evolves as the policy improves.

A central terminological point is that DER replays entire successful episodes, not just isolated terminal events. In the same source, “a successful transition” is any transition in a successful episode; the method “does not isolate just the final transition but replays the entire sequence” (Luo et al., 2020). That distinction matters because the learning signal is propagated through the ordered state-action-reward chain that led to success, rather than only through the terminal reward spike.

The phrase itself is not standardized across the broader arXiv corpus. In particular, “STR” is consistently used only as State Transformer in “Large Trajectory Models are Scalable Motion Predictors and Planners” (Sun et al., 2023), and only as Simultaneous Transmit and Receive in “Comparison of STR and EMLSR Performance in Wi-Fi 7 MLO” (Choorakuzhiyil et al., 7 Jan 2025). Any encyclopedia treatment therefore has to distinguish Successful Transition Replay from these unrelated usages.

2. Formal setting in off-policy reinforcement learning

The DER paper places STR-like replay in a standard episodic RL setting, with an optimal policy $\pi_\theta(a_t \mid s_t)$ minimizing expected trajectory loss,

$$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$

where

$\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$

and

$$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$

Transitions are stored as

$e_t = (s_t, a_t, s_{t+1}, r_t)$

(Luo et al., 2020).

In the robotic assembly tasks used for DER, the observation is 13-dimensional and the action is 6-dimensional. The state is

$$s_t = (x,y,z,q_x,q_y,q_z,q_w, f_x,f_y,f_z,t_x,t_y,t_z),$$

combining Cartesian pose and force/torque sensing, while the action is

$a_t = (v_x,v_y,v_z,w_x,w_y,w_z),$

a Cartesian velocity command (Luo et al., 2020).

Success is task-specific but explicit. For peg-in-hole and lap-joint, the reward is

$$r = \begin{cases} -|g - x|, & |g - x| > \epsilon,\ -|g - x| + R, & |g - x| \le \epsilon, \end{cases}$$

with terminal reward $R = 1000$ for peg-in-hole and $R = 100$ for lap-joint (Luo et al., 2020). An episode is successful if the object reaches the goal region $$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$0 and receives the large positive reward. Under an STR interpretation, this yields a binary success predicate at the episode level, while all transitions within that episode become success-marked replay candidates.

DER is designed to be plug-and-play with arbitrary off-policy algorithms such as DDPG, DQN, and distributed Ape-X DDPG (Luo et al., 2020). The underlying update equations remain standard. For DDPG-style learning, the critic target is

$$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$1

with loss

$$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$2

and the actor objective is

$$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$3

DER changes the training distribution by changing what is sampled, not by changing these update equations (Luo et al., 2020).

3. Replay-buffer structures and sampling bias

DER implements success emphasis through replay structure. Each replay buffer $$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$4 is split into a demonstration zone with capacity $$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$5 and a regular zone for all other transitions, while a global success pool $$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$6 stores complete successful episodes from all workers (Luo et al., 2020). Periodically, each replay buffer samples one successful episode from $$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$7 and loads its transitions into the demonstration zone, discarding the oldest demonstrations when necessary. Before training, buffers may additionally be initialized with zero, one, or multiple human demonstrations.

The paper evaluates four replay structures for each task (Luo et al., 2020):

Structure	Initialization	Dynamic behavior with DER
No Human Demonstrations	Demo zones start empty	Agent successes populate $$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$8 and then the demo zones
Same One-shot Demo in All Buffers	One human demo episode in every buffer	Agent successes replace or augment demos
All Human Shots in All Buffers	All human demonstrations in every buffer	Demo zone remains dynamic
Each Buffer with a Different One-shot Demo	Each buffer gets one unique human demo	Agent successes still refresh demo zones

Sampling is further biased by prioritized replay. Each transition has priority

$$\min_{\pi_\theta}\; \mathbb{E}_{\tau \sim \pi_\theta}\bigl(l(\tau)\bigr),$$9

with absolute TD error used for the calculation, and sampling probability

$\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$0

with $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$1 in the hyperparameters (Luo et al., 2020). Successful transitions are not given a separate priority formula. Their advantage is structural: they occupy reserved capacity in the demonstration zone and are continually refreshed with successful episodes. Prioritized replay then operates over the union of demonstration and regular zones.

This combination produces two biases toward success. One is a capacity bias, since demonstrations and agent successes cannot be displaced immediately by ordinary failures. The other is a TD-error bias, since successful transitions near the terminal reward can also attract large priority values. The DER source states this explicitly as the interaction between a dedicated demonstration zone and prioritized experience replay (Luo et al., 2020).

4. Empirical evidence in robotic assembly

DER is evaluated on robotic tight-fitting joint assembly tasks based on force/torque and Cartesian pose observations: peg-in-hole and lap-joint (Luo et al., 2020). The paper reports that Dynamic Experience Replay is “a crucial ingredient” that either “largely shortens the training time in these challenging environments or solves the tasks that the vanilla Ape-X DDPG cannot solve” (Luo et al., 2020).

For peg-in-hole, the reported outcomes depend on replay structure. With No Human Demos, both vanilla Ape-X DDPG and DER reach high success rates, but DER reaches that level roughly twice as fast. With One Shot in All Buffers, DER yields significantly higher average success rates than vanilla. With All Shots in All Buffers, DER again outperforms vanilla with markedly higher success rates and better reward curves. With Each Buffer Taking One Shot, performance is similar and DER does not bring a clear benefit (Luo et al., 2020).

For lap-joint, which is described as harder, with tight tolerance and no chamfer, success rates are generally lower across settings. DER improves performance for No Human Demos and All Shots in All Buffers, while One Shot in All Buffers and Each Buffer Taking One Shot show no consistent gain relative to vanilla Ape-X DDPG (Luo et al., 2020). The same source states that in some settings the vanilla algorithm fails to solve the task or is very slow, while DER either solves it within the training budget or significantly boosts success rate and reward.

The tasks illuminate why successful transitions matter. The reward is effectively sparse: most of the signal is negative distance, with a large positive spike only inside a small goal region. The contact dynamics are complex, small pose errors can produce jamming or frictional failure, and the tolerances are on the order of millimeters and degrees (Luo et al., 2020). In such conditions, random exploration is unlikely to generate successful episodes early, so replay mechanisms that preserve rare success traces become disproportionately valuable.

DER also provides simulation-to-real evidence. A policy trained purely in simulation on the lap-joint task with the No Human Demos + DER structure was deployed on a KUKA KR60 robot with a real force/torque sensor and grippers, succeeding 3 out of 3 trials on the physical robot (Luo et al., 2020). A plausible implication is that repeated replay of successful trajectories can help stabilize policies around insertion behaviors that are robust enough to transfer across the simulation-to-real gap.

5. Related replay paradigms

A sequence-level antecedent appears in “Experience Replay Using Transition Sequences,” which proposes selecting and replaying sequences of transitions rather than isolated samples (Karimpanal et al., 2017). The method identifies successful sequences that lead to high rewards and large TD errors, stores them in a replay library, and constructs additional virtual transition sequences by combining parts of different trajectories that intersect in state space. Replayed end-to-end, these sequences allow value-function information to “trickle down to larger sections of the state/state-action space” (Karimpanal et al., 2017).

That paper formalizes a successful transition sequence $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$2 by storing a recent trajectory prefix ending at a high-reward event together with absolute TD errors, and prioritizes admission into the library by the maximum absolute TD error in the candidate sequence (Karimpanal et al., 2017). It therefore instantiates a version of STR at the sequence level: success is tied to high reward, while replay emphasis is modulated by learning potential as measured by TD error.

The reported quantitative gains are large in sparse off-policy settings. In the modified navigation environment, with difficulty ratio $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$3, regular Q-learning yields $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$4, while transition-sequence replay with default memory parameters yields $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$5 (Karimpanal et al., 2017). In mountain car, with $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$6, regular Q-learning yields $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$7 and sequence replay yields $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$8 (Karimpanal et al., 2017). These results support the broader STR proposition that replaying successful trajectories as ordered chains can accelerate propagation of value in rare-reward regimes.

Outside RL, “REPOT: Recoverable Program-of-Thought via Checkpoint Repair” introduces a deterministic verified replay that walks an LLM-generated plan through an environment to the first invalid transition, extracts the maximal verified prefix, and then resumes from the verified checkpoint with one repair call (Mazaheri, 28 May 2026). The paper does not use the phrase “Successful Transition Replay,” but its replay primitive preserves all valid transitions up to the failure boundary. It reports gains of +3 to +11pp across four closed-model configurations on PuzzleZoo-775 and shows on Derail-550 that every condition with checkpoint information clears $\tau = \{s_0, a_0, s_1, a_1, \dots, s_T, a_T\},$9 on GPT-medium and $$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$0 on Gemini, versus $$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$1 for error-only feedback (Mazaheri, 28 May 2026). This suggests a broader, cross-domain meaning of STR as replay that preserves the successful portion of a trajectory and conditions subsequent computation on that verified prefix.

A further conceptual extension appears in “Replay and compositional computation,” which argues that replay can implement compositional computation by assembling role-bound entities into relational structures that yield qualitatively new knowledge (Kurth-Nelson et al., 2022). In that framing, successful replay is not limited to accurate sampling of $$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$2; it can also involve composing known relational patterns into novel but structurally valid trajectories or compounds (Kurth-Nelson et al., 2022).

6. Limitations, misconceptions, and design implications

The main practical limitation in DER-style STR is overfitting to success. The source explicitly notes a “risk of overfitting to successes”: if the success buffer is too large or sampling is too biased, the policy may underexplore and fail to generalize to new initial configurations or perturbations (Luo et al., 2020). DER addresses this by keeping the success-oriented zone small relative to the full replay buffer; the example given is roughly 1% of a 2M-transition buffer, or 20,000 transitions (Luo et al., 2020).

A second limitation is sensitivity to demonstration quality and replay structure. In lap-joint, DER does not improve performance under some buffer structures, and the source identifies one possible reason as demonstrations that are suboptimal or mismatched to training observations because there is no haptic feedback in the demonstrations (Luo et al., 2020). Hyperparameters such as capacity $$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$3, refresh rate, and prioritized replay parameters $$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$4 and $$\pi_\theta(\tau) = p(s_0)\prod_{t=1}^T p(s_t \mid s_{t-1}, a_{t-1})\, \pi_\theta(a_t \mid s_t).$$5 also affect outcome (Luo et al., 2020).

A common misconception is to equate STR with a single established named algorithm. The provided literature does not support that interpretation. Rather, DER is a concrete realization in distributed off-policy RL (Luo et al., 2020), transition-sequence replay is a related sequence-based realization (Karimpanal et al., 2017), and RePoT provides an analogous verified-prefix mechanism in LLM planning (Mazaheri, 28 May 2026). By contrast, “STR” in autonomous-driving trajectory modeling and Wi-Fi 7 networking refers to entirely different constructs (Sun et al., 2023, Choorakuzhiyil et al., 7 Jan 2025).

Taken together, these works support a stable encyclopedia definition: Successful Transition Replay is a family of replay strategies that detect successful trajectories, retain them under privileged storage or verification rules, and resample them so that rare but high-value experience has disproportionate influence on subsequent learning or repair. In sparse, contact-rich robotics this can shorten training time or convert failure into solvability (Luo et al., 2020); in sequence-based off-policy learning it can improve average return by propagating value over long chains (Karimpanal et al., 2017); and in verifier-backed LLM planning it can transform all-or-nothing execution into checkpointed recovery from partial success (Mazaheri, 28 May 2026).