HiER: Highlight Experience Replay in Deep RL

• Highlight Experience Replay (HiER) is an experience replay method that uses a dual-buffer system to selectively store and replay high-cumulative-reward episodes.
• It integrates with approaches like HER and PER, employing adaptive mixing and curriculum components (HiER+) to improve learning efficiency and robustness.
• Experimental results on robotic manipulation tasks demonstrate faster convergence, reduced variance, and superior success rates compared to traditional replay strategies.

Highlight Experience Replay (HiER) is a methodological advance in experience replay for deep reinforcement learning (RL) that exploits a dual-buffer system to prioritize and more frequently replay the most informative or high-value episodes, drawing inspiration from human learning’s retention of salient experiences. The primary goal is to improve both the efficiency and quality of learning, especially in challenging settings characterized by continuous state and action spaces, sparse rewards, and the absence of expert demonstrations. HiER has been proposed as a modular strategy that can be integrated with or without established techniques such as Hindsight Experience Replay (HER) and Prioritized Experience Replay (PER), and its most comprehensive instantiation, HiER+, further incorporates a curriculum learning component for data collection (Horváth et al., 2023).

1. Conceptual Motivation and Principle

At its core, HiER differentiates itself from traditional experience replay, in which all experience transitions are stored in a single buffer and accessed uniformly or probabilistically (e.g., via TD error), by maintaining an additional, dedicated buffer for “highlight” experiences. In HiER, after each episode, the undiscounted cumulative reward $R = \sum_{i=0}^{T} r_i$ is computed, and if this sum exceeds a specified threshold $\lambda$, all transitions from that episode are placed into the highlight buffer (in addition to being recorded in the standard buffer). This mechanism enables HiER to automatically accumulate demonstration-like episodes—those with exceptional returns—without reliance on human or external expert trajectories.

The replay process then samples mini-batches from both the standard buffer ($\mathcal{B}_{\text{ser}}$) and the highlight buffer ($\mathcal{B}_{\text{hier}}$) according to a mixing ratio $\xi$. This mixing can be fixed or adaptively determined, for example by setting

$$\xi_k = L_{\text{hier},k}^{\alpha_p} / (L_{\text{hier},k}^{\alpha_p} + L_{\text{ser},k}^{\alpha_p})$$

where $L_{\cdot,k}$ denotes the mean TD-error in the corresponding buffer and $\alpha_p$ modulates prioritization strength. This explicit stratification and prioritization of experiences aims to expose the agent more frequently to “highlight” transitions that encode either success or substantial progress, leading to accelerated learning and potentially improved robustness (Horváth et al., 2023).

2. Buffer Construction, Sampling, and Schedule Adaptation

HiER’s implementation involves several carefully designed steps:

• Episode Processing: Each episode is buffered separately. On termination, the sum of its rewards is assessed.
• Threshold Gatekeeping: If $R > \lambda$, the episode is appended to both the standard and highlight buffers.
• Schedule Adaptation: The threshold $\lambda$ can be statically fixed, scheduled according to a predefined progression (e.g., linear annealing), or adaptively updated throughout training.
• Sampling: During each policy or network update, samples are drawn from both buffers in proportions dictated by $\xi$. When PER is employed, $\xi$ can be updated on each step using mean TD-errors as above to dynamically adjust the degree of emphasis on highlight transitions.

The overall flow is captured in Algorithm 1 (“HiER+”) in (Horváth et al., 2023), outlining initialization, episodic data collection, buffer updates, sampling, and weight update routines. HiER is agnostic to the base RL algorithm and fully compatible with both HER (which augments the buffer with virtual, goal-relabelled transitions) and PER (which prioritizes experiences by TD-error).

3. Integration with Other Replay and Curriculum Techniques

HiER is engineered to function as an independent replay enhancement or in concert with other sample-efficiency techniques:

• With Hindsight Experience Replay: Episodes relabelled via HER are augmented in both buffers. The highlight buffer mechanism is applied on top of this, yielding a compound benefit: HER densifies the reward landscape, while HiER ensures replay frequency favors high-reward learning signals.
• With Prioritized Experience Replay: PER’s sampling probabilities can interact with HiER’s buffer structure through adaptive mixing as a function of TD-error statistics. This synergy aims to align TD-error-based prioritization with high-value episode targeting, balancing exploitation and exploring informative failures.
• HiER+ (Integration with Curriculum Learning): HiER+ couples the highlight buffer approach with “easy-2-hard initial state entropy” (E2H-ISE) curriculum learning. In E2H-ISE, the initial state–goal distribution $\mu_0$ is parametrically scaled (e.g., via a factor $c$ transitioning from deterministic to stochastic), resulting in a gradual increase of environment difficulty. The HiER buffer then provides focused exploitation of high-reward experiences, while E2H-ISE modulates data collection for effective exploration.

4. Experimental Validation and Comparative Performance

HiER and HiER+ have been empirically evaluated on Panda-Gym robotic manipulation benchmarks—Push, Slide, and Pick-and-place—characterized by sparse rewards and continuous domains (Horváth et al., 2023). In these tasks:

• Quantitative results indicate that HiER+ achieves high evaluation success rates (Push: 1.0, Slide: 0.83, Pick-and-place: 0.69), outperforming baselines (e.g., 0.97, 0.38, 0.27 for standard methods).
• Ablation studies show both HiER and E2H-ISE individually provide improvements, while their combination (HiER+) further accelerates learning and yields more robust policy convergence (success curves with consistently tighter confidence intervals).
• The buffer threshold $\lambda$ schedule and mixing ratio $\xi$ are shown to be flexible, with both predetermined and adaptive versions yielding positive results.

These findings demonstrate that replaying high-cumulative-reward episodes facilitates a higher frequency of demonstration-quality transitions, expediting learning in RL agents operating under challenging reward conditions and without demonstration data.

5. Theoretical and Practical Implications

HiER’s dual-buffer design introduces several notable properties and implications:

• Automatic Demonstration Extraction: HiER automatically surfaces demonstration-like transitions absent any human- or expert-provided samples, an advantage in domains such as robotics where demonstrations are expensive or unavailable.
• End-to-End Compatibility: As a modular add-on, HiER is compatible with HER, PER, and curriculum learning schemes, functioning orthogonally to the underlying algorithmic substrate.
• Data Efficiency and Sample Quality: By supplementing uniform or TD-error-based sampling with frequency-weighted “highlight” experiences, HiER empirically achieves improved data efficiency and generalization, as shown through multiple robot benchmarks.
• Robustness and Generalizability: The paper reports that HiER can lead not only to improvements in mean and median success rates, but also to reductions in the variance of learning curves across training seeds, suggesting enhanced robustness.

6. Broader Applications and Prospects

HiER and its variants are directly pertinent to:

• Robotic manipulation and control with sparse rewards: Push, slide, pick-and-place, and multi-goal tasks.
• Transfer learning and sim-to-real policy transfer, given the practical value of replaying high-reward trajectories found in simulation.
• Other domains with reward sparsity and no demonstrations: Industrial automation, search-and-rescue robotics, medical and surgical robot learning, and certain classes of autonomous driving or process control.

Future research directions include refining the criteria for what constitutes a highlight (e.g., employing alternative scoring functions beyond reward sum), adaptive scheduling for buffer thresholds and curriculum difficulty, and integrating HiER with multi-agent or sim-to-real transfer pipelines (Horváth et al., 2023).

7. Comparison to Other Highlighting and Buffer Strategies

HiER is distinguished from methods such as PER (which emphasizes transitions by TD error) or HER (which densifies rewards via goal relabeling) by its explicit episodic performance gating and dual-buffer mechanism. The concept aligns with and extends ideas in Experience Replay Optimization (ERO) (Zha et al., 2019), Introspective Experience Replay (IER) (Kumar et al., 2022), and other adaptive replay schemes—the common thread being selective replay of transitions empirically shown or hypothesized to drive learning progress.

Whereas methods like ERO implement a learnable priority score via policy gradient, and IER “looks back” before surprising events, HiER’s criterion is cumulative reward, allowing for automatic demonstration mining and explicit control over sample frequency. Its modularity and demonstrated efficacy in challenging robotic environments make HiER a flexible and impactful component in the modern RL toolkit.

---

Summary Table: HiER in Context

Method	Replay Selection Principle	Buffer Structure	Domain Integration
PER	TD-error prioritization	Single buffer	Most RL domains
HER	Goal relabeling of failures	Single buffer	Goal-based, sparse reward RL
HiER	High-episode-reward filtering	Standard + highlight	Sparse reward, robotics
HiER+	HiER + curriculum (E2H-ISE)	Standard + highlight	Difficult/sparse RL tasks
ERO	Learnable “replay policy”	Single buffer	Continuous control

This table contrasts HiER’s episode-level, cumulative-reward criterion and dual-buffer architecture to established sample prioritization and goal-augmentation strategies, with HiER+ further adding curriculum data collection.