Actor Prioritized Experience Replay

Abstract: A widely-studied deep reinforcement learning (RL) technique known as Prioritized Experience Replay (PER) allows agents to learn from transitions sampled with non-uniform probability proportional to their temporal-difference (TD) error. Although it has been shown that PER is one of the most crucial components for the overall performance of deep RL methods in discrete action domains, many empirical studies indicate that it considerably underperforms actor-critic algorithms in continuous control. We theoretically show that actor networks cannot be effectively trained with transitions that have large TD errors. As a result, the approximate policy gradient computed under the Q-network diverges from the actual gradient computed under the optimal Q-function. Motivated by this, we introduce a novel experience replay sampling framework for actor-critic methods, which also regards issues with stability and recent findings behind the poor empirical performance of PER. The introduced algorithm suggests a new branch of improvements to PER and schedules effective and efficient training for both actor and critic networks. An extensive set of experiments verifies our theoretical claims and demonstrates that the introduced method significantly outperforms the competing approaches and obtains state-of-the-art results over the standard off-policy actor-critic algorithms.

• The paper introduces LA3P, which adjusts prioritization to overcome high TD error challenges in actor-critic reinforcement learning.
• It employs inverse sampling, shared transitions, and tailored loss adjustments to better align actor updates with critic evaluations.
• Empirical results on MuJoCo and Box2D benchmarks show improved learning efficiency and policy stability, with optimal performance at lambda = 0.5.

Actor Prioritized Experience Replay

The paper "Actor Prioritized Experience Replay" explores the integration of Prioritized Experience Replay (PER) into continuous control reinforcement learning (RL) environments, specifically actor-critic methods. It proposes a novel approach to ameliorating the drawbacks of PER in continuous domains through a systematic reevaluation of experience replay strategy. The study is notably focused on how to effectively prioritize and sample experiences for the actor and critic networks to optimize learning.

Introduction and Background

PER is an extension to the experience replay mechanism, where experiences (transitions) are sampled with a probability proportional to their temporal-difference (TD) error. While PER has been effective in discrete action spaces, facilitating faster learning by focusing on transitions with the largest errors, its application to continuous domains often results in suboptimal performance, particularly when paired with actor-critic algorithms. The paper posits that actor networks struggle with high TD error transitions, causing divergence between approximate and true policy gradients.

Theoretical Analysis and Findings

The authors provide a theoretical foundation explaining why actor networks are inefficiently trained with high TD error transitions, arguing that such transitions yield an inaccurate policy gradient. This discrepancy stems from the fact that those errors indicate the critic's uncertainty, which subsequently misguides the update directions for the actor. Through rigorous derivations, they connect the estimation error in the Q-value function with the divergence in the actor's policy gradient, offering a significant insight into the limitations of existing PER frameworks.

Proposed Method: LA3P

In response to the identified shortcomings, the paper introduces the Loss-Adjusted Approximate Actor Prioritized Experience Replay (LA3P) algorithm. This method involves multiple innovations:

1. Inverse Sampling for Actor Networks: The actor is trained using transitions with low TD errors, aiming to align more closely with the critic's better-understood dynamics.
2. Shared Transitions: Ensuring a subset of transitions is uniformly sampled and used for updates in both actor and critic networks to maintain aligned learning and uphold the actor-critic dependencies.
3. Loss Adjustments: Adopting modified loss functions (LAP and PAL) to mitigate the bias introduced by PER, correcting the prioritization scheme without adverse impacts on gradient estimation.
4. Computational Feasibility: Despite the added complexity (with added cost primarily in creating an additional sum-tree for inverse sampling), the potential for SIMD parallel processing as a mitigation strategy is highlighted, ensuring the method remains practical for large-scale applications.

These components are incorporated into a structured algorithm that orchestrates updates through a combination of different sampling strategies, detailed in Algorithm 1 within the paper.

Empirical Validation

The framework is subjected to robust evaluations across numerous continuous control tasks using the MuJoCo and Box2D environments, benchmarked against standard off-policy actor-critic methods like TD3 and SAC.

• Performance Figures: Notable improvements in learning efficiency and final performance metrics are observed, especially in environments requiring extensive learned policy stability (Figures 1 and 2).
• Ablation Studies: The necessity of each component, such as the inverse sampling and shared transitions, is experimentally corroborated by comparing variations of the LA3P algorithm without these features. Figure 1 highlights these findings while Figure 2 demonstrates sensitivity to the hyperparameter $\lambda$, echoing the optimal setting of $\lambda = 0.5$ across tasks.

Conclusion

The research contributes a carefully articulated remediation to the inadequacies of PER in continuous control RL, particularly enhancing the stability and performance of actor-critic methods. The LA3P approach exemplifies an extendable strategy toward better utilization of experience prioritization, with potential for broader application beyond those examined. Further exploration in extending these ideas to other RL paradigms and more complex scenarios may yield additional promising avenues for AI development.