Interference from Cross-Task Experience Replay in Lifelong RL

Determine whether replaying samples from previously encountered tasks interferes with the online gradient updates when learning a new task in lifelong reinforcement learning settings that employ experience replay (e.g., reservoir sampling).

Background

In the experiments on the 2D navigation task, the authors compare several baselines, including a replay-based approach that uses reservoir sampling to manage a long-term memory. They observe that this baseline achieves slightly higher average return than simple fine-tuning but exhibits more oscillatory returns during lifelong training.

Based on this observation, they explicitly conjecture that cross-task replay may impose interference on the online updates for the current task. This raises a concrete and important question about the stability of replay-based continual reinforcement learning methods and whether interleaving samples from prior tasks can degrade adaptation to new tasks.

References

We conjecture that replaying samples from other tasks can impose some interference on the online updates when learning a new task.

— A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning  (2205.10787 - Wang et al., 2022) in Section 4.1 (Simple 2D Navigation)

This was beneficial for overall stability, but it also degraded performance in the harder exploration games such as Pitfall. We were not able to find an easy remedy for this so we did not investigate further in this direction.

— Human-level Atari 200x faster  (2209.07550 - Kapturowski et al., 2022) in Appendix, Section 'Other Things we Tried', subsection 'Mixture of Online and Replay Data'