Avalanche RL: a Continual Reinforcement Learning Library (2202.13657v2)

Abstract: Continual Reinforcement Learning (CRL) is a challenging setting where an agent learns to interact with an environment that is constantly changing over time (the stream of experiences). In this paper, we describe Avalanche RL, a library for Continual Reinforcement Learning which allows to easily train agents on a continuous stream of tasks. Avalanche RL is based on PyTorch and supports any OpenAI Gym environment. Its design is based on Avalanche, one of the more popular continual learning libraries, which allow us to reuse a large number of continual learning strategies and improve the interaction between reinforcement learning and continual learning researchers. Additionally, we propose Continual Habitat-Lab, a novel benchmark and a high-level library which enables the usage of the photorealistic simulator Habitat-Sim for CRL research. Overall, Avalanche RL attempts to unify under a common framework continual reinforcement learning applications, which we hope will foster the growth of the field.

• The paper introduces Avalanche RL as a unified framework that bridges continual and reinforcement learning research.
• It offers a modular design with curated benchmarks, training strategies, and evaluation tools for sequential, evolving tasks.
• Its automated parallelism and integration with Continual Habitat Lab enhance scalability and enable realistic, adaptive learning experiments.

Avalanche RL: A Tool for Advancing Continual Reinforcement Learning

The paper presents Avalanche RL, a library developed to support research in Continual Reinforcement Learning (CRL) by facilitating the training and evaluation of reinforcement learning agents across a sequence of non-stationary tasks. At its core, Avalanche RL builds on the Avalanche framework, which is well established in the Continual Learning (CL) domain. It extends Avalanche's capabilities to Reinforcement Learning (RL), incorporating a comprehensive suite of pre-implemented strategies, environments, and benchmarks to ease the burden of initializing and transitioning between tasks.

Key Components and Design

Avalanche RL consists of several essential modules: Benchmarks, Training, Evaluation, Models, and Logging. These modules collectively enable researchers to define, train on, and evaluate a sequence of continually evolving tasks in RL environments. Key features of these modules include:

• Benchmarks and Stream of Environments: This module offers a task stream abstraction, where experiences are derived from different environments. This setup differs significantly from traditional static datasets, as experiences must be generated through the active interaction between agents and environments. This interaction is crucial for CRL benchmarks, allowing flexibly defined environments that agents can continuously learn from.
• Training and Strategy Customization: Training reinforcement learning models is facilitated through modular strategies. These strategies can operate on existing RL frameworks but extend them through plugins, allowing advanced customizations and hybrid continual learning behaviors. The system makes it straightforward to incorporate strategies like EWC and replay buffers, promoting the composition and innovation of CRL methodologies.
• Evaluation and Logging: Avalanche RL not only performs state-of-the-art algorithm implementations but also includes robust mechanisms for evaluation and monitoring. This includes logging and tracking key performance metrics, such as reward accumulation and resource utilization, which are essential for a deeper understanding of agent learning dynamics.

Parallelism and Computational Efficiency

One of the distinguishing attributes of Avalanche RL is its automated parallelism capability, specifically through the use of vectorized environments. This mechanism enables the efficient parallelization of agent-environment interactions, leveraging CPU and GPU resources for scalability. The integration of Ray supports these parallel interactions, facilitating distributed computing setups essential for large-scale continual learning experiments.

Continual Habitat Lab: A New CRL Benchmark

The paper also introduces Continual Habitat Lab, an extension of the Habitat framework from FAIR, specialized for CRL. It provides the infrastructure to enable agents to interact with photorealistic simulations, operating under a sequence of tasks and scenes. This framework emphasizes the ability to model environment dynamics and task variations, making it a valuable tool for research into CRL applications in realistic, complex environments.

Implications for CRL Research

The creation of Avalanche RL addresses a critical gap in the CRL field: the lack of integrated, shared frameworks that combine the robustness of RL algorithms with continual learning insights. By merging proven CL techniques with modern RL practices, Avalanche RL offers a pathway to explore adaptive learning strategies that bridge multiple tasks and environments.

Future Directions

As part of its ongoing development, future plans for Avalanche RL include the integration of more sophisticated RL algorithms, such as PPO, TRPO, and SAC. Additionally, expanding the support for a wider variety of simulators, including those used in robotics and gaming, could broaden the applicability and robustness of CRL benchmarks. These enhancements are poised to significantly enrich the CRL research space, providing a consolidated platform for examination and innovation in the ways agents learn and adapt in evolving environments.

In conclusion, Avalanche RL and its companion Continual Habitat Lab provide a strategically structured and highly modular framework that encourages innovation and reproducibility in the field of continual reinforcement learning. This tool represents a meaningful contribution to advancing the state of research, encouraging collaboration, and fostering new developments in achieving versatile, adaptive learning systems.