NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

Abstract: As AI agents leave the lab and venture into the real world as autonomous vehicles, delivery robots, and cooking robots, it is increasingly necessary to design and comprehensively evaluate algorithms that tackle the ``open-world''. To this end, we introduce NovelGym, a flexible and adaptable ecosystem designed to simulate gridworld environments, serving as a robust platform for benchmarking reinforcement learning (RL) and hybrid planning and learning agents in open-world contexts. The modular architecture of NovelGym facilitates rapid creation and modification of task environments, including multi-agent scenarios, with multiple environment transformations, thus providing a dynamic testbed for researchers to develop open-world AI agents.

• The paper introduces NovelGym, a modular ecosystem that tests hybrid planning and learning agents using novelty injection to evaluate adaptability.
• It utilizes a comprehensive evaluation protocol featuring metrics such as pre-novelty performance, adaptation duration, and post-adaptation efficiency across various novelty scenarios.
• Experiments reveal that hybrid models adapt faster than pure reinforcement learning agents, highlighting potential for real-world autonomous applications.

NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds

Introduction

The paper "NovelGym: A Flexible Ecosystem for Hybrid Planning and Learning Agents Designed for Open Worlds" introduces NovelGym, an adaptable platform aimed at testing reinforcement learning (RL) and hybrid planning agents in open-world settings. The motivation behind developing NovelGym is the increasing demand for AI systems that can efficiently navigate and adapt within open-world environments, characterized by the presence of novel and unforeseen elements. As agents transition from closed laboratory settings to the real-world applications – such as autonomous vehicles and delivery robots – it becomes crucial to devise algorithms that can handle the complexities intrinsic to open-world scenarios, including recognizing and adapting to new concepts that were not part of their initial training.

NovelGym provides a modular architecture which enables the rapid development and testing of agents' ability to perceive and adapt to novel environmental conditions without requiring exhaustive pre-learning of every possible scenario.

Figure 1: NovelGym environment representation. The figure shows a gridworld environment with various entities, as described in the legend. The red box highlights the novelty in the environment.

System Design and Environment

At its core, NovelGym is designed to simulate gridworld environments that can be adapted to test various novelty scenarios. The system comprises several key components, including dynamic environment modules and agent modules that allow for the seamless integration and interaction of agents within an environment that may contain both known and unknown elements. This modular setup is depicted in the system design of the NovelGym ecosystem.

Figure 2: System design of the NovelGym ecosystem. Blue highlights the environment modules, and purple highlights the agent modules.

Agents in NovelGym are equipped with sensors, allowing them to generate observations and interact with the environment via a set of predefined actions. The environment supports multi-agent systems, ensuring that researchers can explore collaborative and competitive multi-agent dynamics. Novelty injection, a key feature of NovelGym, is facilitated by environment transformations, allowing researchers to inject or modify elements within the environment to simulate novel scenarios.

Evaluation Protocol and Metrics

NovelGym proposes a comprehensive protocol for evaluating novelty-aware agents. This involves an initial training phase in a stable, non-novel environment, followed by a novelty injection phase wherein environmental changes challenge the agent's adaptability. The process culminates in an adaptation phase where agents refine their strategies, and a final evaluation phase assesses how well agents have adapted.

Through a set of defined metrics – including pre-novelty performance, novelty impact, adaptation duration, post-adaptation efficiency, and others – NovelGym facilitates detailed analysis of an agent's ability to handle novelties. These metrics provide insights into the immediate impact of novelty on performance, the efficiency and success of adaptation strategies, and any potential long-term benefits or detriments.

Figure 3: Illustration of performance metrics for open-world agents.

Experiments and Results

The paper details experiments across five key novelty scenarios within NovelGym, including both beneficial and detrimental novelties. Agents, implemented using various architectures including reinforcement learning with Intrinsic Curiosity Module (ICM), exhibit varied levels of adaptability and performance across the scenarios. The results demonstrate significant differences in how agents cope with novelties, with hybrid models often showing quicker adaptation compared to pure RL models.

NovelGym successfully highlights the challenges and opportunities in designing robust AI agents capable of operating in open-world settings. By capturing intricate details such as immediate impacts and long-term adaptation efficiency, the ecosystem provides a robust framework for future research.

Conclusion

NovelGym provides an essential platform for advancing research in open-world learning by offering robust tools to design, implement, and test agents in a controlled yet highly flexible environment. The implementation of NovelGym opens avenues for exploring the integration of planning and learning paradigms, procedural novelty generation, and more complex multi-agent interactions. This platform sets the stage for the development of agents capable of more efficiently dealing with real-world complexities, promoting more seamless interaction between autonomous systems and their dynamic environments. Future work will focus on leveraging the modular capabilities of NovelGym to investigate human-in-the-loop learning, the framework's expansion to encompass more intricate multi-agent dynamics, and the integration of continual learning processes.