Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction (2408.11816v2)

Abstract: In the face of difficult exploration problems in reinforcement learning, we study whether giving an agent an object-centric mapping (describing a set of items and their attributes) allow for more efficient learning. We found this problem is best solved hierarchically by modelling items at a higher level of state abstraction to pixels, and attribute change at a higher level of temporal abstraction to primitive actions. This abstraction simplifies the transition dynamic by making specific future states easier to predict. We make use of this to propose a fully model-based algorithm that learns a discriminative world model, plans to explore efficiently with only a count-based intrinsic reward, and can subsequently plan to reach any discovered (abstract) states. We demonstrate the model's ability to (i) efficiently solve single tasks, (ii) transfer zero-shot and few-shot across item types and environments, and (iii) plan across long horizons. Across a suite of 2D crafting and MiniHack environments, we empirically show our model significantly out-performs state-of-the-art low-level methods (without abstraction), as well as performant model-free and model-based methods using the same abstraction. Finally, we show how to learn low level object-perturbing policies via reinforcement learning, and the object mapping itself by supervised learning.

• The paper introduces the MEAD algorithm, which uses an object-centric abstraction (Ab-MDP) to boost semantic transition predictions and improve sample efficiency.
• The paper demonstrates robust transferability, achieving effective zero-shot and few-shot learning across diverse environments.
• The paper outlines advanced planning capabilities with count-based intrinsic rewards that enable long-horizon exploration and outperform conventional approaches.

Analyzing Object-Centric Abstractions in Reinforcement Learning

Recent advancements in reinforcement learning (RL) have highlighted the potential of incorporating structured hierarchical abstractions to address complex exploration and model learning tasks. An intriguing paper, "Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstraction," introduces a framework leveraging an object-centric approach to enhance RL agent's learning efficiency by abstracting items and their attributes.

Overview

The paper presents a model-based reinforcement learning algorithm employing a novel abstracted hierarchical framework dubbed as Abstracted Item-Attribute Markov Decision Process (Ab-MDP). The core of this framework is integrating semantic-level abstractions into the RL paradigm, wherein agents perceive the environment not merely as raw data but as a structured collection of objects and their attributes. This abstraction significantly simplifies the transition dynamics, allowing the prediction of future states to be specific and tractable.

Key Findings

The research showcases a model-based algorithm—MEAD—which fundamentally focuses on learning and utilizing discriminative world models:

1. Efficiency in Task Solving: The introduced model demonstrates the ability to efficiently solve individual tasks using a discriminative learning objective that enhances semantic-level transition predictions. This results in noticeable improvements in sample efficiency compared to existing low-level state representations, as evidenced by experiments conducted across multiple environments, including 2D crafting and MiniHack tasks.
2. Transferability: MEAD shows notable success in transferring knowledge zero-shot and few-shot across varying object types and environments. Such transferability hints at the robustness and generality of the object-centric abstractions leveraged by MEAD, vital for applying RL in diverse real-world applications.
3. Advanced Planning with Abstraction: The model outperforms current model-free and model-based approaches by planning actions over long time horizons, utilizing a count-based intrinsic reward mechanism that facilitates exploration even in environments which are traditionally challenging to explore.
4. Learning Low-Level Components: The paper also presents strategies for factually learning low-level object-perturbation policies, as well as generating object-centric map predictions without needing explicit provision, thus expanding the adaptability of the solution.

Implications and Future Directions

The implications of these findings are both practical and theoretical:

• Simplified Learning in Structured Tasks: By abstracting the environment into meaningful object representations, MEAD reduces the complexity of the learning task, which can lead to significant computational savings and improved policy learning rates.
• Scalability and Generalization: The ability to transfer learning across varying tasks underscores MEAD’s potential to generalize learned policies across domains, a critical advancement towards scalable RL applications.
• Framework Flexibility: Future research could explore expanding the abstraction capabilities by integrating additional semantic layers or domain-specific knowledge, potentially enhancing both interpretability and performance of RL agents.
• Real-World Applications: Investigating the application of such an abstraction-based framework in domains like robotics, autonomous systems, and video game environments could further attest to its practical impact.

In summary, the work establishes a promising direction for incorporating structured semantic abstractions into reinforcement learning, offering a compelling avenue for further exploration of abstraction-based RL models that can learn and adapt efficiently across diverse environments.