Open-Ended Learning Leads to Generally Capable Agents (2107.12808v2)

Abstract: In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards. We show that through constructing an open-ended learning process, which dynamically changes the training task distributions and training objectives such that the agent never stops learning, we achieve consistent learning of new behaviours. The resulting agent is able to score reward in every one of our humanly solvable evaluation levels, with behaviour generalising to many held-out points in the universe of tasks. Examples of this zero-shot generalisation include good performance on Hide and Seek, Capture the Flag, and Tag. Through analysis and hand-authored probe tasks we characterise the behaviour of our agent, and find interesting emergent heuristic behaviours such as trial-and-error experimentation, simple tool use, option switching, and cooperation. Finally, we demonstrate that the general capabilities of this agent could unlock larger scale transfer of behaviour through cheap finetuning.

An Overview of Open-Ended Learning and Generally Capable Agents

The paper "Open-Ended Learning Leads to Generally Capable Agents," authored by the Open-Ended Learning Team et al. from DeepMind, explores the development of agents capable of handling a multitude of tasks in a procedurally generated space. Several key contributions of this research delve into constructing a diverse, rich task space and employing a learning process aimed at achieving general capability across tasks.

Core Contributions and Methodologies

This paper introduces the XLand framework comprising two vital components: worlds and games. Worlds denote the initial conditions in which tasks are embedded, encompassing static topological arrangement and dynamic objects. Games, conversely, are configurations that define agents' goals associated with the world's state dynamics, allowing for procedural generation of diverse tasks within multi-agent setups encompassing competitive and cooperative games. Such variety in configuration achieves vastness and smoothness in tasks, crucial for testing robustness and adaptability of agents. The tasks, represented as a Cartesian product of worlds, games, and co-player policies, are characterized by their complexity and the interaction dynamics they necessitate among agents.

The research underscores the inadequacy of existing reinforcement learning methods when fixed to singular objectives or limited training task distributions. Agents herein are trained via an open-ended process that dynamically alters the distribution of tasks throughout the learning trajectory. By reshaping this trajectory, the agents constantly adapt, learn new behaviors, and improve performance even across typically intractable tasks.

The framework uses a novel task representation, leveraging a normalised percentile approach to gauge agent performance. This metric evaluates agents over task distribution percentiles rather than isolated metrics, providing a robust mechanism for assessing catastrophic failure, competence, and coverage. The iterative honing of task distributions ensures consistent training signal and ongoing challenge exposure for agents.

Key Results and Findings

The paper reports that trained agents could engage with all tasks considered humanly possible within the evaluation set, demonstrating emergent behaviors such as tool use and adaptive strategy changes under changing conditions. Moreover, the agents exhibit rapid finetuning potential on downstream tasks after vast initial training, suggesting a scalable transferability advantage inherent to the learned policies.

Training methods such as self-reward play are pivotal, encouraging agents to continually explore and exploit the dynamic space, driving cooperation among agents and minimizing conflict in multi-agent contexts. Ablation studies highlight the significance of components like population-based training, dynamic task generation, and recurrent network architecture in augmenting agent capabilities.

Implications and Future Directions

The paper suggests considerable implications for scaling artificial intelligence towards comprehensive generalization across diverse environments and tasks. The research methodology encourages exploring vast spaces while maintaining training efficiency through the iterative process and dynamic task generation.

The promising performance of agents opens avenues for employing such frameworks in real-world scenarios where adaptability across unseen tasks is critical. Future developments may focus on refining procedural generation spaces further and exploring even broader capability metrics or more complex multi-agent dynamics.

In summary, the paper presents a compelling case for open-ended learning paradigms tailored to develop robust, adaptable AI agents in procedurally extensive environments, offering a significant leap forward in multi-agent reinforcement learning domains.