General Intelligence Requires Rethinking Exploration (2211.07819v1)

Abstract: We are at the cusp of a transition from "learning from data" to "learning what data to learn from" as a central focus of AI research. While the first-order learning problem is not completely solved, large models under unified architectures, such as transformers, have shifted the learning bottleneck from how to effectively train our models to how to effectively acquire and use task-relevant data. This problem, which we frame as exploration, is a universal aspect of learning in open-ended domains, such as the real world. Although the study of exploration in AI is largely limited to the field of reinforcement learning, we argue that exploration is essential to all learning systems, including supervised learning. We propose the problem of generalized exploration to conceptually unify exploration-driven learning between supervised learning and reinforcement learning, allowing us to highlight key similarities across learning settings and open research challenges. Importantly, generalized exploration serves as a necessary objective for maintaining open-ended learning processes, which in continually learning to discover and solve new problems, provides a promising path to more general intelligence.

• The paper redefines exploration by unifying treatment across static datasets in supervised learning and simulated environments in reinforcement learning.
• It proposes a dual-level framework that separates within-environment and cross-environment exploration to mitigate limitations like concept drift and the sim2real gap.
• The study highlights new directions for AI research, emphasizing dynamic curriculum design and data-centric approaches to advance general intelligence.

Overview of "General Intelligence Requires Rethinking Exploration"

The paper "General Intelligence Requires Rethinking Exploration" by Minqi Jiang, Tim Rocktäschel, and Edward Grefenstette presents a novel perspective on the necessity of redefining exploration within the field of AI to advance towards general intelligence. The authors argue that as AI progresses from learning from static data to determining which data should be prioritized in learning, exploration assumes a central role in the learning process. The paper contends that exploration, traditionally associated with reinforcement learning (RL), is a critical aspect across all learning systems, including supervised learning (SL). Through the concept of generalized exploration, the authors aim to unify exploration-driven learning across SL and RL and outline the challenges in an open-ended domain such as the real world.

The introduction of generalized exploration represents a significant conceptual shift. It critiques the current paradigms where ML systems remain constrained by static datasets or limited simulators. The authors suggest that exploration encompasses two levels of abstraction: exploration within a specific environment and the broader exploration across environments. This framework emphasizes continual learning in an open-ended fashion, driving an agent towards increasingly general intelligence (IGI).

Detailed Insights

The authors begin by highlighting the limitations of the current ML paradigm, where SL relies on static, finite datasets and RL operates within static simulators. These constraints limit the systems' ability to generalize beyond their training environments. The paper points out that despite advancements in large-scale models like transformers, essential data-related issues, such as data acquisition and utilization, remain underexplored.

In SL, the static nature of datasets results in models failing to adapt to dynamic changes in the world, suffering from concept drift and covariate shifts. Similarly, RL's reliance on simulators leads to a sim2real gap, where models trained in simulation do not generalize effectively in real-world settings. Generalized exploration aims to address these limitations, enabling models to actively seek informative data that expands their capabilities, thus facilitating open-ended learning.

The framework introduces exploration across the environment space as a solution. Specifically, it proposes a curriculum-driven exploration in RL, where parameters of the environments are adjusted to maintain learning potential. The goal is to explore configurations that maximize learning opportunities, aligning with the view that exploration should drive the agent towards new challenges at the boundary of its existing capabilities.

Implications and Future Directions

Generalized exploration posits significant implications for the trajectory of AI research. By focusing on active collection and exploration beyond static confines, the paper argues for a shift towards a data-centric approach in AI. This perspective is envisioned to catalyze advancements in continuously improving models that align with the efficiency and relevance of their training data against dynamic real-world contexts.

The authors discuss several open research questions essential for realizing generalized exploration, including how to choose the domain for paper, designing scalable generators for training data, interfacing with open-ended task spaces, and defining benchmarks for open-ended learning. The exploration criterion, with its components of learning potential, diversity, and grounding, guides these inquiries, aiming to establish a principled process for active data generation.

The paper concludes that the framework of generalized exploration provides a path towards AGI by enabling the development of systems that are increasingly general in intelligence. The outlined strategy advocates a paradigm shift from classical AI approaches, where neither static data nor isolated exploration in fixed domains sufficiently captures the complexity required for such generalized intelligence. Instead, an iterative process of exploration and learning is emphasized, fundamentally transforming how AI systems are designed and operated.