Open-Endedness is Essential for Artificial Superhuman Intelligence (2406.04268v1)

Abstract: In recent years there has been a tremendous surge in the general capabilities of AI systems, mainly fuelled by training foundation models on internetscale data. Nevertheless, the creation of openended, ever self-improving AI remains elusive. In this position paper, we argue that the ingredients are now in place to achieve openendedness in AI systems with respect to a human observer. Furthermore, we claim that such open-endedness is an essential property of any artificial superhuman intelligence (ASI). We begin by providing a concrete formal definition of open-endedness through the lens of novelty and learnability. We then illustrate a path towards ASI via open-ended systems built on top of foundation models, capable of making novel, humanrelevant discoveries. We conclude by examining the safety implications of generally-capable openended AI. We expect that open-ended foundation models will prove to be an increasingly fertile and safety-critical area of research in the near future.

Open-Endedness is Essential for Artificial Superhuman Intelligence: A Technical Examination

The paper "Open-Endedness is Essential for Artificial Superhuman Intelligence" by Hughes et al. examines the concept of open-endedness as a critical property for the development of Artificial Superhuman Intelligence (ASI). The authors assert that achieving open-ended, ever self-improving AI systems has remained elusive despite the advances in training foundation models with large-scale internet data. This paper provides a formal definition of open-endedness through the dimensions of novelty and learnability and discusses the implications for safety and future AI research.

Formal Definition of Open-Endedness

The authors offer a concrete formalization of open-endedness via the twin properties of novelty and learnability. Novelty is defined as the capability of a system to generate increasingly unpredictable artifacts over time, whereas learnability ensures that an observer can improve their predictive model by examining the artifacts generated thus far. This dual criterion ensures that the generated artifacts are both surprising and informative to the observer.

Formally, the system $S$ produces a sequence of artifacts $X_t$ indexed by time $t$. An observer $O$ utilizes a statistical model $\hat{X}$ to predict future artifacts based on the observed sequence up to time $t$. Novelty demands that the expectation of the prediction loss $\ell(t,T)$ increases over time, while learnability requires that the loss decreases with access to more history. This ensures that the system continually presents new and interpretable information to the observer.

Interdependency between Foundation Models and Open-Endedness

The authors argue that foundation models and open-endedness are orthogonal dimensions that, when combined, hold significant promise. Foundation models like LLMs have demonstrated their capacity to efficiently guide search spaces for discovering human-relevant artifacts. Open-ended algorithms can leverage this potential, enabling continuous generation and refinement of novel artifacts.

Despite the scale of current foundation models, there is an impending limitation of available high-quality textual and visual data. Future progress necessitates systems that can create, refute, and refine their own knowledge base dynamically, learning what data is valuable for further learning.

Implications and Future Directions

The paper identifies several immediate research directions:

1. Reinforcement Learning (RL): Integration of RL with open-endedness, where agents could set their own goals and generate experiences that drive perpetual learning. Applying evolutionary algorithms and automated curricula can enhance exploration.
2. Task Generation: Open-endedness can be facilitated by evolving task difficulties matching the agent's capabilities to ensure a perpetual challenge. This can be implemented via web-based environments, leveraging the internet as a dynamic information repository.
3. Self-Improvement: The role of self-generated feedback and iterative refinement is crucial. Foundation models can enhance task efficacy by critiquing and revising their outputs, guiding themselves towards better performance.
4. Evolutionary Algorithms: Employing evolutionary methods can help generate novel and functional artifacts. For instance, text-to-concept generation by LLMs can act as mutation operators guiding creative processes.

Safety and Ethical Considerations

The paper emphasizes the inherent safety and ethical challenges in developing open-ended systems. The authors suggest mechanisms such as informed human oversight to guide and direct the output of open-ended AI responsibly. The evolving nature of artifacts demands that human observers remain capable of interpreting and learning from these outputs, aligning them with beneficial societal outcomes.

Conclusion

The authors conclude that open-ended foundation models represent a crucial frontier in AI research. These models provide a pathway to achieving ASI by continuously generating novel, learnable, and human-relevant artifacts. While the potential benefits are significant, so too are the safety concerns, which must be proactively managed to ensure that such systems remain aligned with human values and capabilities. Responsible and ethical development practices will be critical as the field advances toward increasingly autonomous and capable AI systems.