OpenAGI: When LLM Meets Domain Experts (2304.04370v6)

Abstract: Human Intelligence (HI) excels at combining basic skills to solve complex tasks. This capability is vital for AI and should be embedded in comprehensive AI Agents, enabling them to harness expert models for complex task-solving towards AGI. LLMs show promising learning and reasoning abilities, and can effectively use external models, tools, plugins, or APIs to tackle complex problems. In this work, we introduce OpenAGI, an open-source AGI research and development platform designed for solving multi-step, real-world tasks. Specifically, OpenAGI uses a dual strategy, integrating standard benchmark tasks for benchmarking and evaluation, and open-ended tasks including more expandable models, tools, plugins, or APIs for creative problem-solving. Tasks are presented as natural language queries to the LLM, which then selects and executes appropriate models. We also propose a Reinforcement Learning from Task Feedback (RLTF) mechanism that uses task results to improve the LLM's task-solving ability, which creates a self-improving AI feedback loop. While we acknowledge that AGI is a broad and multifaceted research challenge with no singularly defined solution path, the integration of LLMs with domain-specific expert models, inspired by mirroring the blend of general and specialized intelligence in humans, offers a promising approach towards AGI. We are open-sourcing the OpenAGI project's code, dataset, benchmarks, evaluation methods, and the UI demo to foster community involvement in AGI advancement: https://github.com/agiresearch/OpenAGI.

Introduction

The intersection of human expertise and artificial intelligence presents a new horizon for problem-solving capabilities. A recent conceptual milestone in this domain is the development of what is known as OpenAGI, a research and development platform crafted to tackle real-world, multi-step tasks with a level of proficiency that approaches human intelligence. OpenAGI is built upon the integration of LLMs with various domain-specific models, aptly termed as expert models.

Core Concepts

OpenAGI operates utilizing a dual approach that combines traditional benchmark tasks and open-ended problem-solving for more creative tasks. With regard to benchmark tasks, OpenAGI leverages a diverse collection of expert models ranging from sentiment analysis to image super-resolution. These tasks are formulated as natural language queries, which are then interpreted and executed by the LLM to derive appropriate solutions. This strategy ensures a consistent platform for evaluating the planning and execution faculties of the AI system.

An intrinsic part of OpenAGI's intent is to mirror the intricate blend of general and specialized intelligence observed in humans. To capture this, OpenAGI introduces the concept of Reinforcement Learning from Task Feedback (RLTF). RLTF employs task performance feedback to refine the AI's decision-making process, aligning it ever closer to optimization and task-specific nuancing.

Challenges and Responses

While OpenAGI represents a significant leap forward, it is not without challenges. Extending the platform's capabilities, dealing with nonlinear task planning, and quantitative evaluations of multi-modal tasks are some hurdles that require inventive solutions. The RLTF, for instance, is a direct response to streamline the LLM's planning strategies, enabling more accurate and contextually relevant action plans.

Conclusion

The research highlights that even though AGI remains a vast field with much ground to cover, the approach of intertwining LLMs with domain expertise catalyzes the journey towards more generalizable AI solutions. Furthermore, this open-source platform welcomes community involvement, aimed at fostering synergy in the AGI sphere. By quantitatively capturing LLM's planning abilities and fostering a feedback-informed learning environment, OpenAGI not only carves a niche for challenging AI tasks but also sets a collaborative precedent for future advancements in the field.