RestGPT: Connecting Large Language Models with Real-World RESTful APIs (2306.06624v2)

Abstract: Tool-augmented LLMs have achieved remarkable progress in tackling a broad range of tasks. However, existing methods are mainly restricted to specifically designed tools and fail to fulfill complex instructions, having great limitations when confronted with real-world scenarios. In this paper, we explore a more realistic scenario by connecting LLMs with RESTful APIs, which adhere to the widely adopted REST software architectural style for web service development. To address the practical challenges of tackling complex instructions, we propose RestGPT, which exploits the power of LLMs and conducts a coarse-to-fine online planning mechanism to enhance the abilities of task decomposition and API selection. RestGPT also contains an API executor tailored for calling RESTful APIs, which can meticulously formulate parameters and parse API responses. To fully evaluate the performance of RestGPT, we propose RestBench, a high-quality benchmark which consists of two real-world scenarios and human-annotated instructions with gold solution paths. Experiments show that RestGPT is able to achieve impressive results in complex tasks and has strong robustness, which paves a new way towards AGI. RestGPT and RestBench is publicly available at https://restgpt.github.io/.

• The paper introduces RestGPT, a novel framework with a coarse-to-fine online planning mechanism for integrating Large Language Models with any real-world RESTful API via Planner, API Selector, and Executor modules.
• Evaluation on the new RestBench benchmark demonstrates RestGPT's superior performance, achieving success rates of 75% on TMDB and over 70% on Spotify for complex, multi-step tasks.
• RestGPT enables new LLM applications by facilitating access to vast real-world data via diverse API interactions, highlighting potential for advancements towards artificial general intelligence.

An Examination of RestGPT: Integrating LLMs with Real-World RESTful APIs

The paper presents an innovative approach to enhancing LLMs by connecting them with real-world RESTful APIs, proposing a novel framework named RestGPT. RestGPT aims to overcome the limitations of existing tool-augmented LLMs, which are often confined to interacting with a small set of specifically designed APIs and tools. This research focuses on expanding the applicability of LLMs by integrating them with any RESTful API-defined application, thereby facilitating their extensibility and usability in real-world scenarios.

Core Contributions and Methodology

RestGPT introduces a coarse-to-fine online planning mechanism that significantly enhances the LLM's ability to decompose tasks and select APIs accurately. The RestGPT architecture consists of three primary modules: a Planner, an API Selector, and an Executor. Each module leverages the capabilities of LLMs to execute distinct roles:

1. Planner: Generates high-level natural-language-based plans for addressing user queries, allowing for iterative task decomposition.
2. API Selector: Maps the high-level plan to specific API calls using available API endpoint descriptions, effectively creating a fine-grained action plan for execution.
3. Executor: Executes the API plans through its components, Caller and Parser, and adeptly handles parameter formulation and response parsing using the OpenAPI Specification (OAS) documentation.

By aligning components with separate sections of the OAS, RestGPT efficiently handles complex instructions and provides robust execution of RESTful API plans.

Evaluation and Benchmarking

To evaluate the effectiveness of RestGPT, the authors introduce a benchmark named RestBench, which encompasses two real-world scenarios: TMDB (a movie database) and Spotify (a music player application). Each scenario is supported by a curated set of annotated instructions and corresponding API solution paths.

The experimental results demonstrate RestGPT's superior performance compared to several baselines, including offline, ReAct, and Reflexion planning methodologies. RestGPT achieved a success rate of 75% for TMDB and over 70% for Spotify. This success illustrates the framework's ability to handle complex, multi-step tasks more efficiently and flexibly than alternative approaches.

Implications and Future Directions

RestGPT's integration of RESTful APIs with LLMs not only opens avenues for novel applications of LLMs in real-world settings but also emphasizes the potential of LLMs in accessing and leveraging vast amounts of data provided by various API services. The paper facilitates further research into augmenting LLMs with broader and more diverse API interactions, potentially leading to advancements in artificial general intelligence (AGI).

For future work, extending the capabilities of RestGPT to include a wider range of APIs and scenarios could further validate and enhance its practical applications. Furthermore, addressing the identified limitations and errors in planning and decision-making processes within the modules could improve the robustness of such frameworks, paving the way for comprehensive deployment across various domains requiring complex reasoning and decision-making capabilities.