MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL (2312.11242v5)

Abstract: Recent LLM-based Text-to-SQL methods usually suffer from significant performance degradation on "huge" databases and complex user questions that require multi-step reasoning. Moreover, most existing methods neglect the crucial significance of LLMs utilizing external tools and model collaboration. To address these challenges, we introduce MAC-SQL, a novel LLM-based multi-agent collaborative framework. Our framework comprises a core decomposer agent for Text-to-SQL generation with few-shot chain-of-thought reasoning, accompanied by two auxiliary agents that utilize external tools or models to acquire smaller sub-databases and refine erroneous SQL queries. The decomposer agent collaborates with auxiliary agents, which are activated as needed and can be expanded to accommodate new features or tools for effective Text-to-SQL parsing. In our framework, We initially leverage GPT-4 as the strong backbone LLM for all agent tasks to determine the upper bound of our framework. We then fine-tune an open-sourced instruction-followed model, SQL-Llama, by leveraging Code Llama 7B, to accomplish all tasks as GPT-4 does. Experiments show that SQL-Llama achieves a comparable execution accuracy of 43.94, compared to the baseline accuracy of 46.35 for vanilla GPT-4. At the time of writing, MAC-SQL+GPT-4 achieves an execution accuracy of 59.59 when evaluated on the BIRD benchmark, establishing a new state-of-the-art (SOTA) on its holdout test set (https://github.com/wbbeyourself/MAC-SQL).

An Expert Overview of MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL

The paper on MAC-SQL posits a novel multi-agent collaborative framework designed to address the complexities and performance challenges associated with Text-to-SQL tasks, particularly on substantial databases and questions demanding intricate multi-step reasoning. Through leveraging LLMs, the MAC-SQL framework introduces a decomposition strategy facilitated by collaborative agent interactions to refine Text-to-SQL parsing.

Core Contribution

The paper identifies a significant performance degradation in existing LLM-based Text-to-SQL systems when applied to large-scale databases and complicated questions. To combat this, MAC-SQL proposes a framework composed of multiple collaborative agents:

• Decomposer Agent: Core to the framework, the decomposer employs few-shot chain-of-thought reasoning to generate SQL queries.
• Selector Agent: Performs initial filtration by breaking down a large database into relevant sub-databases to reduce noise from extraneous information.
• Refiner Agent: Utilizes external tools for SQL execution and feedback to address and amend SQL query errors.

The employment of GPT-4 as the backbone LLM reveals the upper performance bounds of the framework, establishing a baseline for further investigation with other models, such as SQL-Llama built upon Code Llama 7B. SQL-Llama is fine-tuned to emulate GPT-4’s performance, achieving an execution accuracy of 43.94, a promising result compared against GPT-4’s 46.35 baseline.

Numerical Results and Claims

In demonstrating MAC-SQL’s efficacy, the framework attains a state-of-the-art execution accuracy rate of 59.59 on the BIRD benchmark’s holdout test set, heralding it as a competitive model in Text-to-SQL parsing. This result highlights the framework's superior capability in handling complex data environments and intricate reasoning tasks, advancing beyond traditional systems' scope primarily scoped for spider-like datasets.

Implications and Future Research

The practical implications of MAC-SQL are substantial for database accessibility, especially when querying extensive databases without SQL expertise. Theoretically, it presents a new paradigm in handling LLM-based tasks, utilizing collaborative agent networks to expand upon standard LLM capacities.

The paper suggests future avenues could explore further enhancements in the collaboration dynamics among agents, optimization for varied LLM architectures, and additional fine-tuning approaches to improve SQL generation further. Understanding the dynamics between agent collaboration and its impact on overall system performance can inform broader developments in LLM-based frameworks across other domains.

By exploring these insights, MAC-SQL not only represents an advancement in Text-to-SQL methods but also paves the way for more resilient multi-agent frameworks capable of tackling real-world, large-scale database challenges. As AI continues to integrate and adapt to more comprehensive data systems, this research fosters foundational stepping stones for subsequent breakthroughs.