PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency (2403.09732v4)

Abstract: Recent advancements in Text-to-SQL (Text2SQL) emphasize stimulating the LLMs (LLM) on in-context learning, achieving significant results. Nevertheless, they face challenges when dealing with verbose database information and complex user intentions. This paper presents a two-stage framework to enhance the performance of current LLM-based natural language to SQL systems. We first introduce a novel prompt representation, called reference-enhanced representation, which includes schema information and randomly sampled cell values from tables to instruct LLMs in generating SQL queries. Then, in the first stage, question-SQL pairs are retrieved as few-shot demonstrations, prompting the LLM to generate a preliminary SQL (PreSQL). After that, the mentioned entities in PreSQL are parsed to conduct schema linking, which can significantly compact the useful information. In the second stage, with the linked schema, we simplify the prompt's schema information and instruct the LLM to produce the final SQL. Finally, as the post-refinement module, we propose using cross-consistency across different LLMs rather than self-consistency within a particular LLM. Our methods achieve new SOTA results on the Spider benchmark, with an execution accuracy of 87.6%.

Enhanced Text-to-SQL Conversion Using PET-SQL with Cross-Consistency

Introduction to PET-SQL Framework

The novel "Prompt-enhanced Two-stage Text-to-SQL Framework with Cross-consistency" (PET-SQL) introduces a two-stage approach aimed at overcoming intrinsic difficulties faced by LLMs in text-to-SQL conversions. This framework is designed to optimize SQL query generation without the training of specialized models, leveraging advances in LLMs for in-context learning. Initial results report a state-of-the-art execution accuracy of 87.6% on the Spider benchmark.

Key Contributions

The primary achievements of this paper can be distilled into three innovative approaches:

1. Reference-Enhanced Prompt Representation: This involves an advanced prompt setup that includes schema and cell value references which provide a more comprehensive context to the LLM, leading to more accurate SQL generation.
2. PreSQL-based Schema Linking: Instead of direct schema prediction, this approach employs an intermediate SQL generation phase (PreSQL) that helps in creating a more focused and simplified schema context for the final SQL generation (FinSQL).
3. Cross-Consistency across LLMs: Moving away from self-consistency within a single LLM, this method uses a voting mechanism across multiple models to refine the SQL outputs, enhancing reliability and reducing errors introduced by model hallucinations.

Methodology and Implementation

The PET-SQL methodology is detailed in sequences:

• Stage 1: Involves the generation of Preliminary SQL (PreSQL) using a novel prompting strategy which includes few-shot demonstrations based on semantic similarity and a reference-enhanced representation. This step purposes to create an initial, rough SQL draft and begin schema linking.
• Stage 2: Utilizes the linked schema to refine the initial draft into the final SQL query (FinSQL). This part reassesses and simplifies the schema utilized in the prompt, decreasing the context's complexity and potentially increasing the LLM's performance.

The paper critically examines the cross-consistency concept, aiming to utilize the diversity across different LLMs to achieve better accuracy and consistency in the generated SQL queries.

Experimental Results

• Spider Benchmark: PET-SQL achieved a benchmark-setting execution accuracy of 87.6%, which is a tangible improvement over its closest competitor DAIL-SQL which archived 86.6%.
• LLM Variance: Tests across multiple LLMs, including both specialized SQL LLMs like SQLCoder-34B and generalist models like GPT4, showed consistent improvements, underlining the robustness and effectiveness of the PET-SQL framework.

Theoretical Implications and Practical Applications

The framework significantly contributes to both the academic understanding and practical implementation of LLM-based text-to-SQL systems. The introduction of a two-stage processing with an enhanced prompting strategy provides a robust template for addressing complex user intentions and verbose database schemas. Practically, such advancement lowers the entry barrier for interacting with databases, broadening the user base to include individuals without formal SQL training.

Future Directions

Considering future enhancements, exploring the integration of more dynamic in-context examples based on real-time database changes could offer further improvements. Additionally, implementing adaptive learning where the LLM learns from the query history of the database could personalize and refine SQL query generation further.

In sum, PET-SQL not only sets a new standard on the Spider leaderboard but also opens new pathways for research into more efficient and accurate text-to-SQL conversions using LLMs.