DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2304.11015v3)

Abstract: There is currently a significant gap between the performance of fine-tuned models and prompting approaches using LLMs on the challenging task of text-to-SQL, as evaluated on datasets such as Spider. To improve the performance of LLMs in the reasoning process, we study how decomposing the task into smaller sub-tasks can be effective. In particular, we show that breaking down the generation problem into sub-problems and feeding the solutions of those sub-problems into LLMs can be an effective approach for significantly improving their performance. Our experiments with three LLMs show that this approach consistently improves their simple few-shot performance by roughly 10%, pushing the accuracy of LLMs towards SOTA or surpassing it. On the holdout test set of Spider, the SOTA, in terms of execution accuracy, was 79.9 and the new SOTA at the time of this writing using our approach is 85.3. Our approach with in-context learning beats many heavily fine-tuned models by at least 5%. Additionally, when evaluated on the BIRD benchmark, our approach achieved an execution accuracy of 55.9%, setting a new SOTA on its holdout test set.

Overview of DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction

The research paper titled "DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction" addresses the existing performance gap between fine-tuned models and prompting strategies utilizing LLMs for the text-to-SQL task. This problem is evaluated using benchmarks such as Spider and BIRD. The authors propose a novel approach that decomposes the task into smaller sub-problems, which are effectively leveraged by LLMs, leading to improved performance.

Methodology and Modules

The proposed methodology involves a multi-module approach that decomposes the text-to-SQL task to enhance the capability of LLMs in generating correct SQL queries:

1. Schema Linking Module: This module is designed to identify references to database schema and condition values within natural language queries. It is pivotal for ensuring domain generalizability and addressing schema ambiguity errors.
2. Classification and Decomposition Module: Queries are classified into three categories: easy, non-nested complex, and nested complex, based on their structural properties. This classification assists in tailoring the prompting strategy to the complexity of the query, thereby improving accuracy.
3. SQL Generation Module: Three separate prompting strategies are employed based on query complexity. An intermediate representation, specifically NatSQL, is used for more complex queries to bridge the gap between natural language and SQL syntax, facilitating the generation process.
4. Self-Correction Module: This module enhances the reliability of the generated SQL queries by employing a self-correction mechanism. The model iteratively improves its output by identifying and rectifying errors.

Experimental Results

The proposed method demonstrates significant performance improvements across several models, as evidenced by experiments on Spider and BIRD datasets:

• On the Spider dataset, the execution accuracy achieved was 85.3% using GPT-4, establishing new state-of-the-art (SOTA) performance metrics. The performance of the method utilizing CodeX Davinci was also notable at 78.2%.
• The BIRD dataset results further underscore the effectiveness of DIN-SQL, achieving a new SOTA execution accuracy of 55.9%.

The consistent improvements across LLMs by approximately 10% over simple few-shot prompting underscore the efficacy of the decomposition strategy.

Implications and Future Directions

The research successfully demonstrates that decomposing complex natural language processing tasks can significantly enhance the performance of LLMs. The use of intermediate representations like NatSQL, coupled with self-correction methodologies, opens up more avenues for exploring prompt engineering techniques. Future research could focus on refining automated and adaptive demonstration generation or optimizing cost and latency issues associated with large-scale queries.

Additionally, the approach's flexibility and applicability to models without dependency on database content offer a viable solution for real-world applications where database access is restricted.

Conclusion

The paper presents a comprehensive and effective framework for leveraging LLMs in text-to-SQL conversion tasks through strategic task decomposition and error-correction. The work provides substantial contributions in the field of natural language interfaces to databases, marking significant progress towards bridging the gap between model prompting and traditional fine-tuning approaches.