Using LLM to select the right SQL Query from candidates (2401.02115v1)
Abstract: Text-to-SQL models can generate a list of candidate SQL queries, and the best query is often in the candidate list, but not at the top of the list. An effective re-rank method can select the right SQL query from the candidate list and improve the model's performance. Previous studies on code generation automatically generate test cases and use them to re-rank candidate codes. However, automatic test case generation for text-to-SQL is an understudied field. We propose an automatic test case generation method that first generates a database and then uses LLMs to predict the ground truth, which is the expected execution results of the ground truth SQL query on this database. To reduce the difficulty for LLMs to predict, we conduct experiments to search for ways to generate easy databases for LLMs and design easy-to-understand prompts. Based on our test case generation method, we propose a re-rank method to select the right SQL query from the candidate list. Given a candidate list, our method can generate test cases and re-rank the candidate list according to their pass numbers on these test cases and their generation probabilities. The experiment results on the validation dataset of Spider show that the performance of some state-of-the-art models can get a 3.6\% improvement after applying our re-rank method.
- Re-ranking words to improve interpretability of automatically generated topics. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 43–54, Gothenburg, Sweden. Association for Computational Linguistics.
- Global reasoning over database structures for text-to-SQL parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3659–3664, Hong Kong, China. Association for Computational Linguistics.
- Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.
- Codet: Code generation with generated tests.
- Evaluating large language models trained on code.
- Text-to-sql empowered by large language models: A benchmark evaluation.
- Dynamic hybrid relation network for cross-domain context-dependent semantic parsing.
- Bertrand-dr: Improving text-to-sql using a discriminative re-ranker.
- BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
- Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql.
- A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability.
- Explaining wrong queries using small examples. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD ’19, page 503–520, New York, NY, USA. Association for Computing Machinery.
- OpenAI. 2023. Gpt-4 technical report.
- Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction.
- Rasat: Integrating relational structures into pretrained seq2seq model for text-to-sql.
- Evaluating the text-to-sql capabilities of large language models.
- Ohad Rubin and Jonathan Berant. 2021. Smbop: Semi-autoregressive bottom-up semantic parsing.
- Reflexion: Language agents with verbal reinforcement learning.
- Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921. Association for Computational Linguistics.
- N-best hypotheses reranking for text-to-sql systems.
- Understanding image retrieval re-ranking: A graph neural network perspective.
- Active programming by example with a natural language prior.
- Semantic evaluation for text-to-SQL with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396–411, Online. Association for Computational Linguistics.