Open-SQL Framework: Enhancing Text-to-SQL on Open-source Large Language Models (2405.06674v1)
Abstract: Despite the success of LLMs in Text-to-SQL tasks, open-source LLMs encounter challenges in contextual understanding and response coherence. To tackle these issues, we present \ours, a systematic methodology tailored for Text-to-SQL with open-source LLMs. Our contributions include a comprehensive evaluation of open-source LLMs in Text-to-SQL tasks, the \openprompt strategy for effective question representation, and novel strategies for supervised fine-tuning. We explore the benefits of Chain-of-Thought in step-by-step inference and propose the \openexample method for enhanced few-shot learning. Additionally, we introduce token-efficient techniques, such as \textbf{Variable-length Open DB Schema}, \textbf{Target Column Truncation}, and \textbf{Example Column Truncation}, addressing challenges in large-scale databases. Our findings emphasize the need for further investigation into the impact of supervised fine-tuning on contextual learning capabilities. Remarkably, our method significantly improved Llama2-7B from 2.54\% to 41.04\% and Code Llama-7B from 14.54\% to 48.24\% on the BIRD-Dev dataset. Notably, the performance of Code Llama-7B surpassed GPT-4 (46.35\%) on the BIRD-Dev dataset.
- An encoder-decoder framework translating natural language to database queries. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, pages 3977–3983.
- Shuaichen Chang and Eric Fosler-Lussier. 2023. How to prompt llms for text-to-sql: A study in zero-shot, single-domain, and cross-domain settings.
- Recent advances in text-to-sql: A survey of what we have and what we expect. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2166–2187.
- A survey for in-context learning. CoRR, abs/2301.00234.
- C3: zero-shot text-to-sql with chatgpt. CoRR, abs/2307.07306.
- Text-to-sql empowered by large language models: A benchmark evaluation. CoRR, abs/2308.15363.
- A case-based reasoning framework for adaptive prompting in cross-domain text-to-sql. CoRR, abs/2304.13301.
- Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, pages 4524–4535.
- Lora: Low-rank adaptation of large language models.
- S2sql: Injecting syntax to question-schema interaction graph encoder for text-to-sql parsers. In Findings of the Association for Computational Linguistics, pages 1254–1262.
- Re-examining the role of schema linking in text-to-sql. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6943–6954.
- Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. AAAI-23, page 13067–13075. AAAI Press.
- Graphix-t5: Mixing pre-trained transformers with graph-aware layers for text-to-sql parsing. In 37th AAAI Conference on Artificial Intelligence, pages 13076–13084.
- Can LLM already serve as A database interface? A big bench for large-scale database grounded text-to-sqls. CoRR, abs/2305.03111.
- A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability. CoRR, abs/2303.13547.
- Multi-hop relational graph attention network for text-to-sql parsing. In International Joint Conference on Neural Networks, pages 1–8.
- What makes good in-context examples for gpt-3? In Proceedings of Deep Learning Inside Out: The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114.
- Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. CoRR, abs/2305.12586.
- OpenAI. 2023a. GPT-4 technical report. CoRR, abs/2303.08774.
- OpenAI. 2023b. Introducing chatgpt. https://openai.com/blog/chatgpt. Last accessed on 2023-07-24.
- OpenAI. 2023c. Sql translate. https://platform.openai.com/examples/default-sql-translate. Last accessed on 2023-07-24.
- Addressing limitations of encoder-decoder based approach to text-to-sql. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1593–1603.
- Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction.
- RASAT: integrating relational structures into pretrained seq2seq model for text-to-sql. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3215–3229.
- PICARD: parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901.
- Sql-palm: Improved large language model adaptation for text-to-sql. CoRR, abs/2306.00739.
- Llama: Open and efficient foundation language models. CoRR, abs/2302.13971.
- Llama2: Open foundation and fine-tuned chat models. CoRR.
- RAT-SQL: relation-aware schema encoding and linking for text-to-sql parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578.
- Proton: Probing schema linking information from pre-trained language models for text-to-sql parsing. In The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1889–1898.
- Sql-to-text generation with graph-to-sequence model. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 931–936.
- Tabert: Pretraining for joint understanding of textual and tabular data. Cornell University - arXiv,Cornell University - arXiv.
- HIE-SQL: history information enhanced network for context-dependent text-to-sql semantic parsing. In Findings of the Association for Computational Linguistics, pages 2997–3007.
- Xiaojun Chen (100 papers)
- Tianle Wang (30 papers)
- Tianhao Qiu (1 paper)
- Jianbin Qin (13 papers)
- Min Yang (239 papers)