PromptMind Team at EHRSQL-2024: Improving Reliability of SQL Generation using Ensemble LLMs (2405.08839v1)
Abstract: This paper presents our approach to the EHRSQL-2024 shared task, which aims to develop a reliable Text-to-SQL system for electronic health records. We propose two approaches that leverage LLMs for prompting and fine-tuning to generate EHRSQL queries. In both techniques, we concentrate on bridging the gap between the real-world knowledge on which LLMs are trained and the domain specific knowledge required for the task. The paper provides the results of each approach individually, demonstrating that they achieve high execution accuracy. Additionally, we show that an ensemble approach further enhances generation reliability by reducing errors. This approach secured us 2nd place in the shared task competition. The methodologies outlined in this paper are designed to be transferable to domain-specific Text-to-SQL problems that emphasize both accuracy and reliability.
- Anthropic and others. 2024. The claude 3 model family: Opus, sonnet, haiku. https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. Online; accessed March 2024.
- STAR: SQL guided pre-training for context-dependent text-to-SQL parsing. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1235–1247, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- C3: Zero-shot text-to-sql with chatgpt. Preprint, arXiv:2307.07306.
- Multispider: Towards benchmarking multilingual text-to-sql semantic parsing. In AAAI Conference on Artificial Intelligence.
- Domain-specific language model pretraining for biomedical natural language processing.
- Retrieval-augmented gpt-3.5-based text-to-sql framework with sample-aware prompting and dynamic revision chain. In Neural Information Processing, pages 341–356, Singapore. Springer Nature Singapore.
- Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances in Neural Information Processing Systems, 35:15589–15601.
- Overview of the ehrsql 2024 shared task on reliable text-to-sql modeling on electronic health records. In Proceedings of the 6th Clinical Natural Language Processing Workshop, Mexico City, Mexico. Association for Computational Linguistics.
- Retrieval-augmented generation for knowledge-intensive NLP tasks. CoRR, abs/2005.11401.
- Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36.
- Xianming Li and Jing Li. 2023. Angle-optimized text embeddings. arXiv preprint arXiv:2309.12871.
- A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability. Preprint, arXiv:2303.13547.
- Mteb: Massive text embedding benchmark. arXiv preprint arXiv:2210.07316.
- Gpt-4 technical report. Preprint, arXiv:2303.08774.
- Mohammadreza Pourreza and Davood Rafiei. 2023. Din-sql: Decomposed in-context learning of text-to-sql with self-correction. arXiv preprint arXiv:2304.11015.
- PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901. Association for Computational Linguistics.
- RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578, Online. Association for Computational Linguistics.
- Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics.
- Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.
- Satya K Gundabathula (1 paper)
- Sriram R Kolar (2 papers)