Domain Adaptation of a State of the Art Text-to-SQL Model: Lessons Learned and Challenges Found (2312.05448v1)
Abstract: There are many recent advanced developments for the Text-to-SQL task, where the Picard model is one of the the top performing models as measured by the Spider dataset competition. However, bringing Text-to-SQL systems to realistic use-cases through domain adaptation remains a tough challenge. We analyze how well the base T5 LLM and Picard perform on query structures different from the Spider dataset, we fine-tuned the base model on the Spider data and on independent databases (DB). To avoid accessing the DB content online during inference, we also present an alternative way to disambiguate the values in an input question using a rule-based approach that relies on an intermediate representation of the semantic concepts of an input question. In our results we show in what cases T5 and Picard can deliver good performance, we share the lessons learned, and discuss current domain adaptation challenges.
- Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.12773.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the ACL: Human Language Technologies, Volume 1 (Long and Short Papers).
- Towards robustness of text-to-sql models against synonym substitution. arXiv preprint arXiv:2106.01065.
- Exploring underexplored limitations of cross-domain text-to-sql generalization. arXiv preprint arXiv:2109.05157.
- Text-to-SQL in the wild: A naturally-occurring dataset based on stack exchange data. In Proceedings of the Workshop on Natural Language Processing for Programming (NLP4Prog), pages 77–87. ACL.
- From zero to hero: On the limitations of zero-shot language transfer with multilingual Transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483–4499, Online. ACL.
- Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing. In Findings of the ACL: EMNLP 2020, pages 4870–4888. ACL.
- Towards a theory of natural language interfaces to databases.
- Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
- PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901. ACL.
- Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost.
- Exploring unexplored generalization challenges for cross-database semantic parsing.
- Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS, page 6000–6010. Curran Associates Inc.
- Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966.
- Query synthesis from natural language. International Conference on Object-Oriented Programming.
- Model-based interactive semantic parsing: A unified framework and a text-to-sql case study. IJCNLP.
- Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases. arXiv preprint arXiv:1909.05378.
- Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3911–3921. ACL.
- Sparc: cross-domain semantic parsing in context. arXiv preprint arXiv:1906.02285.
- Irene Manotas (4 papers)
- Octavian Popescu (2 papers)
- Ngoc Phuoc An Vo (4 papers)
- Vadim Sheinin (7 papers)