Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation (2402.17144v1)

Published 27 Feb 2024 in cs.DB and cs.AI

Abstract: The Natural Language Interface to Databases (NLIDB) empowers non-technical users with database access through intuitive natural language (NL) interactions. Advanced approaches, utilizing neural sequence-to-sequence models or large-scale LLMs, typically employ auto-regressive decoding to generate unique SQL queries sequentially. While these translation models have greatly improved the overall translation accuracy, surpassing 70% on NLIDB benchmarks, the use of auto-regressive decoding to generate single SQL queries may result in sub-optimal outputs, potentially leading to erroneous translations. In this paper, we propose Metasql, a unified generate-then-rank framework that can be flexibly incorporated with existing NLIDBs to consistently improve their translation accuracy. Metasql introduces query metadata to control the generation of better SQL query candidates and uses learning-to-rank algorithms to retrieve globally optimized queries. Specifically, Metasql first breaks down the meaning of the given NL query into a set of possible query metadata, representing the basic concepts of the semantics. These metadata are then used as language constraints to steer the underlying translation model toward generating a set of candidate SQL queries. Finally, Metasql ranks the candidates to identify the best matching one for the given NL query. Extensive experiments are performed to study Metasql on two public NLIDB benchmarks. The results show that the performance of the translation models can be effectively improved using Metasql.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (73)
  1. I. Androutsopoulos, G. D. Ritchie, and P. Thanisch, “Natural language interfaces to databases - an introduction,” Nat. Lang. Eng., no. 1, 29–81, 1995.
  2. F. Benzi, D. Maio, and S. Rizzi, “VISIONARY: a viewpoint-based visual language for querying relational databases,” J. Vis. Lang. Comput., no. 2, 117–145, 1999.
  3. G. Bhalotia, A. Hulgeri, C. Nakhe, S. Chakrabarti, and S. Sudarshan, “Keyword searching and browsing in databases using BANKS,” in ICDE, R. Agrawal and K. R. Dittrich, Eds., 2002, 431–440.
  4. X. Xu, C. Liu, and D. Song, “Sqlnet: Generating structured queries from natural language without reinforcement learning,” CoRR, 2017.
  5. J. Guo, Z. Zhan, Y. Gao, Y. Xiao, J. Lou, T. Liu, and D. Zhang, “Towards complex text-to-sql in cross-domain database with intermediate representation,” in ACL, A. Korhonen, D. R. Traum, and L. Màrquez, Eds., 2019, 4524–4535.
  6. B. Bogin, J. Berant, and M. Gardner, “Representing schema structure with graph neural networks for text-to-sql parsing,” in ACL, 2019, 4560–4565.
  7. B. Bogin, M. Gardner, and J. Berant, “Global reasoning over database structures for text-to-sql parsing,” in EMNLP-IJCNLP, 2019, 3657–3662.
  8. B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, “RAT-SQL: relation-aware schema encoding and linking for text-to-sql parsers,” in ACL, 2020, 7567–7578.
  9. P. Shi, P. Ng, Z. Wang, H. Zhu, A. H. Li, J. Wang, C. N. dos Santos, and B. Xiang, “Learning contextual representations for semantic parsing with generation-augmented pre-training,” in AAAI, 2021, 13 806–13 814.
  10. O. Rubin and J. Berant, “Smbop: Semi-autoregressive bottom-up semantic parsing,” CoRR, 2020.
  11. R. Cao, L. Chen, Z. Chen, Y. Zhao, S. Zhu, and K. Yu, “LGESQL: line graph enhanced text-to-sql model with mixed local and non-local relations,” in ACL/IJCNLP, 2021, 2541–2555.
  12. H. Li, J. Zhang, C. Li, and H. Chen, “Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql,” in AAAI, 2023.
  13. OpenAI, “GPT-4 technical report,” CoRR, 2023.
  14. M. Pourreza and D. Rafiei, “DIN-SQL: decomposed in-context learning of text-to-sql with self-correction,” CoRR, vol. abs/2304.11015, 2023.
  15. R. Sun, S. Ö. Arik, H. Nakhost, H. Dai, R. Sinha, P. Yin, and T. Pfister, “Sql-palm: Improved large language model adaptation for text-to-sql,” CoRR, vol. abs/2306.00739, 2023.
  16. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. R. Radev, “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” in EMNLP, 2018, 3911–3921.
  17. A. Fan, M. Lewis, and Y. N. Dauphin, “Hierarchical neural story generation,” in ACL, 2018, 889–898.
  18. K. Gimpel, D. Batra, C. Dyer, and G. Shakhnarovich, “A systematic exploration of diversity in machine translation,” in EMNLP, 2013, 1100–1111.
  19. J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan, “A diversity-promoting objective function for neural conversation models,” in NAACL, 2016, 110–119.
  20. J. Li and D. Jurafsky, “Mutual information and diverse decoding improve neural machine translation,” CoRR, vol. abs/1601.00372, 2016.
  21. M. Ravaut, S. R. Joty, and N. F. Chen, “Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization,” in ACL, 2022, 4504–4524.
  22. D. Jiang, B. Y. Lin, and X. Ren, “Pairreranker: Pairwise reranking for natural language generation,” CoRR, vol. abs/2212.10555, 2022.
  23. W. Shen, Y. Gong, Y. Shen, S. Wang, X. Quan, N. Duan, and W. Chen, “Joint generator-ranker learning for natural language generation,” in ACL, 2023, 7681–7699.
  24. Y. Fu, W. Ou, Z. Yu, and Y. Lin, “MIGA: A unified multi-task generation framework for conversational text-to-sql,” CoRR, vol. abs/2212.09278, 2022.
  25. Y. Fan, Z. He, T. Ren, D. Guo, C. Lin, R. Zhu, G. Chen, Y. Jing, K. Zhang, and X. Wang, “Gar: A generate-and-rank approach for natural language to sql translation,” in ICDE, 2023.
  26. Y. Fan, T. Ren, Z. He, X. S. Wang, Y. Zhang, and X. Li, “Gensql: A generative natural language interface to database systems,” in ICDE, 2023, 3603–3606.
  27. X. Zheng, H. Lin, X. Han, and L. Sun, “Toward unified controllable text generation via regular expression instruction,” CoRR, 2023.
  28. M. Kim, H. Lee, K. M. Yoo, J. Park, H. Lee, and K. Jung, “Critic-guided decoding for controlled text generation,” in ACL, 2023, 4598–4612.
  29. H. Zhang, H. Song, S. Li, M. Zhou, and D. Song, “A survey of controllable text generation using transformer-based pre-trained language models,” CoRR, 2022.
  30. N. Gupta and M. Lewis, “Neural compositional denotational semantics for question answering,” in EMNLP, 2018, 2152–2161.
  31. A. Talmor and J. Berant, “The web as a knowledge-base for answering complex questions,” in NAACL-HLT, 2018, 641–651.
  32. H. Zhang, J. Cai, J. Xu, and J. Wang, “Complex question decomposition for semantic parsing,” in ACL, 2019, 4477–4486.
  33. S. Min, V. Zhong, L. Zettlemoyer, and H. Hajishirzi, “Multi-hop reading comprehension through question decomposition and rescoring,” in ACL, 2019, 6097–6109.
  34. T. Wolfson, M. Geva, A. Gupta, Y. Goldberg, M. Gardner, D. Deutch, and J. Berant, “Break it down: A question understanding benchmark,” Trans. Assoc. Comput. Linguistics, 183–198, 2020.
  35. Y. Zhang, J. Deriu, G. Katsogiannis-Meimarakis, C. Kosten, G. Koutrika, and K. Stockinger, “Sciencebenchmark: A complex real-world benchmark for evaluating natural language to SQL systems,” CoRR, 2023.
  36. X. V. Lin, R. Socher, and C. Xiong, “Bridging textual and tabular data for cross-domain text-to-sql semantic parsing,” in EMNLP, 2020, 4870–4888.
  37. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in ICLR, 2015.
  38. I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in NeurIPS, 2014, 3104–3112.
  39. R. Zhang, T. Yu, H. Er, S. Shim, E. Xue, X. V. Lin, T. Shi, C. Xiong, R. Socher, and D. R. Radev, “Editing-based SQL query generation for cross-domain context-dependent questions,” in EMNLP-IJCNLP, 2019, 5337–5348.
  40. Y. Gan, X. Chen, J. Xie, M. Purver, J. R. Woodward, J. H. Drake, and Q. Zhang, “Natural SQL: making SQL easier to infer from natural language specifications,” in EMNLP, 2021, 2030–2042.
  41. N. Rajkumar, R. Li, and D. Bahdanau, “Evaluating the text-to-sql capabilities of large language models,” CoRR, vol. abs/2204.00498, 2022.
  42. A. Liu, X. Hu, L. Wen, and P. S. Yu, “A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability,” CoRR, vol. abs/2303.13547, 2023.
  43. Z. Gu, J. Fan, N. Tang, S. Zhang, Y. Zhang, Z. Chen, L. Cao, G. Li, S. Madden, and X. Du, “Interleaving pre-trained language models and large language models for zero-shot NL2SQL generation,” CoRR, vol. abs/2306.08891, 2023.
  44. R. F. Nogueira, W. Yang, K. Cho, and J. Lin, “Multi-stage document ranking with BERT,” CoRR, 2019.
  45. L. Gao, Z. Dai, and J. Callan, “Rethink training of BERT rerankers in multi-stage retrieval pipeline,” in ECIR, D. Hiemstra, M. Moens, J. Mothe, R. Perego, M. Potthast, and F. Sebastiani, Eds., 2021, 280–286.
  46. J. Austin, A. Odena, M. I. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. J. Cai, M. Terry, Q. V. Le, and C. Sutton, “Program synthesis with large language models,” CoRR, 2021.
  47. K. Cobbe, V. Kosaraju, M. Bavarian, J. Hilton, R. Nakano, C. Hesse, and J. Schulman, “Training verifiers to solve math word problems,” CoRR, 2021.
  48. Y. Li, D. H. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. de Masson d’Autume, I. Babuschkin, X. Chen, P. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavukcuoglu, and O. Vinyals, “Competition-level code generation with alphacode,” CoRR, 2022.
  49. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, “Language models are few-shot learners,” in NeurIPS, 2020.
  50. K. Lee, M. Chang, and K. Toutanova, “Latent retrieval for weakly supervised open domain question answering,” in ACL, 2019, 6086–6096.
  51. V. Karpukhin, B. Oguz, S. Min, P. S. H. Lewis, L. Wu, S. Edunov, D. Chen, and W. Yih, “Dense passage retrieval for open-domain question answering,” in EMNLP, 2020, 6769–6781.
  52. L. Xiong, C. Xiong, Y. Li, K. Tang, J. Liu, P. N. Bennett, J. Ahmed, and A. Overwijk, “Approximate nearest neighbor negative contrastive learning for dense text retrieval,” in ICLR, 2021.
  53. J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” in NAACL-HLT, 2019, 4171–4186.
  54. C. Liu, Z. Mao, T. Zhang, H. Xie, B. Wang, and Y. Zhang, “Graph structured network for image-text matching,” in CVPR, 2020, 10 918–10 927.
  55. Z. Fan, Z. Wei, Z. Li, S. Wang, H. Shan, X. Huang, and J. Fan, “Constructing phrase-level semantic labels to form multi-grained supervision for image-text retrieval,” in ICMR, 2022, 137–145.
  56. Z. Cao, T. Qin, T. Liu, M. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” in ICML, 2007, 129–136.
  57. G. Koutrika, A. Simitsis, and Y. E. Ioannidis, “Explaining structured queries in natural language,” in ICDE, 2010, 333–344.
  58. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Summarizing source code using a neural attention model,” in ACL, 2016.
  59. K. Xu, L. Wu, Z. Wang, Y. Feng, and V. Sheinin, “Sql-to-text generation with graph-to-sequence model,” in EMNLP, 2018, 931–936.
  60. P. Pobrotyn and R. Bialobrzeski, “Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting,” CoRR, 2021.
  61. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR, 2015.
  62. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” CoRR, 2019.
  63. D. A. Hull, “Xerox TREC-8 question answering track report,” in TREC, 1999.
  64. J. M. Zelle and R. J. Mooney, “Learning to parse database queries using inductive logic programming,” in AAAI, 1996, 1050–1055.
  65. A. Simitsis, G. Koutrika, and Y. E. Ioannidis, “Précis: from unstructured keywords as queries to structured databases as answers,” PVLDB, no. 1, 117–149, 2008.
  66. L. S. Zettlemoyer and M. Collins, “Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars,” in UAI, 2005, 658–666.
  67. F. Li and H. V. Jagadish, “Constructing an interactive natural language interface for relational databases,” PVLDB, no. 1, 73–84, 2014.
  68. D. Saha, A. Floratou, K. Sankaranarayanan, U. F. Minhas, A. R. Mittal, and F. Özcan, “ATHENA: an ontology-driven system for natural language querying over relational data stores,” PVLDB, no. 12, 1209–1220, 2016.
  69. C. Baik, H. V. Jagadish, and Y. Li, “Bridging the semantic gap with SQL query logs in natural language interfaces to databases,” in ICDE, 2019, 374–385.
  70. J. Guo, Z. Zhan, Y. Gao, Y. Xiao, J. Lou, T. Liu, and D. Zhang, “Towards complex text-to-sql in cross-domain database with intermediate representation,” in ACL, 2019, 4524–4535.
  71. T. Yu, C. Wu, X. V. Lin, B. Wang, Y. C. Tan, X. Yang, D. R. Radev, R. Socher, and C. Xiong, “Grappa: Grammar-augmented pre-training for table semantic parsing,” CoRR, 2020.
  72. Y. Fan, T. Ren, D. Guo, Z. Zhao, Z. He, X. S. Wang, Y. Wang, and T. Sui, “An integrated interactive framework for natural language to SQL translation,” in WISE, vol. 14306, 2023, 643–658.
  73. H. Fu, C. Liu, B. Wu, F. Li, J. Tan, and J. Sun, “Catsql: Towards real world natural language to SQL applications,” Proc. VLDB Endow., vol. 16, no. 6, 1534–1547, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Yuankai Fan (3 papers)
  2. Zhenying He (10 papers)
  3. Tonghui Ren (3 papers)
  4. Can Huang (43 papers)
  5. Yinan Jing (6 papers)
  6. Kai Zhang (542 papers)
  7. X. Sean Wang (14 papers)
Citations (6)
View on Semantic Scholar