Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

OpenTab: Advancing Large Language Models as Open-domain Table Reasoners (2402.14361v2)

Published 22 Feb 2024 in cs.LG

Abstract: LLMs trained on large volumes of data excel at various natural language tasks, but they cannot handle tasks requiring knowledge that has not been trained on previously. One solution is to use a retriever that fetches relevant information to expand LLM's knowledge scope. However, existing textual-oriented retrieval-based LLMs are not ideal on structured table data due to diversified data modalities and large table sizes. In this work, we propose OpenTab, an open-domain table reasoning framework powered by LLMs. Overall, OpenTab leverages table retriever to fetch relevant tables and then generates SQL programs to parse the retrieved tables efficiently. Utilizing the intermediate data derived from the SQL executions, it conducts grounded inference to produce accurate response. Extensive experimental evaluation shows that OpenTab significantly outperforms baselines in both open- and closed-domain settings, achieving up to 21.5% higher accuracy. We further run ablation studies to validate the efficacy of our proposed designs of the system.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Feverous: Fact extraction and verification over unstructured and structured information. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, volume 1, 2021.
  2. Tutorial proposal: Retrieval-based language models and applications. In The 61st Annual Meeting of the Association for Computational Linguistics (ACL): Tutorial Abstracts, pp.  41, 2023.
  3. Improving language models by retrieving from trillions of tokens. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pp.  2206–2240. PMLR, 2022.
  4. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877–1901, 2020.
  5. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp.  1870–1879, 2017.
  6. Evaluating large language models trained on code, 2021a.
  7. Retrieval augmented via execution guidance in open-domain table qa. In Proceedings of the 2022 5th International Conference on Algorithms, Computing and Artificial Intelligence (ACAI), pp.  1–6, 2022.
  8. Wenhu Chen. Large language models are few(1)-shot table reasoners. In Findings of the Association for Computational Linguistics (EACL), pp.  1120–1130, 2023.
  9. Open question answering over tables and text. In The Ninth International Conference on Learning Representations (ICLR), 2021b.
  10. Binding language models in symbolic languages. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
  11. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929–3938. PMLR, 2020.
  12. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00303, 2022.
  13. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp.  4320–4333, 2020.
  14. Open domain question answering over tables via dense retrieval. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pp.  512–519, 2021.
  15. Dual-channel evidence fusion for fact verification over texts and tables. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pp.  5232–5242, 2022.
  16. Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics (ACL), pp.  1049–1065, 2023.
  17. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022.
  18. OmniTab: Pretraining with natural and synthetic data for few-shot table-based question answering. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pp.  932–942, 2022.
  19. Dense passage retrieval for open-domain question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.  6769–6781, 2020.
  20. Open-WikiTable : Dataset for open domain question answering with complex reasoning over table. In Findings of the Association for Computational Linguistics (ACL), pp.  8285–8297, 2023.
  21. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  22. A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability, 2023a.
  23. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023b.
  24. TAPEX: Table pre-training via learning a neural SQL executor. In The Tenth International Conference on Learning Representations (ICLR), 2022.
  25. OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023.
  26. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp.  1470–1480, 2015.
  27. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp.  3982–3992, 2019.
  28. A thorough examination on zero-shot dense retrieval. arXiv preprint arXiv:2204.12755, 2022.
  29. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009a.
  30. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389, 2009b.
  31. Joint verification and reranking for open fact checking over tables. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp.  6787–6799, 2021.
  32. Sql-palm: Improved large language model adaptation for text-to-sql, 2023.
  33. Table retrieval may not necessitate table-specific model design. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pp.  36–46, 2022.
  34. UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.  602–631, 2022.
  35. Xi Ye and Greg Durrett. The unreliability of explanations in few-shot prompting for textual reasoning. Advances in Neural Information Processing Systems (NeurIPS), 35:30378–30392, 2022.
  36. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp.  174–184, 2023.
  37. Tacube: Pre-computing data cubes for answering numerical-reasoning questions over tabular data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.  2278–2291, 2022.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (8)
  1. Kezhi Kong (12 papers)
  2. Jiani Zhang (21 papers)
  3. Zhengyuan Shen (7 papers)
  4. Balasubramaniam Srinivasan (12 papers)
  5. Chuan Lei (16 papers)
  6. Christos Faloutsos (88 papers)
  7. Huzefa Rangwala (57 papers)
  8. George Karypis (110 papers)
Citations (12)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets