Papers
Topics
Authors
Recent
Search
2000 character limit reached

WikiTableEdit: A Benchmark for Table Editing by Natural Language Instruction

Published 5 Mar 2024 in cs.AI | (2403.02962v1)

Abstract: Tabular data, as a crucial form of data representation, exists in diverse formats on the Web. When confronted with complex and irregular tables, manual modification becomes a laborious task. This paper investigates the performance of LLMs in the context of table editing tasks. Existing research mainly focuses on regular-shaped tables, wherein instructions are used to generate code in SQL, Python, or Excel Office-script for manipulating the tables. Nevertheless, editing tables with irregular structures, particularly those containing merged cells spanning multiple rows, poses a challenge when using code. To address this, we introduce the WikiTableEdit dataset. Leveraging 26,531 tables from the WikiSQL dataset, we automatically generate natural language instructions for six distinct basic operations and the corresponding outcomes, resulting in over 200,000 instances. Subsequently, we evaluate several representative LLMs on the WikiTableEdit dataset to demonstrate the challenge of this task. The dataset will be released to the community to promote related researches.

Authors (3)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (15)
  1. Falcon-40B: an open large language model with state-of-the-art performance. 2023.
  2. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1533–1544, 2013.
  3. Wikitablet: A large-scale data-to-text dataset for generating wikipedia article sections, 2021.
  4. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, 2022.
  5. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries, 2023.
  6. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81, 2004.
  7. What to talk about and how? selective generation using lstms with coarse-to-fine alignment, 2016.
  8. Gpt-4 technical report, 2023.
  9. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA, 2002. Association for Computational Linguistics.
  10. Compositional semantic parsing on semi-structured tables. CoRR, abs/1508.00305, 2015.
  11. Instructexcel: A benchmark for natural language instruction in excel, 2023.
  12. Llama 2: Open foundation and fine-tuned chat models, 2023.
  13. Challenges in data-to-document generation, 2017.
  14. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415, 2016.
  15. Seq2sql: Generating structured queries from natural language using reinforcement learning, 2017.
Citations (1)

Summary

No one has generated a summary of this paper yet.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Continue Learning

We haven't generated follow-up questions for this paper yet.