Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Effective Distillation of Table-based Reasoning Ability from LLMs (2309.13182v2)

Published 22 Sep 2023 in cs.CL

Abstract: LLMs have demonstrated remarkable performance across a wide range of natural language processing tasks. However, their enormous parameter size and extremely high requirements for compute power pose challenges for their practical deployment. Recent research has revealed that specific capabilities of LLMs, such as numerical reasoning, can be transferred to smaller models through distillation. Some studies explore the potential of leveraging LLMs to perform table-based reasoning. However, there has been no prior work focusing on table reasoning skills in smaller models specifically tailored for scientific table-to-text generation tasks. In this paper, we propose a novel table-based reasoning distillation approach, with the aim of distilling LLMs into tailored smaller models. Our experimental results have shown that a 220 million parameter model (Flan-T5-base) fine-tuned using distilled data, not only achieves a significant improvement compared to traditionally fine-tuned baselines, but also surpasses specific LLMs on a scientific table-to-text generation dataset. Our code is available at https://github.com/Bernard-Yang/DistillTableCoT.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Feverous: Fact extraction and verification over unstructured and structured information. ArXiv, abs/2106.05707.
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In IEEvaluation@ACL.
  3. Language models are few-shot learners. ArXiv, abs/2005.14165.
  4. Star: Sql guided pre-training for context-dependent text-to-sql parsing. ArXiv, abs/2210.11888.
  5. Wenhu Chen. 2023. Large Language Models are few(1)-shot Table Reasoners. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1120–1130, Dubrovnik, Croatia. Association for Computational Linguistics.
  6. Logical Natural Language Generation from Open-Domain Tables. ArXiv:2004.10404 [cs].
  7. TABFACT: A LARGE-SCALE DATASET FOR TABLE- BASED FACT VERIFICATION.
  8. Logic2Text: High-Fidelity Natural Language Generation from Logical Forms. arXiv:2004.14579 [cs]. ArXiv: 2004.14579.
  9. Binding Language Models in Symbolic Languages. ArXiv:2210.02875 [cs].
  10. Explanatory and actionable debugging for machine learning: A tableqa demonstration. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
  11. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.02311.
  12. Scaling instruction-finetuned language models. ArXiv, abs/2210.11416.
  13. Inducing and Using Alignments for Transition-based AMR Parsing. ArXiv:2205.01464 [cs].
  14. Specializing Smaller Language Models towards Multi-Step Reasoning. ArXiv:2301.12726 [cs].
  15. Carlos Gemmell and Jeffrey Stephen Dalton. 2023. Generate, transform, answer: Question specific tool synthesis for tabular data. ArXiv, abs/2303.10138.
  16. Infotabs: Inference on tables as semi-structured data. In Annual Meeting of the Association for Computational Linguistics.
  17. TAPAS: Weakly Supervised Table Parsing via Pre-training. arXiv:2004.02349 [cs]. ArXiv: 2004.02349.
  18. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531.
  19. Large Language Models Are Reasoning Teachers. ArXiv:2212.10071 [cs].
  20. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. ArXiv, abs/2305.02301.
  21. Omnitab: Pretraining with natural and synthetic data for few-shot table-based question answering. In North American Chapter of the Association for Computational Linguistics.
  22. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
  23. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics.
  24. PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5531–5546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  25. TAPEX: TABLE PRE-TRAINING VIA LEARNING A NEURAL SQL EXECUTOR.
  26. Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View. arXiv:2102.08585 [cs]. ArXiv: 2102.08585.
  27. SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables. ArXiv:2305.13186 [cs].
  28. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.17651.
  29. Teaching Small Language Models to Reason. ArXiv:2212.08410 [cs].
  30. SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables.
  31. FeTaQA: Free-form Table Question Answering. Transactions of the Association for Computational Linguistics, 10:35–49.
  32. Enhancing few-shot text-to-sql capabilities of large language models: A study on prompt design strategies. arXiv preprint arXiv:2305.12586.
  33. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Annual Meeting of the Association for Computational Linguistics.
  34. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. ArXiv.
  35. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
  36. Bleurt: Learning robust metrics for text generation. In ACL.
  37. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
  38. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ArXiv:2203.11171 [cs].
  39. Self-consistency improves chain of thought reasoning in language models. ArXiv, abs/2203.11171.
  40. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
  41. Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning. ArXiv:2301.13808 [cs].
  42. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. ArXiv, abs/1809.08887.
  43. Glm-130b: An open bilingual pre-trained model. ArXiv, abs/2210.02414.
  44. Bertscore: Evaluating text generation with bert. ArXiv, abs/1904.09675.
  45. Reastap: Injecting table reasoning skills during pre-training via synthetic reasoning examples. In Conference on Empirical Methods in Natural Language Processing.
  46. Large Language Models are Effective Table-to-Text Generators, Evaluators, and Feedback Providers. ArXiv:2305.14987 [cs].
  47. Seq2sql: Generating structured queries from natural language using reinforcement learning. ArXiv, abs/1709.00103.
  48. LEAST-TO-MOST PROMPTING ENABLES COMPLEX REASONING IN LARGE LANGUAGE MODELS.
  49. PaD: Program-aided Distillation Specializes Large Models in Reasoning. ArXiv:2305.13888 [cs].
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Bohao Yang (16 papers)
  2. Chen Tang (94 papers)
  3. Kun Zhao (97 papers)
  4. Chenghao Xiao (21 papers)
  5. Chenghua Lin (127 papers)
Citations (18)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - Bernard-Yang/DistillTableCoT (3 stars)