2000 character limit reached
Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models (2306.08952v2)
Published 15 Jun 2023 in cs.CL and cs.AI
Abstract: Reasoning about time is of fundamental importance. Many facts are time-dependent. For example, athletes change teams from time to time, and different government officials are elected periodically. Previous time-dependent question answering (QA) datasets tend to be biased in either their coverage of time spans or question types. In this paper, we introduce a comprehensive probing dataset \tempreason to evaluate the temporal reasoning capability of LLMs. Our dataset includes questions of three temporal reasoning levels. In addition, we also propose a novel learning framework to improve the temporal reasoning capability of LLMs, based on temporal span extraction and time-sensitive reinforcement learning. We conducted experiments in closed book QA, open book QA, and reasoning QA settings and demonstrated the effectiveness of our approach. Our code and data are released on https://github.com/DAMO-NLP-SG/TempReason.
- SemEval-2016 task 12: Clinical TempEval. In Proceedings of SemEval.
- Translating embeddings for modeling multi-relational data. In Proceedings of NIPS.
- An annotation framework for dense event ordering. In Proceedings of ACL.
- A dataset for answering time-sensitive questions. In Proceedings of NIPS.
- A dataset for hyper-relational extraction and a cube-filling approach. In Proceedings of EMNLP.
- HyTE: Hyperplane-based temporally aware knowledge graph embedding. In Proceedings of EMNLP.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL.
- Time-Aware Language Models as Temporal Knowledge Bases. Transactions of ACL.
- Are large language models good data annotators? a study on gpt-3, chatgpt and gpt-4. In Proceedings of ACL.
- Reasoning implicit sentiment with chain-of-thought prompting. In Proceedings of ACL.
- Retrieval augmented language model pre-training. In Proceedings of ICML.
- Joint event and temporal relation extraction with shared representations and structured prediction. In Proceedings of EMNLP.
- Tempquestions: A benchmark for temporal question answering. In Proceedings of WWW.
- Tequila: Temporal question answering over knowledge bases. In Proceedings of CIKM.
- Complex temporal question answering on knowledge graphs. In Proceedings of CIKM.
- Towards time-aware knowledge graph completion. In Proceedings of COLING.
- Dense passage retrieval for open-domain question answering. In Proceedings of EMNLP.
- Realtime qa: What’s the answer right now? In Proceedings of EMNLP.
- Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In Proceedings of ICLR.
- Natural questions: a benchmark for question answering research. Transactions of ACL.
- Tensor decompositions for temporal knowledge base completion. In Proceedings of ICLR.
- Streamingqa: A benchmark for adaptation to new knowledge over time in question answering models. In Proceedings of ICML.
- Roberta: A robustly optimized bert pretraining approach. In arXiv preprint arXiv:1907.11692.
- Tempoqr: temporal question reasoning over knowledge graphs. In Proceedings of AAAI, volume 36, pages 5825–5833.
- TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of EMNLP.
- Training language models to follow instructions with human feedback. In arXiv preprint arXiv:2203.02155.
- The timebank corpus. In Corpus linguistics.
- Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR.
- Hossein Rajaby Faghihi and Parisa Kordjamshidi. 2021. Time-stamped language model: Teaching language models to understand the flow of events. In Proceedings of NAACL.
- SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP.
- How much knowledge can you pack into the parameters of a language model? In Proceedings of EMNLP.
- Question answering over temporal knowledge graphs. In Proceedings of ACL.
- Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06347.
- Improving time sensitivity for question answering over temporal knowledge graphs. In Proceedings of ACL. Association for Computational Linguistics.
- SemEval-2010 task 13: TempEval-2. In Proceedings of SemEval.
- Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Proceedings of CACM.
- Knowledge graph embedding by translating on hyperplanes. In Proceedings of AAAI.
- Finetuned language models are zero-shot learners. In Proceedings of ICLR.
- Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium.
- Multi-source test-time adaptation as dueling bandits for extractive question answering. In Proceedings of ACL.
- Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of EMNLP.
- Verify-and-edit: A knowledge-enhanced chain-of-thought framework. In Proceedings of ACL.
- Qingyu Tan (9 papers)
- Hwee Tou Ng (44 papers)
- Lidong Bing (144 papers)