Temporal QA Benchmarks Overview

• Temporal QA benchmarks are datasets and protocols designed to rigorously evaluate automated systems' ability to handle dynamic time facts, event ordering, and implicit as well as explicit temporal reasoning.
• They span multiple modalities including textual documents, knowledge graphs, structured tables, videos, and time series, enabling comprehensive evaluation across diverse data formats.
• Evaluation metrics such as exact match, token-level F1, and specialized time accuracy reveal challenges like multi-hop complexity, knowledge drift, and differences between synthetic and real-world data.

Temporal question answering (QA) benchmarks are standardized datasets and evaluation protocols designed to rigorously assess and advance the temporal reasoning capabilities of automated QA systems. Temporal QA benchmarks are critical in domains where answers depend on temporal facts, dynamic contexts, event ordering, or time-anchored relationships, and they now span a broad range of document, knowledge graph, table, timeline, video, and time series settings. These resources provide formal definitions, challenging question types (often requiring multi-hop and implicit time reasoning), and precise evaluation metrics to drive the development of models capable of handling the complexities of temporally structured information.

1. Taxonomy of Temporal Question Answering Benchmarks

Temporal QA benchmarks can be classified along several orthogonal axes:

A. Modality:

• Textual/Document-based: Datasets like TimeQA (Chen et al., 2021), ArchivalQA (Wang et al., 2021), ChronoQA (Chen et al., 17 Aug 2025), and ComplexTempQA (Gruber et al., 2024) focus on news, encyclopedic, or web documents.
• Knowledge Graph (KG)–based: Benchmarks such as TimeQuestions (Jia et al., 2021), TempReason (Tan et al., 2023), TempQA-WD (Neelam et al., 2022), and ForecastTKGQuestions (Ding et al., 2022).
• Table/Structured Data: Resources including TempTabQA (Gupta et al., 2023), TimelineQA (Tan et al., 2023), and TDBench (Kim et al., 4 Aug 2025).
• Video/Multimodal: NExT-QA (Xiao et al., 2021), TLQA (Swetha et al., 13 Jan 2025), Perceive, Query & Reason (Amoroso et al., 2024).
• Time Series: TSAQA (Jing et al., 30 Jan 2026).

B. Temporal Reasoning Type:

• Explicit time anchoring: "Who was president in 1990?" as in TimeQA.
• Implicit constraints: "Who was president before the war?" as in Tiq (Jia et al., 2024).
• Event/event or date/event comparisons: Event ordering, overlap, before/after relations.
• Present-anchored (continually updating): PAT-Questions (Meem et al., 2024).
• Multi-hop and aggregation: Complex reasoning chains or aggregate counts (Complex-TR (Tan et al., 2023), ComplexTempQA).

C. Question Format and Domain:

• Span extraction (Answer spans in texts): TimeQA, ArchivalQA.
• Generation (free-text answers): Many video QA and open-ended benchmarks.
• Table or structured answer: TimelineQA, TempTabQA.
• Multilingual and multi-granularity: HistoryBankQA (Mandal et al., 16 Sep 2025) extends both linguistic and temporal depth.

This taxonomy reflects a shift from single-span or single-hop settings toward larger, more expressive, and more temporally nuanced QA tasks.

2. Dataset Construction and Annotation Methodologies

Temporal QA benchmarks are constructed through diverse pipelines, often involving substantial automatic extraction followed by rigorous filtering and validation:

• Document-based QA: ArchivalQA (Wang et al., 2021) initiates with Wikipedia "Year" pages for seed events, retrieves relevant news articles, generates candidate QAs via masked entity templates, and applies specificity, ambiguity, and quality filtering with neural classifiers, yielding over 532 k non-ambiguous QA pairs over a 20-year archive.
• KG-based QA: TimeQuestions (Jia et al., 2021) extracts temporal questions from established Freebase/DBpedia QA datasets, projects these to Wikidata, and annotates them for temporal intent (explicit, implicit, ordinal, temporal-answer).
• Present-anchored QA: PAT-Questions (Meem et al., 2024) leverages SPARQL queries on current snapshots of Wikidata, and includes a scriptable, self-updating mechanism for continuous gold-answer refresh.
• Implicit temporal questions: The Tiq benchmark (Jia et al., 2024) systematically generates implicit questions by pairing text/KG snippets with compatible time scopes and neural rephrasing, then ensures coverage of temporal constraint types (overlap, before, after).
• Table-based QA: TempTabQA (Gupta et al., 2023) obtains tables from Wikipedia infoboxes, crowdsources multi-step temporal questions that require reasoning over multiple rows, and ensures complexity and answerability via expert adjudication.
• Time series QA: TSAQA (Jing et al., 30 Jan 2026) generates synthetic and real-world temporal analysis tasks with labeled segments, classes, and transformations, and applies standardized, large-scale sampling across domains.

Benchmarks such as ComplexTempQA (Gruber et al., 2024) and TimelineQA (Tan et al., 2023) automate large-scale, multi-hop temporal QA generation using Wikipedia/Wikidata scraping, logical forms, and templated question construction.

3. Temporal Reasoning Categories, Tasks, and Formalizations

Temporal QA benchmarks operationalize multiple reasoning categories:

Event/Time-point anchoring:

• Questions require alignment with explicit dates, time intervals, or implicit event references.
• Formally, answers must satisfy constraints such as $t_q \in [t_s, t_e]$ (TimeQA (Chen et al., 2021)).

Temporal ordering or comparison:

• Event-event, time-time, and event-time relations—e.g., ordering (“before,” “after”), overlap, inclusion.
• Formal operations: $$\text{before}(I, J): \text{end}(I) \leq \text{start}(J)$$, $$\text{overlap}([s_1,e_1],[s_2,e_2]): \max(s_1,s_2) \leq \min(e_1,e_2)$$ (Neelam et al., 2022).

Multi-hop and aggregation:

• Multi-hop: Chained relations, e.g., “Who was the head coach of the team that X played for in 2012?” (PAT-Questions (Meem et al., 2024), Complex-TR (Tan et al., 2023), TLQA (Swetha et al., 13 Jan 2025)).
• Counting/aggregation: Number of events/entities satisfying temporal predicates within intervals (ComplexTempQA (Gruber et al., 2024), TimelineQA (Tan et al., 2023)).

Implicit and present-anchored reasoning:

• Implicit: No explicit temporal cue—date must be inferred from context (Tiq (Jia et al., 2024)).
• Present-anchored: Answers depend on the world state at query-time, necessitating continual update or retrieval (PAT-Questions).

Forecasting temporal QA:

• ForecastTKGQuestions (Ding et al., 2022) uniquely addresses future-oriented queries over temporal KGs, enforcing “past-only” input constraints.

Video/Multimodal temporal logic:

• TLQA (Swetha et al., 13 Jan 2025) adopts temporal logic operators (eventually, always, until, before, after, co-occur, strict/loose order) and generates both Boolean and multi-choice QA over action/state sequences.

4. Evaluation Protocols, Metrics, and Baseline Results

Temporal QA benchmarks utilize a variety of precision-oriented and faithfulness-oriented evaluation metrics:

Canonical metrics:

• Exact Match (EM): Strict span equality (Chen et al., 2021, Wang et al., 2021, Gruber et al., 2024).
• Token-level F1: For partial credit on extractive answers.
• Set-Accuracy: For multi-answer questions; the answer set must exactly match the gold set (Tan et al., 2023).
• Answer-F1: Generalized to handle multi-answer, multi-hop outputs (Tan et al., 2023).

Specialized/Novel metrics:

• Time accuracy: Requires models to not only generate the correct answer but faithfully cite the relevant time reference in their explanations (TDBench (Kim et al., 4 Aug 2025)).
• Faithfulness: Fraction of correct answers that also satisfy temporal consistency with gold evidence (Jia et al., 2024).
• Task-specific: For time series—puzzling score (sequence ordering), macro-average across tasks (TSAQA (Jing et al., 30 Jan 2026)).

Baseline results (selected):

Benchmark	Best Model/Setting	EM (%)	F1 (%)	Notes
TimeQA	FiD (NQ+TimeQA)	60.5	67.9	Humans: 89.0/93.3
ArchivalQA	BERTserini-NYT-TempRes	56.3	68.9	Four subsplits analyzed
PAT-Questions	GPT-3.5+RAG (single-hop)	15.5	16.5	Only ~7–9% EM on multi-hop
Complex-TR	T5-B PIT-SFT (ReasonQA, L2 1-hop)	87.0	91.7	Set-Accuracy/Multi-Answer
TDBench	LLaMA-3.1-70B, Wikipedia domain	56.9	N/A	Time-accuracy: 35.2%
TSAQA	Gemini-2.5-Flash	65.08	—	Instruction tuning boosts OS
TLQA (Video)	SeViLA (MC), VideoLLaVA (Bool QA)	55.7	52.6	16-temporal logic operators
HistoryBankQA	GPT4o (FactQA, English)	49	—	Multilingual, multi-century
ComplexTempQA	GPT-3.5 (zero-shot EM)	0.49	7.74	100M+ pairs; task is unsolved

Zero-shot performance remains below 1% EM on the most complex, multi-hop and aggregation-rich benchmarks (Gruber et al., 2024), underscoring unsolved challenges.

5. Unique Challenges and Limitations Across Temporal QA Benchmarks

Temporal Robustness and Knowledge Drift:

• LLMs trained on static corpora struggle with present-anchored and emerging facts (PAT-Questions, TDBench).
• Multi-hop and multi-answer settings induce significant generalization gaps and error cascades (Complex-TR, ComplexTempQA).

Handling Implicit and Diverse Temporal Constraints:

• Implicit temporal constraints are rarely supported directly in existing systems; benchmarks such as Tiq expose this deficiency (Jia et al., 2024).
• Explicit and implicit temporal expressions, as well as a mix of ordinal, ranking, and aggregation requirements, require more contextually aware semantic parsers and retrievers (TimeQuestions, TempTabQA).

Evaluation Gaps:

• EM and F1 often overestimate model competence in multi-answer and compositional settings.
• Time accuracy and faithfulness metrics reveal persistent hallucination and inconsistency in time references, even when final answers are correct (TDBench).

Synthetic vs. Real-World Data:

• Synthetic benchmarks (TimelineQA, ComplexTempQA) offer controllable, scalable scope but risk underrepresenting natural phrasing and event structure.
• Real-world data sets (ArchivalQA, ChronoQA, HistoryBankQA) offer richer temporal diversity but require more extensive validation.

6. Relationship to Adjacent Research and Future Directions

Temporal QA benchmarks play a foundational role in the evaluation, diagnosis, and advancement of temporal reasoning in NLP and related fields. They connect to:

• Temporal Information Extraction: Relation to event detection, temporal tagging, and time expression normalization.
• Temporal Knowledge Graph Construction: KGs and temporal databases provide ground truth for multi-hop and time-anchored queries (TempQA-WD, ForecastTKGQuestions, TDBench).
• Video and Time Series Reasoning: Direct generalization into multimodal QA, temporal logic in vision (TLQA), and time-series causal inference (TSAQA).
• Retrieval-Augmented Generation: Importance of dynamically retrievable, temporally-aligned evidence document sets (ChronoQA).
• Continual and Online Evaluation: Need for self-updating, streaming, and present-anchored benchmarks (PAT-Questions, TDBench).

Key directions highlighted in benchmark papers include extending coverage to implicit and multi-lingual settings (Mandal et al., 16 Sep 2025), improving faithfulness and time-accuracy measurement, hybridizing symbolic-neural methods (e.g., integrating temporal SQL or KGs into neural pipelines), and pushing to greater reasoning complexity, aggregation, and compositionality (Gruber et al., 2024).

Current benchmarks establish a rigorous reference for systematic challenge in temporal reasoning, and advances in benchmark design will likely drive the next generation of time-aware and temporally robust QA systems.