TDBench: Temporal QA Benchmark

• TDBench is a benchmark for time-sensitive factual question answering that uses temporal databases, SQL, and temporal functional dependencies to generate and verify QA pairs.
• It evaluates model capabilities in temporal reasoning and alignment through 13 interval relations and metrics like answer, time, and answer-time accuracy.
• Its scalable, schema-driven pipeline reduces manual biases by generating diverse questions across domains such as Wikipedia and domain-specific datasets.

TDBench is a systematic benchmark for evaluating LLMs on time-sensitive factual question answering (TSQA). It leverages temporal databases and database design techniques—temporal SQL, Temporal Functional Dependencies (TFDs), and temporal joins—to scalably generate diverse, temporally constrained QA pairs and to automatically verify both the final answers and the time references used in model explanations. The benchmark studies factual TSQA through two model capabilities, temporal reasoning and temporal alignment, and introduces a fine-grained evaluation metric called time accuracy, which assesses the validity of time references in model explanations alongside traditional answer accuracy (Kim et al., 4 Aug 2025).

1. Problem setting and motivation

The benchmark focuses on factual TSQA, where facts evolve over time and a model must answer questions whose correctness depends on temporal context. In the formulation adopted by TDBench, this involves temporal reasoning—understanding temporal contexts such as “before 2019,” “exactly half a year prior to …,” and “currently”—and temporal alignment—answering with up-to-date facts as the world evolves (Kim et al., 4 Aug 2025).

TDBench was introduced in response to three limitations in prior TSQA benchmarks. First, manual curation bottlenecks make benchmark construction expensive and slow to update as facts change. Second, small, fixed templates restrict diversity of temporal contexts and factual types; the paper explicitly contrasts this with prior template counts such as 9 templates in Dhingra et al. 2022, 16 in Margatina et al. 2023, and 27 in DyKnow. Third, heavy reliance on Wikipedia/Wikidata and, in some cases, free-text tables, makes complex multi-hop construction difficult and constrains applicability to application-specific domains (Kim et al., 4 Aug 2025).

The benchmark addresses these limitations by harnessing structured temporal databases with explicit valid-time intervals, systematic template-free question generation, automatic verification of both final answers and time references, and applicability beyond Wikipedia/Wikidata to domain-specific databases and synthetic data. This suggests a shift from benchmark construction as manual prompt authoring to benchmark construction as temporal query generation over fixed schemas (Kim et al., 4 Aug 2025).

2. Database foundations and temporal semantics

The underlying data model is uni-temporal valid time. Facts are tracked with two attributes, start and end, which encode when each fact holds. An example relation is:

• Leader(country, role, name, gender, start, end)

with instances such as USA, President, Bush, M, 2001, 2009, USA, President, Obama, M, 2009, 2017, and U.K., Monarch, Elizabeth, F, 1952, 2022 (Kim et al., 4 Aug 2025).

A central design choice is the use of Temporal Functional Dependencies. A TFD, denoted $X \to^T Y$, requires the dependency to hold at every timepoint. The formal definition given is:

$$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$

where $\rho_{t_1}(r)$ selects tuples valid at $t_1$, and $\tau_{t_2}(\cdot)$ further restricts to tuples visible at $t_2$ (Kim et al., 4 Aug 2025).

TDBench also formalizes temporal natural join. For instances $r$ and $s$ of schemas $R$ and $S$, the temporal natural join $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$0 matches on shared attributes, concatenates unique attributes, and retains only rows whose valid-time intervals overlap:

$$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$1

This database foundation is not incidental. TFDs provide a principled way to select valid attribute pairs for TSQA: if country, role →^T name holds, then the identity of a role-holder is uniquely determined by (country, role) at any timepoint, preventing ambiguity and reducing spurious multi-answer artifacts. In multi-hop settings, TFDs propagate through joins using FD/TFD inference rules; the paper gives the example that Olympic’s game_edition → country combined with Leader’s country, role →^T name yields game_edition, role →^T name (Kim et al., 4 Aug 2025).

Temporal constraints are encoded over the 13 mutually exclusive Allen interval relations: before, after, meet, met-by, overlap, overlapped-by, equal, start, started-by, finish, finished-by, during, and contains. For current facts, the benchmark treats overlap with end IS NULL as the up-to-date condition (Kim et al., 4 Aug 2025).

3. Benchmark construction pipeline

The construction pipeline has three stages: knowledge selection via TFDs, temporal SQL query generation through the Genqueries algorithm, and SQL-to-text conversion with answer grounding (Kim et al., 4 Aug 2025).

In the first stage, if a TFD $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$2 is satisfied in a relation $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$3, the benchmark builds questions from $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$4 and uses $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$5 as the answer because $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$6 determines $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$7 at all time points. The paper’s canonical example is country, role →^T name, which yields questions such as “Who is the [role] of [country]?” with answer [name] (Kim et al., 4 Aug 2025).

In the second stage, Genqueries constructs temporal SQL rather than relying on hand-authored natural-language templates. For each tuple $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$8 and each TFD $$\forall\ t_1, t_2,\ \forall\ u_1, u_2 \in \tau_{t_2}(\rho_{t_1}(r)),\ u_1[X] = u_2[X] \Rightarrow u_1[Y] = u_2[Y]$$9, it builds a base query SELECT Y_i FROM s WHERE X_i = u.X_i, samples an interval $\rho_{t_1}(r)$0, and augments the query with a relation-specific SQL condition corresponding to one of the 13 Allen relations. The pseudocode is presented as:

• Initialize $\rho_{t_1}(r)$1
• For each tuple $\rho_{t_1}(r)$2 in $\rho_{t_1}(r)$3
  • For each TFD $\rho_{t_1}(r)$4
  • $\rho_{t_1}(r)$5
  • For each temporal relation $\rho_{t_1}(r)$6
  • Sample interval $\rho_{t_1}(r)$7
  • $\rho_{t_1}(r)$8 using the SQL condition from the relation
  • Append $\rho_{t_1}(r)$9 to $t_1$0
• Return $t_1$1 (Kim et al., 4 Aug 2025)

The benchmark’s temporal SQL includes relation-specific predicates such as inequalities for before/after, exact offsets for meet/met-by, equality constraints for equal, mixed equalities and inequalities for start and finish families, and strict containment for during and contains. The meet relation example is encoded as a.end = b.end − b.length, corresponding to the temporal context “A president who ends exactly half a year before March 20, 2001” (Kim et al., 4 Aug 2025).

In the third stage, TDBench uses GPT-4o as a SQL-to-text translator to produce diverse natural-language phrasings, with 91.5% correctness under manual inspection. Answers are computed by executing the SQL, so linguistic diversity comes from the LLM but answer grounding comes from the database (Kim et al., 4 Aug 2025).

Several examples illustrate the construction:

• Meet example:

SELECT name, end FROM Leader WHERE Country = 'Brazil' AND Role = 'President' AND date(end) = date('2019-05-01', '-4 month') Answer: Michel Temer Time reference: 2019-01-01 (end) A model rationale should include “ended on January 1, 2019” (Kim et al., 4 Aug 2025).

• Contains example:

SELECT name, start, end FROM Leader WHERE Country='Japan' AND Role='Emperor' AND Start < '2019-10-01' AND End > '2024-08-26' Answer: Akihito Time references: 1989-01-07 (start), 2019-05-01 (end) (Kim et al., 4 Aug 2025).

• Current overlap example:

SELECT name, start FROM Leader WHERE Country='Netherlands' AND Role='King' AND End IS NULL Answer: Willem-Alexander Time reference: 2013-04-30 (start) (Kim et al., 4 Aug 2025).

• Multi-hop temporal join example:

SELECT name, start, end FROM Leader ⋈^T Olympic WHERE game_edition='7th Winter' AND Role='President' Answer: Giovanni Gronchi Time references: 1955-05-11, 1962-05-11 (Kim et al., 4 Aug 2025).

The benchmark’s multi-hop generation follows directly from temporal joins over fixed schemas, rather than manual bridge-entity engineering. A plausible implication is that the cost of extending benchmark scope is shifted from question authoring to schema and dependency specification (Kim et al., 4 Aug 2025).

4. Dataset composition and task structure

TDBench supports arbitrary temporal databases and is demonstrated on ILGA legal data, World Bank carbon pricing, UNESCO heritage, Netflix shows, Wikipedia-like temporal databases, and synthetic medical records (Kim et al., 4 Aug 2025).

The main experiments use 6,177 questions from Wikipedia domains and 1,704 from domain-specific data, for a total of 7,881 questions. An additional 1,350 synthetic medical questions are used in supplementary analysis. Task construction is divided into temporal alignment, temporal reasoning, and multi-hop question answering (Kim et al., 4 Aug 2025).

Component	Count	Notes
Wikipedia domains	6,177	Main experiments
Domain-specific	1,704	Main experiments
Synthetic medical	1,350	Additional analysis
Temporal alignment	2,079	“current” facts
Temporal reasoning	6,756	full set of 13 relations
Multi-hop	396	via joins

The sources include Wikipedia-like temporal databases covering Countries, Athletes, Organizations, and Olympics, as well as domain-specific platforms covering ILGA legal, World Bank carbon tax, UNESCO heritage, and Netflix shows. The benchmark therefore spans legal, environmental, cultural, and entertainment domains, in addition to synthetic medical data (Kim et al., 4 Aug 2025).

The temporal coverage of the closed-book evaluations spans 1985–2025. The benchmark also supports three answer cardinality categories: Multiple answers, Unique answer, and None. This support for “None” is explicit and is described as differing from most benchmarks that assume unique answers (Kim et al., 4 Aug 2025).

Two evaluation settings are defined. In open-book evaluation, row-level table context, including relevant and irrelevant rows, is appended to the prompt, testing temporal reasoning given structured evidence. In closed-book evaluation, models rely on internal world knowledge only (Kim et al., 4 Aug 2025).

5. Evaluation methodology and empirical findings

TDBench evaluates responses with three metrics. Answer accuracy (A) is the proportion of responses with correct final answers. Time accuracy (T) is the proportion of responses whose explanation correctly includes the required time references, defined per temporal relation. If both start and end are relevant, partial credit of 50% is given when only one is correct. Answer-Time accuracy (AT) is the proportion of responses where both answer and required time references are correct, with no partial credit on time (Kim et al., 4 Aug 2025).

The formal definition of time accuracy uses a time reference set $t_1$2 and a question-reference function $t_1$3 derived from the underlying temporal SQL relation:

$t_1$4

This measures whether the model’s explanation includes the correct time references in content, regardless of formatting (Kim et al., 4 Aug 2025).

Explanation assessment is automated. Relation-specific criteria determine which timestamps to check for each question; models are prompted to include time references in their rationale; and an LLM-based judge, Deepseek-R1-14B, verifies whether the correct dates appear in the explanation, allowing for format variations such as “26 Jan 2025.” The paper reports 91.1% accuracy under manual inspection for this judge, and 95.5% for GPT-4o as judge, though GPT-4o is not used as the main judge in order to avoid bias (Kim et al., 4 Aug 2025).

The experiments evaluate eight LLMs—GPT-3.5, GPT-4, GPT-4o, Llama3.1‑70B, Mixtral‑8×7B, Gemma2‑27B, Qwen2‑72B, and Granite3.1‑8B—with temperature set to 0. SQL-to-text translation uses a small temperature of 0.3 to increase paraphrastic variety (Kim et al., 4 Aug 2025).

The central empirical finding is that time hallucinations are frequent. Across Wikipedia domains, integrating time accuracy reveals an average 21.7-point drop from $t_1$5 to $t_1$6. The paper highlights GPT-4o with $t_1$7 and $t_1$8 ($t_1$9), Llama3.1‑70B with $\tau_{t_2}(\cdot)$0 and $\tau_{t_2}(\cdot)$1 ($\tau_{t_2}(\cdot)$2), and Gemma2‑27B with $\tau_{t_2}(\cdot)$3 and $\tau_{t_2}(\cdot)$4 ($\tau_{t_2}(\cdot)$5). The stated interpretation is that many answers judged correct under answer-only evaluation are accompanied by invalid time references (Kim et al., 4 Aug 2025).

Domain-specific performance varies. In Law, Carbon Tax, Heritage, and Netflix, the performance leaders differ; GPT‑4o is reported as strong in Law, Carbon, and Heritage, whereas Llama3.1‑70B is strong in Netflix. For Law, the average drop is 42.8 points, from $\tau_{t_2}(\cdot)$6 to $\tau_{t_2}(\cdot)$7 (Kim et al., 4 Aug 2025).

Performance also varies by answer cardinality. In open-book Wikipedia settings, “None” questions produce lower scores and reflect challenges in rejecting spurious matches. For unique-answer questions, Qwen2‑72B reaches $\tau_{t_2}(\cdot)$8. For multiple-answer questions, GPT-4 has $\tau_{t_2}(\cdot)$9, $t_2$0, and $t_2$1 (Kim et al., 4 Aug 2025).

Across temporal relations, models perform best on equal relations such as “in 2025” and worse on overlap and contain, which require precise dual-boundary reasoning. Start-anchored constraints, including start and started-by, are generally easier than end-anchored constraints such as finish and finished-by (Kim et al., 4 Aug 2025).

The closed-book time-span analysis shows a gradual increase from 1985 to 2010, with a noticeable bump in 1990–1995 across all models, followed by a drop in most recent years, 2020–2025. The paper describes this as likely reflecting shared training data distribution (Kim et al., 4 Aug 2025).

Multi-hop evaluation is performed over Olympic×Country joins. For 3-hop questions, GPT-4o achieves $t_2$2 and $t_2$3, while Llama3.1‑70B achieves $t_2$4 and $t_2$5; Granite3.1‑8B shows severe second-hop failures with $t_2$6. The benchmark uses step-wise hallucination rates $t_2$7 to identify which hop fails, conditioned on earlier correct hops (Kim et al., 4 Aug 2025).

The paper also compares reasoning aids. Chain-of-thought and Time‑CoT generally outperform RAG, which often retrieves temporally misaligned or generic passages. In a 3-hop example, GPT‑4o reaches $t_2$8 with Time‑CoT, $t_2$9 with CoT, and $r$0 with RAG; Llama3.1‑70B reaches $r$1 with CoT, $r$2 with Time‑CoT, and $r$3 with RAG (Kim et al., 4 Aug 2025).

Finally, benchmark correctness is checked against human verification on 125 samples per task. Agreement scores are reported as temporal alignment $r$4, temporal reasoning $r$5, and multi-hop $r$6 (Kim et al., 4 Aug 2025).

6. Positioning, limitations, and disambiguation

TDBench is positioned as complementary to Wikipedia/Wikidata-based TSQA evaluations rather than as a replacement. Its stated advantages are coverage of 13 interval relations rather than a typical equal/current focus, elimination of fixed template restrictions and manual authoring, evaluation of both answers and time references, and applicability to application-specific temporal databases beyond Wikidata/Wikipedia (Kim et al., 4 Aug 2025).

The benchmark also has clear limits. It assumes that the underlying temporal databases are clean, so inaccuracies in source data can affect QA correctness. Its explanation evaluation targets temporal references in rationales, but models may still hallucinate other non-temporal parts of the explanation. The paper identifies integrating automated data cleaning and extending evaluation to broader factual verification and causality checks as future work (Kim et al., 4 Aug 2025).

A recurrent source of confusion is nomenclature. The name “TDBench” is also used for at least two unrelated benchmarks in other subfields: a benchmark for vision-LLMs on top-down image understanding (Hou et al., 1 Apr 2025), and a tabular data distillation benchmark introduced alongside TDColER (Kang et al., 23 Jan 2025). By contrast, the TDBench discussed here is specifically a benchmark for factual time-sensitive question answering in LLMs, grounded in temporal databases, temporal SQL, TFDs, and temporal joins (Kim et al., 4 Aug 2025).

In that sense, the benchmark’s most distinctive contribution is methodological rather than merely curatorial. It operationalizes TSQA evaluation through executable temporal database semantics, and it treats time references in explanations as first-class evaluation targets. This suggests that, for temporally grounded QA, answer-only scoring is structurally incomplete unless the temporal rationale itself is validated (Kim et al., 4 Aug 2025).