Time-LongQA: Temporal Long-Form QA Benchmark

• The paper introduces Time-LongQA, a benchmark that evaluates temporal reasoning by requiring models to retrieve and synthesize year-specific evidence from evolving corporate annual reports.
• It employs temporally-aware retrieval and query decomposition techniques to manage single-time, dual-time, and multi-time queries, achieving up to 25% improvement on complex temporal challenges.
• Time-LongQA provides a real-world, long-context evaluation setting that distinguishes evolving factual states from static corpora, thereby mitigating model leakage and ensuring robust assessments.

Time-LongQA is a benchmark dataset for long-form, temporally-aware question answering over real evolving knowledge. It was introduced with Temporal GraphRAG (T-GRAG) as a resource designed to test temporal reasoning across changing factual states rather than over static corpora, and it is built from public corporate annual reports so that retrieval-augmented systems must distinguish, compare, and synthesize year-specific evidence across a longitudinal document collection (Li et al., 3 Aug 2025).

1. Definition and benchmark scope

Time-LongQA is presented as a benchmark for assessing question answering over evolving knowledge, with an emphasis on temporal reasoning under long-context conditions. Its central premise is that many facts in a document collection are not timeless: figures, policies, personnel, and other corporate statements vary from year to year, so correct answering depends on retrieving the appropriate temporal slice and reconciling multiple time-indexed states when a query spans several years (Li et al., 3 Aug 2025).

The benchmark is explicitly positioned around temporal depth and evolution. In the formulation used for Time-LongQA, questions are not limited to one-off factual lookup; they include single-time retrieval, cross-year comparison, and multi-time analysis. This design targets failure modes such as temporal ambiguity and outdated retrieval, which arise when systems treat an evolving corpus as if it were temporally flat (Li et al., 3 Aug 2025).

A plausible implication is that Time-LongQA serves two evaluation purposes simultaneously. First, it measures whether a system can find relevant evidence in long reports. Second, it measures whether the system can maintain temporal consistency when the same entity or metric is described differently across years.

2. Source corpus and construction pipeline

The source corpus consists of publicly available corporate annual reports from Audi covering the years 2012 to 2023. The benchmark is described as using this real-world, longitudinal corpus to ensure that most content is not part of the training data of current LLMs, thereby minimizing model leakage and providing a fair testing ground for retrieval-augmented systems (Li et al., 3 Aug 2025).

For preprocessing, the reports were converted from PDF to Markdown using the MinerU tool. The construction procedure retains pure text while discarding layout and noise. After conversion, reports were divided into text blocks of 2,000 tokens per block, with the stated goal of aligning temporal content for more focused retrieval (Li et al., 3 Aug 2025).

This construction choice is significant because the underlying documents are not synthetic and are not organized as isolated fact triples. Instead, the benchmark inherits the discursive and redundant character of annual reports, while still imposing a temporal segmentation regime that supports retrieval experiments. The use of a homogeneous corporate corpus also means that temporal reasoning is tested within a stable domain vocabulary rather than across heterogeneous web text.

3. Dataset structure and question categories

Time-LongQA contains 2,292 question-answer pairs. Its internal organization follows four question categories that differentiate whether a query refers to one time point, two time points, several time points, or no explicit temporal constraint (Li et al., 3 Aug 2025).

Category	Count	Description
QSingle	1,538	Single-time-constrained questions
QDual	524	Dual-time-constrained questions
QMulti	113	Questions referencing three or more time points
QNon	117	Questions with no explicit temporal constraint

QSingle targets facts at a single year. QDual targets comparison across two distinct years. QMulti targets questions that reference three or more time points. QNon includes cases with no explicit temporal constraint, but the benchmark notes that these were manually validated to avoid temporal conflicts or outdated answers (Li et al., 3 Aug 2025).

The benchmark also introduces the notion of Temporal Evolution Knowledge for multi-time questions. In this setting, “key points” are methodically extracted to track the change of specific factual content across reports, such as vehicle deliveries, financials, and personnel changes. QDual and QMulti are then generated by finding similar key points across years using cosine similarity and producing question-answer pairs about temporal trends or comparisons. By contrast, QSingle and QNon are generated with GPT-4 under strong temporal grounding and then manually validated (Li et al., 3 Aug 2025).

4. Temporal reasoning profile

Time-LongQA is designed to stress-test systems along several axes of temporal reasoning. The benchmark explicitly requires retrieval of facts tied to specific time points, comparison between distinct time periods, analysis spanning multiple temporally evolving facts, and ambiguity detection in cases where year-to-year variation is substantial (Li et al., 3 Aug 2025).

The example given in the benchmark is a comparison question: “How did Audi Group total deliveries in 2012 and 2022 compare?” The associated evidence comes from different yearly reports and contains distinct numerical statements for each year. This illustrates the intended reasoning pattern: the system must isolate the relevant annual statements, preserve their temporal attribution, and then synthesize a comparative answer (Li et al., 3 Aug 2025).

This reasoning profile distinguishes simple temporal filtering from temporal synthesis. In QSingle, the main challenge is identifying the correct annual state. In QDual and QMulti, the challenge expands to coordinated retrieval across multiple years, followed by comparison, trend analysis, or multi-hop synthesis over temporally evolving facts. A plausible implication is that errors can arise at several layers: retrieval of the wrong year, conflation of multiple yearly states, or incorrect aggregation across years.

5. Evaluation protocol

Time-LongQA uses an LLM-based evaluation procedure. Qwen2.5-72B serves as the reference judge, comparing system-generated answers to ground truth. Each sample is scored three times, and a majority vote determines correctness. The principal reported metric is accuracy, referred to as “LLMscore,” defined as the fraction of answers considered correct (Li et al., 3 Aug 2025).

The evaluation prompts follow the DocBench protocol. The benchmark summary also states that all methods are evaluated with controlled token-count limits and the same embedding models, which is intended to preserve comparability across retrieval and generation pipelines (Li et al., 3 Aug 2025).

The choice of LLM-based judging is consequential in this setting because many answers involve comparative or synthesized responses rather than a single extractive span. This suggests that the benchmark values semantic correctness under temporal constraints more than literal span overlap, although the benchmark’s reported top-line number remains accuracy via the judge.

6. Experimental findings and benchmark behavior

Time-LongQA is used to evaluate base LLMs, vanilla RAG, GraphRAG, and T-GRAG. The reported result pattern is that base LLMs perform poorly, vanilla RAG improves substantially, GraphRAG performs well in non-temporal cases but struggles with temporal queries, and T-GRAG obtains the strongest results, especially as temporal complexity increases (Li et al., 3 Aug 2025).

Model family	QSingle	QDual	QMulti	QNon
Base LLMs	2–13	5–10	3–12	3–9
Vanilla RAG	45–65	22–51	17–46	53–69
GraphRAG	36–62	9–38	9–34	54–80
T-GRAG	63–82	49–68	54–75	59–81

The benchmark summary highlights especially large gaps on temporally complex queries. On QDual, T-GRAG achieves 63.74%–68.32%, whereas GraphRAG reaches 20.03%–37.91%. On QMulti, T-GRAG scores about 70%, whereas GraphRAG is reported at about 15–33% (Li et al., 3 Aug 2025).

The ablation analysis attached to the benchmark attributes substantial gains to explicitly temporal retrieval and decomposition. Adding the temporal retriever $R_{time}$ improves accuracy on temporal queries by about 19.3%. Temporal Query Decomposition improves multiyear comparison accuracy by up to about 25%. The benchmark also reports robustness under tighter retrieval budgets: vanilla RAG drops by more than 50% when the number of retrieved units is reduced, whereas T-GRAG drops by about 20% (Li et al., 3 Aug 2025).

These findings support the benchmark’s core claim that temporal conflict and redundancy are not peripheral issues in long-text QA over evolving knowledge. They are central error sources that materially affect system performance.

7. Position within temporal and long-context QA research

Time-LongQA occupies a distinct position among temporal QA resources in the supplied literature. Unlike TimelineQA, which generates synthetic lifelogs of imaginary people and evaluates atomic, multi-hop, aggregate, and temporal queries over structured episodes, Time-LongQA is built from real corporate annual reports and centers on temporal evolution in a homogeneous longitudinal corpus (Tan et al., 2023). Unlike the TLQA benchmark for time-referenced list-based question answering, which evaluates list completeness and temporal alignment using Wikipedia infoboxes in political and sports domains, Time-LongQA emphasizes long-form retrieval and synthesis across annual-report documents (Dumitru et al., 26 Jun 2025). Unlike TDBench, which systematically generates factual time-sensitive QA pairs from temporal databases using temporal SQL and temporal functional dependencies, Time-LongQA is document-centric rather than database-centric (Kim et al., 4 Aug 2025).

It also differs from broader temporal reasoning benchmarks such as TIME, which spans Wiki, News, and Dialog and covers 38,522 QA pairs across 3 levels and 11 fine-grained sub-tasks, because Time-LongQA narrows the setting to evolving corporate knowledge and long reports (Wei et al., 19 May 2025). At the same time, it shares an important long-context concern with Loong: both benchmarks reject the use of irrelevant filler as the main source of context length and instead focus on settings where long documents matter to the answer (Wang et al., 2024).

A common misconception is to treat Time-LongQA as merely another temporal QA dataset. The benchmark’s distinguishing feature is not just that questions mention time; it is that answers must be grounded in a corpus where the same factual dimension changes across years, making temporal disambiguation an intrinsic part of retrieval and generation (Li et al., 3 Aug 2025). This suggests that Time-LongQA is best understood as a benchmark at the intersection of temporal QA, long-context QA, and temporally aware retrieval.