Timeline-Bench

• Timeline-Bench connects we various datasets for evaluating systems transforming temporally organized evidence into outputs like ordered events and forecasts.
• The benchmark encompasses diverse task families, including timeline extraction, summarization, question answering, video timeline modeling, and more.
• Timeline-Bench should utilize both human and automated evaluation based on metrics such as temporal structure, content, grounding, and granularity.

Timeline-Bench is an Editor’s term for a proposed family of evaluations spanning timeline construction, temporal reasoning, timeline summarization, timeline visualization, temporal question answering, process timelines, and time-series forecasting. No supplied paper defines a single benchmark formally named “Timeline-Bench”; instead, the term connects complementary resources including TimeSET, TimelineQA, DTELS-Bench, TLS-I, YouTube-News-Timeline, TemporalBench, TempusBench, and other timeline-oriented datasets. These resources differ substantially in input modality, temporal direction, output representation, annotation scheme, and evaluation criteria. Accordingly, Timeline-Bench is best understood as a benchmark architecture rather than a single established dataset.

1. Scope and conceptual foundations

Timeline-Bench encompasses systems that transform temporally organized evidence into one or more of the following outputs: ordered events, timestamp–summary pairs, event clusters, temporal answers, visual timeline structures, process graphs, forecasts, or event-conditioned predictions. The common requirement is that temporal information is not incidental: systems must identify, preserve, infer, compare, or exploit relationships among events distributed over time.

The principal task families are:

1. Timeline extraction and ordering: detect salient events, assign temporal anchors, resolve event identity, and construct an ordered or partially ordered timeline. Multilingual and cross-lingual timeline extraction additionally requires entity linking, semantic-role interoperability, and cross-language event coreference (Laparra et al., 2017).
2. Timeline summarization: generate chronological time–summary pairs from timestamped documents. Conventional TLS separates date selection from date summarization, while archival TLS extends the setting to heterogeneous, OCR-derived, incomplete, and historically long collections (Gutehrlé et al., 2023).
3. Dynamic-granularity summarization: generate timelines whose detail level follows explicit or natural-language instructions, such as “five nodes” or “a coarse-grained timeline.” DTELS-Bench evaluates informativeness, granular consistency, factuality, and coherence (Zhang et al., 2024).
4. Timeline question answering: answer atomic, multi-hop, temporal, geographic, and aggregate questions over structured personal episodes or lifelogs. TimelineQA distinguishes single-episode span extraction from table-like aggregation and multi-episode reasoning (Tan et al., 2023).
5. Multimedia timeline modeling: partition topical news videos into chronologically ordered event nodes without receiving the textual timeline at inference time. YouTube-News-Timeline evaluates node matching, video assignment, positional displacement, and pairwise ordering (Liu et al., 2023).
6. Temporal forecasting and interpretation: evaluate historical structure interpretation, context-free forecasting, contextual temporal reasoning, and event-conditioned prediction over numerical series (Weng et al., 5 Feb 2026). Forecasting-specific evaluation also requires contamination control, rolling-origin testing, standardized tuning, and multidimensional statistical metadata (Goktas et al., 13 Apr 2026).

These task families should not be collapsed into one metric. A model can perform well at numerical extrapolation while failing event-conditioned reasoning, or generate fluent summaries while assigning claims to incorrect dates. Temporal competence is therefore multidimensional.

2. Input modalities and output representations

Timeline-Bench should distinguish among textual, tabular, visual, video, process-log, and numerical time-series inputs.

Textual and document collections

A conventional TLS input is a collection of timestamped documents,

$\mathcal{D}=\{d_1,d_2,\ldots,d_{|D|}\},$

with output entries

$p_i=(t_i,s_i),$

where $t_i$ is a date or temporal unit and $s_i$ is a summary (Gutehrlé et al., 2023). Open-domain TLS adds query-driven retrieval, heterogeneous sources, irrelevant documents, and timestamp grounding (Hu et al., 22 Jun 2025).

TimeSET instead provides a single document with salient event mentions and a graph of temporal relations. Its relations include [AFTER](https://www.emergentmind.com/topics/after), BEFORE, and COEX, where COEX indicates that two events occur around the same time but their internal order is unspecified (Hasegawa et al., 2024). This representation is a partial order rather than a forced total sequence.

Structured lifelogs

TimelineQA represents episodes as time-and-space-associated records with start and end times, locations, entities, attributes, and possible nesting within super-episodes. Atomic questions can be answered from one episode; multi-hop and aggregate questions require filtering, joining, sorting, counting, summing, averaging, or selecting extrema over multiple episodes (Tan et al., 2023).

Images and videos

Infographic understanding treats a bitmap timeline as a structured visual object. The model predicts global information—representation, scale, layout, and orientation—and local information—category, bounding box, and pixel mask—before reconstructing an editable template (Zhu-Tian et al., 2019). Video timeline modeling instead maps an unordered set of topical videos to an ordered partition of event nodes:

$(C_1,C_2,\ldots,C_K),$

where each video belongs to exactly one node (Liu et al., 2023).

Process and experimental timelines

Timeline-based process discovery maps activities to case-relative temporal positions while retaining directly-follows edges. For an activity $a$, its mean relative occurrence time is:

$$g_L(a)=\operatorname{avg}(\text{relative occurrence times of }a).$$

The resulting layout exposes approximate phases, delays, and temporal grouping, but does not provide exact activity durations or formal concurrency semantics (Kaur et al., 2023).

Wigner Time represents experimental procedures as data tables of timed updates rather than globally coupled real-time programs. Rows contain variables, resolved times, physical values, and context labels; named anchors and relative origins permit compositional stage construction (Clark et al., 5 Sep 2026). This suggests a benchmark dimension involving temporal-program modification and semantic preservation, although Wigner Time itself is not a benchmark.

3. Task decomposition and benchmark protocols

A rigorous Timeline-Bench should expose intermediate subtasks rather than evaluate only final outputs.

Event and temporal-structure extraction

Required subtasks may include event-span detection, saliency selection, temporal-expression extraction, date normalization, event-time assignment, event coreference, participant linking, and temporal-relation prediction. TimeSET explicitly separates salient-event selection from document-level partial ordering, avoiding exhaustive annotation of every temporal pair (Hasegawa et al., 2024).

Multilingual extraction requires additional components: target-entity linking through DBpedia, predicate and role mapping through PredicateMatrix, normalized TIMEX3 values, and cross-lingual event identity. Its central distinction is between IDENTITY and SIMULTANEOUS; two events occurring at the same time must not automatically be merged (Laparra et al., 2017).

Selection, clustering, and summarization

TLS systems select important dates or events, cluster cross-document mentions, reduce redundancy, and produce extractive or abstractive descriptions. Archive TimeLine Summarization proposes importance features based on frequency, contemporary references, retrospective references, causality, and domain importance (Gutehrlé et al., 2023).

UTS models event dependencies through a graph-based event encoder and uses event-level attention, time-position memory, sequential extraction, and an abstractive–extractive consistency objective (Chen et al., 2023). Its position embeddings represent event order rather than explicit calendar time, so direct use with irregular or partially ordered benchmark data would require adaptation.

Dynamic-granularity systems must additionally satisfy an instruction-conditioned abstraction requirement. DTELS defines granularity by the amount of event information omitted between neighboring nodes, rather than by node count alone (Zhang et al., 2024). Its evaluation aligns generated and reference nodes using semantic entailment and timestamp proximity, with global matching solved by the Hungarian algorithm.

Question answering and aggregation

TimelineQA defines atomic answers as information recoverable from one episode and multi-hop answers as operations over a set of episodes. Aggregate questions require operations such as:

$$\operatorname{Ans}(q)=\operatorname{Agg}\{f(e):e\in E_q\},$$

where $E_q$ is the filtered evidence set and $\operatorname{Agg}$ may be count, sum, average, maximum, minimum, or argmax (Tan et al., 2023).

The benchmark demonstrates that retrieval completeness is essential. A system can retrieve relevant episodes yet still produce an incorrect count or average if even a small number of required records are missing.

Forecasting and event conditioning

TemporalBench separates:

• T1: historical structure interpretation;
• T2: context-free forecasting;
• T3: contextual temporal reasoning;
• T4: event-conditioned prediction.

This decomposition distinguishes numerical extrapolation from interpreting regimes, lags, covariates, and interventions (Weng et al., 5 Feb 2026). TempusBench complements this approach with rolling-origin forecasting, contamination-aware datasets, standardized hyperparameter tuning, and visualization of comparative performance (Goktas et al., 13 Apr 2026).

4. Evaluation metrics

No single score is sufficient. Timeline-Bench should report task-specific metrics and preserve the distinction between content, temporal structure, grounding, and efficiency.

Temporal ordering and identity

For partial-order data, relation-level precision, recall, and $p_i=(t_i,s_i),$0 should be computed over transitive closures where appropriate. TimeSET compares temporal graphs using closure-based overlap and supports COEX relations (Hasegawa et al., 2024).

For cross-lingual extraction, strict scoring evaluates both temporal relations and event identity, while relaxed scoring evaluates event types after redundant coreferent mentions are removed (Laparra et al., 2017). This distinction prevents simultaneous but independent events from being incorrectly treated as duplicate mentions.

Timeline-node alignment and granularity

DTELS uses an information score combining entailment and timestamp proximity:

$p_i=(t_i,s_i),$1

where

$p_i=(t_i,s_i),$2

Generated and reference nodes are aligned before quality measurement. Granular consistency is then evaluated over adjacent timeline edges, allowing the benchmark to distinguish accurate individual summaries from an incorrectly abstracted sequence (Zhang et al., 2024).

Factuality and provenance

Source-grounded factuality should be evaluated independently from reference similarity. DTELS decomposes generated and source summaries into event atoms and evaluates entailment precision against supporting articles (Zhang et al., 2024). Archive-oriented evaluation likewise emphasizes links from generated entries to primary documents and contextual sources (Gutehrlé et al., 2023).

Textual overlap and human judgment

ROUGE-1, ROUGE-2, and ROUGE-L remain common in TLS, but they do not adequately measure date assignment, chronology, factuality, redundancy, or granularity. Alignment-based ROUGE and Date F1 address some of these limitations (Chen et al., 2023).

Human evaluation should assess informativeness, readability, fidelity, coherence, contextual adequacy, and usefulness. DTELS reports correlations between automatic metrics and human judgments for informativeness, granular consistency, factuality, and coherence, but its evaluation relies substantially on GPT-based decomposition and judging (Zhang et al., 2024).

Video timeline metrics

YouTube-News-Timeline evaluates node-level precision and recall through maximum bipartite matching at an IoU threshold of $p_i=(t_i,s_i),$3. It also reports normalized video-level Hamming distance, absolute positional displacement, and pairwise agreement over relations of the form:

$p_i=(t_i,s_i),$4

These metrics measure grouping and order but not semantic quality of generated event descriptions (Liu et al., 2023).

Forecasting metrics

For numerical forecasting, MAE and sMAPE are appropriate for point predictions, while CRPS, quantile score, and WIS evaluate probabilistic forecasts. TempusBench additionally reports average win rate and skill score relative to a seasonal-naive baseline (Goktas et al., 13 Apr 2026). TemporalBench reports MAE, sMAPE, and domain-specific weighted multivariate metrics, together with multiple-choice accuracy and response success rate (Weng et al., 5 Feb 2026).

Forecasting evaluation should include rolling-origin splits, horizon-wise errors, calibration, failure rates, and statistical uncertainty. Random train/test shuffling is inappropriate for temporally ordered prediction.

5. Datasets, reproducibility, and contamination

Existing resources illustrate complementary strengths and limitations.

TimeSET contains 50 English Wikinews articles, 356 salient events, 314 temporal relations, and 654 event arguments. It is small, single-document, English-only, and news-focused, but its partial-order annotation is designed for contextual temporal reasoning (Hasegawa et al., 2024).

TimelineQA contains 3,000 synthetic lifelogs, 128,023,476 entries, 600,000 atomic questions, and multi-hop splits involving approximately 10 million, 5 million, and 5 million evidence records for training, validation, and testing. Its controlled generation enables exact answers but limits linguistic and social realism (Tan et al., 2023).

DTELS-Bench contains 543 Chinese news topics, 55,432 articles, 2,858 sources, and three principal granularity levels: original, ten-node, and five-node timelines (Zhang et al., 2024). TLS-I contains 1,189 Chinese news topics, 3,567 timeline instances, more than 100,000 reported source articles, and 12 domains, although one table reports 1,252,017 articles, creating an internal count discrepancy (Hu et al., 22 Jun 2025).

YouTube-News-Timeline contains approximately 12,411 timelines, 103,382 nodes, and 302,984 video instances, divided into training, validation, and test splits. Its labels are generated through web-timeline extraction, NewsEmbed retrieval, and temporal filtering rather than exhaustive manual video annotation (Liu et al., 2023).

A benchmark intended for current foundation models should explicitly document pretraining overlap, source versions, temporal cutoffs, licenses, preprocessing, and hidden-test policy. TempusBench identifies dataset contamination as a major threat and proposes datasets outside known TSFM pretraining corpora, but its manuscript does not provide a complete corpus audit or exact overlap methodology (Goktas et al., 13 Apr 2026). Bench to the Future addresses a related problem through pastcasting: agents receive a frozen, historically bounded evidence corpus for questions whose outcomes are already known (FutureSearch et al., 11 Jun 2025).

A living benchmark may be required because static data increasingly become training data. Such a benchmark should add temporally newer questions or test series, preserve historical information states, and report whether evaluation items are eligible relative to each model’s training cutoff. Snapshot content, search indexes, ranking behavior, and document metadata should be frozen when retrieval is part of the task.

6. Limitations, controversies, and interpretation

Several methodological controversies recur across timeline benchmarks.

Total order versus partial order: Documents often do not specify a unique order. COEX and unknown relations should not be conflated, and a system should not be penalized for leaving incomparable events unordered (Hasegawa et al., 2024).

Publication time versus event time: An article’s date may differ from the date of the described event. Archive TLS explicitly treats document creation dates as fallback anchors rather than universally correct event times (Gutehrlé et al., 2023).

Event identity versus simultaneity: Multiple mentions can describe one event, while independent events can share a date. Identity-aware evaluation is therefore necessary (Laparra et al., 2017).

Fluency versus factuality: Topic-only or abstractive systems can produce coherent but unsupported narratives. DTELS reports that coherence and factuality must be measured separately (Zhang et al., 2024).

Node count versus semantic granularity: Five nodes do not necessarily constitute a coarse-grained timeline if the nodes retain excessive detail or fail to aggregate appropriate event groups (Zhang et al., 2024).

Synthetic control versus ecological validity: TimelineQA provides exact evidence and answers but uses imaginary people and template-based descriptions. TimelineQA therefore diagnoses retrieval and reasoning under controlled conditions rather than full real-world lifelog understanding (Tan et al., 2023).

Automated labels versus authoritative ground truth: YouTube-News-Timeline labels inherit web-timeline editorial choices and embedding-based retrieval errors (Liu et al., 2023). GPT-assisted annotation in DTELS and TLS-I introduces model-dependent decomposition, selection, and evaluation bias (Zhang et al., 2024, Hu et al., 22 Jun 2025).

Temporal reasoning versus forecasting: Numerical forecast accuracy does not imply contextual temporal understanding. TemporalBench reports that models can perform adequately on numerical forecasting while failing contextual comparison, lag reasoning, or event-conditioned prediction (Weng et al., 5 Feb 2026).

Benchmark ranking versus diagnostic value: A single aggregate ranking obscures domain, horizon, modality, and task-specific failures. TempusBench emphasizes multidimensional reporting, rolling evaluation, common tuning, and visualization rather than one universal winner (Goktas et al., 13 Apr 2026).

7. Recommended architecture for Timeline-Bench

A comprehensive Timeline-Bench should be organized as a modular suite with shared provenance and task-specific tracks.

1. Event track: event detection, saliency, temporal-expression normalization, event-time assignment, participant linking, and event coreference.
2. Order track: pairwise temporal relations, partial orders, interval relations, COEX, before/after consistency, and transitive closure.
3. Summary track: extractive and abstractive timestamp–summary generation, with semantic alignment, factuality, chronology, redundancy, and readability metrics.
4. Granularity track: explicit node counts, coarse/fine natural-language instructions, example-based abstraction, and edge-level granular consistency.
5. QA track: atomic span extraction, temporal filtering, multi-hop joins, list answers, counts, averages, argmax, and first/last-event reasoning.
6. Multimedia track: infographic parsing, template reconstruction, video-to-event-node assignment, and visual timeline usability.
7. Process track: case-relative temporal normalization, activity-phase discovery, waiting-time analysis, time-aware graph layout, and robustness to loops and variants.
8. Forecasting track: historical interpretation, context-free forecasting, contextual temporal reasoning, event-conditioned prediction, probabilistic calibration, and rolling-origin evaluation.

Each item should provide structured metadata, source provenance, temporal granularity, uncertainty, event identity, admissible partial orders, evidence links, and annotation disagreement. Results should be stratified by domain, language, modality, document length, event density, temporal ambiguity, retrieval condition, and evidence completeness.

The benchmark should report a vector of scores rather than a single headline number:

$p_i=(t_i,s_i),$5

Any composite score should be secondary, explicitly weighted, and accompanied by raw measurements, confidence intervals, failure rates, and computational cost. This design follows the common lesson across the supplied research: temporal competence is not reducible to chronological sorting, lexical overlap, numerical accuracy, or fluent generation alone.