TemporalBench: A Temporal Reasoning Benchmark

• TemporalBench is a diagnostic multi-domain benchmark that evaluates LLM-based temporal reasoning and forecasting across structured time series tasks.
• It decomposes evaluation into four task tiers (T1–T4), focusing on structural understanding, context-free forecasting, contextual reasoning, and event-conditioned prediction.
• The benchmark leverages real-world datasets from retail, healthcare, energy, and physical systems, providing detailed metrics to expose challenges in temporal and event-aware modeling.

TemporalBench is a diagnostic multi-domain benchmark designed to assess reasoning and forecasting behavior of LLM-based agents in time series tasks under varying degrees of contextual and event-driven complexity. By decomposing temporal prediction and understanding into distinct, progressively information-rich categories, it provides fine-grained evaluation of not only numerical forecasting accuracy but also the agent's qualitative and contextual interpretive abilities (Weng et al., 5 Feb 2026).

1. Formal Definition and Core Objectives

TemporalBench models the evaluation of temporal reasoning in LLM-based agents via a structured, four-tier taxonomy, using notation as follows. For a time series $x_{1:T}\in\mathbb{R}^T$, possibly with associated textual context $c$ and event description $e$, an event time $t_e$ is chosen to split the series into historical segment $H = x_{1:t_e}$ and future segment $F = x_{t_e+1:t_e+h}$ of horizon $h$. Tasks are defined as:

• $T_1$ ("Historical Structure Interpretation"): Given $H$, output discrete labels $y^1$ for temporal structure—trend, volatility, seasonality, anomaly type.
• $c$0 ("Context-Free Forecasting"): Given $c$1, predict $c$2 (numerical forecast) and qualitative MCQ labels $c$3 about changes.
• $c$4 ("Contextual Temporal Reasoning"): Given $c$5, answer MCQ $c$6 probing six capabilities: alignment, slicing, difference, lag, structure, interaction.
• $c$7 ("Event-Conditioned Prediction"): Given $c$8, predict $c$9 and MCQ $e$0 about outcomes under event $e$1.

This framework allows the formulation: $e$2

The benchmark's main diagnostic goal is to test whether accurate forecasting (low MAE/sMAPE in $e$3) necessarily reflects genuine temporal "understanding" in $e$4, or if systematic weaknesses manifest under contextual/eventually altered regimes.

2. Dataset Construction and Design

TemporalBench comprises four real-world domains:

• Retail (FreshRetailNet-50K): Univariate daily censored sales, sparse promotional events; covariates include discount, holidays, weather, time-of-day.
• Healthcare (MIMIC-IV): Multivariate ICU vitals (heart rate, temperature, SpO$e$5), with physiological and time-of-day covariates.
• Energy (PSML): Dense half-hourly power loads, meteorological covariates.
• Physical Systems (Causal Chambers): System pressures, flows, and currents under mechanical interventions.

Each domain's time series instances are split at $e$6, selected either by algorithmic event detection (e.g., clinical event logs, change-point detection) or artificial event injection within allowable range. For every instance, all four task families ($e$7–$e$8) are instantiated, yielding:

• 191 series,
• 2,775 MCQ tasks,
• 382 numerical forecasting tasks.

Task counts and average context tokenization are calibrated domain-wise (see data for detailed figures).

3. Task Taxonomy and Evaluation Protocols

Input–Output Specifications:

Task	Input	Output	Focus
T₁	$e$9	$t_e$0: structure class (MCQ)	Structural understanding
T₂	$t_e$1	$t_e$2 (numeric), $t_e$3 (qualitative MCQ)	Extrapolation
T₃	$t_e$4, $t_e$5	$t_e$6 (MCQ: C₁–C₆)	Contextual reasoning
T₄	$t_e$7, $t_e$8, $t_e$9	$H = x_{1:t_e}$0, $H = x_{1:t_e}$1 (event-conditioned MCQ)	Event-driven forecast

Evaluation Metrics:

• MCQ (T₁, T₃, T₂/T₄ qualitative): Accuracy

$H = x_{1:t_e}$2

• Forecasting (T₂, T₄): MAE and sMAPE

$H = x_{1:t_e}$3

• MIMIC-IV uses overall weighted versions (OW_sMAPE, OW_RMSSE) to control scale.

4. Global Metrics and Leaderboard

TemporalBench computes both per-domain and global aggregated scores:

• Domain-averaged: $H = x_{1:t_e}$4
• Leaderboard global score: average over normalized scores for each domain and task: normalized accuracy for qualitative, min-max or rank-based normalization for forecasting errors.

All scores and baselines are hosted on a public leaderboard for standardized comparison: TemporalBench Leaderboard.

5. Baseline Agent Results and Error Profiles

Agents evaluated (primarily GPT-4o backbones) include direct single-LLM prompting, TimeSeriesScientist (domain specialist), AgentScope, MetaGPT, and CAMEL (general scaffolding):

Task	Single LLM	AgentScope/CAMEL	TimeSeriesScientist
T₁ Structural ACC	63.6%	~62.5%	33.5%
T₂ Qual. Forecast ACC	52.3%	12.1%	56.8%
T₃ Contextual ACC	2.9–13.6%	2.9–13.6%	–
T₄ Event MCQs ACC	≤35%	≤35%	–

• Forecasting (MAE/sMAPE): Single LLM and AgentScope both achieve $H = x_{1:t_e}$5 MAE and $H = x_{1:t_e}$6 sMAPE on FreshRetailNet, but AgentScope produces highly unstable sMAPE due to extreme errors.
• More than 40% of general agent errors arise from forecast-length violations.
• Numeric predictions can be valid in form yet have very high error (numerical anomalies).
• Agent scaffolding helps reduce format/horizon mistakes but does not remedy deeper contextual failures.

6. Diagnostic Insights Revealed by TemporalBench

• Achieving $H = x_{1:t_e}$7 structural understanding (T₁) is possible, but does not translate to contextual or event-aware reasoning (T₃, T₄), with T₃ rarely exceeding $H = x_{1:t_e}$8 accuracy.
• Numerical forecasting accuracy is often decoupled from interpretation; agents may achieve low MAE yet remain near-random on qualitative MCQs for the same instances.
• Event descriptions $H = x_{1:t_e}$9 provide little gain in qualitative accuracy (improvement $F = x_{t_e+1:t_e+h}$0), indicating deficient integration of exogenous event semantics.
• Input history length has a non-monotonic effect; excessive history can degrade performance.
• Augmenting inputs with classical feature sets (e.g., catch22, simple plots) sometimes helps in structured domains, but offers inconsistent benefit otherwise.
• Pseudo-equation: If $F = x_{t_e+1:t_e+h}$1 but $F = x_{t_e+1:t_e+h}$2 random, then forecasting accuracy does not imply temporal understanding.

7. Data Resources and Reproducibility

Comprehensive resources including datasets (across all domains, events, prompts, and labels), public leaderboard, evaluation scripts, prompt templates, event-injection and label-generation code (with fixed seeds for replicability), are hosted at https://huggingface.co/datasets/Melady/TemporalBench and https://huggingface.co/spaces/Melady/TemporalBench_Leaderboard. Default preprocessing parameters (e.g., event injection probability $F = x_{t_e+1:t_e+h}$3, robust statistics such as median/MAD/IQR) and code versions are provided to ensure transparent, end-to-end reproduction of results across backbones.

---

TemporalBench thus systematizes the evaluation of temporal reasoning in LLM-based agents, showing that robust forecasting does not equate to genuine temporal or event-conditioned understanding, and highlighting major open challenges concealed by prior forecasting-only assessment paradigms (Weng et al., 5 Feb 2026).