--- title: 'TemporalBench: A Temporal Reasoning Benchmark' url: https://www.emergentmind.com/topics/temporalbench type: topic --- # TemporalBench: A Temporal Reasoning Benchmark TemporalBench is a diagnostic multi-domain benchmark designed to assess reasoning and forecasting behavior of large language model (LLM)-based agents in time series tasks under varying degrees of contextual and event-driven complexity. By decomposing temporal prediction and understanding into distinct, progressively information-rich categories, it provides fine-grained evaluation of not only numerical forecasting accuracy but also the agent's qualitative and contextual interpretive abilities [2602.13272]. ## 1. Formal Definition and Core Objectives TemporalBench models the evaluation of temporal reasoning in LLM-based agents via a structured, four-tier taxonomy, using notation as follows. For a time series $x_{1:T}\in\mathbb{R}^T$, possibly with associated textual context $c$ and event description $e$, an event time $t_e$ is chosen to split the series into historical segment $H = x_{1:t_e}$ and future segment $F = x_{t_e+1:t_e+h}$ of horizon $h$. Tasks are defined as: - $T_1$ ("Historical Structure Interpretation"): Given $H$, output discrete labels $y^1$ for temporal structure—trend, volatility, seasonality, anomaly type. - $T_2$ ("Context-Free Forecasting"): Given $H$, predict $\hat F$ (numerical forecast) and qualitative MCQ labels $y^2$ about changes. - $T_3$ ("Contextual Temporal Reasoning"): Given $H, c$, answer MCQ $y^3$ probing six capabilities: alignment, slicing, difference, lag, structure, interaction. - $T_4$ ("Event-Conditioned Prediction"): Given $H, c, e$, predict $\hat F$ and MCQ $y^4$ about outcomes under event $e$. This framework allows the formulation: \[ \begin{align*} &f_1: H \to y^1 \\ &f_2: H \to (\hat F_2, y^2) \\ &f_3: (H, c) \to y^3 \\ &f_4: (H, c, e) \to (\hat F_4, y^4) \end{align*} \] The benchmark's main diagnostic goal is to test whether accurate forecasting (low MAE/sMAPE in $T_2/T_4$) necessarily reflects genuine temporal "understanding" in $T_1/T_3$, or if systematic weaknesses manifest under contextual/eventually altered regimes. ## 2. Dataset Construction and Design TemporalBench comprises four real-world domains: - **Retail (FreshRetailNet-50K):** Univariate daily censored sales, sparse promotional events; covariates include discount, holidays, weather, time-of-day. - **Healthcare (MIMIC-IV):** Multivariate ICU vitals (heart rate, temperature, SpO$_2$), with physiological and time-of-day covariates. - **Energy (PSML):** Dense half-hourly power loads, meteorological covariates. - **Physical Systems (Causal Chambers):** System pressures, flows, and currents under mechanical interventions. Each domain's time series instances are split at $t_e$, selected either by algorithmic event detection (e.g., clinical event logs, change-point detection) or artificial event injection within allowable range. For every instance, all four task families ($T_1$–$T_4$) are instantiated, yielding: - 191 series, - 2,775 MCQ tasks, - 382 numerical forecasting tasks. Task counts and average context tokenization are calibrated domain-wise (see data for detailed figures). ## 3. Task Taxonomy and Evaluation Protocols **Input–Output Specifications:** | Task | Input | Output | Focus | |------|--------------------|---------------------------------------------|-------------------------| | T₁ | $H$ | $y^1$: structure class (MCQ) | Structural understanding| | T₂ | $H$ | $\hat F$ (numeric), $y^2$ (qualitative MCQ) | Extrapolation | | T₃ | $H$, $c$ | $y^3$ (MCQ: C₁–C₆) | Contextual reasoning | | T₄ | $H$, $c$, $e$ | $\hat F$, $y^4$ (event-conditioned MCQ) | Event-driven forecast | **Evaluation Metrics:** - MCQ (T₁, T₃, T₂/T₄ qualitative): Accuracy \[ \mathrm{ACC} = \frac{1}{N}\sum_{i=1}^N 1[\hat y_i = y_i] \] - Forecasting (T₂, T₄): MAE and sMAPE \[ \mathrm{MAE} = \frac{1}{h}\sum_{j=1}^h |F_j - \hat F_j|,\quad \mathrm{sMAPE} = \frac{100\%}{h}\sum_{j=1}^h \frac{2|F_j - \hat F_j|}{|F_j| + |\hat F_j|} \] - MIMIC-IV uses overall weighted versions (OW_sMAPE, OW_RMSSE) to control scale. ## 4. Global Metrics and Leaderboard TemporalBench computes both per-domain and global aggregated scores: - **Domain-averaged:** $\overline{\mathrm{ACC}}_{domain} = \frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\mathrm{ACC}_t$ - **Leaderboard global score:** average over normalized scores for each domain and task: normalized accuracy for qualitative, min-max or rank-based normalization for forecasting errors. All scores and baselines are hosted on a public leaderboard for standardized comparison: [TemporalBench Leaderboard](https://huggingface.co/spaces/Melady/TemporalBench_Leaderboard). ## 5. Baseline Agent Results and Error Profiles Agents evaluated (primarily GPT-4o backbones) include direct single-LLM prompting, TimeSeriesScientist (domain specialist), AgentScope, MetaGPT, and CAMEL (general scaffolding): | Task | Single LLM | AgentScope/CAMEL | TimeSeriesScientist | |-----------------------|------------|------------------|--------------------| | T₁ Structural ACC | 63.6% | ~62.5% | 33.5% | | T₂ Qual. Forecast ACC | 52.3% | 12.1% | 56.8% | | T₃ Contextual ACC | 2.9–13.6% | 2.9–13.6% | – | | T₄ Event MCQs ACC | ≤35% | ≤35% | – | - Forecasting (MAE/sMAPE): Single LLM and AgentScope both achieve $\sim0.12$ MAE and $1.27$ sMAPE on FreshRetailNet, but AgentScope produces highly unstable sMAPE due to extreme errors. - More than 40% of general agent errors arise from forecast-length violations. - Numeric predictions can be valid in form yet have very high error (numerical anomalies). - Agent scaffolding helps reduce format/horizon mistakes but does not remedy deeper contextual failures. ## 6. Diagnostic Insights Revealed by TemporalBench - Achieving $\sim60\%$ structural understanding (T₁) is possible, but does not translate to contextual or event-aware reasoning (T₃, T₄), with T₃ rarely exceeding $10\%$ accuracy. - Numerical forecasting accuracy is often decoupled from interpretation; agents may achieve low MAE yet remain near-random on qualitative MCQs for the same instances. - Event descriptions $e$ provide little gain in qualitative accuracy (improvement $<5\%$), indicating deficient integration of exogenous event semantics. - Input history length has a non-monotonic effect; excessive history can degrade performance. - Augmenting inputs with classical feature sets (e.g., catch22, simple plots) sometimes helps in structured domains, but offers inconsistent benefit otherwise. - Pseudo-equation: If $\mathrm{MAE}(f_2) \leq \epsilon$ but $\mathrm{ACC}(f_3) \ll$ random, then forecasting accuracy does not imply temporal understanding. ## 7. Data Resources and Reproducibility Comprehensive resources including datasets (across all domains, events, prompts, and labels), public leaderboard, evaluation scripts, prompt templates, event-injection and label-generation code (with fixed seeds for replicability), are hosted at [https://huggingface.co/datasets/Melady/TemporalBench](https://huggingface.co/datasets/Melady/TemporalBench) and [https://huggingface.co/spaces/Melady/TemporalBench_Leaderboard](https://huggingface.co/spaces/Melady/TemporalBench_Leaderboard). Default preprocessing parameters (e.g., event injection probability $p_n \approx 0.4$, robust statistics such as median/MAD/IQR) and code versions are provided to ensure transparent, end-to-end reproduction of results across backbones. --- TemporalBench thus systematizes the evaluation of temporal reasoning in LLM-based agents, showing that robust forecasting does not equate to genuine temporal or event-conditioned understanding, and highlighting major open challenges concealed by prior forecasting-only assessment paradigms [2602.13272]. Source: https://www.emergentmind.com/topics/temporalbench