TempusBench: Temporal Benchmark Framework

• TempusBench is a polysemous benchmarking label used to evaluate time-series forecasting models, cloud TSMS, and parametric timed automata.
• It introduces standardized evaluation protocols that address issues like dataset contamination, limited metadata, and unfair hyperparameter tuning.
• The framework leverages both synthetic and real-world datasets, providing detailed task taxonomies and visualization interfaces for enhanced interpretability.

Searching arXiv for "TempusBench" and closely related benchmark papers to ground the article in the cited literature. TempusBench is a name used in the arXiv literature for benchmarking infrastructures concerned with temporal data, but its most explicit and recent use denotes an open-source evaluation framework for time-series foundation models (TSFMs) in forecasting (Goktas et al., 13 Apr 2026). In that usage, TempusBench addresses four shortcomings attributed to prior forecasting evaluation frameworks: reliance on often outdated datasets with weak metadata and possible overlap with TSFM pretraining corpora, task taxonomies that emphasize surface dimensions rather than statistical properties, unfair comparison protocols for models requiring hyperparameter tuning, and limited support for interpretive visualization (Goktas et al., 13 Apr 2026). In a separate database-systems context, the same name is used for an interactive cloud benchmark for time series management systems (TSMS) that compares four systems on advanced analytical operators across real-world datasets (Arora, 2021). Another description uses the label for an expanded benchmark library for parametric timed automata, extending the IMITATOR benchmarks library with new models, properties, metadata, and unsolvable toy benchmarks (André et al., 2021). This plurality of usage makes “TempusBench” a polysemous benchmark label rather than a single universally fixed artifact.

1. Terminological scope and research contexts

The forecasting-oriented TempusBench is presented as an evaluation framework for TSFMs, motivated by the rapid growth of foundation-style forecasting models such as Moirai, TimesFM, Chronos, Lag-Llama, MOMENT, and Toto (Goktas et al., 13 Apr 2026). Its stated objective is to provide a fair, comprehensive, and interpretable basis for comparison across decontaminated datasets, statistically structured benchmark tasks, standardized hyperparameter tuning, and a tensorboard-based visualization interface (Goktas et al., 13 Apr 2026).

A distinct usage appears in a cloud database benchmarking setting. There, TempusBench is described as an interactive benchmark or demo system for comparing time series management systems in the cloud, with emphasis on advanced analytical operators such as normalization, clustering, similarity search, anomaly handling, decomposition, and recovery (Arora, 2021). Its motivating question is practical rather than purely methodological: given a workload and a dataset, which TSMS is likely to perform best (Arora, 2021).

A third usage appears in the literature on parametric timed automata. In that context, TempusBench denotes, in essence, a new and expanded benchmark library evolving from the IMITATOR collection, intended to support the evaluation of parameter-synthesis algorithms over PTAs and their extensions (André et al., 2021). This library adds liveness properties, multi-rate clocks, stopwatches, semantic metadata, JANI translation, and unsolvable toy benchmarks (André et al., 2021).

This distribution of meanings suggests that the name functions less as a canonical benchmark brand than as a recurring label for temporal benchmarking infrastructures in distinct subfields.

2. TempusBench for time-series forecasting

In forecasting, TempusBench is defined as a four-part framework comprising new benchmark datasets not included in existing TSFM pretraining corpora, novel benchmark tasks organized around statistical properties, a standardized model evaluation pipeline with automated hyperparameter search, and a tensorboard-based visualization interface (Goktas et al., 13 Apr 2026). The framework formalizes a forecasting task as

$\forecastprob \doteq (\contextlen, \forecastlen, \numcovars, \numtargets, \covarset, \targetset, \covarts, \targetts),$

where $\contextlen$ is context length, $\forecastlen$ is forecast horizon, $\numtargets$ is the number of target series, and $\numcovars$ is the number of covariate series (Goktas et al., 13 Apr 2026). Within this setup, a point forecaster maps past targets and covariates to future target values, whereas a probabilistic forecaster maps them to a distribution over future values (Goktas et al., 13 Apr 2026).

The framework is explicitly motivated by concerns over contamination in zero-shot evaluation. The paper notes that benchmark datasets often overlap with TSFM pretraining corpora and specifically states that in GIFT-Eval, all assessed TSFMs except Moirai2 include test data in their training corpora (Goktas et al., 13 Apr 2026). TempusBench responds by emphasizing datasets not found in existing TSFM training corpora and by selecting sources with clearer metadata, semantic meaning, and more modern provenance than traditional competition datasets (Goktas et al., 13 Apr 2026).

Its task taxonomy extends beyond horizon length, frequency, and domain. TempusBench introduces tasks centered on movement, data quality, frequency, context length, forecast horizon, seasonality, domain, dataset coverage, and target type (Goktas et al., 13 Apr 2026). The taxonomy includes stationary versus non-stationary series, noisy data versus measurement error, sparse versus dense coverage, and continuous, count, binary, and categorical targets (Goktas et al., 13 Apr 2026). Seasonality categories include cyclical, non-stationary cyclical, regressive, irregular, additive, and multiplicative (Goktas et al., 13 Apr 2026).

Synthetic benchmarks play a central role because they isolate properties that are difficult to disentangle in observational data. The paper describes synthetic cyclic seasonality, cyclic seasonality with non-stationarity, cyclic seasonality with additive trends, and cyclic seasonality with multiplicative and additive trends (Goktas et al., 13 Apr 2026). Example constructions include

$$y(t) = \sin\!\left(\frac{2\pi}{P}t\right) + \epsilon_t$$

with exponential noise, and

$$y_t = \alpha_t \sin(\sigma_t t) + \beta_t \cos\!\left(\frac{t}{\tau_t}\right),$$

where the parameters are resampled at each step to produce non-stationarity [2604.115