SynTSBench: Synthetic Evaluation for Time Series Forecasting

• SynTSBench is a synthetic evaluation framework that decomposes time series into trend, seasonality, and noise to attribute model performance to specific temporal features.
• It systematically varies signal components and introduces controlled anomalies to stress-test deep forecasting models, revealing patterns in robustness and failure.
• The benchmark establishes near-oracle performance baselines, guiding model selection by mapping architectural strengths to distinct temporal patterns.

Searching arXiv for the SynTSBench paper and closely related forecasting benchmarks/models to ground the article in current literature. SynTSBench is a synthetic data-driven evaluation paradigm for time series forecasting that was introduced to assess fundamental modeling capabilities of deep learning models through programmable feature configuration. It was proposed in response to a persistent discrepancy between strong results on standard benchmark datasets and weaker robustness in real-world applications, a discrepancy attributed to both the black-box nature of deep architectures and the limited diagnostic resolution of existing evaluation frameworks. Rather than treating forecasting quality as a single aggregate score, SynTSBench isolates confounding factors and organizes evaluation around temporal feature decomposition, robustness under data irregularities, and theoretical optimum benchmarking (Tan et al., 23 Oct 2025).

1. Motivation and benchmark rationale

The benchmark is motivated by three limitations in prior evaluation practice. First, most real-world benchmarks, including Electricity and ETTh, mix multiple temporal effects simultaneously, making it difficult to isolate why one model outperforms another. Second, prior synthetic-data evaluations often stop at an additive trend + seasonality + i.i.d. noise model with only Gaussian perturbations and simple outlier pulses. Third, almost no studies compute the “best possible” error achievable if the true generative components were known, that is, an oracle predictor (Tan et al., 23 Oct 2025).

These limitations matter because aggregate performance on heterogeneous datasets can obscure whether a model is learning trend, periodicity, or merely exploiting correlations specific to a benchmark. SynTSBench was designed to fill this gap by decomposing time series into interpretable components, systematically stressing models along those dimensions, and comparing them against provable performance bounds.

Core analytical dimension	Function
Temporal feature decomposition and capability mapping	Evaluates capacity to learn specific pattern types
Robustness analysis under data irregularities	Quantifies noise tolerance thresholds and anomaly recovery capabilities
Theoretical optimum benchmarking	Establishes performance boundaries for each pattern type

A central implication is that model selection becomes pattern-conditional rather than benchmark-conditional. This suggests that rankings on standard datasets should not be interpreted as universal indicators of temporal pattern learning capacity.

2. Synthetic generative structure

SynTSBench generates each univariate series $x(t)$ on a grid $t=1,\dots,T$ by an additive latent decomposition (Tan et al., 23 Oct 2025):

$x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$

Within this construction, trend $T(t)$ captures low-frequency drift. The reported typical forms are

$$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$

with slopes $a\in[-1,1]$ or exponential rate $c\in[0,0.01]$.

Seasonality $S(t)$ encodes periodic structure through

$$S(t)\;=\;\sum_{k=1}^K A_k\sin\Bigl(\tfrac{2\pi\,t}{P_k}+\phi_k\Bigr), \quad P_k\in\{12,24,168\},\;A_k\in[0,5].$$

Noise $N(t)$ is a baseline stochastic process, in the simplest case

$t=1,\dots,T$0

By varying $t=1,\dots,T$1 over a grid, SynTSBench creates a lattice of time series each dominated by one pattern or mixture. For each synthetic task, models are trained and their forecasting error is recorded. The result is a “capability map” intended to show which architectures handle linear trends, high-frequency seasonality, or low signal-to-noise regimes more effectively.

The significance of this design lies in its controlled factorization of temporal structure. Instead of inferring model behavior from complex observational datasets, SynTSBench makes the latent composition explicit and programmable. This allows direct attribution of success or failure to specific temporal mechanisms.

3. Capability mapping across temporal patterns

The first analytical dimension, temporal feature decomposition and capability mapping, is concerned with systematic evaluation of model capacities to learn specific pattern types. In practice, SynTSBench reports differentiated behavior across architectures rather than a single dominant model class (Tan et al., 23 Oct 2025).

The benchmark description states that DLinear, characterized there as a decomposed linear model, attains near-oracle performance on pure trends but degrades heavily under multi-periodic $t=1,\dots,T$2. TimesNet, by contrast, excels on moderate-frequency seasonality with $t=1,\dots,T$3 and $t=1,\dots,T$4, but is somewhat fragile when $t=1,\dots,T$5.

These observations formalize a pattern-specific view of inductive bias. A decomposed linear model can perform extremely well when the signal is dominated by low-frequency drift, while architectures effective for periodic structure may lose stability as noise increases. SynTSBench therefore treats forecasting competence as a structured profile rather than a scalar property.

A common misconception in benchmark-driven model comparison is that a model with strong average performance has learned the full spectrum of temporal behaviors present in time series data. SynTSBench counters this by showing that models can be near-oracle on one pattern family and substantially suboptimal on another. The capability map is designed precisely to make such asymmetries visible.

4. Robustness under irregularities

The second analytical dimension extends evaluation beyond baseline Gaussian noise by injecting three classes of irregularities (Tan et al., 23 Oct 2025). The first is additive non-Gaussian noise,

$t=1,\dots,T$6

used to test tail-robustness. The second is structural breaks, implemented as level shifts:

$t=1,\dots,T$7

The third is point anomalies:

$t=1,\dots,T$8

For each corruption level, such as noise magnitude $t=1,\dots,T$9 or break size $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$0, SynTSBench computes a “noise tolerance threshold” $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$1 or $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$2 at which a model’s MSE first exceeds twice its clean-data error. This provides a principled operational definition of robustness rather than relying on ad hoc stress tests.

The reported empirical pattern is that state-of-the-art deep models maintain accuracy up to $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$3 on Gaussian noise, but break down quickly once heavy tails with $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$4 or large shifts with $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$5 appear. Structural break experiments further indicate that none of the deep models recovers the pre-break trend without explicit change-point modules, and their relative gap $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$6 jumps by 200% when $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$7.

This robustness axis broadens the benchmark from pattern learning to failure analysis. A plausible implication is that model comparison on clean synthetic signals is insufficient for applications where heavy-tailed anomalies, abrupt regime changes, or isolated outliers are not exceptional events but ordinary data conditions.

5. Oracle baselines and theoretical optimum benchmarking

The third analytical dimension establishes theoretical performance boundaries for each synthetic pattern (Tan et al., 23 Oct 2025). Assuming oracle knowledge of the true generative components $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$8, $x(t)\;=\;T(t)\;+\;S(t)\;+\;N(t)\,.$9, and $T(t)$0, one can construct the forecast

$T(t)$1

ignoring the zero-mean noise.

Under this assumption, the one-step-ahead MSE lower bound is

$T(t)$2

More generally, for an $T(t)$3-step horizon,

$T(t)$4

The benchmark computes these bounds in closed form for every synthetic pattern and scores models by a “relative gap” $T(t)$5. A value $T(t)$6 means that a model attains the oracle bound, while larger $T(t)$7 indicates room for architectural improvement.

This framework is methodologically important because it changes the semantics of error measurement. Instead of reporting only absolute performance, SynTSBench evaluates how far each model remains from the best mathematically attainable forecast under the stated generative assumptions. That makes it possible to distinguish difficult tasks from ineffective architectures: a high raw MSE can be near-optimal if the irreducible noise is large, whereas a modest raw MSE can still be structurally poor if the oracle bound is much lower.

6. Empirical results, model selection, and extensions

SynTSBench reports several specific findings across trend, seasonality, noise, and break regimes (Tan et al., 23 Oct 2025). On pure trend tasks, DLinear closes over 95% of the gap, with $T(t)$8, whereas generic RNN and transformer-based models hover at $T(t)$9. For single-period seasonality with $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$0, TimesNet and N-HiTS achieve $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$1. When two periods coexist, $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$2, $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$3 grows to $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$4 for all models except CATS, which remains at $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$5. Under moderate Gaussian noise with $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$6, most models see $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$7 double relative to the clean case, and under heavy Student’s-$$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$8 tails with $$T(t) \;=\; a\,t \quad\text{or}\quad T(t)=b\,e^{ct}, \quad a,b,c\in\mathbb{R},$$9, the worst models exceed $a\in[-1,1]$0.

The benchmark also reports external validation on real datasets. On ETTh1, Weather, and Electricity, there is a strong correlation, $a\in[-1,1]$1, between each model’s synthetic pattern gaps and its real data ranking. This suggests that the synthetic diagnostics capture structure relevant to non-synthetic forecasting settings rather than merely reflecting artifacts of the generator.

SynTSBench translates these findings into model-pattern matching guidance. If the data is trend-dominated with low noise, a decomposed linear model such as DLinear or N-HiTS is likely optimal. If strong multi-periodic seasonality is present, TimesNet or CATS should be considered. When heavy-tailed anomalies or structural breaks matter, the benchmark advises incorporating explicit robust modules or change-point detectors.

The reported future directions are equally diagnostic. Large gaps with $a\in[-1,1]$2 on heavy-noise and structural-break tasks identify robust loss functions, including Laplace and Huber, and adaptive segmentation as research frontiers. The benchmark is also proposed as a protocol for testing probabilistic forecasting metrics such as CRPS under the synthetic generator, as well as multivariate non-linear coupling beyond bivariate sinusoids. The codebase is available at the project repository linked in the paper (Tan et al., 23 Oct 2025).

In sum, SynTSBench reframes time series forecasting evaluation from leaderboard comparison to mechanism-specific diagnosis. Its contribution is not merely a new synthetic dataset family, but an interpretable evaluation system in which temporal decomposition, corruption stress testing, and oracle-referenced scoring are combined to characterize where deep forecasting models excel, where they fail, and how those failures map onto architectural design choices.