SynthTS: Synthetic Benchmark for Forecasting

• SynthTS is a synthetic time-series benchmark that decomposes data into controlled, additive components to diagnose learning capacity and robustness.
• It provides a programmable configuration to simulate trends, periodicity, noise, and anomalies, enabling clear quantitative analysis of model performance.
• Empirical findings show that no deep learning model dominates across all temporal features, highlighting the need for nuanced assessment in forecasting.

Searching arXiv for the cited work and related uses of “SynthTS” to disambiguate the term. SynthTS, in the present usage, denotes SynTSBench, the official name of a synthetic, programmable benchmark for evaluating temporal pattern learning in deep learning models for time-series forecasting (Tan et al., 23 Oct 2025). SynTSBench was introduced to address a persistent gap between strong results on standard benchmark datasets and weaker robustness in real-world applications, a gap the authors attribute to the black-box nature of deep learning architectures and to evaluation frameworks that do not provide clear, quantitative insight into what a model has actually learned (Tan et al., 23 Oct 2025). The framework replaces aggregate benchmark scores with controlled synthetic experiments that isolate temporal features, inject irregularities, and compare model outputs with a theoretical optimum under known additive generative assumptions. In this sense, SynTSBench is designed not merely as another leaderboard dataset, but as an interpretable evaluation system for capability diagnosis, robustness analysis, and model selection.

1. Nomenclature and scope

The label “SynthTS” is informal in this context. The authors’ official name for the framework is SynTSBench, and the work is explicitly a synthetic time-series benchmark (Tan et al., 23 Oct 2025). The distinction matters because the term “SynthTS” is used elsewhere in the arXiv literature for unrelated systems, including TTS-synthesized speech augmentation for cross-domain speaker recognition (Huang et al., 2020), a transformer-based system for synthesizer sound matching (Bruford et al., 2024), and the synthetic data engine used to train Tiny-TSM (Birkel, 24 Nov 2025). Within the present topic, however, “SynthTS” should be treated as shorthand for SynTSBench rather than as a separate framework.

SynTSBench is scoped to time-series forecasting model evaluation. Its stated goals are to diagnose which temporal pattern types a model can learn, stress-test robustness to controlled data irregularities, provide theoretical optima for additive synthetic components, and translate synthetic insights into practical model-selection guidelines for real-world data (Tan et al., 23 Oct 2025). This suggests that SynTSBench is positioned as an analytic layer over forecasting research rather than as a replacement for standard real-world benchmarks.

2. Motivation and benchmark philosophy

The benchmark is motivated by the observation that commonly used real-world datasets such as ETTh, Weather, ETTm2, and Electricity conflate multiple temporal phenomena, including trend, periodicity, noise, anomalies, and cross-series effects (Tan et al., 23 Oct 2025). Under those conditions, a single aggregate score can hide systematic strengths and weaknesses. A model may, for example, be strong at trend modeling while remaining vulnerable to particular noise types or structural breaks. SynTSBench therefore treats confounding as the central methodological problem.

Its design principles are explicitly interpretability first, programmability, breadth of models, and bridging to practice (Tan et al., 23 Oct 2025). Interpretability is pursued through simple, controlled additive components so that attribution remains clear. Programmability allows users to configure pattern types, strengths, and irregularities. Breadth is reflected in a diverse benchmark suite that includes models such as N-BEATS, N-HiTS, and CATS in addition to prior baselines. Bridging to practice is addressed through correlation analyses between synthetic capability maps and performance on standard real-world datasets.

A central claim of the framework is that synthetic generation enables causal attribution of failure modes. If a score changes when only the noise distribution changes, while other aspects of the data-generating process remain fixed, then the change is plausibly attributable to robustness rather than to representational limits in trend or seasonality learning (Tan et al., 23 Oct 2025).

3. Programmable synthetic construction

SynTSBench is built around data generation modules, pattern libraries, and a configuration interface (Tan et al., 23 Oct 2025). The data generation modules include additive synthetic components for trends and periodic patterns, a multivariate cross-variable dependency module, and irregularity modules for noise, anomalies, and structural breaks or level shifts. The pattern libraries include trend components such as linear and exponential trends, periodic or sinusoidal components, and extended anomaly types such as structural breaks and level shifts.

The multivariate component supports both linear and non-linear relationships. The rebuttal explicitly notes non-linear forms such as

$y_t = \sin(x_{t-1}) + \epsilon_t.$

This is important because it extends the benchmark beyond purely univariate additive decomposition and allows capability mapping over cross-series dependencies (Tan et al., 23 Oct 2025).

The configuration interface allows users to choose pattern types, amplitudes, frequencies, dependency structures, noise distributions, anomaly schedules, and sequence lengths. At a high level, the pipeline proceeds by specifying the dataset configuration, generating clean components per time index, summing them into a base series, injecting irregularities, applying cross-series mappings in multivariate settings, and then splitting the resulting data into train, validation, and test sets for downstream model fitting and evaluation (Tan et al., 23 Oct 2025).

The benchmark’s additive construction is central. The rebuttal mentions a formal derivation of the theoretical optimum “for our additive synthetic components” under oracle assumptions. Although the exact decomposition equation is not reproduced in the supplied text, the implied structure is an additive model in which the observed series is the sum of component processes plus noise. This additive organization is what makes feature-level attribution and optimum benchmarking tractable.

4. Evaluation protocol and analytical dimensions

SynTSBench defines three core analytical dimensions: temporal feature decomposition and capability mapping, robustness analysis under data irregularities, and theoretical optimum benchmarking (Tan et al., 23 Oct 2025). Together, these dimensions form the framework’s evaluation logic.

Temporal feature decomposition and capability mapping evaluates a model on controlled datasets that isolate pattern types such as trends, periodicity or sinusoidal patterns, cross-variable dependencies, robustness to different noise distributions, and structural breaks or level shifts. The output is a diagnostic profile summarizing model error across these isolated settings. The significance of this profile is that it makes architecture-level strengths and weaknesses explicit rather than latent in a single benchmark average.

Robustness analysis under data irregularities focuses on the revised benchmark’s implemented perturbations: Gaussian noise, Student’s $t$ heavy-tailed noise, simple pulses, and structural breaks or level shifts (Tan et al., 23 Oct 2025). The primary quantitative measures are deterministic forecasting metrics, namely MSE and MAE. The authors discuss alternatives such as CRPS for probabilistic forecasting, but retain MSE and MAE for this work. A common misconception would be to read SynTSBench as a general irregular-sampling or missingness benchmark; the supplied text explicitly states that specific SNR formulas, missingness or irregular sampling via an observation mask $m_t$, and thresholds for anomaly recovery are not defined in the paper or rebuttal.

Theoretical optimum benchmarking supplies a performance boundary under oracle knowledge of the additive synthetic generative process. The revised manuscript adds a formal appendix deriving how this optimum is computed under MSE/MAE for the additive synthetic components (Tan et al., 23 Oct 2025). The rebuttal does not reproduce closed-form formulas such as normal equations, DFT estimators, or AR/ARMA predictors; the benchmark instead uses the oracle-derived optimum as a reference bound aligned with the chosen losses. This reference makes it possible to ask not only whether one model is better than another, but how close each model comes to the mathematically attainable limit under a known generating mechanism.

5. Empirical findings and real-world transfer

The principal empirical finding is that no single model universally dominates across all temporal feature types and robustness conditions (Tan et al., 23 Oct 2025). More strongly, the experiments show that current deep learning models do not universally approach optimal baselines across all types of temporal features. This directly supports the benchmark’s premise that aggregate benchmark success does not imply uniform competence across the latent subskills of forecasting.

The rebuttal highlights several architecture-specific findings. DLinear is reported as strong at trend modeling but vulnerable to high-frequency noise. TimesNet is described as excellent at periodic pattern extraction. The addition of N-BEATS, N-HiTS, and CATS sharpens comparative insights across the evaluation dimensions (Tan et al., 23 Oct 2025). These examples are significant because they illustrate the intended use of capability maps: not as generic rankings, but as structured profiles of architectural bias.

A further result is transferability. The authors report correlation analyses showing that the strengths and weaknesses identified by SynTSBench align with empirical behavior on ETTh, ETTm2, Weather, and Electricity (Tan et al., 23 Oct 2025). Models that excel on synthetic trend components also perform best on real datasets known to exhibit those properties. This suggests that the synthetic diagnostics are not merely artificial stress tests, but a usable intermediate representation between mechanistic understanding and practical forecasting performance.

6. Practical use and model-selection function

SynTSBench is designed to support a concrete workflow for model assessment and selection (Tan et al., 23 Oct 2025). The workflow begins by choosing an evaluation focus, such as trend, periodicity, cross-variable dependency, noise type, or anomaly and structural-break behavior. The user then configures dataset parameters, including amplitudes, frequencies, noise-distribution parameters, anomaly schedules, number of variables, non-linear dependency forms, and sequence length. Candidate models are trained under a consistent protocol, evaluated with MSE and MAE, and then compared both against one another and against the theoretical optimum.

The interpretation stage is the framework’s operational core. Capability maps are used to match model families to data properties. The supplied text gives concrete examples: one may prefer TimesNet for multi-periodicity or consider DLinear for strong trend with moderate noise (Tan et al., 23 Oct 2025). The recommended practice is to start with simple, isolated components for clean attribution and then add complexity incrementally, such as heavy-tailed noise or structural breaks, to stress-test robustness.

In positioning terms, SynTSBench complements standard benchmarks rather than displacing them. Its role is to reveal what a model learns under controlled conditions, whereas a real-world aggregate score often cannot do so because of confounding among temporal phenomena (Tan et al., 23 Oct 2025). This suggests a division of labor in evaluation: SynTSBench for diagnosis and mechanism-level comparison, conventional datasets for final empirical validation under naturally entangled conditions.

7. Limitations, misconceptions, and future directions

The authors acknowledge that initial synthetic data can be simpler than real-world complexity, and the revised benchmark partly addresses this by adding Student’s $t$ noise and structural breaks or level shifts (Tan et al., 23 Oct 2025). They also note that cross-variable evaluation was initially limited and was subsequently expanded to include more variables and non-linear dependencies. Metric scope remains narrow: the framework currently emphasizes MSE/MAE, while CRPS and extreme-event-focused measures are identified as future extensions.

Several misconceptions are explicitly ruled out by the supplied text. SynTSBench, in its current reported form, does not define detailed anomaly-recovery thresholds, does not provide missingness or irregular-sampling mechanisms through an observation mask, and does not reproduce closed-form optimum formulas in the rebuttal (Tan et al., 23 Oct 2025). It should therefore be understood as a benchmark for controlled temporal-pattern evaluation under additive synthetic generation, rather than as a complete simulation environment for every form of real-world data pathology.

The stated future directions are to enrich anomaly types, extend multivariate and non-linear dependencies, broaden evaluation metrics beyond deterministic losses, and continue validating synthetic insights against additional real-world datasets (Tan et al., 23 Oct 2025). The code is available at the authors’ repository, which supports the framework’s intended use as a reusable benchmarking instrument rather than a one-off experimental scaffold. In that form, SynTSBench occupies a specific methodological niche: it makes temporal pattern learning, robustness, and distance to oracle performance measurable in a way that standard benchmark leaderboards generally do not.