TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults

Abstract: Time series forecasting (TSF) underpins consequential decisions in energy, transportation, finance, and healthcare, yet TSF models are almost universally ranked by a single number (e.g., average error) on clean held-out data, under the implicit assumption that it predicts deployed reliability. However, real faults are not i.i.d noise but structured events with temporal shape, broken cross-variable dependencies, regime change coupled with missingness, and causal propagation across a sensing pipeline. Treating TSF robustness as a data-quality problem, we present TS-Fault, a benchmark that evaluates forecasting models under explicit, parameterized fault scenarios with controllable semantic difficulty. TS-Fault organizes recurring failures into four modes along two orthogonal axes (observation- vs mechanism-level; univariate vs multivariate) and injects each fault into the most prediction-critical window via a unified importance score. This design enables robustness to be tested against the structures models actually rely on, rather than reduced to generic noise sensitivity. We evaluate 21 models across 6 datasets, 4 modes, and 5 difficulty levels under a paired clean/corrupt protocol. The results reveal three findings that contradict common leaderboard intuition: (i) clean-data accuracy anti-correlates with robustness; (ii) clean rankings are preserved under observation-level faults but reshuffled under mechanism-level faults; and (iii) all catastrophic failures occur under mechanism-level faults, with foundation models achieving the highest clean-data accuracy yet exhibiting the greatest fragility. The code is publicly available at https://github.com/Ray-zyy/TS-Fault.

• The paper introduces TS-Fault, a benchmark that evaluates time series models under semantically parameterized structural faults.
• It employs a fault-operator framework with four fault modes and controlled difficulty levels to simulate realistic deployment scenarios.
• Empirical results show models with high clean accuracy are highly vulnerable to mechanism-level faults, stressing the need for robust evaluation.

TS-Fault: A Benchmark for Structured Fault Robustness in Time Series Forecasting

Motivation and Problem Statement

Time series forecasting (TSF) models are pervasive in critical applications such as energy management, healthcare, finance, and transportation. While evaluation has traditionally been based on average error metrics (e.g., MSE, MAE) under clean, held-out data, these criteria assume i.i.d. noise and do not probe model reliability under deployment-relevant faults. Real-world faults manifest as structured events: temporal patterns, disrupted cross-variable dependencies, regime changes accompanied by missingness, and causal error propagation. These cannot be reduced to random noise, standard data corruption, or adversarial perturbations. The paper introduces TS-Fault, a benchmarking suite that evaluates TSF methods under controlled, semantically parameterized structural fault scenarios, explicitly addressing robustness in a way that is absent from prior benchmarks.

TS-Fault Framework and Fault Taxonomy

TS-Fault adopts a fault-operator framework, where the object of evaluation is not average performance on clean data, but degradation under explicit, mechanism-based faults. Faults are injected according to interpretable parameters and controllable difficulty levels into prediction-critical input windows, enabling counterfactual analysis of when and how a model fails.

The benchmark organizes faults along two axes: scope (observation-level versus mechanism-level) and variable scope (univariate versus multivariate), yielding four structured fault modes:

1. Mode I: Time-Warped Shock (Observation × Univariate) Localized, temporally misaligned value events (e.g., sensor spikes with delayed emission).
2. Mode II: Dependency-Fracture Shock (Observation × Multivariate) Synchronized events that disrupt dynamic cross-series dependencies (e.g., a fractured co-movement among variables).
3. Mode III: Regime-Transition Missingness (Mechanism × Univariate) Regime change in process dynamics (trend, seasonality, volatility) with state-dependent, block-missing inputs, reflecting operational transitions coupled to sensor dropout.
4. Mode IV: Cascading Sensor-to-System Failure (Mechanism × Multivariate) Faults originating at upstream sensors, propagating causally through dependent variables, degrading downstream state and observability.

Crucially, each fault is injected in the most prediction-critical window—identified via a unified importance score aggregating change-point detection, period relevance, volatility, and direct occlusion influence—to avoid degenerate random-noise stress testing.

Instances are parameterized by scenario vectors $\Theta$, which encode semantically meaningful features (e.g., magnitude, duration, propagation strength). A difficulty function $\kappa(\Theta)$ provides fine-grained control from mild to catastrophic faults, enabling not only average-case but also worst-case risk measurement.

Experimental Protocol and Model Evaluation

The benchmark comprises 21 representative forecasting models, including classical statistical baselines (Naive, ARIMA, ETS), lightweight linear and basis-expansion models, recurrent and convolutional architectures (LSTM, GRU, TCN), state-of-the-art decomposition and attention-based Transformers, and prominent zero-shot time series foundation models (TimesFM, Chronos, Moirai). Evaluation encompasses 6 diverse multivariate datasets, 4 modes, and 5 difficulty levels, yielding controlled paired clean/corrupt evaluation across 120 configurations per model.

Metrics include absolute degradation ($\Delta$MSE), robustness ratio ($r$), relative degradation (RD), and the Spearman rank correlation of model ordering between clean and faulted contexts. Catastrophic failures are defined as cases where the faulted error is at least an order of magnitude larger than the clean error ($r \geq 10$).

Empirical Findings

Contradicting Default Assumptions:

The results reveal several findings that refute common beliefs in TSF evaluation:

• Clean Accuracy is Anti-Correlated with Robustness:

The global correlation between clean-data accuracy and fault robustness ranking is significantly negative ($\rho = -0.544$, $p=0.011$). Models that excel on clean data (notably SOTA and foundation models) tend to be the most brittle under structured faults.

• Ranking Preservation is Fault-Mode Dependent:

Observation-level faults (Modes I/II) preserve the clean-model ordering almost perfectly ($\rho > 0.92$), while mechanism-level faults (Modes III/IV) destroy any predictive value of clean ordering ($\rho < 0.06$). All catastrophic failures are concentrated in mechanism-level scenarios.

• Foundation Models Are Strong but Fragile:

Leading zero-shot foundation models (TimesFM, Chronos, Moirai) achieve the best clean errors but exhibit the highest degradation and catastrophic failure rates under structural faults, particularly Modes III/IV. Their clean accuracy buys no robustness, amplifying model-selection risk.

• Robustness is Architectural, Not Dataset-Specific:

The fragility patterns persist across datasets of varying dimensionality and sampling granularity, indicating that vulnerability is rooted in model class, not domain instantiation.

• Robustness is Controllable and Stratified:

The degradation grows monotonically (by design) with difficulty, separating models into tiers: recurrent models (LSTM/GRU) are most robust (ratio $\sim$1), followed by convolutional and decomposition-based models, with linear, attention-based SOTA, and foundation models being most sensitive to increasing fault difficulty.

Numerical Highlights

• Catastrophic Failure Concentration:

All 884 catastrophic failures (defined as $\kappa(\Theta)$0) occur in mechanism-level modes, with none in observation-level modes.

• Relative Degradation Ranges:

Median RD under Mode III (regime missingness) reaches up to $\kappa(\Theta)$1 for TimesFM; even well-performing classical baselines frequently show degradation in the tens of thousands of percent under these conditions.

• Model-Selection Instability:

Several models undergo immense rank shifts when moving from clean to robustness-centric evaluation (e.g., iTransformer falls from 3rd to 21st; TimesFM from 2nd to 18th).

Implications and Theoretical Insights

Model-Selection Risk and Leaderboard Illusions:

Optimizing or selecting based solely on clean-data error can systematically prefer models that are most vulnerable to deployment-relevant, mechanism-level structured faults. This implies a "reverse" model-selection effect: clean leaderboards frequently indicate the models most likely to fail disastrously when faults matter.

Benchmarking Practice:

TS-Fault demonstrates that a single-number summary is insufficient for TSF evaluation. Robustness to interpretable structural faults should be an explicit axis in reporting and model development, next to clean-data accuracy.

Time Series Foundation Model Design:

Foundation models' reliance on strong priors and absence of adaptation renders them unusually brittle to regime changes and cross-variable disruptions. Robustness in these settings requires architectural or training paradigms explicitly incorporating structured perturbations and adaptation mechanisms.

Methodological Transparency and Goodharting:

Because TS-Fault exposes its scenario parameters, iterative overfitting to the benchmark is plausible. However, compositionality of scenario parameters allows the benchmark to hold out unseen combinations, maintaining its generalization utility over time.

Future Directions

• Adaptive and Robust Training:

Architecture search and training protocols should directly incorporate structured fault scenarios, with a robustness component guiding optimization.

• Evaluation in Broader Domains:

Application to financial, clinical, and industrial control domains remains to be pursued with tailored structural fault models.

• Integration into Model Reporting:

Structured robustness metrics, stratified by fault mode and difficulty, should be standard in TSF model reporting, akin to model cards for responsible machine learning.

• Online Adaptation Mechanisms:

Explicit separation and modeling of mechanism-level shifts (including online adaptation) could close the robustness gap exposed by TS-Fault.

Conclusion

TS-Fault exposes fundamental limitations of single-metric evaluation in time series forecasting. Its fault-operator framework reveals that the strongest models on clean data are often the most vulnerable under realistic deployment faults, and that these vulnerabilities are both mode- and architecture-dependent. Rigorous robustness evaluation against semantically structured, parameterized scenarios must be standard practice for moving TSF to deployment-critical domains. The TS-Fault benchmark, code, and synthetic scenarios provide a reproducible pathway towards this goal, supporting research into architectures and methods that are both accurate and structurally robust.

Citation:

"TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults" (2606.18539)