TS-Fault: Robustness Benchmark

• TS-Fault is a benchmark that evaluates time series forecasters via structured fault injection, modeling realistic data-quality issues.
• It defines faults along observation and mechanism axes, incorporating modes like temporal misalignment and cascading sensor failures.
• Empirical findings reveal an inverse relationship between clean accuracy and robustness, with attention and foundation models showing higher fragility.

Searching arXiv for the cited papers to ground the article in current literature. TS-Fault is a robustness benchmark for time series forecasting that evaluates forecasters under explicit, parameterized fault scenarios with controllable semantic difficulty, rather than ranking them only by average error on clean held-out data. It treats robustness as a data-quality problem: real failures are modeled as structured events with temporal shape, broken cross-variable dependencies, regime change coupled with missingness, and causal propagation across a sensing pipeline. In current usage, the named benchmark refers to the framework introduced in "TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults" (Zhao et al., 16 Jun 2026); the label also appears in adjacent literature as shorthand for transformer-specific fault analysis, most notably in DEFault++ (Jahan et al., 30 Apr 2026).

1. Problem formulation and formal scope

Conventional time series forecasting evaluation assumes that clean held-out error represents deployed reliability. TS-Fault rejects that assumption and instead formalizes faults as structured transformations of the input history $X\in\mathbb{R}^{L\times C}$, while leaving the forecast target $Y\in\mathbb{R}^{H\times C}$ unchanged. A forecaster is a map $f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$, and a fault operator is defined as

$$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$

where $\Theta$ encodes interpretable parameters such as onset, duration, affected channels, and propagation. A fault family is

$\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$

The benchmark evaluates structured instances

$$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$

with difficulty score $\delta=\kappa(\Theta)$ (Zhao et al., 16 Jun 2026).

TS-Fault defines both conditional evaluation and family-level risk. For a specific scenario,

$$\mathrm{eval}(f,\Theta)=\mathbb{E}\big[\ell(f(\tilde X),\tilde Y)\mid \Theta\big].$$

For a family $\mathcal{F}$, the benchmark distinguishes worst-case and average risk: $Y\in\mathbb{R}^{H\times C}$0

$Y\in\mathbb{R}^{H\times C}$1

The reported tables use $Y\in\mathbb{R}^{H\times C}$2, while maxima capture worst-case behavior (Zhao et al., 16 Jun 2026).

2. Taxonomy of structural faults

TS-Fault organizes faults along two orthogonal axes: observation-level versus mechanism-level, and univariate versus multivariate. This yields four modes (Zhao et al., 16 Jun 2026).

Mode	Axes	Mechanism
I	Observation × Univariate	Time-Warped Shock
II	Observation × Multivariate	Dependency-Fracture Shock
III	Mechanism × Univariate	Regime-Transition Missingness
IV	Mechanism × Multivariate	Cascading Sensor-to-System Failure

Mode I models a localized event in a single channel followed by temporal misalignment. Within a prediction-critical window $Y\in\mathbb{R}^{H\times C}$3, TS-Fault injects a structured shock

$Y\in\mathbb{R}^{H\times C}$4

and then applies a warp

$Y\in\mathbb{R}^{H\times C}$5

with

$Y\in\mathbb{R}^{H\times C}$6

Its parameter tuple is $Y\in\mathbb{R}^{H\times C}$7 (Zhao et al., 16 Jun 2026).

Mode II models a shared multivariate event whose cross-variable dependencies are falsified. For channels $Y\in\mathbb{R}^{H\times C}$8, TS-Fault estimates local lead-lag coupling by

$Y\in\mathbb{R}^{H\times C}$9

selects a dependency-critical subset $f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$0, injects a shared shock $f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$1,

$f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$2

and perturbs follower lag and gain: $f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$3

$f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$4

This can invert co-movement into anti-movement (Zhao et al., 16 Jun 2026).

Mode III alters the data-generating mechanism itself. The clean series is decomposed as

$f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$5

After a switch time $f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$6, the regime becomes

$f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$7

$f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$8

$f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}$9

with smooth transition

$$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$0

Missingness is state-dependent: $$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$1 so the model loses observations precisely around the transition it most needs to resolve (Zhao et al., 16 Jun 2026).

Mode IV models a root sensor fault that propagates into downstream channels. Directed influence is estimated by

$$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$2

Strong drivers define roots $$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$3, strong victims define downstream nodes $$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$4. Root channels receive sensor faults such as bias drift, saturation, coarse quantization, or stuck-at value. Propagation uses gains $$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$5, delays $$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$6, and kernels

$$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$7

to produce downstream perturbation

$$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$8

and state drift

$$\mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X),$$9

Secondary dropout then depends on cascade magnitude (Zhao et al., 16 Jun 2026).

3. Fault injection and semantic difficulty

A central design choice is that faults are not injected randomly. TS-Fault first finds the most prediction-critical window by a unified importance score

$\Theta$0

The four subscores correspond to change-point structure, periodic structure, volatility, and direct predictive influence under occlusion (Zhao et al., 16 Jun 2026).

The change-point term fits light local predictors to the left and right halves of a candidate window and measures bidirectional cross-prediction error: $\Theta$1 The periodicity term uses autocorrelation,

$\Theta$2

the volatility term uses wavelet detail energy, and the occlusion term measures forecast deviation when a window is removed: $\Theta$3 Mode-specific weights $\Theta$4 emphasize different structures: volatile predictive windows for Mode I, and near-change windows for Mode III (Zhao et al., 16 Jun 2026).

Each mode also has a decomposed semantic difficulty score,

$\Theta$5

with five released levels $\Theta$6. The generator samples $\Theta$7 so that $\Theta$8, scores candidate windows, selects $\Theta$9 from the top-$\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$0, applies the mode-specific operator, and returns $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$1. The paper states that computation scales roughly linearly with the number of windows and channels and is easily parallelized (Zhao et al., 16 Jun 2026).

4. Benchmark composition and evaluation protocol

TS-Fault is instantiated on six multivariate long-term forecasting datasets: ETTh1, ETTh2, ETTm1, ETTm2, Electricity, and Weather. The channel counts are 7 for the ETT datasets, 321 for Electricity, and 21 for Weather; the sampling granularities are 1 hour for ETTh1/2 and Electricity, 15 minutes for ETTm1/2, and 10 minutes for Weather. The forecast setting uses input length $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$2 and horizon $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$3 (Zhao et al., 16 Jun 2026).

The benchmark evaluates 21 models spanning statistical baselines, linear models, recurrent and convolutional models, decomposition-based transformers, recent attention-based architectures, and three foundation models. The full set is Naive, SeasonalNaive, ARIMA, ETS, DLinear, NLinear, N-BEATS, LSTM, GRU, TCN, Autoformer, FEDformer, PatchTST, iTransformer, TimeXer, TimeMixer, TimesNet, Non-stationary Transformer, TimesFM, Chronos, and Moirai. Statistical and deep models are trained per dataset with a shared recipe; the foundation models are evaluated zero-shot (Zhao et al., 16 Jun 2026).

Evaluation follows a paired clean/corrupt protocol. For each clean pair $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$4, TS-Fault produces a corresponding $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$5, so degradation reflects fault sensitivity rather than differences in target difficulty. The study covers 6 datasets, 4 modes, and 5 difficulty levels, producing 120 configurations with 20 paired windows each (Zhao et al., 16 Jun 2026).

Robustness is measured by clean loss $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$6, faulted loss $\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$7, absolute degradation

$\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$8

robustness ratio

$\mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}.$9

and relative degradation

$$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$0

A catastrophic failure is any configuration with $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$1 (Zhao et al., 16 Jun 2026).

5. Empirical findings

The benchmark’s main empirical result is that clean-data accuracy anti-correlates with robustness. Across all 21 models, the Spearman correlation between clean accuracy rank and robustness rank is $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$2 with $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$3; over the 18 non-foundation models it remains negative at $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$4 with $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$5. The published rank shifts are large: iTransformer moves from 3rd on clean accuracy to 21st on robustness, while TCN, LSTM, and GRU move from 16th, 14th, and 12th on clean accuracy to 1st, 2nd, and 3rd on robustness (Zhao et al., 16 Jun 2026).

The second result is that robustness depends sharply on whether faults are observation-level or mechanism-level. Clean rankings are preserved under Mode I and Mode II, with Spearman $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$6 and $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$7, but essentially destroyed under Mode III and Mode IV, with $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$8 and $$(X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y,$$9. This means that a clean-data leaderboard remains informative for local observation corruption, but ceases to predict behavior when the underlying process changes or a causal cascade develops (Zhao et al., 16 Jun 2026).

The third result is that all catastrophic failures occur under mechanism-level faults. The paper reports 884 catastrophic failures in total: 0 under Mode I, 0 under Mode II, 537 under Mode III, and 347 under Mode IV. Mode III accounts for 85.9% of its 625 evaluated cells, and Mode IV for 55.5% of its cells. Median degradation under Modes I and II stays at or below 18% for all models, whereas Mode III and Mode IV produce degradations ranging from hundreds to tens of thousands of percent (Zhao et al., 16 Jun 2026).

Foundation models are the strongest on clean data and the most fragile under fault. TimesFM attains clean MSE $\delta=\kappa(\Theta)$0 and clean MAE $\delta=\kappa(\Theta)$1, yet its faulted MSE rises to $\delta=\kappa(\Theta)$2 and its global robustness ratio reaches $\delta=\kappa(\Theta)$3. Chronos and Moirai show similarly extreme ratios, $\delta=\kappa(\Theta)$4 and $\delta=\kappa(\Theta)$5. In per-mode median relative degradation, TimesFM reaches $\delta=\kappa(\Theta)$6 on Mode III and $\delta=\kappa(\Theta)$7 on Mode IV. On the 321-channel Electricity dataset, TimesFM reaches a ratio of $\delta=\kappa(\Theta)$8 (Zhao et al., 16 Jun 2026).

Difficulty behaves monotonically. Across all four modes, degradation increases from $\delta=\kappa(\Theta)$9 to $$\mathrm{eval}(f,\Theta)=\mathbb{E}\big[\ell(f(\tilde X),\tilde Y)\mid \Theta\big].$$0. The reported sensitivity slopes $$\mathrm{eval}(f,\Theta)=\mathbb{E}\big[\ell(f(\tilde X),\tilde Y)\mid \Theta\big].$$1 separate model families into low-sensitivity recurrent models such as GRU and LSTM, mid-sensitivity models such as TCN and N-BEATS, and high-sensitivity attention and foundation models, with TimesFM highest at $$\mathrm{eval}(f,\Theta)=\mathbb{E}\big[\ell(f(\tilde X),\tilde Y)\mid \Theta\big].$$2 (Zhao et al., 16 Jun 2026).

6. Related usages, methodological context, and extensions

The benchmark sits in a wider fault-analysis landscape that also emphasizes explicit mechanisms over generic perturbation. In transformer diagnosis, DEFault++ uses the expression “TS-Fault” for transformer-specific fault analysis and organizes detection, categorization, and root-cause diagnosis through a Fault Propagation Graph and the DEFault-bench benchmark of 3,739 labeled instances across seven transformer models and nine downstream tasks (Jahan et al., 30 Apr 2026). This parallel usage is terminological rather than identical in scope, but it reflects the same shift from generic noise sensitivity to fault-specific evaluation.

A broader methodological continuity is visible across adjacent work. MetaFI defines a model-driven, simulator-independent fault simulation framework with Statistical Fault Injection, Direct Fault Injection, and Exhaustive Fault Injection at mixed RTL/GL granularity (Kaja et al., 2022). LLM-assisted automotive fault injection translates functional safety requirements into representative, high-coverage fault test cases for real-time Hardware-in-the-Loop execution (Abboush et al., 24 Nov 2025). AFETM combines adaptive function-level trace monitoring, target fault injection, and graph convolutional diagnosis for intra-component fault localization (Zhang et al., 2022). Temporal Fault Tree Analysis extends Boolean FTA with PAND and SAND so that sequence dependencies can be analyzed without transformation into state space (Schilling, 2015). Taken together, these works suggest a common methodological trend: fault analysis is increasingly built around explicit fault operators, structured injection, and mechanism-aware diagnosis rather than unstructured disturbance models.

Within that trend, TS-Fault’s specific contribution is to make structural robustness a first-class benchmarking target for forecasting. The paper’s own forward-looking directions are consistent with that role: online adaptation and change-point-aware models, mechanism-level robustness guarantees in semantic parameter spaces, and compositional fault operators of the form $$\mathrm{eval}(f,\Theta)=\mathbb{E}\big[\ell(f(\tilde X),\tilde Y)\mid \Theta\big].$$3 (Zhao et al., 16 Jun 2026).