--- title: 'TS-Fault: Robustness Benchmark' url: https://www.emergentmind.com/topics/ts-fault type: topic --- # TS-Fault: Robustness Benchmark Searching arXiv for the cited papers to ground the article in current literature. TS-Fault is a robustness benchmark for time series forecasting that evaluates forecasters under explicit, parameterized fault scenarios with controllable semantic difficulty, rather than ranking them only by average error on clean held-out data. It treats robustness as a data-quality problem: real failures are modeled as structured events with temporal shape, broken cross-variable dependencies, regime change coupled with missingness, and causal propagation across a sensing pipeline. In current usage, the named benchmark refers to the framework introduced in "TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults" [2606.18539]; the label also appears in adjacent literature as shorthand for transformer-specific fault analysis, most notably in DEFault++ [2604.28118]. ## 1. Problem formulation and formal scope Conventional time series forecasting evaluation assumes that clean held-out error represents deployed reliability. TS-Fault rejects that assumption and instead formalizes faults as structured transformations of the input history \(X\in\mathbb{R}^{L\times C}\), while leaving the forecast target \(Y\in\mathbb{R}^{H\times C}\) unchanged. A forecaster is a map \(f:\mathbb{R}^{L\times C}\to\mathbb{R}^{H\times C}\), and a fault operator is defined as \[ \mathcal{T}_\Theta:\mathbb{R}^{L\times C}\to\mathbb{R}^{L\times C},\qquad \tilde X=\mathcal{T}_\Theta(X), \] where \(\Theta\) encodes interpretable parameters such as onset, duration, affected channels, and propagation. A fault family is \[ \mathcal{F}=\{\mathcal{T}_\Theta:\Theta\in\Phi\}. \] The benchmark evaluates structured instances \[ (X,Y)\xrightarrow{\mathcal{T}_\Theta}(\tilde X,\tilde Y,\Theta,\delta),\qquad \tilde Y=Y, \] with difficulty score \(\delta=\kappa(\Theta)\) [2606.18539]. TS-Fault defines both conditional evaluation and family-level risk. For a specific scenario, \[ \mathrm{eval}(f,\Theta)=\mathbb{E}\big[\ell(f(\tilde X),\tilde Y)\mid \Theta\big]. \] For a family \(\mathcal{F}\), the benchmark distinguishes worst-case and average risk: \[ \mathrm{WC}(f,\mathcal{F})=\sup_{\Theta\in\Phi}\mathbb{E}_{(X,Y)\sim P}[\ell(f(\mathcal{T}_\Theta(X)),Y)], \] \[ \mathrm{AVG}(f,\mathcal{F})=\mathbb{E}_{\Theta\sim\pi_\Phi}\mathbb{E}_{(X,Y)\sim P}[\ell(f(\mathcal{T}_\Theta(X)),Y)]. \] The reported tables use \(\mathrm{AVG}\), while maxima capture worst-case behavior [2606.18539]. ## 2. Taxonomy of structural faults TS-Fault organizes faults along two orthogonal axes: observation-level versus mechanism-level, and univariate versus multivariate. This yields four modes [2606.18539]. | Mode | Axes | Mechanism | |---|---|---| | I | Observation × Univariate | Time-Warped Shock | | II | Observation × Multivariate | Dependency-Fracture Shock | | III | Mechanism × Univariate | Regime-Transition Missingness | | IV | Mechanism × Multivariate | Cascading Sensor-to-System Failure | Mode I models a localized event in a single channel followed by temporal misalignment. Within a prediction-critical window \(W^\star\), TS-Fault injects a structured shock \[ b_t=a_t m_t,\qquad \tilde x_t=x_t+\alpha b_t,\quad t\in W^\star, \] and then applies a warp \[ x'_t=\tilde x_{\phi(t)}, \] with \[ \phi(t)= \begin{cases} t+\Delta, & \text{shift}\\ s+\alpha_w(t-s), & \text{scale}\\ t+\Delta\sin\!\big(2\pi(t-s)/|W^\star|\big)+\xi_t, & \text{nonlinear}. \end{cases} \] Its parameter tuple is \(\Theta_{\mathrm{I}}=(\Theta_{\mathrm{evt}},\Theta_{\mathrm{wrp}},\Theta_{\mathrm{ctx}},\Theta_{\mathrm{cpl}})\) [2606.18539]. Mode II models a shared multivariate event whose cross-variable dependencies are falsified. For channels \(i,j\), TS-Fault estimates local lead-lag coupling by \[ R_{ij}(W)=\max_{|\tau|\le \tau_{\max}}\big|\mathrm{Corr}(x^{(i)}_t,x^{(j)}_{t-\tau})\big|, \qquad G_i(W)=\sum_{j\neq i}R_{ij}(W), \] selects a dependency-critical subset \(S\), injects a shared shock \(u_t\), \[ \tilde x^{(i)}_t=x^{(i)}_t+g_i u_t,\quad i\in S, \] and perturbs follower lag and gain: \[ \tau'_j=\hat\tau_j+\Delta\tau_j,\qquad g'_j=\hat g_j+\Delta g_j, \] \[ {x'}^{(j)}_t=x^{(j)}_t+g'_j u_{t-\tau'_j}. \] This can invert co-movement into anti-movement [2606.18539]. Mode III alters the data-generating mechanism itself. The clean series is decomposed as \[ x_t=T_t+S_t+R_t. \] After a switch time \(\tau\), the regime becomes \[ \tilde T_t=T_t+\Delta_\beta(t-\tau)+\Delta_b, \] \[ \tilde S_t=a_s S_{\phi_s(t)},\qquad \phi_s(t)=\tau+\tfrac{P}{P'}(t-\tau)+\psi, \] \[ \tilde R_t=c_r R_t, \] with smooth transition \[ \omega_t=\sigma((t-\tau)/w_\tau),\qquad z_t=(1-\omega_t)x_t+\omega_t\bar z_t. \] Missingness is state-dependent: \[ p^{(i)}_{\mathrm{start}}(t)=\sigma\!\big(a_0+a_1 h_t+a_2 v^{(i)}_t+a_3 \varrho^{(i)}_t+a_4 c_i\big), \] so the model loses observations precisely around the transition it most needs to resolve [2606.18539]. Mode IV models a root sensor fault that propagates into downstream channels. Directed influence is estimated by \[ \Delta_{i\to j}(W)= \frac{\mathrm{Err}^{(j)}_{\mathrm{self}}-\mathrm{Err}^{(i\to j)}_{\mathrm{aug}}} {\mathrm{Err}^{(j)}_{\mathrm{self}}+\varepsilon}. \] Strong drivers define roots \(R\), strong victims define downstream nodes \(D\). Root channels receive sensor faults such as bias drift, saturation, coarse quantization, or stuck-at value. Propagation uses gains \(\gamma_{rj}\), delays \(\Delta_{rj}\), and kernels \[ k_{rj}(\ell)=e^{-\ell/h_{rj}}/Z_{rj}, \] to produce downstream perturbation \[ \zeta^{(j)}_t=\sum_{r\in R}\gamma_{rj}\sum_{\ell=0}^{n_k}k_{rj}(\ell)e^{(r)}_{t-\Delta_{rj}-\ell}, \] and state drift \[ z'^{(j)}_t=z^{(j)}_t+\zeta^{(j)}_t. \] Secondary dropout then depends on cascade magnitude [2606.18539]. ## 3. Fault injection and semantic difficulty A central design choice is that faults are not injected randomly. TS-Fault first finds the most prediction-critical window by a unified importance score \[ S(W)=\lambda_1 S_{\mathrm{cp}}(W)+\lambda_2 S_{\mathrm{per}}(W)+\lambda_3 S_{\mathrm{var}}(W)+\lambda_4 S_{\mathrm{pred}}(W). \] The four subscores correspond to change-point structure, periodic structure, volatility, and direct predictive influence under occlusion [2606.18539]. The change-point term fits light local predictors to the left and right halves of a candidate window and measures bidirectional cross-prediction error: \[ S_{\mathrm{cp}}(W)=\frac{1}{|W_R|}\sum_{t\in W_R}\|x_t-f_L(x_{t})\|_2^2. \] The periodicity term uses autocorrelation, \[ \mathrm{ACF}(k)= \frac{\sum_t (x_t-\bar x)(x_{t+k}-\bar x)} {\sum_t (x_t-\bar x)^2}, \] the volatility term uses wavelet detail energy, and the occlusion term measures forecast deviation when a window is removed: \[ S_{\mathrm{pred}}(W)= \sum_{h=1}^{H}\alpha_h\|\hat y_{L+h}-\hat y^{(\backslash W)}_{L+h}\|_2^2. \] Mode-specific weights \(\lambda_k\) emphasize different structures: volatile predictive windows for Mode I, and near-change windows for Mode III [2606.18539]. Each mode also has a decomposed semantic difficulty score, \[ \delta=\kappa(\Theta)=\sum_k \beta_k D_k, \] with five released levels \(d_{02},d_{04},d_{06},d_{08},d_{10}\). The generator samples \(\Theta\) so that \(\kappa_i(\Theta)\approx \delta_s\), scores candidate windows, selects \(W^\star\) from the top-\(K\), applies the mode-specific operator, and returns \((\tilde X,\tilde Y,\Theta,\delta)\). The paper states that computation scales roughly linearly with the number of windows and channels and is easily parallelized [2606.18539]. ## 4. Benchmark composition and evaluation protocol TS-Fault is instantiated on six multivariate long-term forecasting datasets: ETTh1, ETTh2, ETTm1, ETTm2, Electricity, and Weather. The channel counts are 7 for the ETT datasets, 321 for Electricity, and 21 for Weather; the sampling granularities are 1 hour for ETTh1/2 and Electricity, 15 minutes for ETTm1/2, and 10 minutes for Weather. The forecast setting uses input length \(L=336\) and horizon \(H=96\) [2606.18539]. The benchmark evaluates 21 models spanning statistical baselines, linear models, recurrent and convolutional models, decomposition-based transformers, recent attention-based architectures, and three foundation models. The full set is Naive, SeasonalNaive, ARIMA, ETS, DLinear, NLinear, N-BEATS, LSTM, GRU, TCN, Autoformer, FEDformer, PatchTST, iTransformer, TimeXer, TimeMixer, TimesNet, Non-stationary Transformer, TimesFM, Chronos, and Moirai. Statistical and deep models are trained per dataset with a shared recipe; the foundation models are evaluated zero-shot [2606.18539]. Evaluation follows a paired clean/corrupt protocol. For each clean pair \((X,Y)\), TS-Fault produces a corresponding \((\tilde X,Y)\), so degradation reflects fault sensitivity rather than differences in target difficulty. The study covers 6 datasets, 4 modes, and 5 difficulty levels, producing 120 configurations with 20 paired windows each [2606.18539]. Robustness is measured by clean loss \(L(f)\), faulted loss \(\mathrm{AVG}(f,\mathcal{F}_i)\), absolute degradation \[ \Delta\mathrm{MSE}_i(f)=\mathrm{AVG}(f,\mathcal{F}_i)-L(f), \] robustness ratio \[ r_i(f)=\frac{\mathrm{AVG}(f,\mathcal{F}_i)}{L(f)}, \] and relative degradation \[ \mathrm{RD}_i(f)=(r_i(f)-1)\times 100\%. \] A catastrophic failure is any configuration with \(r_i\ge 10\) [2606.18539]. ## 5. Empirical findings The benchmark’s main empirical result is that clean-data accuracy anti-correlates with robustness. Across all 21 models, the Spearman correlation between clean accuracy rank and robustness rank is \(\rho=-0.544\) with \(p=0.011\); over the 18 non-foundation models it remains negative at \(\rho=-0.509\) with \(p=0.031\). The published rank shifts are large: iTransformer moves from 3rd on clean accuracy to 21st on robustness, while TCN, LSTM, and GRU move from 16th, 14th, and 12th on clean accuracy to 1st, 2nd, and 3rd on robustness [2606.18539]. The second result is that robustness depends sharply on whether faults are observation-level or mechanism-level. Clean rankings are preserved under Mode I and Mode II, with Spearman \(\rho=0.925\) and \(\rho=0.952\), but essentially destroyed under Mode III and Mode IV, with \(\rho=0.032\) and \(\rho=0.055\). This means that a clean-data leaderboard remains informative for local observation corruption, but ceases to predict behavior when the underlying process changes or a causal cascade develops [2606.18539]. The third result is that all catastrophic failures occur under mechanism-level faults. The paper reports 884 catastrophic failures in total: 0 under Mode I, 0 under Mode II, 537 under Mode III, and 347 under Mode IV. Mode III accounts for 85.9% of its 625 evaluated cells, and Mode IV for 55.5% of its cells. Median degradation under Modes I and II stays at or below 18% for all models, whereas Mode III and Mode IV produce degradations ranging from hundreds to tens of thousands of percent [2606.18539]. Foundation models are the strongest on clean data and the most fragile under fault. TimesFM attains clean MSE \(0.516\) and clean MAE \(0.436\), yet its faulted MSE rises to \(162.7\) and its global robustness ratio reaches \(555.2\). Chronos and Moirai show similarly extreme ratios, \(512.5\) and \(365.6\). In per-mode median relative degradation, TimesFM reaches \(87{,}000\%\) on Mode III and \(10{,}000\%\) on Mode IV. On the 321-channel Electricity dataset, TimesFM reaches a ratio of \(2{,}090\) [2606.18539]. Difficulty behaves monotonically. Across all four modes, degradation increases from \(d_{02}\) to \(d_{10}\). The reported sensitivity slopes \(\tfrac{d_{10}}{d_{02}}\) separate model families into low-sensitivity recurrent models such as GRU and LSTM, mid-sensitivity models such as TCN and N-BEATS, and high-sensitivity attention and foundation models, with TimesFM highest at \(27.9\) [2606.18539]. ## 6. Related usages, methodological context, and extensions The benchmark sits in a wider fault-analysis landscape that also emphasizes explicit mechanisms over generic perturbation. In transformer diagnosis, DEFault++ uses the expression “TS-Fault” for transformer-specific fault analysis and organizes detection, categorization, and root-cause diagnosis through a Fault Propagation Graph and the DEFault-bench benchmark of 3,739 labeled instances across seven transformer models and nine downstream tasks [2604.28118]. This parallel usage is terminological rather than identical in scope, but it reflects the same shift from generic noise sensitivity to fault-specific evaluation. A broader methodological continuity is visible across adjacent work. MetaFI defines a model-driven, simulator-independent fault simulation framework with Statistical Fault Injection, Direct Fault Injection, and Exhaustive Fault Injection at mixed RTL/GL granularity [2204.13183]. LLM-assisted automotive fault injection translates functional safety requirements into representative, high-coverage fault test cases for real-time Hardware-in-the-Loop execution [2511.19132]. AFETM combines adaptive function-level trace monitoring, target fault injection, and graph convolutional diagnosis for intra-component fault localization [2210.07486]. Temporal Fault Tree Analysis extends Boolean FTA with PAND and SAND so that sequence dependencies can be analyzed without transformation into state space [1505.04511]. Taken together, these works suggest a common methodological trend: fault analysis is increasingly built around explicit fault operators, structured injection, and mechanism-aware diagnosis rather than unstructured disturbance models. Within that trend, TS-Fault’s specific contribution is to make structural robustness a first-class benchmarking target for forecasting. The paper’s own forward-looking directions are consistent with that role: online adaptation and change-point-aware models, mechanism-level robustness guarantees in semantic parameter spaces, and compositional fault operators of the form \(\mathcal{T}_{\Theta_k}\circ\cdots\circ\mathcal{T}_{\Theta_1}\) [2606.18539]. Source: https://www.emergentmind.com/topics/ts-fault