Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting

Abstract: While previous research in multivariate time series forecasting has focused on developing complex holistic models, this work advocates for a shift toward a granular, component-level understanding of their impacts. We propose TSCOMP, the first large-scale benchmark that systematically deconstructs deep forecasting methods into their core, fine-grained components--spanning series preprocessing, encoding strategies, network architectures including specific and large time-series models, and optimization methods. Using constrained orthogonal experimental design and extensive evaluations, we conduct multi-view analyses that reveal component effectiveness across different backbones, data characteristics, and their interactions. Beyond providing insights, this benchmark establishes a fine-grained performance corpus comprising over 20,000 model-dataset evaluations, which supports the learning of automated component selection, enabling zero-shot model construction on new datasets. Our experiments demonstrate that the corpus-driven approach, despite its simplicity, consistently outperforms state-of-the-art methods, validating the soundness of our evaluation design and confirming that systematic component selection surpasses manually designed complex architectures. All code and the performance corpus are publicly available at https://github.com/SUFE-AILAB/TSCOMP.

• The paper demonstrates that series normalization drives over 60% of performance variance, emphasizing the need for component-level analysis.
• It presents TSCOMP, a modular framework that deconstructs forecasting pipelines into preprocessing, encoding, network, and optimization components.
• The study shows that automated, zero-shot model construction using meta-learning outperforms holistic approaches in reproducibility and adaptability.

Systematic Component-level Benchmarking for Deep Multivariate Time-Series Forecasting

Motivation and Problem Scope

The paper "Beyond Holistic Models: Systematic Component-level Benchmarking of Deep Multivariate Time-Series Forecasting" (2605.26562) identifies a methodological limitation in current research on multivariate time-series forecasting (MTSF): existing benchmarks and architectures are predominantly evaluated as indivisible entities, obscuring the relative contributions of their internal modules. As the landscape expands to include MLPs, advanced Transformers, LLMs, and Time Series Foundation Models (TSFMs), empirical analysis reveals divergent performance profiles, especially when exposed to heterogeneous data domains and temporal characteristics. However, without granular component-level evaluation, it is infeasible to attribute efficacy to specific architectural or preprocessing innovations, which inhibits principled architectural design and transferability.

TSCOMP Framework: Hierarchical Decomposition and Benchmarking

The authors propose TSCOMP, a hierarchical component-level benchmarking and model construction system that deconstructs deep MTSF pipelines into a modular pool spanning series preprocessing, encoding, network architecture, and optimization strategies. Each pipeline stage is further partitioned into component dimensions (e.g., normalization, decomposition, channel structuring, tokenization, attention, loss).

Figure 1: TSCOMP modularizes SOTA models into a component pool, performs bottom-up multi-level evaluation, and supports zero-shot model construction via a pre-trained meta-predictor for adaptive component selection.

The evaluation protocol imposes a constrained orthogonal experimental design, achieving maximal pairwise component coverage while discarding structurally invalid configurations. The design space ultimately encompasses thousands of valid models, with large-scale experimentation yielding an empirical corpus—over 20,000 model-dataset evaluations—establishing a high-resolution component–data–performance linkage.

Empirical Insights: Component, Dimension, and Pipeline Contribution

Component-level Analysis

Statistical modeling (GLMMs, ANOVA) isolates the marginal effects of each module, controlling for dataset and task variance. Among major findings:

• Series normalization (especially variants such as RevIN and Stationary Norm) is the single largest driver of performance, accounting for over 60% of total explained variance (far exceeding any network architectural feature).
• Series decomposition (classical trend/seasonality removal, frequency-based splits) frequently degrades performance, particularly in architectures relying on cross-channel inductive biases.
• Channel independence (processing variables separately) outperforms dependent modeling in low-correlation regimes but should not be universally applied, as it introduces inefficacy on datasets with high inter-variable correlation.
• For tokenization, Orthogonal and Inverted encoding schemes offer robust improvements, particularly in settings lacking explicit sequence modeling.

Figure 2: Ridgeline plots visualize the performance distribution of various backbone architectures, highlighting the heterogeneity across model types.

Dimension and Pipeline-level Analysis

At the meta-level, Series Preprocessing dominates the pipeline, accounting for 66.6% of variance in forecasting performance. Series Encoding is next (18.3%), exceeding Network Architecture (8.0%) and Optimization (7.1%). This pipeline-stage dominance is metric-invariant and robust across scenarios and cross-validated splits.

Architecture-specific Patterns

Component importance shifts markedly across backbone types:

• MLP architectures are reliant on preprocessing; network tuning offers marginal gains.
• RNNs exhibit outsize benefit from channel independence and sequence-length optimization, but are susceptible to performance collapse under aggressive decomposition.
• Transformers derive significant value from specialized loss functions and orthogonal tokenizations; decomposition impairs attention expressivity.
• TSFMs and LLMs display orthogonal sensitivities, for example, LLMs are robust to mixing/patching strategies originally designed for non-attentional models, while TSFMs’ performance is modulated primarily by encoding strategies and less by decomposition.

Figure 3: Visual interaction analysis between architecture and attention type, demonstrating that lightweight MLPs with sparse attention can outperform more complex configurations.

Data-specific Adaptivity

Systematic evaluation across datasets with variable sample size, nonstationarity, correlation, and distributional drift reveals conditional efficacy:

• MLPs are preferable on datasets with larger samples and weak inter-variable correlation.
• Instance normalization (RevIN) underperforms on datasets with pronounced shift; Stationary Norm is optimal in such settings.
• Channel independence is actively harmful on highly correlated datasets.
• Mechanism-context synergy surfaces: For example, auto-correlation based attention mechanisms outperform on series with high temporal autocorrelation.

Figure 4: Architecture performance as a function of sample size, showing that MLP models benefit disproportionately from larger datasets relative to LLM or TSFM baselines.

Corpus-driven Automated Model Construction

A core contribution is the automated, meta-learning-driven synthesis of MTSF models:

• The TSCOMP performance corpus supports training a meta-predictor that, given dataset-level meta-features (TabPFN embeddings), can recommend optimal component configurations in a zero-shot manner.
• Empirically, the top-5 configurations recommended by the meta-predictor fall within the top quartile of actual performance 98% of the time.

  Figure 5: Distribution of selection quality for the meta-predictor’s top-5 recommendations, indicating extremely high consistency with true best configurations.

This approach not only enables rapid pipeline instantiation without task-specific retraining but, crucially, outperforms state-of-the-art (SOTA) deep learning architectures, AutoML systems, and large pre-trained TSFMs in both long-term and short-term forecasting tasks. The results hold in both MSE, MAE, and scale-invariant metrics, for both full-shot and zero-shot evaluation settings.

Practical and Theoretical Implications

• Component combinatorics eclipse backbone innovation: Precise selection of basic modules (normalization, encoding, loss) yields greater accuracy gains than introducing architectural complexity or scale in most regimes.
• Automated pipeline design becomes tractable: Through systematic corpus construction and meta-learning, it is possible to algorithmically generate domain-adaptive, data-driven MTSF solutions, substantially reducing the required manual model search.
• Benchmarking should shift from model-level to component-level protocols: Holistic performance reporting is insufficient for scientific progress; component-wise attribution is essential for interpretable, transferable, and reproducible research.

Outlook and Future Directions

The authors propose to evolve TSCOMP into an LLM-agent-powered workflow that continuously assimilates new architectures and component innovations from emerging literature. This direction would yield a live, self-updating corpus and automated synthesis system, maintaining alignment with the frontier of deep time-series modeling research, and opening prospects for personalized, data-driven forecasting pipelines scalable across domains and tasks.

Conclusion

This work provides the first large-scale, fine-grained benchmarking and automated construction framework for deep MTSF pipelines, demonstrating that systematic component selection is both more effective and principled than reliance on holistic architectural design or scale. The open-sourced corpus and codebase have the potential to establish a new standard for evaluative rigor and reproducibility within the time series forecasting community.