TSI-Bench: Benchmarking Time Series Imputation (2406.12747v2)

Abstract: Effective imputation is a crucial preprocessing step for time series analysis. Despite the development of numerous deep learning algorithms for time series imputation, the community lacks standardized and comprehensive benchmark platforms to effectively evaluate imputation performance across different settings. Moreover, although many deep learning forecasting algorithms have demonstrated excellent performance, whether their modelling achievements can be transferred to time series imputation tasks remains unexplored. To bridge these gaps, we develop TSI-Bench, the first (to our knowledge) comprehensive benchmark suite for time series imputation utilizing deep learning techniques. The TSI-Bench pipeline standardizes experimental settings to enable fair evaluation of imputation algorithms and identification of meaningful insights into the influence of domain-appropriate missing rates and patterns on model performance. Furthermore, TSI-Bench innovatively provides a systematic paradigm to tailor time series forecasting algorithms for imputation purposes. Our extensive study across 34,804 experiments, 28 algorithms, and 8 datasets with diverse missingness scenarios demonstrates TSI-Bench's effectiveness in diverse downstream tasks and potential to unlock future directions in time series imputation research and analysis. All source code and experiment logs are released at https://github.com/WenjieDu/AwesomeImputation.

• The paper introduces a standardized benchmark suite for comparing deep learning-based time series imputation methods.
• It evaluates 28 imputation models across 8 datasets, highlighting model performance variations due to diverse missingness patterns.
• The study demonstrates that forecasting backbones can outperform traditional imputation techniques, driving robust data preprocessing.

Overview of TSI-Bench: Benchmarking Time Series Imputation

The paper "TSI-Bench: Benchmarking Time Series Imputation" addresses a critical gap in the time series research community by providing a comprehensive benchmarking suite for evaluating time series imputation (TSI) methods, particularly those leveraging deep learning techniques. The work aims to standardize and streamline the evaluation process, enabling fair comparisons across various imputation algorithms, and ultimately enhancing the robustness and effectiveness of TSI methods in different application domains.

Introduction

Time series data are prevalent across numerous domains such as air quality monitoring, healthcare, and energy systems. However, missing data, caused by sensor malfunctions, environmental interference, and other factors, pose significant challenges to accurate data analysis and model performance. Imputation aims to estimate and fill these gaps, presenting a crucial preprocessing step to ensure data reliability and completeness for downstream tasks.

TSI-Bench Suite

The TSI-Bench framework is built upon the PyPOTS Ecosystem, an open-source toolkit designed to facilitate extensive imputation, classification, and forecasting tasks. Key components of the suite include:

• TSDB: A centralized database housing a variety of time series datasets.
• PyGrinder: A flexible tool for simulating different missingness patterns in time series data.
• BenchPOTS: A standardized preprocessing pipeline enabling reproducible benchmarking.
• PyPOTS: The core imputation tool that integrates various imputation algorithms.

Datasets

TSI-Bench evaluates imputation methods using 8 datasets across air quality, traffic, electricity, and healthcare domains, each exhibiting specific characteristics and missing patterns. These datasets serve as diverse testing grounds to assess the generalizability and robustness of the imputation algorithms.

Models

The benchmark suite includes 28 diverse models spanning several architecture types:

• Transformers: iTransformer, SAITS, Nonstationary, Autoformer, etc.
• RNNs: BRITS, MRNN, GRUD.
• CNNs: TimesNet, MICN, SCINet.
• GNNs: StemGNN.
• MLPs: FiLM, DLinear, Koopa.
• Generative Models: CSDI, US-GAN, GP-VAE.
• Traditional Methods: Mean, Median, LOCF, Linear.

The forecasting backbones are also adapted for imputation by incorporating the embedding strategy and training methodology of SAITS, demonstrating the interoperability and versatility of advanced time series models.

Missingness Patterns and Evaluation Metrics

Imputation performance is evaluated under various missingness patterns (e.g., point, subsequence, block) using metrics such as MAE, MSE, and MRE. A comprehensive hyper-parameter optimization process is conducted to ensure fair comparisons. Downstream task evaluations in classification, regression, and forecasting provide insights into the practical implications of imputation.

Experimental Results

The paper's extensive experiments reveal significant findings:

1. Dataset Dependency: Imputation performance varies significantly across different datasets and application domains, underscoring the need for domain-specific model tuning.
2. Model Architecture: Transformer-based models generally perform well, but simpler architectures like MLPs can offer competitive performance with lower computational costs.
3. Forecasting Backbones: Forecasting models adapted for imputation often outperform traditional imputation-specific methods, suggesting the benefits of leveraging advanced forecasting techniques for imputation tasks.
4. Missingness Patterns: Different patterns of missing data (point, block, subsequence) greatly influence imputation accuracy.

Downstream Task Perspective

Imputation enhances downstream task performance, particularly in classification and regression. Imputed data often leads to more accurate and reliable models in subsequent analysis, highlighting the imputation's critical role as a preprocessing step.

Conclusion and Future Directions

TSI-Bench successfully establishes a standardized, comprehensive benchmark for time series imputation, facilitating rigorous evaluation and comparison of various imputation methods. The benchmark suite offers valuable insights for researchers to develop more robust imputation techniques adaptable to diverse application scenarios. Future developments of TSI-Bench aim to integrate more advanced models and datasets, further enhancing its utility as a prevalent tool for time series imputation research and application.