ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data

Abstract: Unsupervised domain adaptation methods aim to generalize well on unlabeled test data that may have a different (shifted) distribution from the training data. Such methods are typically developed on image data, and their application to time series data is less explored. Existing works on time series domain adaptation suffer from inconsistencies in evaluation schemes, datasets, and backbone neural network architectures. Moreover, labeled target data are often used for model selection, which violates the fundamental assumption of unsupervised domain adaptation. To address these issues, we develop a benchmarking evaluation suite (AdaTime) to systematically and fairly evaluate different domain adaptation methods on time series data. Specifically, we standardize the backbone neural network architectures and benchmarking datasets, while also exploring more realistic model selection approaches that can work with no labeled data or just a few labeled samples. Our evaluation includes adapting state-of-the-art visual domain adaptation methods to time series data as well as the recent methods specifically developed for time series data. We conduct extensive experiments to evaluate 11 state-of-the-art methods on five representative datasets spanning 50 cross-domain scenarios. Our results suggest that with careful selection of hyper-parameters, visual domain adaptation methods are competitive with methods proposed for time series domain adaptation. In addition, we find that hyper-parameters could be selected based on realistic model selection approaches. Our work unveils practical insights for applying domain adaptation methods on time series data and builds a solid foundation for future works in the field. The code is available at \href{https://github.com/emadeldeen24/AdaTime}{github.com/emadeldeen24/AdaTime}.

• The paper introduces ADATIME, a suite that standardizes evaluation of unsupervised domain adaptation methods on time series data.
• It employs consistent datasets, backbone architectures, and evaluation strategies to compare both visual and time series-specific adaptation methods.
• The study highlights the importance of joint distribution alignment and robust metrics like macro F1-score to effectively address domain shift challenges.

Overview of "ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data"

The paper entitled "ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data" provides a comprehensive framework for the evaluation of unsupervised domain adaptation (UDA) methods specifically tailored for time series data. The impetus for developing this benchmarking suite arises from the existing inconsistencies in the evaluation of domain adaptation methods on time series, with prior studies predominantly focused on image data. The authors identify a lack of consistency across datasets, neural network backbones, and evaluation schemes, which motivates the inception of AdaTime.

Problem Addressed

The primary aim of unsupervised domain adaptation is to generalize well on an unlabeled target domain using labeled data from a different, yet related, source domain. This is particularly challenging for time series data, where existing studies have been limited and characterized by inconsistent experimental setups. Addressing these inconsistencies, the authors propose AdaTime, a benchmarking suite that standardizes various aspects of UDA evaluation on time series data.

Methodological Contributions

The paper develops AdaTime to provide a uniform platform for evaluating UDA algorithms on time series data. The core components of this suite include:

1. Standardization of Datasets and Evaluation Schemes: AdaTime includes representative datasets from real-world applications, ensuring fair cross-domain adaptation scenarios. The benchmarking framework utilizes five well-known datasets, each with varying characteristics, to evaluate methodologies comprehensively.
2. Backbone Network Unification: Three different backbone architectures (1D-CNN, 1D-ResNet, and TCN) are employed, promoting consistency across evaluations while accommodating varied computational complexities and dataset scales.
3. Incorporation of Visual UDA Methods: State-of-the-art visual domain adaptation methods, alongside methods designed for time series, are incorporated, broadening the perspective on potential cross-domain applicability.
4. Realistic Model Selection Approaches: The authors explore practical model selection strategies that do not rely on labeled target domain data, including source risk (SRC) and Deep Embedded Evaluation Risk (DEV). They also propose a few-shot approach (FST) that leverages minimal labeled data from the target domain, achieving competitive performance with traditional target risk (TGT) approaches.

Experimental Insights

Results across the five datasets reveal several insights:

• Competitive Performance of Visual UDA Methods: The adaptability of visual domain adaptation methods to time series signals, achieving comparable performance to existing time series-specific approaches, underscores their potential as robust baselines.
• Effectiveness of Joint Distribution Alignment: Approaches aligning both marginal and conditional distributions consistently yield better results than those addressing only marginal distributions. This indicates the importance of fine-grained alignment in mitigating the domain shift.
• Relevance of Evaluation Metrics: The study highlights the inadequacy of accuracy as a performance metric for imbalanced datasets, advocating for macro F1-score to ensure more reliable evaluation.

Theoretical and Practical Implications

By removing extraneous variables from evaluations and providing a unified benchmark, AdaTime permits a clearer attribution of performance gains to UDA methodologies themselves. This positions it as a valuable toolkit for researchers exploring domain adaptation in time series contexts.

Moreover, the exploration of various model selection strategies imparts practical guidelines for real-world applications of TS-UDA, helping practitioners navigate the inherent challenges of unlabeled target domains.

Future Directions

Future work could extend this benchmarking approach to encompass other time series tasks such as regression and forecasting, as well as to explore domain adaptation scenarios with mismatched source and target classes (partial and open-set adaptation).

The AdaTime suite lays a foundational framework for further refinements and advancements in time series domain adaptation, promising to impact fields such as healthcare and manufacturing, where adaptability to temporal shifts is paramount.