Self-Supervised Contrastive Pre-Training For Time Series via Time-Frequency Consistency (2206.08496v3)

Abstract: Pre-training on time series poses a unique challenge due to the potential mismatch between pre-training and target domains, such as shifts in temporal dynamics, fast-evolving trends, and long-range and short-cyclic effects, which can lead to poor downstream performance. While domain adaptation methods can mitigate these shifts, most methods need examples directly from the target domain, making them suboptimal for pre-training. To address this challenge, methods need to accommodate target domains with different temporal dynamics and be capable of doing so without seeing any target examples during pre-training. Relative to other modalities, in time series, we expect that time-based and frequency-based representations of the same example are located close together in the time-frequency space. To this end, we posit that time-frequency consistency (TF-C) -- embedding a time-based neighborhood of an example close to its frequency-based neighborhood -- is desirable for pre-training. Motivated by TF-C, we define a decomposable pre-training model, where the self-supervised signal is provided by the distance between time and frequency components, each individually trained by contrastive estimation. We evaluate the new method on eight datasets, including electrodiagnostic testing, human activity recognition, mechanical fault detection, and physical status monitoring. Experiments against eight state-of-the-art methods show that TF-C outperforms baselines by 15.4% (F1 score) on average in one-to-one settings (e.g., fine-tuning an EEG-pretrained model on EMG data) and by 8.4% (precision) in challenging one-to-many settings (e.g., fine-tuning an EEG-pretrained model for either hand-gesture recognition or mechanical fault prediction), reflecting the breadth of scenarios that arise in real-world applications. Code and datasets: https://github.com/mims-harvard/TFC-pretraining.

Self-Supervised Contrastive Pre-Training for Time Series via Time-Frequency Consistency

The paper presents a novel approach for self-supervised pre-training on time series data, leveraging a concept termed Time-Frequency Consistency (TF-C). This methodology is designed to cope with the challenge of representing varying temporal dynamics across different domains, an issue prevalent in time series data derived from disparate sources, such as clinical diagnostics, activity monitoring, and mechanical systems.

Problem and Motivation

Time series data are inherently complex due to their temporal characteristics, which can include varying patterns and irregularities. Existing domain adaptation techniques mitigate domain shifts but often require access to labeled target domain data, which is infeasible during pre-training. This paper introduces a strategy that does not necessitate target domain examples during pre-training. The key idea revolves around ensuring that embeddings derived from both time and frequency representations of the same time series sample remain proximate in a latent space.

Methodology

The authors propose a pre-training model composed of time-based and frequency-based encoders, coupled with projectors to map data into a shared latent space. Central to the model is the TF-C property, which asserts that time and frequency embeddings of a sample, and their augmentations, should be closely aligned. This approach employs contrastive learning to train embeddings by minimizing distances between time-based and frequency-based augmentations.

The pre-training process generates robust representations by employing contrastive loss to enforce consistency within individual domains and a novel consistency loss across them. The innovation lies in developing distinct augmentations for both domains, thereby capitalizing on the rich attribute set offered by both timeseries' temporal and spectral properties.

Results and Implications

Evaluations conducted on eight diversified datasets demonstrate the efficacy of the TF-C approach, with the model surpassing baseline methods by an average of 15.4% in F1 score and by 8.4% in precision for complex one-to-many setups. These results underscore the model's robustness and generalizability.

The implications of this research are significant for applications requiring accurate and transferable representations of time series data. By establishing a method that does not require target domain visibility, the TF-C framework enhances flexibility in deploying pre-trained models across various scenarios, thus contributing to fields like healthcare, machinery fault detection, and more.

Future Directions

Future research could explore additional invariant properties of time series for pre-training. Moreover, optimizing the handling of irregularly sampled time series and refining the augmentation strategies, particularly in the frequency domain, warrants further investigation. This could include exploring phase perturbations or more nuanced perturbation distributions, enhancing the model's generalization capabilities.

Overall, the proposed TF-C framework presents a substantial advancement in the pre-training of time series data, providing a resilient foundation for subsequent transfer learning tasks.