Out-of-Distribution Representation Learning for Time Series Classification (2209.07027v4)

Abstract: Time series classification is an important problem in real world. Due to its non-stationary property that the distribution changes over time, it remains challenging to build models for generalization to unseen distributions. In this paper, we propose to view the time series classification problem from the distribution perspective. We argue that the temporal complexity attributes to the unknown latent distributions within. To this end, we propose DIVERSIFY to learn generalized representations for time series classification. DIVERSIFY takes an iterative process: it first obtains the worst-case distribution scenario via adversarial training, then matches the distributions of the obtained sub-domains. We also present some theoretical insights. We conduct experiments on gesture recognition, speech commands recognition, wearable stress and affect detection, and sensor-based human activity recognition with a total of seven datasets in different settings. Results demonstrate that DIVERSIFY significantly outperforms other baselines and effectively characterizes the latent distributions by qualitative and quantitative analysis. Code is available at: https://github.com/microsoft/robustlearn.

• The paper presents Diversify, a min-max adversarial framework that iteratively identifies and minimizes worst-case latent distribution gaps in time series data.
• It uses pseudo domain-class labels and self-supervised pseudo-labeling to update features and achieve robust domain-invariant representation learning.
• Experiments show Diversify outperforms state-of-the-art methods in EMG, speech, and HAR datasets, improving accuracy by up to 4.3%.

Out-of-distribution Representation Learning for Time Series Classification

The paper focuses on the complex problem of time series classification, specifically addressing the challenges posed by the non-stationary nature of time series data. The proposed solution, labeled as "Diversify," aims to enhance out-of-distribution (OOD) representation learning by targeting latent dynamic distributions inherent within time series data.

Methodology Overview

The core innovation of this paper is a min-max adversarial framework designed to tackle the OOD problem by characterizing and leveraging latent distributional shifts within the training data. The methodology can be distilled into a few key components:

1. Iterative Process: Diversify iteratively identifies the 'worst-case' latent distribution scenario through adversarial training and subsequently reduces the distributional gap between these latent distributions.
2. Fine-grained Feature Update: Utilizing pseudo domain-class labels, the method updates its features, taking into account both domain and class variations, leading to a more nuanced feature extraction.
3. Latent Distribution Characterization: This step involves self-supervised pseudo-labeling to identify domains without predefined labels, maximizing segment-wise distribution gaps to preserve the diversity of latent distributions.
4. Domain-invariant Representation Learning: By employing adversarial training inspired by DANN, the method minimizes distribution divergence across the characterized latent domains, enhancing the generalization to unseen data.

Experimental Results

The paper provides a thorough evaluation across several challenging datasets, including gesture recognition (EMG), speech commands, wearable stress and affect detection (WESAD), and sensor-based human activity recognition (HAR). Diversify consistently demonstrates superior performance over existing OOD methods such as GILE and AdaRNN, particularly emphasizing its capability to handle complex temporal and spatial distribution shifts.

• EMG Data: Diversify achieved an average improvement of 4.3% over the next best method, highlighted by its ability to tackle varying EMG signals' distribution shifts.
• Speech and Wearable Stress Recognition: The framework consistently outperformed others, displaying robustness across domains, even with varying models.
• HAR: Diversify excelled in cross-person, cross-position, and cross-dataset generalization tasks, showcasing its strength in mitigating distributional variances.

Implications and Future Directions

This contribution presents both practical and theoretical advancements. Practically, Diversify offers a versatile, robust solution for OOD problems in time series, which is critical for real-world applications like health monitoring and human activity recognition. Theoretically, the work supplements existing knowledge on domain generalization by framing the problem within a distributional lens.

The potential future directions include automating the determination of latent distribution numbers (K) and extending the framework beyond classification to forecasting applications. Additionally, exploring the applicability of Diversify's principles to other domains such as image and text data could further validate its utility.

Conclusion

The paper effectively advances the field of time series classification by addressing an underexplored area—the dynamic distribution shifts inherent in time-series data. Diversify not only provides a compelling solution to OOD representation learning but also opens new avenues for understanding and tackling distributional challenges across various domains.