Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
119 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TimesURL: Self-supervised Contrastive Learning for Universal Time Series Representation Learning (2312.15709v1)

Published 25 Dec 2023 in cs.LG

Abstract: Learning universal time series representations applicable to various types of downstream tasks is challenging but valuable in real applications. Recently, researchers have attempted to leverage the success of self-supervised contrastive learning (SSCL) in Computer Vision(CV) and Natural Language Processing(NLP) to tackle time series representation. Nevertheless, due to the special temporal characteristics, relying solely on empirical guidance from other domains may be ineffective for time series and difficult to adapt to multiple downstream tasks. To this end, we review three parts involved in SSCL including 1) designing augmentation methods for positive pairs, 2) constructing (hard) negative pairs, and 3) designing SSCL loss. For 1) and 2), we find that unsuitable positive and negative pair construction may introduce inappropriate inductive biases, which neither preserve temporal properties nor provide sufficient discriminative features. For 3), just exploring segment- or instance-level semantics information is not enough for learning universal representation. To remedy the above issues, we propose a novel self-supervised framework named TimesURL. Specifically, we first introduce a frequency-temporal-based augmentation to keep the temporal property unchanged. And then, we construct double Universums as a special kind of hard negative to guide better contrastive learning. Additionally, we introduce time reconstruction as a joint optimization objective with contrastive learning to capture both segment-level and instance-level information. As a result, TimesURL can learn high-quality universal representations and achieve state-of-the-art performance in 6 different downstream tasks, including short- and long-term forecasting, imputation, classification, anomaly detection and transfer learning.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (48)
  1. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075.
  2. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271.
  3. Mind the Gap when Conditioning Amortised Inference in Sequential Latent-Variable Models. In International Conference on Learning Representations.
  4. Are all negatives created equal in contrastive instance discrimination? ArXiv, abs/2010.06682.
  5. An analysis of inference with the universum. Advances in neural information processing systems, 20.
  6. FrAug: Frequency Domain Augmentation for Time Series Forecasting. arXiv preprint arXiv:2302.09292.
  7. A simple framework for contrastive learning of visual representations. In International conference on machine learning, 1597–1607. PMLR.
  8. A recurrent latent variable model for sequential data. Advances in neural information processing systems, 28.
  9. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica, 6(6): 1293–1305.
  10. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery, 34(5): 1454–1495.
  11. Denton, E. L.; et al. 2017. Unsupervised learning of disentangled representations from video. Advances in neural information processing systems, 30.
  12. Time-Series Representation Learning via Temporal and Contextual Contrasting. In International Joint Conference on Artificial Intelligence.
  13. Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification. arXiv preprint arXiv:2208.06616.
  14. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33: 21271–21284.
  15. Noise-Contrastive Estimation of Unnormalized Statistical Models, with Applications to Natural Image Statistics. Journal of machine learning research, 13(2).
  16. Universum-Inspired Supervised Contrastive Learning. In Web and Big Data: 6th International Joint Conference, APWeb-WAIM 2022, Nanjing, China, November 25–27, 2022, Proceedings, Part II, 459–473. Springer.
  17. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16000–16009.
  18. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33: 21798–21809.
  19. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, 4171–4186.
  20. Structured inference networks for nonlinear state space models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
  21. Similarity Preserving Representation Learning for Time Series Clustering. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, IJCAI’19, 2845–2851. AAAI Press. ISBN 9780999241141.
  22. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. Advances in neural information processing systems, 32.
  23. Non-stationary multivariate time series prediction with selective recurrent neural networks. In Pacific rim international conference on artificial intelligence, 636–649. Springer.
  24. Scinet: Time series modeling and forecasting with sample convolution and interaction. Advances in Neural Information Processing Systems, 35: 5816–5828.
  25. The Time-Sequence Prediction via Temporal and Contextual Contrastive Representation Learning. In Pacific Rim International Conference on Artificial Intelligence.
  26. Non-stationary Transformers: Exploring the Stationarity in Time Series Forecasting. In Advances in Neural Information Processing Systems.
  27. Time Series Contrastive Learning with Information-Aware Augmentations. In Proceedings of the AAAI Conference on Artificial Intelligence.
  28. Learning representations for time series clustering. Advances in neural information processing systems, 32.
  29. TimeNet: Pre-trained deep recurrent neural network for time series classification. arXiv preprint arXiv:1706.08838.
  30. A Benchmark Dataset for Time Series Anomaly Detection. https://yahooresearch.tumblr.com/post/114590420346/a-benchmark-dataset-for-time-series-anomaly.
  31. N-BEATS: Neural basis expansion analysis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437.
  32. Unsupervised learning of sentence embeddings using compositional n-gram features. arXiv preprint arXiv:1703.02507.
  33. Time-series anomaly detection service at microsoft. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, 3009–3017.
  34. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592.
  35. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28.
  36. Anomaly detection in streams with extreme value theory. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 1067–1075.
  37. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. In International Conference on Learning Representations.
  38. Data augmentation of wearable sensor data for parkinson’s disease monitoring using convolutional neural networks. In Proceedings of the 19th ACM international conference on multimodal interaction, 216–220.
  39. Vapnik, V. 2006. Transductive Inference and Semi-Supervised Learning. Semi-Supervised Learning, 453–472.
  40. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, 2794–2802.
  41. CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting. In International Conference on Learning Representations.
  42. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In The Eleventh International Conference on Learning Representations.
  43. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34: 22419–22430.
  44. Unsupervised anomaly detection via variational auto-encoder for seasonal kpis in web applications. In Proceedings of the 2018 world wide web conference, 187–196.
  45. Negative sampling for contrastive representation learning: A review. arXiv preprint arXiv:2206.00212.
  46. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 8980–8987.
  47. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2114–2124.
  48. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, 11106–11115.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Jiexi Liu (7 papers)
  2. Songcan Chen (74 papers)
Citations (20)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now