Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash 93 tok/s
Gemini 2.5 Pro 52 tok/s Pro
GPT-5 Medium 17 tok/s
GPT-5 High 14 tok/s Pro
GPT-4o 97 tok/s
GPT OSS 120B 455 tok/s Pro
Kimi K2 194 tok/s Pro
2000 character limit reached

United We Pretrain, Divided We Fail! Representation Learning for Time Series by Pretraining on 75 Datasets at Once (2402.15404v1)

Published 23 Feb 2024 in cs.LG

Abstract: In natural language processing and vision, pretraining is utilized to learn effective representations. Unfortunately, the success of pretraining does not easily carry over to time series due to potential mismatch between sources and target. Actually, common belief is that multi-dataset pretraining does not work for time series! Au contraire, we introduce a new self-supervised contrastive pretraining approach to learn one encoding from many unlabeled and diverse time series datasets, so that the single learned representation can then be reused in several target domains for, say, classification. Specifically, we propose the XD-MixUp interpolation method and the Soft Interpolation Contextual Contrasting (SICC) loss. Empirically, this outperforms both supervised training and other self-supervised pretraining methods when finetuning on low-data regimes. This disproves the common belief: We can actually learn from multiple time series datasets, even from 75 at once.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2019.
  2. PAITS: Pretraining and augmentation for irregularly-sampled time series. ArXiv:2308.13703, 2023.
  3. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M.-F., and Lin, H.-T. (eds.), Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
  4. Multi-view self-supervised learning for multivariate variable-channel time series. ArXiv:2307.09614, 2023.
  5. A simple framework for contrastive learning of visual representations. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
  6. TimeMAE: Self-supervised representations of time series with decoupled masked autoencoders. ArXiv:2303.00320, 2023.
  7. The UCR time series archive. IEEE/CAA Journal Automatica Sinica (JAS), 2019.
  8. A Cluster Separation Measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1979.
  9. Demšar, J. Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research (JMLR), 2006.
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
  11. SimMTM: A simple pre-training framework for masked time-series modeling. ArXiv:2302.00861, 2023.
  12. Time-Series Representation Learning via Temporal and Contextual Contrasting. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2020.
  13. Unsupervised scalable representation learning for multivariate time series. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2019.
  14. Homogeneous Transfer Active Learning for Time Series Classification. In Proceedings of the International Conference on Machine Learning and Applications (ICMLA), 2021.
  15. Large Language Models Are Zero-Shot Time Series Forecasters. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
  16. An empirical survey of data augmentation for time series classification with neural networks. PLOS ONE, 2021.
  17. Time-LLM: Time Series Forecasting by Reprogramming Large Language Models. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
  18. ConvTimeNet: A pre-trained deep convolutional neural network for time series classification. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2019.
  19. CLOCS: Contrastive learning of cardiac signals across space, time, and patients. In Meila, M. and Zhang, T. (eds.), Proceedings of the International Conference on Machine Learning (ICML), 2021.
  20. Ti-MAE: Self-supervised masked time series autoencoders. ArXiv:2301.08871, 2023.
  21. A survey on time-series pre-trained models. ArXiv:2305.10716, 2023.
  22. The M4 Competition: Results, findings, conclusion and way forward. International Journal of Forecasting (IJF), 2018.
  23. A time series is worth 64 words: Long-term forecasting with transformers. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
  24. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2019.
  25. High-Resolution Image Synthesis with Latent Diffusion Models. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  26. Dynamic programming algorithm optimization for spoken word recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP), 1978.
  27. Self-Supervised Pre-training for Time Series Classification. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2021.
  28. Sohn, K. Improved Deep Metric Learning with Multi-class N-pair Loss Objective. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2016.
  29. Exploring contrastive learning in human activity recognition for healthcare. ArXiv:2011.11542, 2020.
  30. On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2019.
  31. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
  32. Lightweight, pre-trained transformers for remote sensing timeseries. ArXiv:2304.14065, 2023.
  33. Representation learning with contrastive predictive coding. ArXiv:1807.03748, 2018.
  34. Attention is All you Need. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2017.
  35. Time series classification from scratch with deep neural networks: A strong baseline. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2017.
  36. Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series. Pattern Recognition Letters, 2022.
  37. Voice2Series: Reprogramming Acoustic Models for Time Series Classification. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
  38. LEAVES: Learning views for time-series data in contrastive learning. ArXiv:2210.07340, 2022.
  39. TS2Vec: Towards Universal Representation of Time Series. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
  40. Are Transformers Effective for Time Series Forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2023.
  41. mixup: Beyond Empirical Risk Minimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
  42. Self-supervised contrastive pre-training for time series via time-frequency consistency. In Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2022.
  43. One Fits All: Power General Time Series Analysis by Pretrained LM. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Ai Generate Text Spark Streamline Icon: https://streamlinehq.com

Paper Prompts

Sign up for free to create and run prompts on this paper using GPT-5.

List Streamline Icon: https://streamlinehq.com Explore 10 Community Prompts
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets