Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UniCL: A Universal Contrastive Learning Framework for Large Time Series Models (2405.10597v1)

Published 17 May 2024 in cs.LG, cs.AI, and cs.CL

Abstract: Time-series analysis plays a pivotal role across a range of critical applications, from finance to healthcare, which involves various tasks, such as forecasting and classification. To handle the inherent complexities of time-series data, such as high dimensionality and noise, traditional supervised learning methods first annotate extensive labels for time-series data in each task, which is very costly and impractical in real-world applications. In contrast, pre-trained foundation models offer a promising alternative by leveraging unlabeled data to capture general time series patterns, which can then be fine-tuned for specific tasks. However, existing approaches to pre-training such models typically suffer from high-bias and low-generality issues due to the use of predefined and rigid augmentation operations and domain-specific data training. To overcome these limitations, this paper introduces UniCL, a universal and scalable contrastive learning framework designed for pretraining time-series foundation models across cross-domain datasets. Specifically, we propose a unified and trainable time-series augmentation operation to generate pattern-preserved, diverse, and low-bias time-series data by leveraging spectral information. Besides, we introduce a scalable augmentation algorithm capable of handling datasets with varying lengths, facilitating cross-domain pretraining. Extensive experiments on two benchmark datasets across eleven domains validate the effectiveness of UniCL, demonstrating its high generalization on time-series analysis across various fields.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (83)
  1. Chronos: Learning the Language of Time Series. arXiv preprint arXiv:2403.07815 (2024).
  2. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 (2018).
  3. Multi-Patch Prediction: Adapting LLMs for Time Series Representation Learning. arXiv preprint arXiv:2402.04852 (2024).
  4. Llm4ts: Two-stage fine-tuning for time-series forecasting with pre-trained llms. arXiv preprint arXiv:2308.08469 (2023).
  5. Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining. 785–794.
  6. A simple framework for contrastive learning of visual representations. In International conference on machine learning. PMLR, 1597–1607.
  7. Gatgpt: A pre-trained large language model with graph attention network for spatiotemporal imputation. arXiv preprint arXiv:2311.14332 (2023).
  8. Timemae: Self-supervised representations of time series with decoupled masked autoencoders. arXiv preprint arXiv:2303.00320 (2023).
  9. The UCR time series archive. IEEE/CAA Journal of Automatica Sinica 6, 6 (2019), 1293–1305.
  10. ROCKET: exceptionally fast and accurate time series classification using random convolutional kernels. Data Mining and Knowledge Discovery 34, 5 (2020), 1454–1495.
  11. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
  12. Simmtm: A simple pre-training framework for masked time-series modeling. Advances in Neural Information Processing Systems 36 (2024).
  13. Time-Series Representation Learning via Temporal and Contextual Contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21. 2352–2359.
  14. Self-supervised time series representation learning by inter-intra relational reasoning. arXiv preprint arXiv:2011.13548 (2020).
  15. Unsupervised scalable representation learning for multivariate time series. Advances in neural information processing systems 32 (2019).
  16. UniTS: Building a Unified Time Series Model. arXiv preprint arXiv:2403.00131 (2024).
  17. Monash Time Series Forecasting Archive. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  18. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013).
  19. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  20. Flowformer: A transformer architecture for optical flow. In European conference on computer vision. Springer, 668–685.
  21. Deep learning for time series classification: a review. Data mining and knowledge discovery 33, 4 (2019), 917–963.
  22. Time-LLM: Time series forecasting by reprogramming large language models. In International Conference on Learning Representations (ICLR).
  23. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXiv:2310.10196 (2023).
  24. Contrastive self-supervised learning for sensor-based human activity recognition. In 2021 IEEE International Joint Conference on Biometrics (IJCB). IEEE, 1–8.
  25. Clocs: Contrastive learning of cardiac signals across space, time, and patients. In International Conference on Machine Learning. PMLR, 5606–5615.
  26. Deep temporal convolution network for time series classification. Sensors 21, 2 (2021), 603.
  27. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval. 95–104.
  28. Ti-mae: Self-supervised masked time series autoencoders. arXiv preprint arXiv:2301.08871 (2023).
  29. Contrastive shapelet learning for unsupervised multivariate time series representation learning. arXiv preprint arXiv:2305.18888 (2023).
  30. Bryan Lim and Stefan Zohren. 2021. Time-series forecasting with deep learning: a survey. Philosophical Transactions of the Royal Society A 379, 2194 (2021), 20200209.
  31. Large Language Model Guided Knowledge Distillation for Time Series Anomaly Detection. arXiv preprint arXiv:2401.15123 (2024).
  32. LSTPrompt: Large Language Models as Zero-Shot Time Series Forecasters by Long-Short-Term Prompting. arXiv preprint arXiv:2402.16132 (2024).
  33. Jiexi Liu and Songcan Chen. 2024. Timesurl: Self-supervised contrastive learning for universal time series representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 13918–13926.
  34. Taming Pre-trained LLMs for Generalised Time Series Forecasting via Cross-modal Knowledge Distillation. arXiv preprint arXiv:2403.07300 (2024).
  35. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International conference on learning representations.
  36. Unitime: A language-empowered unified model for cross-domain time series forecasting. arXiv preprint arXiv:2310.09751 (2023).
  37. AutoTimes: Autoregressive Time Series Forecasters via Large Language Models. arXiv preprint arXiv:2402.02370 (2024).
  38. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems 35 (2022), 9881–9893.
  39. Time series contrastive learning with information-aware augmentations. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 4534–4542.
  40. A survey on time-series pre-trained models. arXiv preprint arXiv:2305.10716 (2023).
  41. MHCCL: masked hierarchical cluster-wise contrastive learning for multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 9153–9161.
  42. Shikai Qiu Nate Gruver, Marc Finzi and Andrew Gordon Wilson. 2023. Large Language Models Are Zero Shot Time Series Forecasters. In Advances in Neural Information Processing Systems.
  43. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In International Conference on Learning Representations.
  44. Saeed Amizadeh Nikolay Laptev, Y. B. 2015. A Benchmark Dataset for Time Series Anomaly Detection. https://yahooresearch.tumblr.com/post/114590420346/a-benchmark-dataset-for-time-series-anomaly
  45. OpenAI. 2022. OpenAI: Introducing ChatGPT. https://openai.com/blog/chatgpt
  46. Contrastive Learning for Unsupervised Domain Adaptation of Time Series. ICLR (2023).
  47. Contrastive learning based self-supervised time-series analysis. Applied Soft Computing 117 (2022), 108397.
  48. TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods. arXiv preprint arXiv:2403.20150 (2024).
  49. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 8748–8763.
  50. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  51. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. http://jmlr.org/papers/v21/20-074.html
  52. Sequential multi-dimensional self-supervised learning for clinical time series. In International Conference on Machine Learning. PMLR, 28531–28548.
  53. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278 (2023).
  54. Time-Series Anomaly Detection Service at Microsoft. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19). Association for Computing Machinery, 3009–3017. https://doi.org/10.1145/3292500.3330680
  55. Recent advances in recurrent neural networks. arXiv preprint arXiv:1801.01078 (2017).
  56. A theoretical analysis of contrastive unsupervised representation learning. In International Conference on Machine Learning. PMLR, 5628–5637.
  57. Financial time series forecasting with deep learning: A systematic literature review: 2005–2019. Applied soft computing 90 (2020), 106181.
  58. TEST: Text prototype aligned embedding to activate LLM’s ability for time series. arXiv preprint arXiv:2308.08241 (2023).
  59. Xue Wang Liang Sun Rong Jin Tian Zhou, Peisong Niu. 2023. One Fits All: Power General Time Series Analysis by Pretrained LM. In NeurIPS.
  60. Unsupervised Representation Learning for Time Series with Temporal Neighborhood Coding. In International Conference on Learning Representations. https://openreview.net/forum?id=8qDwejCuCN
  61. LLaMA: Open and Efficient Foundation Language Models. ArXiv abs/2302.13971 (2023).
  62. Universal Time-Series Representation Learning: A Survey. arXiv preprint arXiv:2401.03717 (2024).
  63. Contrast everything: A hierarchical contrastive framework for medical time-series. Advances in Neural Information Processing Systems 36 (2024).
  64. Robust time series analysis and applications: An industrial perspective. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4836–4837.
  65. Transformers in time series: A survey. arXiv preprint arXiv:2202.07125 (2022).
  66. CoST: Contrastive Learning of Disentangled Seasonal-Trend Representations for Time Series Forecasting. In International Conference on Learning Representations. https://openreview.net/forum?id=PilZY3omXV2
  67. ETSformer: Exponential Smoothing Transformers for Time-series Forecasting. (2022). https://arxiv.org/abs/2202.01381
  68. TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis. In International Conference on Learning Representations.
  69. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems 34 (2021), 22419–22430.
  70. Hao Xue and Flora D Salim. 2023. Promptcast: A new prompt-based learning paradigm for time series forecasting. IEEE Transactions on Knowledge and Data Engineering (2023).
  71. Toward a foundation model for time series data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management. 4400–4404.
  72. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 8980–8987.
  73. Are transformers effective for time series forecasting?. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37. 11121–11128.
  74. A transformer-based framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 2114–2124.
  75. Self-Supervised Learning for Time Series Analysis: Taxonomy, Progress, and Prospects. arXiv preprint arXiv:2306.10125 (2023).
  76. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).
  77. Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures. arXiv preprint arXiv:2207.01186 (2022).
  78. Self-supervised contrastive pre-training for time series via time-frequency consistency. Advances in Neural Information Processing Systems 35 (2022), 3988–4003.
  79. COSTA: covariance-preserving feature augmentation for graph contrastive learning. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 2524–2534.
  80. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics 28, 1 (2017), 162–169.
  81. SimTS: Rethinking Contrastive Representation Learning for Time Series Forecasting. ArXiv abs/2303.18205 (2023).
  82. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 11106–11115.
  83. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning. PMLR, 27268–27286.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Jiawei Li (116 papers)
  2. Jingshu Peng (7 papers)
  3. Haoyang Li (95 papers)
  4. Lei Chen (485 papers)
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets