Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Foundation Time Series Model: To Synthesize Or Not To Synthesize? (2403.02534v1)

Published 4 Mar 2024 in cs.LG

Abstract: The industry is rich in cases when we are required to make forecasting for large amounts of time series at once. However, we might be in a situation where we can not afford to train a separate model for each of them. Such issue in time series modeling remains without due attention. The remedy for this setting is the establishment of a foundation model. Such a model is expected to work in zero-shot and few-shot regimes. However, what should we take as a training dataset for such kind of model? Witnessing the benefits from the enrichment of NLP datasets with artificially-generated data, we might want to adopt their experience for time series. In contrast to natural language, the process of generation of synthetic time series data is even more favorable because it provides full control of series patterns, time horizons, and number of samples. In this work, we consider the essential question if it is advantageous to train a foundation model on synthetic data or it is better to utilize only a limited number of real-life examples. Our experiments are conducted only for regular time series and speak in favor of leveraging solely the real time series. Moreover, the choice of the proper source dataset strongly influences the performance during inference. When provided access even to a limited quantity of short time series data, employing it within a supervised framework yields more favorable results than training on a larger volume of synthetic data. The code for our experiments is publicly available on Github \url{https://github.com/sb-ai-lab/synthesize_or_not}.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021.
  2. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419–22430, 2021.
  3. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International Conference on Machine Learning, pages 27268–27286. PMLR, 2022.
  4. Data preprocessing and augmentation for multiple short time series forecasting with recurrent neural networks. In 36th international symposium on forecasting, 2016.
  5. The m3-competition: results, conclusions and implications. International journal of forecasting, 16(4):451–476, 2000.
  6. Some recent advances in forecasting and control. Journal of the Royal Statistical Society. Series C (Applied Statistics), 17(2):91–109, 1968.
  7. Forecasting at scale. The American Statistician, 72(1):37–45, 2018.
  8. Block hankel tensor arima for multiple short time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5758–5766, 2020.
  9. Probabilistic forecasting with spline quantile function rnns. In The 22nd international conference on artificial intelligence and statistics, pages 1901–1910. PMLR, 2019.
  10. N-beats: Neural basis expansion analysis for interpretable time series forecasting. In International Conference on Learning Representations, 2019.
  11. Learning fast and slow for online time series forecasting. In The Eleventh International Conference on Learning Representations, 2022.
  12. Online time-series forecasting using spiking reservoir. Neurocomputing, 518:82–94, 2023.
  13. Ofter: An online pipeline for time series forecasting. arXiv preprint arXiv:2304.03877, 2023.
  14. Onenet: Enhancing time series forecasting models under concept drift by online ensembling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023a.
  15. Forecastpfn: Synthetically-trained zero-shot forecasting. arXiv preprint arXiv:2311.01933, 2023.
  16. Meta-learning framework with applications to zero-shot time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9242–9250, 2021.
  17. One fits all: Power general time series analysis by pretrained lm. arXiv preprint arXiv:2302.11939, 2023.
  18. Language models are unsupervised multitask learners.
  19. Toward a foundation model for time series data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4400–4404, 2023.
  20. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728, 2023.
  21. Timegpt-1. arXiv preprint arXiv:2310.03589, 2023.
  22. Distribution of residual autocorrelations in autoregressive-integrated moving average time series models. Journal of the American statistical Association, 65(332):1509–1526, 1970.
  23. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  24. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
  25. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271, 2018.
  26. Micn: Multi-scale local and global context modeling for long-term series forecasting. In The Eleventh International Conference on Learning Representations, 2022a.
  27. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  28. A time series is worth 64 words: Long-term forecasting with transformers. In International Conference on Learning Representations, 2023.
  29. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 35:9881–9893, 2022.
  30. Nhits: Neural hierarchical interpolation for time series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6989–6997, 2023.
  31. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121–11128, 2023.
  32. Tsmixer: An all-mlp architecture for time series forecasting. arXiv preprint arXiv:2303.06053, 2023.
  33. Revisiting long-term time series forecasting: An investigation on linear mapping. arXiv preprint arXiv:2305.10721, 2023.
  34. A survey on time-series pre-trained models. arXiv preprint arXiv:2305.10716, 2023.
  35. Self-supervised learning for time series analysis: Taxonomy, progress, and prospects. arXiv preprint arXiv:2306.10125, 2023b.
  36. Ts2vec: Towards universal representation of time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8980–8987, 2022.
  37. Learning latent seasonal-trend representations for time series forecasting. Advances in Neural Information Processing Systems, 35:38775–38787, 2022b.
  38. Large language models are zero-shot time series forecasters. arXiv preprint arXiv:2310.07820, 2023.
  39. Lag-llama: Towards foundation models for time series forecasting. arXiv preprint arXiv:2310.08278, 2023.
  40. A decoder-only foundation model for time-series forecasting. arXiv preprint arXiv:2310.10688, 2023.
  41. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
  42. Towards generating real-world time series data. In 2021 IEEE International Conference on Data Mining (ICDM), pages 469–478. IEEE Computer Society, 2021.
  43. Transformers can do bayesian inference. In International Conference on Learning Representations, 2021.
  44. Tabpfn: A transformer that solves small tabular classification problems in a second. In The Eleventh International Conference on Learning Representations, 2022.
  45. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  46. Monash time series forecasting archive. arXiv preprint, 2021.
  47. Timesnet: Temporal 2d-variation modeling for general time series analysis. arXiv preprint arXiv:2210.02186, 2022.
  48. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  49. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint arXiv:2310.06625, 2023.
  50. Timesnet: Temporal 2d-variation modeling for general time series analysis. In International Conference on Learning Representations, 2023.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now