Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
175 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

TSGBench: Time Series Generation Benchmark (2309.03755v2)

Published 7 Sep 2023 in cs.LG, cs.AI, and cs.DB

Abstract: Synthetic Time Series Generation (TSG) is crucial in a range of applications, including data augmentation, anomaly detection, and privacy preservation. Although significant strides have been made in this field, existing methods exhibit three key limitations: (1) They often benchmark against similar model types, constraining a holistic view of performance capabilities. (2) The use of specialized synthetic and private datasets introduces biases and hampers generalizability. (3) Ambiguous evaluation measures, often tied to custom networks or downstream tasks, hinder consistent and fair comparison. To overcome these limitations, we introduce \textsf{TSGBench}, the inaugural Time Series Generation Benchmark, designed for a unified and comprehensive assessment of TSG methods. It comprises three modules: (1) a curated collection of publicly available, real-world datasets tailored for TSG, together with a standardized preprocessing pipeline; (2) a comprehensive evaluation measures suite including vanilla measures, new distance-based assessments, and visualization tools; (3) a pioneering generalization test rooted in Domain Adaptation (DA), compatible with all methods. We have conducted comprehensive experiments using \textsf{TSGBench} across a spectrum of ten real-world datasets from diverse domains, utilizing ten advanced TSG methods and twelve evaluation measures. The results highlight the reliability and efficacy of \textsf{TSGBench} in evaluating TSG methods. Crucially, \textsf{TSGBench} delivers a statistical analysis of the performance rankings of these methods, illuminating their varying performance across different datasets and measures and offering nuanced insights into the effectiveness of each method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (89)
  1. How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models. In ICML. 290–306.
  2. Generative Time-series Modeling with Fourier Flows. In ICLR.
  3. A Stitch in Time Saves Nine: Enabling Early Anomaly Detection with Correlation Analysis. In ICDE. 1832–1845.
  4. The UEA multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075 (2018).
  5. Libra: A Benchmark for Time Series Forecasting Methods. In ICPE. 189–200.
  6. Donald J Berndt and James Clifford. 1994. Using dynamic time warping to find patterns in time series. In KDD Workshop. 359–370.
  7. Time Series Domain Adaptation via Sparse Associative Structure Alignment. In AAAI. 6859–6867.
  8. Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles. Proc. VLDB Endow. 15, 3 (2021), 611–623.
  9. Luis Candanedo. 2017. Appliances energy prediction. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5VC8G.
  10. Neural Ordinary Differential Equations. In NeurIPS. 6572–6583.
  11. Towards Spatio- Temporal Aware Traffic Time Series Forecasting. In ICDE. 2900–2913.
  12. William Jay Conover and Ronald L Iman. 1979. On multiple-comparisons procedures. Los Alamos Sci. Lab. Tech. Rep. LA-7677-MS 1 (1979), 14.
  13. The UCR Time Series Archive. IEEE/CAA Journal of Automatica Sinica 6, 6 (2019), 1293–1305.
  14. Modeling Continuous Stochastic Processes with Dynamic Normalizing Flows. In NeurIPS. 7805–7815.
  15. TimeVAE: A Variational Auto-Encoder for Multivariate Time Series Generation. arXiv preprint arXiv:2111.08095 (2021).
  16. Towards backdoor attack on deep learning based time series classification. In ICDE. 1274–1287.
  17. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 (2014).
  18. Density estimation using Real NVP. In ICLR.
  19. Adversarial Audio Synthesis. In ICLR.
  20. Real-valued (medical) time series generation with recurrent conditional gans. arXiv preprint arXiv:1706.02633 (2017).
  21. GP-VAE: Deep Probabilistic Multivariate Time Series Imputation. In AISTATS. 1651–1661.
  22. SOM-VAE: Interpretable Discrete Representation Learning on Time Series. In ICLR.
  23. Unsupervised Scalable Representation Learning for Multivariate Time Series. In NeurIPS. 4652–4663.
  24. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J. Amer. Statist. Assoc. 32, 200 (1937), 675–701.
  25. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation 101, 23 (2000), e215–e220.
  26. Generative adversarial nets. In NIPS. 2672–2680.
  27. A Kernel Method for the Two-Sample-Problem. In NIPS. 513–520.
  28. Xin Dong Baoren Xiao Lei Jiang Hang Lou, Jiajie Tao and Hao Ni. 2023. Evaluation of Time Series Generative Models. https://github.com/DeepIntoStreams/Evaluation-of-Time-Series-Generative-Models.git.
  29. TS-Benchmark: A Benchmark for Time Series Databases. In ICDE. 588–599.
  30. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NIPS. 6629–6640.
  31. Datsing: Data augmented time series forecasting with adversarial domain adaptation. In CIKM. 2061–2064.
  32. Jon Hutchins. 2006. Dodgers Loop Sensor. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C51P50.
  33. Deep learning for time series classification: a review. DMKD 33, 4 (2019), 917–963.
  34. Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series. Proc. VLDB Endow. 14, 11 (2021), 2613–2626.
  35. Time-series Generation by Contrastive Imitation. In NeurIPS. 28968–28982.
  36. A Benchmark Study on Time Series Clustering. Machine Learning with Applications 1 (2020), 100001.
  37. PSA-GAN: Progressive self attention GANs for synthetic time series. In ICLR.
  38. GT-GAN: General Purpose Time Series Synthesis with Generative Adversarial Networks. In NeurIPS. 36999–37010.
  39. MIMIC-III, a freely accessible critical care database. Scientific Data 3, 1 (2016), 1–9.
  40. PATE-GAN: Generating synthetic data with differential privacy guarantees. In ICLR.
  41. Eamonn Keogh and Shruti Kasetty. 2002. On the need for time series data mining benchmarks: a survey and empirical demonstration. In KDD. 102–111.
  42. Neural SDEs as Infinite-Dimensional GANs. In ICML. 5453–5463.
  43. Diederik P. Kingma and Prafulla Dhariwal. 2018. Glow: Generative Flow with Invertible 1x1 Convolutions. In NeurIPS. 10236–10245.
  44. Semi-supervised learning with deep generative models. In NIPS. 3581–3589.
  45. Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 (2013).
  46. Modeling Long-and Short-Term Temporal Patterns with Deep Neural Networks. In SIGIR. 95–104.
  47. Revisiting Time Series Outlier Detection: Definitions and Benchmarks. In NeurIPS datasets and benchmarks track (round 1).
  48. Vector Quantized Time Series Generation with a Bidirectional Prior Model. In AISTATS. 7665–7693.
  49. IPS: Instance Profile for Shapelet Discovery for Time Series Classification. In ICDE. 1781–1793.
  50. Causal Recurrent Variational Autoencoder for Medical Time Series Generation. In AAAI. 8562–8570.
  51. Tts-gan: A transformer-based time-series generative adversarial network. In AIME. 133–143.
  52. Regular Time-series Generation using SGM. arXiv preprint arXiv:2301.08518 (2023).
  53. Using GANs for Sharing Networked Time Series Data: Challenges, Initial Promise, and Open Questions. In IMC. 464–483.
  54. Time-Transformer AAE: Connecting Temporal Convolutional Networks and Transformer for Time Series Generation. (2022).
  55. The M5 competition and the future of human expertise in forecasting. Foresight: The International Journal of Applied Forecasting 60 (2021), 33–37.
  56. The M4 Competition: Results, findings, conclusion and way forward. International Journal of Forecasting 34, 4 (2018), 802–808.
  57. DTAIDistance.
  58. Olof Mogren. 2016. C-RNN-GAN: A continuous recurrent neural network with adversarial training. In Constructive Machine Learning Workshop (CML) at NIPS 2016. 1.
  59. Sig-Wasserstein GANs for time series generation. In Proceedings of the Second ACM International Conference on AI in Finance. 1–8.
  60. Conditional Sig-Wasserstein GANs for Time Series Generation. arXiv preprint arXiv:2006.05421 (2020).
  61. TSGM: A Flexible Framework for Generative Modeling of Synthetic Time Series. arXiv preprint arXiv:2305.11567 (2023).
  62. Alan V Oppenheim. 1999. Discrete-time signal processing. Pearson Education India.
  63. Volume under the surface: a new accuracy evaluation measure for time-series anomaly detection. Proc. VLDB Endow. 15, 11 (2022), 2774–2787.
  64. TSB-UAD: an end-to-end benchmark suite for univariate time-series anomaly detection. Proc. VLDB Endow. 15, 8 (2022), 1697–1711.
  65. Emanuel Parzen. 1963. On spectral analysis with missing observations and amplitude modulation. Sankhyā: The Indian Journal of Statistics, Series A (1963), 383–392.
  66. Towards generating real-world time series data. In ICDM. 469–478.
  67. ADATIME: A Benchmarking Suite for Domain Adaptation on Time Series Data. TKDD (2023), 1–18.
  68. T-CGAN: Conditional Generative Adversarial Network for Data Augmentation in Noisy Time Series with Irregular Sampling. arXiv preprint arXiv:1811.08295 (2018).
  69. Sangeeta Rani and Geeta Sikka. 2012. Recent Techniques of Clustering of Time Series Data: A Survey. International Journal of Computer Applications 52, 15 (2012).
  70. Conditional Loss and Deep Euler Scheme for Time Series Generation. In AAAI, Vol. 36. 8098–8105.
  71. Human Activity Recognition Using Smartphones. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C54S4K.
  72. Oliver Roesler. 2013. EEG Eye State. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C57G7J.
  73. Latent Ordinary Differential Equations for Irregularly-Sampled Time Series. In NeurIPS. 5320–5330.
  74. Generating multivariate time series with COmmon Source CoordInated GAN (COSCI-GAN). In NeurIPS. 32777–32788.
  75. Generalizing DTW to the multi-dimensional case requires an adaptive approach. DMKD 31 (2017), 1–31.
  76. Kaleb E Smith and Anthony O Smith. 2020. Conditional GAN for timeseries generation. arXiv preprint arXiv:2006.16477 (2020).
  77. Padmanaba Srinivasan and William J Knottenbelt. 2022. Time-series Transformer Generative Adversarial Networks. arXiv preprint arXiv:2205.11164 (2022).
  78. Maksim Terpilowski. 2019. scikit-posthocs: Pairwise multiple comparison tests in Python. The Journal of Open Source Software 4, 36 (2019), 1169.
  79. Neural Discrete Representation Learning. In NIPS. 6306–6315.
  80. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing Data using t-SNE. JMLR 9, 11 (2008).
  81. AEC-GAN: Adversarial Error Correction GANs for Auto-Regressive Long Time-Series Generation. In AAAI. 10140–10148.
  82. Quant GANs: deep generation of financial time series. Quantitative Finance 20, 9 (2020), 1419–1440.
  83. Multi-source deep domain adaptation with weak supervision for time-series sensor data. In KDD. 1768–1778.
  84. AutoCTS: Automated correlated time series forecasting. Proc. VLDB Endow. 15, 4 (2021), 971–983.
  85. COT-GAN: Generating Sequential Data via Causal Optimal Transport. In NeurIPS. 8798–8809.
  86. A Multifaceted benchmarking of synthetic electronic health record generation models. Nature Communications 13, 1 (2022), 7609.
  87. Time-series Generative Adversarial Networks. In NeurIPS. 5509–5519.
  88. Forecasting fine-grained air quality based on big data. In KDD. 2267–2276.
  89. Deep Latent State Space Models for Time-Series Generation. In ICML. 42625–42643.
Citations (10)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now