Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Utilizing Image Transforms and Diffusion Models for Generative Modeling of Short and Long Time Series (2410.19538v1)

Published 25 Oct 2024 in cs.LG and cs.CV

Abstract: Lately, there has been a surge in interest surrounding generative modeling of time series data. Most existing approaches are designed either to process short sequences or to handle long-range sequences. This dichotomy can be attributed to gradient issues with recurrent networks, computational costs associated with transformers, and limited expressiveness of state space models. Towards a unified generative model for varying-length time series, we propose in this work to transform sequences into images. By employing invertible transforms such as the delay embedding and the short-time Fourier transform, we unlock three main advantages: i) We can exploit advanced diffusion vision models; ii) We can remarkably process short- and long-range inputs within the same framework; and iii) We can harness recent and established tools proposed in the time series to image literature. We validate the effectiveness of our method through a comprehensive evaluation across multiple tasks, including unconditional generation, interpolation, and extrapolation. We show that our approach achieves consistently state-of-the-art results against strong baselines. In the unconditional generation tasks, we show remarkable mean improvements of 58.17% over previous diffusion models in the short discriminative score and 132.61% in the (ultra-)long classification scores. Code is at https://github.com/azencot-group/ImagenTime.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (104)
  1. J. Allen. Short term spectral analysis, synthesis, and modification by discrete Fourier transform. IEEE transactions on acoustics, speech, and signal processing, 25(3):235–238, 1977.
  2. A unified approach to short-time Fourier analysis and synthesis. Proceedings of the IEEE, 65(11):1558–1564, 1977.
  3. S.-I. Amari. Learning patterns and pattern sequences by self-organizing nets of threshold elements. IEEE Transactions on computers, 100(11):1197–1206, 1972.
  4. B. D. Anderson. Reverse-time diffusion equation models. Stochastic Processes and their Applications, 12(3):313–326, 1982.
  5. Unitary evolution recurrent neural networks. In International conference on machine learning. PMLR, 2016.
  6. Forecasting sequential data using consistent Koopman autoencoders. In International Conference on Machine Learning, pages 475–485. PMLR, 2020.
  7. Consistent dynamic mode decomposition. SIAM Journal on Applied Dynamical Systems, 18(3):1565–1585, 2019.
  8. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12945, 2024.
  9. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 1994.
  10. Generative modeling of graphs via joint diffusion of node and edge attributes. arXiv preprint arXiv:2402.04046, 2024.
  11. Multifactor sequential disentanglement via structured Koopman autoencoders. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
  12. Quick and easy time series generation with established image-based GANs. arXiv preprint arXiv:1902.05624, 2019.
  13. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  14. Data driven prediction models of energy use of appliances in a low-energy house. Energy and buildings, 140:81–97, 2017.
  15. Wavegrad: Estimating gradients for waveform generation. In 9th International Conference on Learning Representations, ICLR, 2021.
  16. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
  17. ResGrad: Residual denoising diffusion probabilistic models for text to speech. arXiv preprint arXiv:2212.14518, 2022.
  18. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
  19. On the constrained time-series generation problem. Advances in Neural Information Processing Systems, 36, 2024.
  20. GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32, 2019.
  21. Modeling continuous stochastic processes with dynamic normalizing flows. Advances in Neural Information Processing Systems, 33:7805–7815, 2020.
  22. TimeVAE: a variational auto-encoder for multivariate time series generation. arXiv preprint arXiv:2111.08095, 2021.
  23. Adversarial audio synthesis. In 7th International Conference on Learning Representations, ICLR, 2019.
  24. Lipschitz recurrent neural networks. In International Conference on Learning Representations, 2021.
  25. Real-valued (medical) time series generation with recurrent conditional GANs. arXiv preprint arXiv:1706.02633, 2017.
  26. Empirical mode decomposition as a filter bank. IEEE signal processing letters, 11(2):112–114, 2004.
  27. GP-VAE: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pages 1651–1661. PMLR, 2020.
  28. One-step diffusion distillation via deep equilibrium models. Advances in Neural Information Processing Systems, 36, 2024.
  29. Monash time series forecasting archive. arXiv preprint arXiv:2105.06643, 2021.
  30. It’s raw! audio generation with state-space models. In International Conference on Machine Learning, pages 7616–7633. PMLR, 2022.
  31. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
  32. Professor forcing: A new algorithm for training recurrent networks. Advances in neural information processing systems, 29, 2016.
  33. A. Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.
  34. Automatic speech recognition: An auditory perspective. Speech processing in the auditory system, pages 309–338, 2004.
  35. D. Griffin and J. Lim. Signal estimation from modified short-time Fourier transform. IEEE Transactions on acoustics, speech, and signal processing, 32(2):236–243, 1984.
  36. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021.
  37. X. Guo and L. Zhao. A systematic survey on deep generative models for graph generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5370–5390, 2022.
  38. Classification of time-series images using deep convolutional neural networks. In Tenth international conference on machine vision (ICMV 2017), volume 10696, pages 242–249. SPIE, 2018.
  39. J. Hellermann and S. Lessmann. Leveraging image-based generative adversarial networks for time series generation. arXiv preprint arXiv:2112.08060, 2021.
  40. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  41. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1–33, 2022.
  42. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  43. J. J. Hopfield. Neural networks and physical systems with emergent collective computational abilities. Proceedings of the national academy of sciences, 1982.
  44. GT-GAN: General purpose time series synthesis with generative adversarial networks. Advances in Neural Information Processing Systems, 35:36999–37010, 2022.
  45. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565–26577, 2022.
  46. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401–4410, 2019.
  47. I. Kaufman and O. Azencot. Data representations’ study of latent image manifolds. In International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 15928–15945. PMLR, 2023.
  48. I. Kaufman and O. Azencot. Analyzing deep transformer models for time series forecasting via manifold learning. Transactions on Machine Learning Research, TMLR, 2024.
  49. I. Kaufman and O. Azencot. First-order manifold data augmentation for regression learning. In Forty-first International Conference on Machine Learning, ICML, 2024.
  50. Neural SDEs as infinite-dimensional GANs. In International conference on machine learning, pages 5453–5463. PMLR, 2021.
  51. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696–6707, 2020.
  52. DiffWave: A versatile diffusion model for audio synthesis. In 9th International Conference on Learning Representations, ICLR, 2021.
  53. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 95–104, 2018.
  54. Causal recurrent variational autoencoder for medical time series generation. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI, pages 8562–8570, 2023.
  55. SP-GAN: Sphere-guided 3d shape generation and manipulation. ACM Transactions on Graphics (TOG), 40(4):1–12, 2021.
  56. Time series as images: Vision transformer for irregularly sampled time series. Advances in Neural Information Processing Systems, 36, 2023.
  57. TSGM: Regular and irregular time-series generation using score-based generative models. openreview.com, 2023.
  58. AudioLDM: Text-to-audio generation with latent diffusion models. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 21450–21474. PMLR, 23–29 Jul 2023.
  59. Are GANs created equal? a large-scale study. Advances in neural information processing systems, 31, 2018.
  60. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11461–11471, 2022.
  61. Accuair: Winning solution to air quality prediction for KDD cup 2018. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1842–1850, 2019.
  62. W. S. McCulloch and W. Pitts. A logical calculus of the ideas immanent in nervous activity. The bulletin of mathematical biophysics, 1943.
  63. Long-term daily climate records from stations across the contiguous united states, 2015.
  64. O. Mogren. C-RNN-GAN: Continuous recurrent neural networks with adversarial training. arXiv preprint arXiv:1611.09904, 2016.
  65. I. Naiman and O. Azencot. An operator theoretic approach for analyzing sequence neural networks. In Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI, pages 9268–9276. AAAI Press, 2023.
  66. Generative modeling of regular and irregular time series data via Koopman VAEs. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
  67. Time weaver: A conditional time series generation model. In Forty-first International Conference on Machine Learning, ICML, 2024.
  68. A time series is worth 64 words: Long-term forecasting with transformers. In The Eleventh International Conference on Learning Representations, ICLR, 2023.
  69. Permutation invariant graph generation via score-based generative modeling. In International Conference on Artificial Intelligence and Statistics, pages 4474–4484. PMLR, 2020.
  70. On the difficulty of training recurrent neural networks. In International conference on machine learning. PMLR, 2013.
  71. Grad-TTS: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599–8608. PMLR, 2021.
  72. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
  73. Learning physics for unveiling hidden earthquake ground motions via conditional generative modeling. arXiv preprint arXiv:2407.15089, 2024.
  74. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  75. Latent ordinary differential equations for irregularly-sampled time series. Advances in neural information processing systems, 32, 2019.
  76. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
  77. T. Salimans and J. Ho. Progressive distillation for fast sampling of diffusion models. In The Tenth International Conference on Learning Representations, ICLR, 2022.
  78. Modeling irregular time series with continuous recurrent units. In International Conference on Machine Learning, pages 19388–19405. PMLR, 2022.
  79. L. F. Shampine and S. Thompson. Stiff systems. Scholarpedia, 2(3):2855, 2007.
  80. Predicting in-hospital mortality of ICU patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology, pages 245–248. IEEE, 2012.
  81. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  82. Y. Song and P. Dhariwal. Improved techniques for training consistency models. In 12th International Conference on Learning Representations, ICLR, 2024.
  83. Consistency models. In International Conference on Machine Learning, ICML, volume 202 of Proceedings of Machine Learning Research, pages 32211–32252. PMLR, 2023.
  84. Y. Song and S. Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
  85. Y. Song and S. Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:12438–12448, 2020.
  86. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
  87. F. Takens. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980: proceedings of a symposium held at the University of Warwick 1979/80, pages 366–381. Springer, 2006.
  88. CSDI: Conditional score-based diffusion models for probabilistic time series imputation. Advances in Neural Information Processing Systems, 34:24804–24816, 2021.
  89. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pages 5026–5033. IEEE, 2012.
  90. A. Vahdat and J. Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33:19667–19679, 2020.
  91. WaveNet: A generative model for raw audio. In The 9th ISCA Speech Synthesis Workshop, SSW, page 125. ISCA, 2016.
  92. L. Van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
  93. Attention is all you need. Advances in neural information processing systems, 2017.
  94. M. Vetterli and C. Herley. Wavelets and filter banks: Theory and design. IEEE transactions on signal processing, 40(9):2207–2232, 1992.
  95. Z. Wang and T. Oates. Imaging time-series to improve classification and imputation. In Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI, pages 3939–3945. AAAI Press, 2015.
  96. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in neural information processing systems, 34:22419–22430, 2021.
  97. SwinGNN: Rethinking permutation invariance in diffusion models for graph generation. Trans. Mach. Learn. Res., 2024.
  98. ODE2VAE: Deep generative second order ODEs with bayesian neural networks. Advances in Neural Information Processing Systems, 32, 2019.
  99. Time-series generative adversarial networks. Advances in neural information processing systems, 32, 2019.
  100. X. Yuan and Y. Qiao. Diffusion-TS: Interpretable diffusion for general time series generation. In The Twelfth International Conference on Learning Representations, ICLR, 2024.
  101. Are transformers effective for time series forecasting? In Proceedings of the AAAI conference on artificial intelligence, volume 37, pages 11121–11128, 2023.
  102. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11106–11115, 2021.
  103. Deep latent state space models for time-series generation. In International Conference on Machine Learning, pages 42625–42643. PMLR, 2023.
  104. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting. In International conference on machine learning, pages 27268–27286. PMLR, 2022.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets