Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
144 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
46 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Kernel-U-Net: Multivariate Time Series Forecasting using Custom Kernels (2401.01479v3)

Published 3 Jan 2024 in cs.LG

Abstract: Time series forecasting task predicts future trends based on historical information. Transformer-based U-Net architectures, despite their success in medical image segmentation, have limitations in both expressiveness and computation efficiency in time series forecasting as evidenced in YFormer. To tackle these challenges, we introduce Kernel-U-Net, a flexible and kernel-customizable U-shape neural network architecture. The kernel-U-Net encoder compresses the input series into latent vectors, and its symmetric decoder subsequently expands these vectors into output series. Specifically, Kernel-U-Net separates the procedure of partitioning input time series into patches from kernel manipulation, thereby providing the convenience of customized executing kernels. Our method offers two primary advantages: 1) Flexibility in kernel customization to adapt to specific datasets; and 2) Enhanced computational efficiency, with the complexity of the Transformer layer reduced to linear. Experiments on seven real-world datasets, demonstrate that Kernel-U-Net's performance either exceeds or meets that of the existing state-of-the-art model in the majority of cases in channel-independent settings. The source code for Kernel-U-Net will be made publicly available for further research and application.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Long horizon anomaly prediction in multivariate time series with causal autoencoders. PHM Society European Conference, 7(1):21–31, 2022.
  2. Efficient lossy compression for iot using sz and reconstruction with 1d u-net. Mob. Netw. Appl., 27(3):984–996, jun 2022.
  3. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  4. Breast mass segmentation in ultrasound with selective kernel u-net convolutional neural network. Biomedical Signal Processing and Control, 61:102027, 2020.
  5. Swin-unet: Unet-like pure transformer for medical image segmentation. In Leonid Karlinsky, Tomer Michaeli, and Ko Nishino, editors, Computer Vision – ECCV 2022 Workshops, pages 205–218. Springer Nature Switzerland, 2023.
  6. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition(CVPR), volume 00, pages 248–255, 06 2009.
  7. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  8. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  9. Hierarchical recurrent neural network for skeleton based action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1110–1118, 2015.
  10. Multivariate time-series classification with hierarchical variational graph pooling. Neural Networks, 154:481–490, 2022.
  11. Deep gated recurrent and convolutional network hybrid model for univariate time series classification. International Journal of Advanced Computer Science and Applications, 10(5), 2019.
  12. Multi-level u-net network for image super-resolution reconstruction. Displays, 73:102192, 2022.
  13. Temporal convolutional neural (TCN) network for an effective weather forecasting using time-series data from the local weather station. Soft Computing, 24(21):16453–16482, 2020.
  14. Multivariate time-series classification of sleep patterns using a hybrid deep learning architecture. In 2017 IEEE 19th International Conference on e-Health Networking, Applications and Services (Healthcom), pages 1–6, 2017.
  15. UNet 3+: A full-scale connected UNet for medical image segmentation. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1055–1059, 2020. ISSN: 2379-190X.
  16. HitAnomaly: Hierarchical transformers for anomaly detection in system log. IEEE Transactions on Network and Service Management, 17(4):2064–2076, 2020. Conference Name: IEEE Transactions on Network and Service Management.
  17. U-net vs transformer: Is u-net outdated in medical image registration? In Chunfeng Lian, Xiaohuan Cao, Islem Rekik, Xuanang Xu, and Zhiming Cui, editors, Machine Learning in Medical Imaging, pages 151–160. Springer Nature Switzerland, 2022.
  18. Bidirectional LSTM-RNN-based hybrid deep learning frameworks for univariate time series classification. The Journal of Supercomputing, 77(7):7021–7045, 2021.
  19. Time series forecasting using hybrid ARIMA and ANN models based on DWT decomposition. Procedia Computer Science, 48:173–179, 2015.
  20. Reversible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2021.
  21. Short-term residential load forecasting based on LSTM recurrent neural network. IEEE Transactions on Smart Grid, PP:1–1, 2017.
  22. HDLTex: Hierarchical deep learning for text classification. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pages 364–371, 2017.
  23. Modeling long- and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’18, page 95–104, New York, NY, USA, 2018. Association for Computing Machinery.
  24. Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting, page 11. Curran Associates Inc., Red Hook, NY, USA, 2019.
  25. Anomaly detection for time series using VAE-LSTM hybrid model. In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4322–4326. IEEE, 2020.
  26. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992–10002. IEEE, 2021.
  27. Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting. In International Conference on Learning Representations, 2022.
  28. Non-stationary transformers: Exploring the stationarity in time series forecasting. Advances in Neural Information Processing Systems, 2022.
  29. U-net inspired transformer architecture for far horizon time series forecasting. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2022, Grenoble, France, September 19–23, 2022, Proceedings, Part VI, page 36–52, Berlin, Heidelberg, 2023. Springer-Verlag.
  30. A time series is worth 64 words: Long-term forecasting with transformers. International Conference on Learning Representations, 2023.
  31. U-time: A fully convolutional network for time series segmentation applied to sleep staging. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  32. Multi-site solar power forecasting using gradient boosted regression trees. Solar Energy, 150:423–436, 2017.
  33. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241. Springer International Publishing, 2015.
  34. A RNN based time series approach for forecasting turkish electricity load. In 2018 26th Signal Processing and Communications Applications Conference (SIU), pages 1–4, 2018.
  35. Instance normalization: The missing ingredient for fast stylization, 2017.
  36. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  37. Transformers in time series: A survey. In International Joint Conference on Artificial Intelligence(IJCAI), 2023.
  38. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 22419–22430. Curran Associates, Inc., 2021.
  39. Semantic relation classification via hierarchical recurrent neural network with attention. In Yuji Matsumoto and Rashmi Prasad, editors, Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1254–1263. The COLING 2016 Organizing Committee, 2016.
  40. Weighted res-UNet for high-quality retina vessel segmentation. In 2018 9th International Conference on Information Technology in Medicine and Education (ITME), pages 327–331, 2018. ISSN: 2474-3828.
  41. Are transformers effective for time series forecasting? In Proceedings of the AAAI Conference on Artificial Intelligence, 2023.
  42. Practical blind image denoising via swin-conv-UNet and data synthesis. Mach. Intell. Res., 2023.
  43. Random erasing data augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
  44. UNet++: A nested u-net architecture for medical image segmentation. In Danail Stoyanov, Zeike Taylor, Gustavo Carneiro, Tanveer Syeda-Mahmood, Anne Martel, Lena Maier-Hein, João Manuel R.S. Tavares, Andrew Bradley, João Paulo Papa, Vasileios Belagiannis, Jacinto C. Nascimento, Zhi Lu, Sailesh Conjeti, Mehdi Moradi, Hayit Greenspan, and Anant Madabhushi, editors, Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Lecture Notes in Computer Science, pages 3–11. Springer International Publishing, 2018.
  45. Informer: Beyond efficient transformer for long sequence time-series forecasting. In The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference, volume 35, pages 11106–11115. AAAI Press, 2021.
  46. FEDformer: Frequency enhanced decomposed transformer for long-term series forecasting. In Proceedings of the 39th International Conference on Machine Learning, pages 27268–27286. PMLR, 2022. ISSN: 2640-3498.
  47. 3d u-net: Learning dense volumetric segmentation from sparse annotation. In International Conference on Medical Image Computing and Computer-Assisted Intervention, 2016.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now