A Survey of Deep Learning and Foundation Models for Time Series Forecasting (2401.13912v1)

Abstract: Deep Learning has been successfully applied to many application domains, yet its advantages have been slow to emerge for time series forecasting. For example, in the well-known Makridakis (M) Competitions, hybrids of traditional statistical or machine learning techniques have only recently become the top performers. With the recent architectural advances in deep learning being applied to time series forecasting (e.g., encoder-decoders with attention, transformers, and graph neural networks), deep learning has begun to show significant advantages. Still, in the area of pandemic prediction, there remain challenges for deep learning models: the time series is not long enough for effective training, unawareness of accumulated scientific knowledge, and interpretability of the model. To this end, the development of foundation models (large deep learning models with extensive pre-training) allows models to understand patterns and acquire knowledge that can be applied to new related problems before extensive training data becomes available. Furthermore, there is a vast amount of knowledge available that deep learning models can tap into, including Knowledge Graphs and LLMs fine-tuned with scientific domain knowledge. There is ongoing research examining how to utilize or inject such knowledge into deep learning models. In this survey, several state-of-the-art modeling techniques are reviewed, and suggestions for further work are provided.

Overview of Deep Learning and Foundation Models in Time Series Forecasting

Introduction to Deep Learning in Time Series Forecasting

Time series forecasting is a challenging area of research that benefits significantly from the advancements in deep learning. Recently, a particular interest has been devoted to evaluating the extent to which deep learning models, especially novel architectures such as Transformers and graph neural networks (GNNs), have outperformed traditional approaches. This paper presents a comprehensive survey of the latest developments in deep learning and their implications for time series forecasting, highlighting encoder-decoder structures, attention mechanisms, and GNNs.

Advances in Model Architectures

Deep learning's transformative impact on time series forecasting echoes the trends observed in other domains. State-of-the-art models now incorporate Transformers, known for their self-attention capabilities. However, the growth of models expressly designed for pandemic prediction poses unique challenges in terms of interpretability and adaptability. The paper evaluates various architectural breakthroughs, from attention-based Transformers to graph neural networks that naturally lend themselves to spatial-temporal data, offering insights into their efficacy at both the national and the state levels.

The Rise of Foundation Models for Time Series

A central theme of the paper is the exploration of foundation models—large-scale deep learning models pre-trained on extensive datasets. These models possess the inherent ability to discern intricate patterns, potentially accelerating effectiveness in scenarios where sufficient data is not initially available. The paper explores the criteria for selecting these underlying models and the necessity for domain-specific fine-tuning. It reviews the current literature on the methods employed to incorporate diverse data modalities, as well as the expected payoff from these multifaceted foundation models.

Incorporating Knowledge for Enhanced Forecasting

The survey places significant emphasis on the strategic integration of knowledge into deep learning models. It discusses different methodological strategies for this integration, such as composite loss functions, the injection of knowledge into downstream layers, and the influence of knowledge graphs on model architectures. The paper argues for the potential of knowledge-augmented models to provide more accurate, explainable forecasts, which is particularly pertinent in the context of pandemic forecasting where interpretability is crucial.

Meta Analysis and Future Directions

Finally, the paper presents a meta-analysis evaluating the effectiveness of various modeling techniques across key benchmark datasets. It utilizes metrics such as MSE and MAE to compare performances and presents a ranking of the models discussed. While highlighting the superiority of certain models like PatchTST, it underscores gaps that future research might address. The conclusion points to a seamless blend of multi-modal approaches with LLMs and knowledge graphs as the next frontier in enhancing temporal predictions.

This extensive survey stands as an authoritative reference for researchers aiming to harness the latest deep learning advancements for time series forecasting. It presents a thorough examination of the field's direction, emphasizing the integration of vast amounts of data and domain knowledge to improve forecasting accuracy and reliability.