TEMPO: Prompt-based Generative Pre-trained Transformer for Time Series Forecasting (2310.04948v3)

Abstract: The past decade has witnessed significant advances in time series modeling with deep learning. While achieving state-of-the-art results, the best-performing architectures vary highly across applications and domains. Meanwhile, for natural language processing, the Generative Pre-trained Transformer (GPT) has demonstrated impressive performance via training one general-purpose model across various textual datasets. It is intriguing to explore whether GPT-type architectures can be effective for time series, capturing the intrinsic dynamic attributes and leading to significant accuracy improvements. In this paper, we propose a novel framework, TEMPO, that can effectively learn time series representations. We focus on utilizing two essential inductive biases of the time series task for pre-trained models: (i) decomposition of the complex interaction between trend, seasonal and residual components; and (ii) introducing the design of prompts to facilitate distribution adaptation in different types of time series. TEMPO expands the capability for dynamically modeling real-world temporal phenomena from data within diverse domains. Our experiments demonstrate the superior performance of TEMPO over state-of-the-art methods on zero shot setting for a number of time series benchmark datasets. This performance gain is observed not only in scenarios involving previously unseen datasets but also in scenarios with multi-modal inputs. This compelling finding highlights TEMPO's potential to constitute a foundational model-building framework.

Overview of TEMPO: A Prompt-Based GPT Framework for Time Series Forecasting

This paper introduces and evaluates TEMPO, a novel prompt-based Generative Pre-trained Transformer (GPT) framework specifically designed for time series forecasting. Recognizing the unique characteristics inherent in time series data, such as the presence of trend, seasonality, and residual components, TEMPO proposes a method that leverages these distinctive features to improve predictive accuracy. By integrating concepts from both time series decomposition and prompt-based tuning, TEMPO aims to bridge the gap between the advancements achieved in natural language processing through transformer models and the complex dynamics within time series data.

Methodology

TEMPO is built upon two foundational concepts:

1. Decomposition of Time Series Components:
   • The framework begins by decomposing the time series data into three additive components: trend, seasonal, and residual. This decomposition is accomplished through techniques like locally weighted scatterplot smoothing (STL).
   • Each component is then projected into a hidden space which constructs the input embedding for the GPT, theoretically aligning the time domain with the frequency domain computations.
2. Prompt-Based Tuning:
   • TEMPO integrates a semi-soft prompting approach, allowing for efficient tuning of the GPT. Prompts designed for trend, seasonality, and residual components ensure the reuse of temporal knowledge across different forecasting tasks.
   • A secondary novel design involving a prompt pool is discussed, enhancing adaptability to distribution shifts and leveraging past experiences from similar historical data.

Experiments and Results

TEMPO's performance was evaluated on a diverse set of benchmark datasets including ECL, Traffic, Weather, ETTm1, and several others. The experiments focused on both zero-shot and multimodal contexts, testing the efficacy of the model in scenarios where it has not been trained on the target dataset and where additional textual information provided valuable context.

Key Findings:

• TEMPO consistently outperformed baseline models like GPT-2, T5, and TimesNet across prediction horizons, indicating its superior ability to generalize over unseen time series datasets.
• In multimodal experiments, where contextual information was introduced, TEMPO demonstrated enhanced predictive capacity compared to traditional time series models, showcasing the advantage of integrating external information from news summaries.

Implications and Future Directions

The introduction of TEMPO marks a significant step towards foundational models for time series analysis, suggesting that the integration of pre-trained LLMs could substantially improve forecasting accuracy and robustness. The approach fosters a paradigm shift by moving from traditional deep learning methods to pre-trained models capable of effective representation learning.

Practical Implications:

• TEMPO's adaptability across domains can lead to broader applications in finance, healthcare, and other fields where time series data play a critical role.
• Its zero-shot capabilities provide a promising avenue for applications in real-time forecasting where retraining models is impractical.

Theoretical Implications:

• The method encourages rethinking time series forecasting through the lens of decomposed spectral analysis, supporting the hypothesis that forecasting in the time domain can benefit from insights in the frequency domain.

Future Directions could focus on refining the prompt pool mechanism to dynamically tune the model further as it encounters new datasets or varying temporal signals. Additionally, deeper exploration into alternative decomposing methods and their integration with other GPT-like models might yield even greater improvements in time series forecasting tasks.