Context is Key: A Benchmark for Forecasting with Essential Textual Information (2410.18959v4)

Abstract: Forecasting is a critical task in decision-making across numerous domains. While historical numerical data provide a start, they fail to convey the complete context for reliable and accurate predictions. Human forecasters frequently rely on additional information, such as background knowledge and constraints, which can efficiently be communicated through natural language. However, in spite of recent progress with LLM-based forecasters, their ability to effectively integrate this textual information remains an open question. To address this, we introduce "Context is Key" (CiK), a time-series forecasting benchmark that pairs numerical data with diverse types of carefully crafted textual context, requiring models to integrate both modalities; crucially, every task in CiK requires understanding textual context to be solved successfully. We evaluate a range of approaches, including statistical models, time series foundation models, and LLM-based forecasters, and propose a simple yet effective LLM prompting method that outperforms all other tested methods on our benchmark. Our experiments highlight the importance of incorporating contextual information, demonstrate surprising performance when using LLM-based forecasting models, and also reveal some of their critical shortcomings. This benchmark aims to advance multimodal forecasting by promoting models that are both accurate and accessible to decision-makers with varied technical expertise. The benchmark can be visualized at https://servicenow.github.io/context-is-key-forecasting/v0/.

• The paper introduces the CiK benchmark to evaluate forecasting models that integrate numerical and textual data for enhanced accuracy.
• It presents the novel Region of Interest CRPS (RCRPS) metric that focuses on context-informed evaluation by emphasizing specific time windows.
• Benchmark comparisons reveal that LLM-based methods, especially the Direct Prompt approach, outperform traditional models in leveraging textual context.

Context is Key: A Benchmark for Forecasting with Essential Textual Information

The paper introduces "Context is Key" (CiK), a benchmark designed to evaluate the integration of numerical and textual data in time series forecasting. This work addresses the persistent gap in forecasting models that often rely solely on numerical data, neglecting essential contextual information conveyed through natural language.

Key Contributions:

• Benchmark Overview: CiK pairs numerical data with carefully curated textual context across 71 tasks in seven domains, such as energy and public safety. It aims to assess a model's ability to leverage both types of data for improved forecasting accuracy.
• Evaluation Metrics: The authors introduce the Region of Interest CRPS (RCRPS) metric, emphasizing context-sensitive windows and constraint satisfaction in predictions. This metric extends the CRPS by focusing on context-informed time steps, providing a nuanced view of forecasting performance.
• Forecaster Comparison: Several forecasting approaches are evaluated, including statistical models, time series foundation models, and LLM-based forecasters. Notably, a simple LLM prompting method, termed Direct Prompt, demonstrated superior performance across the CiK benchmark compared to all other methods tested.

Numerical Results:

The benchmark revealed strong performances for LLM-based models, particularly when using the Direct Prompt method. Models such as Llama-3.1-405B-Instruct achieved significant improvements when incorporating textual context, showcasing reductions in RCRPS. Despite these advances, the paper also highlighted crucial limitations, such as the occasional performance degradation due to model failures in processing the context correctly.

Implications:

This benchmark is crucial for advancing multimodal forecasting. It challenges the research community to develop models that are not only accurate but also accessible and context-aware. The ability to integrate natural language context promises enhanced decision-making capabilities within diverse fields, such as energy management and public safety.

Future Directions:

The paper opens avenues for exploring more complex multimodal scenarios, including additional data modalities beyond time series and text. Enhancing model robustness to mitigate significant failures and reducing computational costs of LLMs are vital areas for further exploration. The potential integration of these models into systems facilitating conversational interactions could further democratize advanced forecasting tools.

In conclusion, CiK represents a strategic step towards realistic and contextual machine learning applications in forecasting, pushing the boundaries of what models can achieve by effectively integrating essential contextual knowledge with numerical data.