Language Models Still Struggle to Zero-shot Reason about Time Series (2404.11757v1)

Abstract: Time series are critical for decision-making in fields like finance and healthcare. Their importance has driven a recent influx of works passing time series into LLMs, leading to non-trivial forecasting on some datasets. But it remains unknown whether non-trivial forecasting implies that LLMs can reason about time series. To address this gap, we generate a first-of-its-kind evaluation framework for time series reasoning, including formal tasks and a corresponding dataset of multi-scale time series paired with text captions across ten domains. Using these data, we probe whether LLMs achieve three forms of reasoning: (1) Etiological Reasoning - given an input time series, can the LLM identify the scenario that most likely created it? (2) Question Answering - can a LLM answer factual questions about time series? (3) Context-Aided Forecasting - does highly relevant textual context improve a LLM's time series forecasts? We find that otherwise highly-capable LLMs demonstrate surprisingly limited time series reasoning: they score marginally above random on etiological and question answering tasks (up to 30 percentage points worse than humans) and show modest success in using context to improve forecasting. These weakness showcase that time series reasoning is an impactful, yet deeply underdeveloped direction for LLM research. We also make our datasets and code public at to support further research in this direction at https://github.com/behavioral-data/TSandLanguage

• The paper introduces an innovative evaluation framework that rigorously tests zero-shot time series reasoning capabilities of LMs.
• The paper demonstrates that LMs perform barely above random chance in etiological reasoning and question answering, with models like GPT-4 achieving only 34.7% accuracy compared to 66.1% by humans.
• The paper provides a comprehensive dataset of 230k multiple-choice questions and 8.7k synthetic time series-text pairs, paving the way for future research improvements.

Assessing Time Series Reasoning in LLMs: A Comprehensive Study

Introduction

In recent efforts to enhance the applicability of LLMs (LMs) in real-world domains, the abilities of these models to understand and generate time series data have become a vital area of research. This paper introduces an innovative evaluation framework to rigorously assess time series reasoning across multiple dimensions, including etiological reasoning, question answering, and context-aided forecasting. Despite high expectations, the paper reveals that current LMs, including advanced versions like GPT-4, exhibit limited reasoning capabilities over time series data compared to human performance. This gap highlights significant challenges and opens avenues for future enhancements in this field.

Evaluation Framework and Dataset

The evaluation framework proposed in this paper is designed to test the capacity of LMs to reason about time series data through three distinct reasoning tasks:

1. Etiological Reasoning: Testing whether LMs can hypothesize plausible causes for given time series data.
2. Question Answering: Assessing the model's ability to answer questions correctly that are contingent upon understanding the time series data.
3. Context-Aided Forecasting: Evaluating whether LMs can use contextual text information to enhance forecasting accuracy.

To facilitate this evaluation, the researchers developed a novel dataset comprising 230k multiple-choice questions and 8.7k synthetic time series-text pairs across various scenarios and domains. This extensive dataset underpins a robust testing environment where LMs' reasoning capabilities are systematically challenged against complex, real-world analogous data.

Experimental Findings

Etiological Reasoning

Results indicate that LMs barely perform above random chance in identifying correct scenario descriptions for given time series, with human annotators significantly outperforming the LMs. The best-performing model, GPT-4-Vision, achieved just 34.7% accuracy, starkly lower than the 66.1% human benchmark.

Question Answering

The ability of LMs to answer questions based on time series data was also found largely inadequate. When tested with questions requiring analysis between two different time series, LMs scored nearly at random chance levels, substantially lagging behind the human annotator scores. Notably, even the sophisticated GPT-4 only marginally improved performance with access to the time series data, suggesting a limited understanding of the underlying time series processes.

Context-Aided Forecasting

In forecasting tasks, when LMs were provided with contextual descriptions, their performance showed negligible improvement over forecasts without such context. This was somewhat surprising and demonstrated a significant shortcoming in integrating relevant textual information to predict future time series values accurately.

Implications and Future Directions

The paper unmistakably underscores a profound deficiency in current LMs concerning time series reasoning, despite their adeptness at other forms of data processing. This revelation calls for targeted research efforts focusing on developing models or training approaches that enhance the understanding and predictive capabilities of LMs regarding time series data.

The provided open-source dataset and codebase present an excellent resource for future research, enabling ongoing investigation and potentially fostering advancements in this critical aspect of AI development.

Conclusion

Overall, this paper serves as a benchmark for understanding the current state of LMs in handling time series data and sets a clear mandate for continued research in this area. Improving LMs' proficiency in time series reasoning not only enhances their applicability across various scientific and commercial fields but also elevates their overall utility in automated decision-making systems, where accuracy and reliability are paramount.