Rethinking Multimodal Time-Series Forecasting Evaluation

Abstract: We introduce a new context-enriched, multimodal time series forecasting benchmark, TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on TimesX.

• The paper critiques synthetic benchmarks and exposes their failure to reflect real-world TSF complexities.
• It introduces TimesX, a large-scale, leakage-controlled benchmark that integrates diverse, time-aligned textual and numeric contexts.
• Empirical results reveal that simple ensemble methods outperform complex agentic systems in robust multimodal forecasting.

Detailed Review of "Rethinking Multimodal Time-Series Forecasting Evaluation" (2607.06973)

Motivation and Critique of Existing Multimodal TSF Benchmarks

The paper provides a systematic examination of the limitations in current multimodal time-series forecasting (TSF) benchmarks, particularly in scenarios where both numerical and textual data inform predictions. Existing benchmarks generally suffer from three key issues:

1. Lack of Generalization Due to Synthetic Data: Many recent efforts rely heavily on synthetic time series or instruction-following tasks that do not faithfully capture real-world forecasting complexity. The paper demonstrates empirically that model rankings derived from performance on synthetic datasets do not transfer to real-world settings. For instance, instruction-tuned LLMs and code-generating methods such as Gemini-2.0-Flash and CodeRev dominate in synthetic contexts but fail to outperform standard time-series foundation models (TFMs) such as TimesFM-2.5 in realistic benchmarks, as shown in Figure 1.

Figure 1: Synthetic (CiK) vs real-world (TimesX) performance of top TSF approaches; synthetic benchmarks often distort model rankings, overvaluing instruction-following and code-execution capabilities.

1. Uncontrolled Data Leakage: The majority of prior benchmarks neither isolate data temporally nor provide mechanisms to guarantee leakage-free evaluation, especially for models exposed to up-to-date internet corpora. TimesX enforces strict temporal isolation and timestamp alignment, ensuring all evaluation data is outside the pretraining windows of models under assessment. Empirical analysis reveals that LLM-based forecasters' errors increase by over 13% on samples strictly after their knowledge cutoff, contrasting with stable errors (<2% change) for TFMs, validating the importance of leakage controls.
2. Limited and Uncalibrated Textual Contexts: Previous datasets often restrict text context to metadata, templates, or synthetic annotations, failing to reflect the heterogeneity and nuance of real-world side information relevant for forecasting. TimesX categorically expands context types to include static metadata, calendar events, covariate trends, and precisely timestamped event narratives, thereby more faithfully modeling how human forecasters access contextual information.

The TimesX Benchmark: Construction, Diversity, and Leakage Mitigation

TimesX is proposed as a large-scale, real-world, and extensible multimodal TSF benchmark distinguished by several core properties:

• Scale and Diversity: TimesX encompasses 190 time series variables pulled from 19 domains (e.g., climate, commodities, finance, technology, public health), spanning daily and weekly granularities and multiple geographies. Visualization using PCA and t-SNE over both the numeric and textual representations confirms high feature diversity and domain-separation.

Figure 2: PCA and t-SNE visualization of textual features in TimesX—substantial domain and data source diversity supports cross-domain benchmarking.

Figure 3: PCA and t-SNE visualization of numeric time-series features, indicating feature richness and heterogeneity.

• Automated, Factual, and Time-Aligned Context Construction: The dataset leverages a multi-phase LLM-driven agentic pipeline—hypothesizer, verifier, enricher, synthesizer—that ensures all events are accurately time-aligned, supported by verifiable URLs, and grounded in real-world evidence. The pipeline is extensible, making TimesX automatically refreshable for new evaluation periods.

Figure 4: Multi-agent pipeline for context construction, ensuring timestamp alignment, factual verification, and refreshable benchmark extension.

• Leakage Prevention and Data Release Protocol: Detailed timestamping means evaluation splits are dynamically aligned with model knowledge cutoffs, allowing clean assessments even for future models.

Empirical Study and Methodological Ablations

Experimental Protocol and Baselines

The authors execute over 312,000 independent LLM inferences in a rolling-window, leakage-controlled Zero-Shot TSF setup. The benchmark includes:

• TFMs: State-of-the-art, pre-trained, unimodal models (TimesFM-2.5, Moirai-2.0, Sundial).
• LLMs: General LLMs (Gemini-2.0-Flash, GPT-4o, DeepSeek-V3) prompted with both context and series.
• Composed/Agentic Solutions: Methods in which LLMs revise or post-process TFM outputs (TextRev, CodeRev, FuncRev), and simple averaging ensembles.

Results demonstrate that, contrary to synthetic benchmarks, fine-grained composed solutions, including those involving code revision or function-calling, fail to robustly outperform simple averaging ensembles. The best aggregate performance is consistently achieved by simple ensemble averaging of TFM and LLM predictions, not agentic solutions—contradicting recent claims based on synthetic instruction-following datasets.

Figure 5: Performance boxplot of TimesFM-2.5, Gemini-2.0-Flash, and their composed/ensemble methods; ensembles achieve higher stability and lower error spread.

Context Type Ablation

Systematic ablation across four context types demonstrates that the compounding of event, covariate, and calendar/textual contexts produces the largest accuracy gains for multimodal LLM forecasters. Conversely, adding isolated context types (e.g., only events or only covariates) yields limited marginal improvement, suggesting crucial interaction effects.

Figure 6: Incremental inclusion of all context types (metadata, calendar, covariate, event) provides significant gains across all LLMs relative to any single context type.

Domain-Level and Series Characteristic Analysis

Performance breakdown by domain elucidates that forecasting tasks deeply impacted by exogenous shocks, such as Shopping or Currency, benefit most from multimodal context and LLM-based fusion. Conversely, domains where strong trend/nonstationarity dominate (e.g., commodities), TFMs maintain a comparative edge. Visualizations provided for each domain reveal the distinct seasonality, trend, and event responsiveness across the dataset:

Figure 7: Numeric series visualization for the Arts and Entertainment domain.

Figure 8: Numeric series for the Climate and Environment domain.

...and analogously through all included domains.

Practical and Theoretical Implications

Evaluation Bias: The study strongly cautions against using synthetic or instruction-based datasets as surrogates for real-world forecasting evaluation, due to the clear regime separation in model ranking, error distribution, and practical relevance.

Leakage Control as a Benchmarking Prerequisite: Strict time-aligned data isolation is essential to provide meaningful, future-proof evaluations as foundation models ingest increasingly up-to-date corpora.

Composed and Agentic Systems: Despite the theoretical flexibility of agentic/system-2 approaches (e.g., programmatic revision of predictions), their instability and volatility prevent them from stably outperforming simpler ensemble baselines in realistic settings with noisy, vaguely relevant text context.

Context Compoundability: Real-world performance gains for multimodal TSF accrue only when models can synergistically incorporate diverse, temporally-aligned context types; superficial increases in context length or complexity do not guarantee improvement.

Future Directions: While current results favor zero-shot and simple ensemble fusion, initial experiments with in-context learning (ICL) suggest that training-based and fine-tuning approaches on TimesX can surpass zero-shot models. The benchmark’s extensibility and support for supervised splits position it as a robust platform for investigating data-driven training, advanced fusion, and reasoned generalization in multimodal TSF.

Conclusion

"Rethinking Multimodal Time-Series Forecasting Evaluation" (2607.06973) sets a new methodological standard for TSF evaluation by introducing TimesX—a leakage-free, large-scale, cross-domain benchmark with high-quality, automatically validated textual context. The study provides rigorous evidence that prior synthetic/instructional benchmarks are misaligned with real-world generalization and should not guide model development or deployment. Furthermore, it demonstrates that, in present real-world contexts, simple model ensembling surpasses complex agentic systems, and only multi-type, high-quality context allows LLMs to realize their potential for TSF. This work will serve as a foundation for empirical and methodological advances in the next generation of multimodal time-series forecasting research.