TimesX: Enriched Multimodal Forecasting

• TimesX is a comprehensive benchmark defined by integrating real-world numeric data and aligned textual context to improve forecasting realism and mitigate leakage.
• It aggregates 190 variables across 19 domains with daily and weekly data from 2018 to 2025, allowing training on historical data and leakage-aware evaluation on future periods.
• The benchmark employs a multi-agent pipeline to generate high-quality event context, ensuring robust validation and reliable comparisons across multimodal forecasting methods.

TimesX is a context-enriched, multimodal time series forecasting benchmark built to evaluate forecasting systems that use both historical numerical signals and aligned textual context. It was introduced to address three stated weaknesses of earlier multimodal forecasting benchmarks: poor generalization due to the small scale and synthetic nature of benchmark data, very limited types of textual contexts in the benchmarks, and an inability to mitigate data leakage in evaluation. In its main description, TimesX contains 19 domains and 190 variables—10 variables per domain—with daily and weekly data spanning Jan 2018 to Oct 2025, and it is intended to support training on earlier years and evaluation on later years, while being refreshed every three months (Liu et al., 8 Jul 2026).

1. Definition and benchmark rationale

TimesX is defined around a forecasting regime in which a model must predict future values from a historical time-series window together with textual context. The benchmark is explicitly presented as a response to the claim that earlier multimodal benchmarks can misrepresent real-world performance. The paper argues, first, that small or synthetic benchmarks may not generalize to real-world forecasting. Second, prior benchmarks often expose only limited forms of text. Third, pretrained models create a serious data leakage / contamination problem, especially for LLMs whose exact pretraining corpora are usually not public (Liu et al., 8 Jul 2026).

The benchmark is organized around four design principles: real-world data, leakage mitigation, comprehensive high-quality context, and large-scale evaluation. In the comparison table provided by the paper, TimesX is the only listed benchmark that simultaneously claims to use real data, be leakage-free, and provide meta + calendar + covariates + event context at the reported scale of 190 datasets/variables. The contrast is drawn against MT-Bench, ChatTime, MoTime, Time-MMD, and CiK, each of which lacks at least one of those properties.

A central empirical motivation is that benchmark choice changes the ranking of methods. On the synthetic CiK benchmark, geometric-mean MASE over the five subsets is reported as 0.38 for CodeRev, 0.43 for Gemini-2.0-Flash, and 0.58 for TimesFM-2.5. On TimesX, the paper states that this ranking no longer holds: the pure TFM is better than Gemini alone, and the code-revision approach degrades. This is used to argue that conclusions drawn from synthetic multimodal benchmarks may not transfer to real forecasting.

A common misconception is to treat TimesX as a forecasting model. It is instead a benchmark and evaluation framework. The paper’s contribution is the construction of benchmark data, aligned textual context, leakage-aware evaluation, and a large empirical study of zero-shot multimodal forecasting approaches.

2. Data composition and contextual modalities

The numeric component of TimesX is built from three real-world sources, at two frequencies:

• Weekly Google Search Trends across 12 domains
• Daily commodity prices from a market-data API
• Daily USD exchange rates from a currency-rate API

All variables within a frequency share aligned timestamps. Missing target values are ignored in metric computation; missing input values are linearly interpolated for TFMs when gaps are short. The appendix reports an average maximum contiguous missing gap per sample of 0.7962, and states that 3 anomalous samples with gap $>6$ are removed (Liu et al., 8 Jul 2026).

The 19 benchmark domains are listed as:

• Arts & Entertainment
• Climate & Environment
• Economy
• Electronic Technology
• Finance
• Pets & Animals
• Public Health
• Public Policy & Governance
• Science
• Shopping
• Society Security & Social Good
• Traffic
• Crops & Staples
• Energy & Fuels
• Livestock & Food Products
• Raw Materials & Construction
• Specialty & Advanced Materials
• Strategic & High-Value Materials
• Currency

Examples of benchmark variables include search-trend series such as inflation, unemployment_rate, nvidia, music_festivals, flu_shot, presidential_election, gas_prices, and wildfires; commodity series such as brent-USD-Bbl, gasoline-USD-Gal, cocoa-USD-T, cotton-USD-Lbs, aluminum-USD-T, gold-USD-t_oz, and lithium-CNY-T; and exchange-rate series such as USDtoAUD-ExchangeRate, USDtoINR-ExchangeRate, and USDtoMXN-ExchangeRate.

A defining feature of TimesX is that each target variable is paired with all four context types recognized by the paper.

Context type	Construction
Metadata	Generated from a fixed template
Calendar	Generated with the Python Holidays library
Covariates	Natural-language summaries of other variables in the same domain
Time-stamped events	Produced by a multi-agent data-generation pipeline

The metadata template is given verbatim as: “This time series records [variable name and unit] in the [domain] domain, with a collection frequency of [frequency]. Prediction target period: from [start date] to [end date].” For calendar context, the series is converted to $(\text{timestamp}, \text{value})$ pairs, holidays that overlap the prediction window are queried, and the result is rendered as text such as “Upcoming holidays in the prediction window: Labor Day (2024-09-02).”

Covariate context is generated by treating the other variables in the same domain as potential covariates and summarizing their mean, median, maximum and date, minimum and date, and overall trend direction. The paper emphasizes that this differs from benchmarks in which covariates are provided as raw side channels: in TimesX, they are transformed into textual summaries directly consumable by LLMs (Liu et al., 8 Jul 2026).

The benchmark also includes extended out-of-distribution additions not used in the main in-distribution benchmark: 11 multilingual variables in Afrikaans, French, German, Hindi, Japanese, Korean, Portuguese, Simplified Chinese, Spanish, Swahili, and Turkish, and 5 rare-disease variables.

The paper notes one textual inconsistency: the abstract and core text describe 190 variables and 19 domains, while the appendix occasionally mentions 20 domains / 200 variables. The main description identifies 190 variables across 19 domains as the central benchmark used in the main experiments.

3. Event generation, leakage mitigation, and quality control

The event context is produced by a multi-agent data-generation pipeline with four roles: Hypothesizer, Verifier, Enricher, and Synthesizer. The stated purpose of this pipeline is to satisfy real-world sourcing, temporal leakage control, and fact quality simultaneously (Liu et al., 8 Jul 2026).

In the Hypothesizer stage, the series is partitioned into blocks and peaks or interesting points are identified. An LLM with time-bounded web search proposes event hypotheses $H_B$. Event coverage over detected peaks is defined as

$$\mathrm{cov}(H_B) \;=\; \frac{|\{p\in\mathcal{P}_B: p \text{ is covered by } H_B\}|}{|\mathcal{P}_B|}.$$

The system iteratively adds hypotheses until either $\mathrm{cov}(H_B)\ge \theta$ or $K_{\max}$ is reached, with defaults:

• $K_{\max} = 3$
• $\theta = 90\%$

The paper reports that the peak-based heuristic reduces construction cost by about 21%, and that increasing $K_{\max}$ from 3 to 5 only increases accepted events by about 2%.

In the Verifier stage, the system re-fetches evidence under the same time bound and performs atomic fact checking, date extraction, URL validation, and source-status filtering. It extracts announcement_date, occurrence_date, and event type. The event taxonomy is:

• Scheduled
• Contemporaneous
• Retrospective
• Predictive
• Mixed

The verifier also removes near-duplicates and unresolved contradictions. The prompt explicitly distinguishes content dates from original publish dates, and rejects 404s, login walls, or updated pages that cannot be shown to preserve the original publication state.

In the Enricher stage, if key fields are missing, the system performs bounded deep search with default $L_{\max} = 3$. In the Synthesizer stage, the final fact-checked narrative is written, an authoritative date is chosen, and unresolved cases are discarded. The paper states that the Synthesizer discards about 32.20% of unverified candidate events.

Leakage control is built into every stage. Retrieval is time-bounded; evidence after the end of the event window is rejected; updated pages are accepted only if the original publication date falls in range and original content can be recovered; and only events with announcement dates before the forecast horizon are passed to the forecasting model. The paper therefore frames TimesX as using time isolation rather than source isolation.

The contamination analysis on the Search Trend subset is one of the benchmark’s strongest empirical arguments. Splitting examples before versus after the June 2024 knowledge cutoff of Gemini-2.0-Flash and DeepSeek-V3, the paper reports:

• Gemini-2.0-Flash: MASE 0.514 before cutoff vs 0.594 after cutoff, +14.81%
• DeepSeek-V3: 0.606 vs 0.681, +12.38%
• TimesFM-2.5: 0.563 vs 0.573, +1.78%
• Moirai-2.0: 0.691 vs 0.696, +0.72%

This suggests that LLM benchmark scores are substantially more sensitive to contamination than TFM scores.

The paper also reports several quality checks. A manual audit of 50 sampled events found 94% exact date matches, 4% conservative later offsets, and 2% earlier offsets. Domain-level event density varies substantially: Public Health averages 119.1 events with summary length 848.2, Electronic Technology averages 115.7 with 783.2, Arts & Entertainment averages 105.7 with 711.2, Currency averages 111.0 with 639.6, while Specialty & Advanced Materials averages only 28.8 events with summary length 672.6.

4. Task formulation and evaluation protocol

TimesX is evaluated as zero-shot multimodal forecasting. For each variable, the paper fixes:

• historical look-back window = 96
• forecast horizon = 12
• rolling stride = 4 for weekly data
• rolling stride = 12 for daily data

For each instance, the model receives:

• numeric history
• metadata
• calendar features
• covariate summaries
• the $(\text{timestamp}, \text{value})$0 most recent events whose announcement dates strictly precede the first timestamp of the prediction horizon

The paper fixes $(\text{timestamp}, \text{value})$1 to control prompt length and effectiveness (Liu et al., 8 Jul 2026).

The benchmark is designed to support both train and test usage. One passage states 2018–2022 for training and 2023–2025 for evaluation; another states training split 2018–2023 and testing 2023–2025. The central idea is that earlier years are available for model development, while later years are reserved for leakage-safe evaluation. For the main benchmark comparisons, the paper uses a global evaluation cutoff of 2024-07-01, so only evaluation examples whose forecast horizon starts on or after July 1, 2024 are included. The appendix lists model knowledge cutoffs including GPT-5: Sep 30, 2024, Gemini-2.5-Flash: Jan 2025, Gemini-2.0-Flash: June 2024, GPT-4o: Oct 2023, DeepSeek-R1: prior to June 2024, and DeepSeek-V3: prior to June 2024.

Evaluation is run offline. No web search or external API access is allowed to plain LLMs; generated code runs in a no-network sandbox; function sets explicitly exclude API access; and execution logs were manually checked for test-time leakage.

The main metric is normalized MASE aggregated by geometric mean across variables. The seasonal scale is

$(\text{timestamp}, \text{value})$2

Per-window MASE is

$(\text{timestamp}, \text{value})$3

The seasonal naive forecast is

$(\text{timestamp}, \text{value})$4

Per-dataset normalization is

$(\text{timestamp}, \text{value})$5

and the primary benchmark score is

$(\text{timestamp}, \text{value})$6

A secondary metric is average rank:

$(\text{timestamp}, \text{value})$7

The paper uses seasonality $(\text{timestamp}, \text{value})$8 for weekly data and $(\text{timestamp}, \text{value})$9 for daily data, with $H_B$0 for monthly data if applicable. It reports about 2,434 forecasting samples in the appendix and about 2.5K samples in the main text. Each stochastic method is repeated 10 times, yielding more than 312,000 independent LLM inferences. The benchmark also reports CRPS for uncertainty:

$H_B$1

5. Empirical findings and benchmark implications

The benchmark compares three broad classes of systems: unimodal zero-shot TFMs, multimodal zero-shot LLMs, and composed multimodal solutions such as AvgEns, TextRev, CodeRev, and FuncRev. The core result is that simple average ensembling wins on TimesX (Liu et al., 8 Jul 2026).

The reported geometric-mean normalized MASE and average rank are:

• SeasonalNaive: 1.000, rank 12.196
• Sundial: 0.771, rank 9.556
• Moirai-2.0: 0.722, rank 7.968
• TimesFM-2.5: 0.645, rank 5.757
• AvgEns: TimesFM + Moirai: 0.668, rank 6.45
• DeepSeek-V3: 0.708, rank 7.73
• Gemini-2.0-Flash: 0.650, rank 6.603
• GPT-4o: 0.643, rank 5.466
• FuncRev: TimesFM + Gemini: 0.720, rank 7.665
• CodeRev: TimesFM + Gemini: 0.713, rank 6.968
• TextRev: TimesFM + Gemini: 0.653, rank 5.63
• AvgEns: TimesFM + GPT-4o: 0.627, rank 4.735
• AvgEns: TimesFM + Gemini-2.0-Flash: 0.619, rank 4.249

The paper emphasizes that CodeRev, which is best on synthetic CiK, underperforms both of its components on TimesX and loses to simple averaging. The stated explanation is that synthetic benchmarks reward instruction following and direct codified transformations, whereas real-world contexts are more ambiguous, partial, and nuanced. The paper attributes the failure of revision methods mainly to LLM stochasticity and failure to preserve temporal structure in the original forecast. Boxplot analysis is said to show wider error spread and more severe outliers for revision methods than for TimesFM alone.

The benchmark also studies the value of context itself. With Gemini-2.0-Flash, the context ablation is reported as follows:

Context	MASE	Rank
Meta	0.787	3.704
Meta + Date	0.670	3.048
Meta + Date + Cov	0.674	3.079
Meta + Date + Event	0.674	2.968
Meta + Date + Event + Cov	0.650	2.635

The relative improvement from Meta to Meta + Date + Event + Cov is approximately

$H_B$2

The paper describes this as consistent with an “around 16% better” summary. It also reports that holidays are effective in 12/19 domains with average intra-domain win rate 65.2%, while covariates are effective in 12/19 domains with average intra-domain win rate 57.9%. Largest holiday gains occur in Shopping, Climate & Environment, Public Health, and Public Policy & Governance; largest covariate gains occur in Specialty & Advanced Materials, Raw Materials & Construction, Livestock & Food Products, and Currency.

To isolate context quality, the paper replaces Time-MMD context with TimesX-generated context while keeping the numeric series, prompts, and Gemini-2.0-Flash fixed. The resulting geometric-mean MASE is 0.840 with TimesX context versus 0.906 with Time-MMD context, a 7.3% relative drop in error:

$H_B$3

The benchmark also argues that benchmark scale stabilizes rankings. When subsampling $H_B$4 variables for $H_B$5, the paper reports that at 10–40 variables the uncertainty bands are wide and strongly overlapping, while larger $H_B$6 narrows the bands and stabilizes rankings. A plausible implication is that many earlier multimodal TSF evaluations were underpowered for reliable ranking.

For probabilistic forecasting, the reported CRPS values are:

• GPT-4o: 0.258
• Gemini-2.0-Flash: 0.276
• DeepSeek-V3: 0.277
• Sundial: 0.278
• TimesFM-2.5: 0.293
• Moirai-2.0: 0.327

All three LLMs outperform all three TFMs on CRPS, which the paper interprets as evidence that textual context helps represent uncertainty over multiple plausible futures.

6. Position in the literature and naming distinctions

TimesX occupies a specific place in the recent literature on time series and multimodal forecasting. It is a benchmark for context-enriched multimodal time-series forecasting, not an explainer, not a symbolic representation, and not a Transformer forecasting architecture (Liu et al., 8 Jul 2026).

This distinction matters because several similarly named methods address different problems. TimeX is an inherently interpretable surrogate explainer for time-series models based on model behavior consistency and discrete attribution masks (Queen et al., 2023). TimeX++ is an information-bottleneck-inspired post-hoc explainer that learns explanation-embedded instances intended to remain label-preserving and in-distribution for the original black-box classifier (Liu et al., 2024). TimeXer is a Transformer-based method for time series forecasting with exogenous variables, where an endogenous series is forecast using exogenous series as auxiliary context (Wang et al., 2024). TSAX is a SAX-family symbolic representation for time series classification, not a multimodal forecasting benchmark (Fuad, 2021). TIME is a next-generation benchmark for zero-shot TSFM evaluation built around 50 fresh datasets and 98 forecasting tasks, but it excludes exogenous covariates and is not the same benchmark as TimesX (Qiao et al., 12 Feb 2026).

Later work uses TimesX not just as an evaluation set but as evidence for a particular modeling strategy. PostTime argues that multimodal forecasting on TimesX should be framed as context-guided revision of a strong numerical TSFM prior, rather than direct LLM forecasting. On a refreshed TimesX built with 99 variables over 2022 to 2025, the final Gemma-3-4B + TimesFM-2.5 PostTime system achieves ID: 0.738 nMAE, 0.638 nMSE and OOD: 0.746 nMAE, 0.597 nMSE, improving over TimesFM-2.5 by 6.38% / 12.43% on ID nMAE/nMSE and 3.93% / 11.24% on OOD nMAE/nMSE (Liu et al., 28 May 2026). This suggests that TimesX has already begun to function as a benchmark for studying when textual context should trigger revision, preservation, or effective ignoring of a numerical prior.

The benchmark’s main scientific significance is therefore methodological. It is designed to make multimodal forecasting evaluation more realistic, broader in context coverage, and more trustworthy, and its reported findings reverse several conclusions previously suggested by smaller or synthetic benchmarks.