---
title: 'TimesX: Enriched Multimodal Forecasting'
url: https://www.emergentmind.com/topics/timesx
type: topic
---

# TimesX: Enriched Multimodal Forecasting

TimesX is a **context-enriched, multimodal time series forecasting benchmark** built to evaluate forecasting systems that use both historical numerical signals and aligned textual context. It was introduced to address three stated weaknesses of earlier multimodal forecasting benchmarks: **poor generalization due to the small scale and synthetic nature of benchmark data, very limited types of textual contexts in the benchmarks, and an inability to mitigate data leakage in evaluation**. In its main description, TimesX contains **19 domains and 190 variables**—**10 variables per domain**—with **daily and weekly** data spanning **Jan 2018 to Oct 2025**, and it is intended to support **training on earlier years and evaluation on later years**, while being **refreshed every three months** [2607.06973].

## 1. Definition and benchmark rationale

TimesX is defined around a forecasting regime in which a model must predict future values from a historical time-series window together with textual context. The benchmark is explicitly presented as a response to the claim that earlier multimodal benchmarks can misrepresent real-world performance. The paper argues, first, that **small or synthetic benchmarks may not generalize to real-world forecasting**. Second, prior benchmarks often expose only limited forms of text. Third, pretrained models create a serious **data leakage / contamination** problem, especially for LLMs whose exact pretraining corpora are usually not public [2607.06973].

The benchmark is organized around four design principles: **real-world data**, **leakage mitigation**, **comprehensive high-quality context**, and **large-scale evaluation**. In the comparison table provided by the paper, TimesX is the only listed benchmark that simultaneously claims to use **real** data, be **leakage-free**, and provide **meta + calendar + covariates + event** context at the reported scale of **190 datasets/variables**. The contrast is drawn against **MT-Bench**, **ChatTime**, **MoTime**, **Time-MMD**, and **CiK**, each of which lacks at least one of those properties.

A central empirical motivation is that benchmark choice changes the ranking of methods. On the synthetic **CiK** benchmark, geometric-mean MASE over the five subsets is reported as **0.38** for **CodeRev**, **0.43** for **Gemini-2.0-Flash**, and **0.58** for **TimesFM-2.5**. On TimesX, the paper states that this ranking no longer holds: the pure TFM is better than Gemini alone, and the code-revision approach degrades. This is used to argue that conclusions drawn from synthetic multimodal benchmarks may not transfer to real forecasting.

A common misconception is to treat TimesX as a forecasting model. It is instead a **benchmark** and evaluation framework. The paper’s contribution is the construction of benchmark data, aligned textual context, leakage-aware evaluation, and a large empirical study of zero-shot multimodal forecasting approaches.

## 2. Data composition and contextual modalities

The numeric component of TimesX is built from **three real-world sources**, at **two frequencies**:

- **Weekly Google Search Trends** across **12 domains**
- **Daily commodity prices** from a market-data API
- **Daily USD exchange rates** from a currency-rate API

All variables within a frequency share aligned timestamps. Missing target values are ignored in metric computation; missing input values are linearly interpolated for TFMs when gaps are short. The appendix reports an **average maximum contiguous missing gap per sample** of **0.7962**, and states that **3 anomalous samples** with gap \(>6\) are removed [2607.06973].

The 19 benchmark domains are listed as:

- Arts & Entertainment
- Climate & Environment
- Economy
- Electronic Technology
- Finance
- Pets & Animals
- Public Health
- Public Policy & Governance
- Science
- Shopping
- Society Security & Social Good
- Traffic
- Crops & Staples
- Energy & Fuels
- Livestock & Food Products
- Raw Materials & Construction
- Specialty & Advanced Materials
- Strategic & High-Value Materials
- Currency

Examples of benchmark variables include search-trend series such as `inflation`, `unemployment_rate`, `nvidia`, `music_festivals`, `flu_shot`, `presidential_election`, `gas_prices`, and `wildfires`; commodity series such as `brent-USD-Bbl`, `gasoline-USD-Gal`, `cocoa-USD-T`, `cotton-USD-Lbs`, `aluminum-USD-T`, `gold-USD-t_oz`, and `lithium-CNY-T`; and exchange-rate series such as `USDtoAUD-ExchangeRate`, `USDtoINR-ExchangeRate`, and `USDtoMXN-ExchangeRate`.

A defining feature of TimesX is that each target variable is paired with **all four** context types recognized by the paper.

| Context type | Construction |
|---|---|
| Metadata | Generated from a fixed template |
| Calendar | Generated with the Python Holidays library |
| Covariates | Natural-language summaries of other variables in the same domain |
| Time-stamped events | Produced by a multi-agent data-generation pipeline |

The metadata template is given verbatim as: *“This time series records [variable name and unit] in the [domain] domain, with a collection frequency of [frequency]. Prediction target period: from [start date] to [end date].”* For calendar context, the series is converted to \((\text{timestamp}, \text{value})\) pairs, holidays that overlap the prediction window are queried, and the result is rendered as text such as *“Upcoming holidays in the prediction window: Labor Day (2024-09-02).”*

Covariate context is generated by treating the other variables in the same domain as potential covariates and summarizing their **mean**, **median**, **maximum and date**, **minimum and date**, and **overall trend direction**. The paper emphasizes that this differs from benchmarks in which covariates are provided as raw side channels: in TimesX, they are transformed into textual summaries directly consumable by LLMs [2607.06973].

The benchmark also includes extended out-of-distribution additions not used in the main in-distribution benchmark: **11 multilingual variables** in **Afrikaans, French, German, Hindi, Japanese, Korean, Portuguese, Simplified Chinese, Spanish, Swahili, and Turkish**, and **5 rare-disease variables**.

The paper notes one textual inconsistency: the abstract and core text describe **190 variables and 19 domains**, while the appendix occasionally mentions **20 domains / 200 variables**. The main description identifies **190 variables across 19 domains** as the central benchmark used in the main experiments.

## 3. Event generation, leakage mitigation, and quality control

The event context is produced by a **multi-agent data-generation pipeline** with four roles: **Hypothesizer**, **Verifier**, **Enricher**, and **Synthesizer**. The stated purpose of this pipeline is to satisfy **real-world sourcing, temporal leakage control, and fact quality** simultaneously [2607.06973].

In the **Hypothesizer** stage, the series is partitioned into blocks and peaks or interesting points are identified. An LLM with time-bounded web search proposes event hypotheses \(H_B\). Event coverage over detected peaks is defined as

\[
\mathrm{cov}(H_B) \;=\; \frac{|\{p\in\mathcal{P}_B: p \text{ is covered by } H_B\}|}{|\mathcal{P}_B|}.
\]

The system iteratively adds hypotheses until either \(\mathrm{cov}(H_B)\ge \theta\) or \(K_{\max}\) is reached, with defaults:

- \(K_{\max} = 3\)
- \(\theta = 90\%\)

The paper reports that the peak-based heuristic reduces construction cost by about **21%**, and that increasing \(K_{\max}\) from **3** to **5** only increases accepted events by about **2%**.

In the **Verifier** stage, the system re-fetches evidence under the same time bound and performs atomic fact checking, date extraction, URL validation, and source-status filtering. It extracts `announcement_date`, `occurrence_date`, and event type. The event taxonomy is:

- Scheduled
- Contemporaneous
- Retrospective
- Predictive
- Mixed

The verifier also removes near-duplicates and unresolved contradictions. The prompt explicitly distinguishes **content dates** from **original publish dates**, and rejects 404s, login walls, or updated pages that cannot be shown to preserve the original publication state.

In the **Enricher** stage, if key fields are missing, the system performs bounded deep search with default \(L_{\max} = 3\). In the **Synthesizer** stage, the final fact-checked narrative is written, an authoritative date is chosen, and unresolved cases are discarded. The paper states that the Synthesizer discards about **32.20%** of unverified candidate events.

Leakage control is built into every stage. Retrieval is **time-bounded**; evidence after the end of the event window is rejected; updated pages are accepted only if the original publication date falls in range and original content can be recovered; and only events with **announcement dates before the forecast horizon** are passed to the forecasting model. The paper therefore frames TimesX as using **time isolation** rather than source isolation.

The contamination analysis on the **Search Trend** subset is one of the benchmark’s strongest empirical arguments. Splitting examples before versus after the **June 2024** knowledge cutoff of **Gemini-2.0-Flash** and **DeepSeek-V3**, the paper reports:

- **Gemini-2.0-Flash**: MASE **0.514** before cutoff vs **0.594** after cutoff, **+14.81%**
- **DeepSeek-V3**: **0.606** vs **0.681**, **+12.38%**
- **TimesFM-2.5**: **0.563** vs **0.573**, **+1.78%**
- **Moirai-2.0**: **0.691** vs **0.696**, **+0.72%**

This suggests that LLM benchmark scores are substantially more sensitive to contamination than TFM scores.

The paper also reports several quality checks. A manual audit of **50** sampled events found **94% exact date matches**, **4% conservative later offsets**, and **2% earlier offsets**. Domain-level event density varies substantially: **Public Health** averages **119.1** events with summary length **848.2**, **Electronic Technology** averages **115.7** with **783.2**, **Arts & Entertainment** averages **105.7** with **711.2**, **Currency** averages **111.0** with **639.6**, while **Specialty & Advanced Materials** averages only **28.8** events with summary length **672.6**.

## 4. Task formulation and evaluation protocol

TimesX is evaluated as **zero-shot multimodal forecasting**. For each variable, the paper fixes:

- historical look-back window = **96**
- forecast horizon = **12**
- rolling stride = **4** for weekly data
- rolling stride = **12** for daily data

For each instance, the model receives:

- numeric history
- metadata
- calendar features
- covariate summaries
- the **\(K\)** most recent events whose **announcement dates strictly precede the first timestamp of the prediction horizon**

The paper fixes \(K = 10\) to control prompt length and effectiveness [2607.06973].

The benchmark is designed to support both train and test usage. One passage states **2018–2022 for training** and **2023–2025 for evaluation**; another states **training split 2018–2023** and testing **2023–2025**. The central idea is that earlier years are available for model development, while later years are reserved for leakage-safe evaluation. For the main benchmark comparisons, the paper uses a **global evaluation cutoff of 2024-07-01**, so only evaluation examples whose forecast horizon starts on or after **July 1, 2024** are included. The appendix lists model knowledge cutoffs including **GPT-5: Sep 30, 2024**, **Gemini-2.5-Flash: Jan 2025**, **Gemini-2.0-Flash: June 2024**, **GPT-4o: Oct 2023**, **DeepSeek-R1: prior to June 2024**, and **DeepSeek-V3: prior to June 2024**.

Evaluation is run **offline**. No web search or external API access is allowed to plain LLMs; generated code runs in a **no-network sandbox**; function sets explicitly exclude API access; and execution logs were manually checked for test-time leakage.

The main metric is normalized **MASE** aggregated by geometric mean across variables. The seasonal scale is

\[
Q_{i,w} \;=\; \frac{1}{\tau_{i,w}-m}\sum_{t=m+1}^{\tau_{i,w}} \big|\, y^{(i)}_{t} - y^{(i)}_{t-m} \,\big|.
\]

Per-window MASE is

\[
\mathrm{MASE}_{i,w}(\text{model}) \;=\; \frac{1}{H_{i,w}} \sum_{h=1}^{H_{i,w}} \frac{\big|\, y^{(i)}_{\tau_{i,w}+h} - \widehat{y}^{(i)}_{\tau_{i,w}+h} \,\big|}{Q_{i,w}}.
\]

The seasonal naive forecast is

\[
\widehat{y}^{(i),\mathrm{SNAIVE}}_{\tau_{i,w}+h} \;=\; \mathbf{s}^{(i,w)}_{\,1 + \big((h-1) \bmod m\big)} , \quad h=1,\dots,H_{i,w}.
\]

Per-dataset normalization is

\[
R_i(\text{model}) \;=\; \frac{\sum_{w=1}^{W_i} \mathrm{MASE}_{i,w}(\text{model})} {\sum_{w=1}^{W_i} \mathrm{MASE}_{i,w}(\mathrm{SNAIVE})},
\]

and the primary benchmark score is

\[
\mathrm{GM}(\text{model}) \;=\; \left( \prod_{i=1}^{D} R_i(\text{model}) \right)^{\frac{1}{D}}.
\]

A secondary metric is average rank:

\[
\mathrm{AvgRank}(\text{model}) \;=\; \frac{1}{D}\sum_{i=1}^{D} \mathrm{rank}_i(\text{model}) .
\]

The paper uses seasonality \(m=4\) for weekly data and \(m=7\) for daily data, with \(m=12\) for monthly data if applicable. It reports about **2,434 forecasting samples** in the appendix and **about 2.5K samples** in the main text. Each stochastic method is repeated **10 times**, yielding more than **312,000 independent LLM inferences**. The benchmark also reports **CRPS** for uncertainty:

\[
\operatorname{CRPS}(F,y) = \int_{-\infty}^{+\infty} \bigl(F(z) - \mathbf{1}\{z \ge y\}\bigr)^2 \,\mathrm{d}z .
\]

## 5. Empirical findings and benchmark implications

The benchmark compares three broad classes of systems: **unimodal zero-shot TFMs**, **multimodal zero-shot LLMs**, and **composed multimodal solutions** such as **AvgEns**, **TextRev**, **CodeRev**, and **FuncRev**. The core result is that **simple average ensembling wins** on TimesX [2607.06973].

The reported geometric-mean normalized MASE and average rank are:

- **SeasonalNaive**: **1.000**, rank **12.196**
- **Sundial**: **0.771**, rank **9.556**
- **Moirai-2.0**: **0.722**, rank **7.968**
- **TimesFM-2.5**: **0.645**, rank **5.757**
- **AvgEns: TimesFM + Moirai**: **0.668**, rank **6.45**
- **DeepSeek-V3**: **0.708**, rank **7.73**
- **Gemini-2.0-Flash**: **0.650**, rank **6.603**
- **GPT-4o**: **0.643**, rank **5.466**
- **FuncRev: TimesFM + Gemini**: **0.720**, rank **7.665**
- **CodeRev: TimesFM + Gemini**: **0.713**, rank **6.968**
- **TextRev: TimesFM + Gemini**: **0.653**, rank **5.63**
- **AvgEns: TimesFM + GPT-4o**: **0.627**, rank **4.735**
- **AvgEns: TimesFM + Gemini-2.0-Flash**: **0.619**, rank **4.249**

The paper emphasizes that **CodeRev**, which is best on synthetic **CiK**, underperforms both of its components on TimesX and loses to simple averaging. The stated explanation is that synthetic benchmarks reward instruction following and direct codified transformations, whereas real-world contexts are more ambiguous, partial, and nuanced. The paper attributes the failure of revision methods mainly to **LLM stochasticity** and **failure to preserve temporal structure** in the original forecast. Boxplot analysis is said to show wider error spread and more severe outliers for revision methods than for **TimesFM** alone.

The benchmark also studies the value of context itself. With **Gemini-2.0-Flash**, the context ablation is reported as follows:

| Context | MASE | Rank |
|---|---:|---:|
| Meta | 0.787 | 3.704 |
| Meta + Date | 0.670 | 3.048 |
| Meta + Date + Cov | 0.674 | 3.079 |
| Meta + Date + Event | 0.674 | 2.968 |
| Meta + Date + Event + Cov | **0.650** | **2.635** |

The relative improvement from **Meta** to **Meta + Date + Event + Cov** is approximately

\[
\frac{0.787 - 0.650}{0.787} \approx 17.4\%.
\]

The paper describes this as consistent with an “around 16% better” summary. It also reports that holidays are effective in **12/19 domains** with average intra-domain win rate **65.2%**, while covariates are effective in **12/19 domains** with average intra-domain win rate **57.9%**. Largest holiday gains occur in **Shopping**, **Climate & Environment**, **Public Health**, and **Public Policy & Governance**; largest covariate gains occur in **Specialty & Advanced Materials**, **Raw Materials & Construction**, **Livestock & Food Products**, and **Currency**.

To isolate context quality, the paper replaces **Time-MMD** context with TimesX-generated context while keeping the numeric series, prompts, and **Gemini-2.0-Flash** fixed. The resulting geometric-mean MASE is **0.840** with TimesX context versus **0.906** with Time-MMD context, a **7.3% relative drop** in error:

\[
\frac{0.906 - 0.840}{0.906} \approx 7.3\%.
\]

The benchmark also argues that benchmark scale stabilizes rankings. When subsampling \(K\) variables for \(K \in \{10,20,\dots,190\}\), the paper reports that at **10–40 variables** the uncertainty bands are wide and strongly overlapping, while larger \(K\) narrows the bands and stabilizes rankings. A plausible implication is that many earlier multimodal TSF evaluations were underpowered for reliable ranking.

For probabilistic forecasting, the reported CRPS values are:

- **GPT-4o**: **0.258**
- **Gemini-2.0-Flash**: **0.276**
- **DeepSeek-V3**: **0.277**
- **Sundial**: **0.278**
- **TimesFM-2.5**: **0.293**
- **Moirai-2.0**: **0.327**

All three LLMs outperform all three TFMs on CRPS, which the paper interprets as evidence that textual context helps represent uncertainty over multiple plausible futures.

## 6. Position in the literature and naming distinctions

TimesX occupies a specific place in the recent literature on time series and multimodal forecasting. It is a **benchmark for context-enriched multimodal time-series forecasting**, not an explainer, not a symbolic representation, and not a Transformer forecasting architecture [2607.06973].

This distinction matters because several similarly named methods address different problems. **TimeX** is an inherently interpretable surrogate explainer for time-series models based on **model behavior consistency** and discrete attribution masks [2306.02109]. **TimeX++** is an information-bottleneck-inspired post-hoc explainer that learns explanation-embedded instances intended to remain **label-preserving** and **in-distribution** for the original black-box classifier [2405.09308]. **TimeXer** is a Transformer-based method for **time series forecasting with exogenous variables**, where an endogenous series is forecast using exogenous series as auxiliary context [2402.19072]. **TSAX** is a SAX-family symbolic representation for **time series classification**, not a multimodal forecasting benchmark [2112.12912]. **TIME** is a next-generation benchmark for **zero-shot TSFM evaluation** built around **50 fresh datasets** and **98 forecasting tasks**, but it excludes exogenous covariates and is not the same benchmark as TimesX [2602.12147].

Later work uses TimesX not just as an evaluation set but as evidence for a particular modeling strategy. **PostTime** argues that multimodal forecasting on TimesX should be framed as **context-guided revision of a strong numerical TSFM prior**, rather than direct LLM forecasting. On a refreshed TimesX built with **99 variables** over **2022 to 2025**, the final **Gemma-3-4B + TimesFM-2.5 PostTime** system achieves **ID: 0.738 nMAE, 0.638 nMSE** and **OOD: 0.746 nMAE, 0.597 nMSE**, improving over **TimesFM-2.5** by **6.38% / 12.43%** on ID nMAE/nMSE and **3.93% / 11.24%** on OOD nMAE/nMSE [2605.29401]. This suggests that TimesX has already begun to function as a benchmark for studying when textual context should trigger **revision**, **preservation**, or effective **ignoring** of a numerical prior.

The benchmark’s main scientific significance is therefore methodological. It is designed to make multimodal forecasting evaluation **more realistic, broader in context coverage, and more trustworthy**, and its reported findings reverse several conclusions previously suggested by smaller or synthetic benchmarks.

Source: https://www.emergentmind.com/topics/timesx