---
title: 'MM-TSFlib: Multimodal Forecasting Library'
url: https://www.emergentmind.com/topics/mm-tsflib
type: topic
---

# MM-TSFlib: Multimodal Forecasting Library

MM-TSFlib is an open-source multimodal time-series forecasting library designed to enable rigorous, extensible benchmarking and analysis of numerical and textual time series data. Built as a first-cut reference implementation for the Time-MMD dataset, MM-TSFlib provides a reproducible platform for evaluating the efficacy of augmenting classic time-series forecasting (TSF) models with contextual information extracted from textual sources. Its architecture, evaluation pipeline, and API design collectively address the needs of contemporary time-series analysis research, which increasingly requires integrating multi-domain, multimodal evidence [2406.08627].

## 1. System Architecture and Core Modules

MM-TSFlib is structured around four primary modules, each dedicated to a critical stage of multimodal forecasting:

- **Data Ingestion & Preprocessing**
  - Uses the `Time-MMDDataset` class to load raw CSV or JSON numerical time series, alongside fact and prediction texts that have been preprocessed by large language models (LLMs).
  - The `DataModule` supports chronologically ordered train/validation/test splits and enforces strict non-leakage: no text snippet used at prediction time contains information with timestamps after the numerical input endpoint.
  - Data preprocessing utilities include transformers for imputing missing values, normalization, and sliding window generation.

- **Text Encoder**
  - Wraps any Hugging-Face LLM (e.g., GPT2-small, BERT-base) in a frozen feature extraction mode.
  - Each text snippet is tokenized, passed through the LLM, mean-pooled across token embeddings, and projected via an MLP:
    $$
    \mathbf{h}_\text{text} = W_{\text{proj}}\,\text{pool}(\text{LLM}(\text{text\_tokens})) + b_\text{proj}
    $$
    where $W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm llm}}$.

- **TSF Backbones**
  - Provides a unified interface for over 20 unimodal forecasting architectures, including Transformer, Informer, Autoformer, PatchTST, DLinear, TimesNet, FiLM, among others. Each implements:
    $$
    \hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),
    $$
    where $\mathbf{X}\in\mathbb{R}^{l\times d_{\rm in}}$ is the lookback window, and $\hat{\mathbf{Y}}^{(n)}$ is the numeric prediction.

- **Multimodal Integration & Training**
  - Outputs from the numerical and textual encoders are fused by a learnable linear weighting:
    $$
    \hat{\mathbf{Y}} = \alpha\,\hat{\mathbf{Y}}^{(n)} + (1-\alpha)\,\hat{\mathbf{Y}}^{(t)},
    $$
    where $\hat{\mathbf{Y}}^{(t)} = g_\phi(\mathbf{h}_\text{text})$ is an MLP mapping text features to the forecast output shape. Only $W_{\rm proj}$, $b_{\rm proj}$, $g_\phi$, and $\alpha$ are trained; all LLM and TSF backbone parameters remain frozen.

- **Evaluation Module**
  - Implements mean squared error (MSE), mean absolute error (MAE), and tools for visualizing unimodal versus multimodal performance across domains and horizons [2406.08627].

## 2. Multimodal Data Alignment and Processing Workflow

MM-TSFlib ensures strict temporal and modality alignment throughout its forecasting pipeline:

- **Temporal Alignment**
  - For every sample at time $t$, the numerical lookback window consists of $[t-l+1, \ldots, t]$.
  - Text snippets are filtered to include only those with timestamp intervals $[\tau_s, \tau_e]$ fully contained in $(-\infty, t]$.
  - The $k$ most recent snippets are assembled into the multimodal input $\mathbf{S}$, with $k$ possibly distinct from $l$.

- **Embedding and Projection**
  1. Each snippet is tokenized.
  2. Passed through the LLM to obtain per-token embeddings $\mathbf{T} \in \mathbb{R}^{k \times L \times d_{\rm llm}}$.
  3. Mean pooling yields $\mathbf{S}_{\rm pooled} \in \mathbb{R}^{k \times d_{\rm llm}}$.
  4. Each is linearly projected (with ReLU nonlinearity) to $\mathbf{H}_{\text{text}} \in \mathbb{R}^{k \times d_{\rm out}}$.
  5. An aggregation step (e.g., mean or selection of the last slice) produces a single text representation $\mathbf{h}_\text{text}\in\mathbb{R}^{d_{\rm out}}$.

- **Numerical TSF Backbone**
  - Computes prediction $\hat{\mathbf{Y}}^{(n)}$ as above.

- **Fusion and Training**
  - Fusion is performed using the learnable $\alpha$; only fusion and text-projection parameters are updated during training, minimizing GPU and memory overhead [2406.08627].

## 3. Supported Models and Training Objectives

A wide range of state-of-the-art TSF backbones is supported. The learning objective for all unimodal and multimodal models is, by default, the mean squared error:
$$
\min_{\theta}\;\mathcal{L}_{\rm MSE}
= \frac{1}{N}\sum_{i=1}^N
\left\|\mathbf{Y}_i - \hat{\mathbf{Y}}_i\right\|^2_F
$$
where $N$ is the aggregate of all sliding samples, and $\|\cdot\|_F$ denotes the Frobenius norm. MAE is also supported as an option.

Included models span:
- Transformer, Informer, Reformer, Autoformer, Fedformer, Crossformer, Non-stationary Transformer, iTransformer,
- MLP-based architectures: DLinear, TSMixer, TimeMixer,
- Architecture-agnostic FiLM,
- LLM-based: Time-LLM (using LLMs for direct sequence modeling via reprogramming) [2406.08627].

## 4. API, Configuration, and Example Usage

MM-TSFlib exposes two principal classes for configuration and orchestration:

- **TimeMMDataset** (in `mm_tsflib.data`)
  - Initialization: `TimeMMDataset(root, domain, freq, lookback, horizon, text_window)`
  - Data loading: `get_dataloaders(batch_size, val_split=0.1)` yields train, validation, and test data loaders.

- **MultiModalForecaster** (in `mm_tsflib.model`)
  - Initialization: `MultiModalForecaster(ts_model, ts_hparams, llm_model, proj_dim, fusion)`
  - Methods:
    - `.fit(train_loader, val_loader, epochs, lr_ts, lr_text, log_interval)`
    - `.predict(test_loader)` returns np.ndarray `[n_samples,horizon,d_out]`
    - `.evaluate(test_loader, metrics=["mse","mae"])` returns metric dictionary

Example workflow (Health domain, weekly, lookback=36, horizon=24, text window=8):

```python
from mm_tsflib.data import TimeMMDataset
from mm_tsflib.model import MultiModalForecaster

ds = TimeMMDataset(
    root="data/Time-MMD",
    domain="Health(US)",
    freq="weekly",
    lookback=36,
    horizon=24,
    text_window=8)

train_loader, val_loader, test_loader = ds.get_dataloaders(batch_size=64)

forecaster = MultiModalForecaster(
    ts_model="Autoformer",
    ts_hparams={"d_model":512, "n_heads":8, "e_layers":3, "d_layers":2},
    llm_model="gpt2-small",
    proj_dim=512,
    fusion="weighted_sum")

forecaster.fit(
    train_loader, val_loader,
    epochs=30,
    lr_ts=1e-3,
    lr_text=1e-4,
    log_interval=100)

preds = forecaster.predict(test_loader)
metrics = forecaster.evaluate(test_loader, metrics=["mse","mae"])
print("Test results:", metrics)
```
[2406.08627]

## 5. Evaluation Protocol and Empirical Performance

The library adopts standardized, rigorous evaluation principles:

- **Splitting and Horizons**
  - Supports five horizons per frequency: daily ([48, 96, 192, 336]), weekly ([12, 24, 36, 48]), monthly ([6, 8, 10, 12]).
  - Chronologically ordered splits prevent temporal leakage.
  - Identical hyperparameters are used for both unimodal and multimodal runs to isolate the effect of added modalities.

- **Metrics and Reporting**
  - Primary metric is MSE; MAE is available.
  - Relative multimodal improvement:
    $$
    \mathit{gain} = 1 - \tfrac{\mathrm{MSE}_{\rm multi}}{\mathrm{MSE}_{\rm uni}}
    $$
  - Aggregated over nine domains: mean-MSE reduces from 0.52 (unimodal) to 0.44 (multimodal)—a ~15% average reduction. In text-rich domains such as Health, the reduction reaches 40% (from 0.27 to 0.16).

- **Visualization and Analysis**
  - Utilities for plotting performance gains by backbone and domain are included (see Fig. 3 and 4a, [2406.08627]).

## 6. Hyperparameterization and Experimental Best Practices

Empirical guidelines for maximizing reproducibility and resource efficiency in MM-TSFlib are as follows:

- **Windowing and Batching**
  - Choose lookback, horizon, and text window values matching those from the reference benchmarks for each frequency.
  - Batch size between 32 and 64 is typical.

- **Learning Rates**
  - Use $lr_{ts} \approx 1 \times 10^{-3}$ for the TSF projector, and $lr_{text\_proj} \approx 1 \times 10^{-4}$ for the textual projection components.

- **Fusion and Model Freezing**
  - Initialize fusion weight $\alpha$ at 0.5.
  - Keep LLM and TSF backbone parameters frozen during training for computational scalability (e.g., four to eight domains per run on a single A100 80 GB GPU).

- **Leakage Prevention**
  - Always ensure the end timestamp of all text input is less than or equal to the numerical input endpoint.

- **Model Selection**
  - Small LLMs, such as GPT2-small, suffice for the majority of use-cases; no significant performance boosts were observed with larger models.

- **Monitoring**
  - Track independent loss curves for unimodal and multimodal components. Monitor $\alpha$ to confirm that fusion is contributing to improved forecast accuracy.

A plausible implication is that, by adhering to these protocols and configurations, users can both reproduce results reported for Time-MMD and extend MM-TSFlib to new datasets or backbones for further research [2406.08627].

Source: https://www.emergentmind.com/topics/mm-tsflib