MM-TSFlib: Multimodal Forecasting Library

• MM-TSFlib is an open-source multimodal forecasting library that integrates numerical and textual data to enhance classic time-series models.
• Its modular design features dedicated components for data ingestion, text encoding via LLMs, TSF backbones, and a learnable fusion module.
• The framework enforces strict temporal alignment and uses standardized metrics like MSE and MAE to ensure reproducible, rigorous evaluations.

MM-TSFlib is an open-source multimodal time-series forecasting library designed to enable rigorous, extensible benchmarking and analysis of numerical and textual time series data. Built as a first-cut reference implementation for the Time-MMD dataset, MM-TSFlib provides a reproducible platform for evaluating the efficacy of augmenting classic time-series forecasting (TSF) models with contextual information extracted from textual sources. Its architecture, evaluation pipeline, and API design collectively address the needs of contemporary time-series analysis research, which increasingly requires integrating multi-domain, multimodal evidence (Liu et al., 2024).

1. System Architecture and Core Modules

MM-TSFlib is structured around four primary modules, each dedicated to a critical stage of multimodal forecasting:

• Data Ingestion & Preprocessing
  • Uses the Time-MMDDataset class to load raw CSV or JSON numerical time series, alongside fact and prediction texts that have been preprocessed by LLMs.
  • The DataModule supports chronologically ordered train/validation/test splits and enforces strict non-leakage: no text snippet used at prediction time contains information with timestamps after the numerical input endpoint.
  • Data preprocessing utilities include transformers for imputing missing values, normalization, and sliding window generation.
• Text Encoder
  • Wraps any Hugging-Face LLM (e.g., GPT2-small, BERT-base) in a frozen feature extraction mode.
  • Each text snippet is tokenized, passed through the LLM, mean-pooled across token embeddings, and projected via an MLP:

  $$\mathbf{h}_\text{text} = W_{\text{proj}}\,\text{pool}(\text{LLM}(\text{text\_tokens})) + b_\text{proj}$$

  where $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$.

• TSF Backbones

  • Provides a unified interface for over 20 unimodal forecasting architectures, including Transformer, Informer, Autoformer, PatchTST, DLinear, TimesNet, FiLM, among others. Each implements:

  $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$

  where $\mathbf{X}\in\mathbb{R}^{l\times d_{\rm in}}$ is the lookback window, and $\hat{\mathbf{Y}}^{(n)}$ is the numeric prediction.

• Multimodal Integration & Training

  • Outputs from the numerical and textual encoders are fused by a learnable linear weighting:

  $$\hat{\mathbf{Y}} = \alpha\,\hat{\mathbf{Y}}^{(n)} + (1-\alpha)\,\hat{\mathbf{Y}}^{(t)},$$

  where $$\hat{\mathbf{Y}}^{(t)} = g_\phi(\mathbf{h}_\text{text})$$ is an MLP mapping text features to the forecast output shape. Only $W_{\rm proj}$, $b_{\rm proj}$, $g_\phi$, and $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$0 are trained; all LLM and TSF backbone parameters remain frozen.

• Evaluation Module

  • Implements mean squared error (MSE), mean absolute error (MAE), and tools for visualizing unimodal versus multimodal performance across domains and horizons (Liu et al., 2024).

2. Multimodal Data Alignment and Processing Workflow

MM-TSFlib ensures strict temporal and modality alignment throughout its forecasting pipeline:

• Temporal Alignment
  • For every sample at time $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$1, the numerical lookback window consists of $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$2.
  • Text snippets are filtered to include only those with timestamp intervals $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$3 fully contained in $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$4.
  • The $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$5 most recent snippets are assembled into the multimodal input $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$6, with $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$7 possibly distinct from $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$8.
• Embedding and Projection
  1. Each snippet is tokenized.
  2. Passed through the LLM to obtain per-token embeddings $$W_{\rm proj}\in\mathbb{R}^{d_{\rm out}\times d_{\rm LLM}}$$9.
  3. Mean pooling yields $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$0.
  4. Each is linearly projected (with ReLU nonlinearity) to $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$1.
  5. An aggregation step (e.g., mean or selection of the last slice) produces a single text representation $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$2.

• Numerical TSF Backbone

  • Computes prediction $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$3 as above.
• Fusion and Training
  • Fusion is performed using the learnable $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$4; only fusion and text-projection parameters are updated during training, minimizing GPU and memory overhead (Liu et al., 2024).

3. Supported Models and Training Objectives

A wide range of state-of-the-art TSF backbones is supported. The learning objective for all unimodal and multimodal models is, by default, the mean squared error:

$\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$5

where $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$6 is the aggregate of all sliding samples, and $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$7 denotes the Frobenius norm. MAE is also supported as an option.

Included models span:

• Transformer, Informer, Reformer, Autoformer, Fedformer, Crossformer, Non-stationary Transformer, iTransformer,
• MLP-based architectures: DLinear, TSMixer, TimeMixer,
• Architecture-agnostic FiLM,
• LLM-based: Time-LLM (using LLMs for direct sequence modeling via reprogramming) (Liu et al., 2024).

4. API, Configuration, and Example Usage

MM-TSFlib exposes two principal classes for configuration and orchestration:

• TimeMMDataset (in mm_tsflib.data)
  • Initialization: TimeMMDataset(root, domain, freq, lookback, horizon, text_window)
  • Data loading: get_dataloaders(batch_size, val_split=0.1) yields train, validation, and test data loaders.
• MultiModalForecaster (in mm_tsflib.model)
  • Initialization: MultiModalForecaster(ts_model, ts_hparams, llm_model, proj_dim, fusion)
  • Methods:
  • .fit(train_loader, val_loader, epochs, lr_ts, lr_text, log_interval)
  • .predict(test_loader) returns np.ndarray [n_samples,horizon,d_out]
  • .evaluate(test_loader, metrics=["mse","mae"]) returns metric dictionary

Example workflow (Health domain, weekly, lookback=36, horizon=24, text window=8):

$\mathbf{X}\in\mathbb{R}^{l\times d_{\rm in}}$3 (Liu et al., 2024)

5. Evaluation Protocol and Empirical Performance

The library adopts standardized, rigorous evaluation principles:

• Splitting and Horizons
  • Supports five horizons per frequency: daily ([48, 96, 192, 336]), weekly ([12, 24, 36, 48]), monthly ([6, 8, 10, 12]).
  • Chronologically ordered splits prevent temporal leakage.
  • Identical hyperparameters are used for both unimodal and multimodal runs to isolate the effect of added modalities.
• Metrics and Reporting
  • Primary metric is MSE; MAE is available.
  • Relative multimodal improvement:

  $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$8 - Aggregated over nine domains: mean-MSE reduces from 0.52 (unimodal) to 0.44 (multimodal)—a ~15% average reduction. In text-rich domains such as Health, the reduction reaches 40% (from 0.27 to 0.16).

• Visualization and Analysis

  • Utilities for plotting performance gains by backbone and domain are included (see Fig. 3 and 4a, (Liu et al., 2024)).

6. Hyperparameterization and Experimental Best Practices

Empirical guidelines for maximizing reproducibility and resource efficiency in MM-TSFlib are as follows:

• Windowing and Batching
  • Choose lookback, horizon, and text window values matching those from the reference benchmarks for each frequency.
  • Batch size between 32 and 64 is typical.
• Learning Rates
  • Use $\hat{\mathbf{Y}}^{(n)} = f_\theta(\mathbf{X}),$9 for the TSF projector, and $\mathbf{X}\in\mathbb{R}^{l\times d_{\rm in}}$0 for the textual projection components.
• Fusion and Model Freezing
  • Initialize fusion weight $\mathbf{X}\in\mathbb{R}^{l\times d_{\rm in}}$1 at 0.5.
  • Keep LLM and TSF backbone parameters frozen during training for computational scalability (e.g., four to eight domains per run on a single A100 80 GB GPU).
• Leakage Prevention
  • Always ensure the end timestamp of all text input is less than or equal to the numerical input endpoint.
• Model Selection
  • Small LLMs, such as GPT2-small, suffice for the majority of use-cases; no significant performance boosts were observed with larger models.
• Monitoring
  • Track independent loss curves for unimodal and multimodal components. Monitor $\mathbf{X}\in\mathbb{R}^{l\times d_{\rm in}}$2 to confirm that fusion is contributing to improved forecast accuracy.

A plausible implication is that, by adhering to these protocols and configurations, users can both reproduce results reported for Time-MMD and extend MM-TSFlib to new datasets or backbones for further research (Liu et al., 2024).