Aurora: Multimodal Time Series Analysis

• Multimodal Time Series Aurora is a framework that integrates numerical, visual, and textual data to enhance forecasting through joint modality fusion.
• It employs innovative techniques such as temporal patch alignment, modality-guided self-attention, and prototype-guided flow matching to improve predictive accuracy.
• Empirical results demonstrate significant gains in classification, anomaly detection, and forecasting benchmarks, highlighting its cross-domain applicability.

Multimodal Time Series Aurora refers to a class of general-purpose time series analysis and forecasting models that integrate information from multiple modalities—such as numerical series, images, text, audio, and tables—to improve predictive and generative capabilities, especially in cross-domain contexts. State-of-the-art instantiations utilize principled architectural innovations involving visual rendering of time series, temporal-patch alignment, modality-guided self-attention, advanced tokenization and distillation, and prototype-guided generative flow modeling. This approach subsumes foundation model reuse, multimodal extension, and cross-modality interaction within a unified generative and interpretative framework, as exemplified by the “Aurora” system and related models.

1. Mathematical Data Representation and Multimodal Rendering

A multimodal time series Aurora system operates on multivariate time series $X \in \mathbb{R}^{C \times T}$, where $C$ is the number of channels (e.g., sensor readings) and $T$ is the temporal dimension. To exploit structure observed by human analysts, $X$ is mapped into a composite visual form: each channel $x^{(c)} \in \mathbb{R}^T$ is rendered as a color-coded line plot $$p^{(c)}: \{1,\ldots,T\} \to \mathbb{R}^2 \to I^{(c)} \in \mathbb{R}^{H \times W \times 3}$$, with RGB channel separation for visual disambiguation. Channels are stacked (typically horizontally) to form a composite image $$I = \mathrm{concat}_{c=1..C} I^{(c)} \in \mathbb{R}^{H \times W \times 3}$$, preserving cross-channel spatial dependencies (Liu et al., 8 Oct 2025).

For additional modalities used in, e.g., aurora forecasting (magnetometer data, all-sky images, satellite tables, logs, audio), all modalities are preprocessed and tokenized via their respective encoders (ViT/CNN for images, TabTransformer for tables, LLM tokenizers for text, spectrogram encoders for audio) (Liu et al., 14 Mar 2025). Each produces a modality-specific representation, which is subsequently mapped into a shared embedding space for joint modeling.

2. Temporal-Aware Patch Alignment and Tokenization

To support temporal reasoning, the rendered image $I$ is partitioned into non-overlapping $P \times P$ patches, inducing a set of visual patch tokens $$Z = \{z_i\}_{i=1}^N \in \mathbb{R}^{N \times (P^2 \cdot 3)}$$, where $N = HW/P^2$. Temporal correspondence is established by mapping each patch's horizontal index to the appropriate time bin: $$f(i) = \left\lfloor \frac{\mathrm{column}(i) \cdot T}{W} \right\rfloor,$$ where $\mathrm{column}(i) = ((i-1) \mod (W/P))$. Patches are grouped by time bin $t$ into sets $S_t$, and each group is aggregated (mean or max) to yield vectors $v_t$, then interpolated to match the number of time-series tokens $N_\mathrm{ts}$. The image and time-series representations are thus temporally aligned at fine granularity (Liu et al., 8 Oct 2025).

Other modalities undergo patching and embedding specific to their structure. For text, BERT-based tokenization is typical; for tables, sequences are reshaped as tabular rows; and for audio, time windows are converted to spectrograms and embedded (Wu et al., 26 Sep 2025, Liu et al., 14 Mar 2025).

3. Model Architectures and Cross-Modal Fusion Strategies

Aurora models exhibit a dual-branch architecture, comprising a vision branch and a numerical branch. The vision branch (e.g., ViT or CNN backbone) ingests the composite image and outputs $$V = \mathrm{VisionEnc}(I) \in \mathbb{R}^{N \times d_v}$$, projected to LLM embedding size $d$ via linear layers. The numerical branch tokenizes and projects the raw time series $X$ into $$T = \mathrm{Tokenizer}(X) \in \mathbb{R}^{N_\mathrm{ts} \times d}$$, following normalization (e.g., RevIN). Additional modalities are encoded in parallel using modality-specific architectures.

Fusion mechanisms include:

• Early fusion: elementwise addition $Z = T + V_\mathrm{align}$ to form a multimodal sequence, fed into a pretrained multimodal LLM (e.g., GPT-2 with frozen attention/FFN, fine-tuned LayerNorm and position embeddings).
• Cross-modal attention: visual and language sequences compute mutual attention, as in $Q_v = V_\mathrm{align} W_q$, $K_t = T W_k$, with attention-mapped features $A = \mathrm{softmax}(Q_v K_t^\top / \sqrt{d})$, and integration via concatenation/addition (Liu et al., 8 Oct 2025).

Aurora's prototype-guided flow matching (Wu et al., 26 Sep 2025) introduces a bank of learnable periodic/trend prototypes $$\mathcal{P} \in \mathbb{R}^{M \times p^{\mathrm{time}}}$$. During inference, the system retrieves prototypes based on current context, initializes generation with $\tilde{\mathcal{P}}$, and solves for the future trajectory using a conditional flow-matching network: $$y^{(t)} = t\,y^{(1)} + (1-t)\,y^{(0)}, \quad v^\theta_t(y|h), \quad \mathcal{L}(\theta, h) = \mathbb{E}_{t, y^{(0)}, y^{(1)}} \|v^\theta_t(y^{(t)}|h) - (y^{(1)}-y^{(0)})\|^2.$$ This generative setup enables probabilistic and conditional forecasting over complex distributions with multimodal guidance.

4. Cross-Modality Interaction and Foundation Model Reuse

Multimodal Aurora models leverage three strategies for exploiting multisource information (Liu et al., 14 Mar 2025):

• Foundation Model Reuse ("Time as X"): converts time series into another modality's representation (e.g., image or text) and processes it with large, pretrained models (ViT, BERT, etc.), achieving efficient knowledge transfer.
• Multimodal Extension ("Time+X"): fuses time series and other modalities within a shared model, using early, late, or intermediate fusion (gated, cross-attention). This supports tasks that require joint reasoning (e.g., aurora forecasting based on combined magnetometer, camera, and tabular data).
• Cross-Modality Interaction ("Time → Text", "Text → Time"): supports tasks such as time series captioning, retrieval, and generation, using joint-embedding contrastive losses or encoder-decoder LLMs.

These capabilities expand the utility of Aurora beyond standard time series prediction to settings involving natural-language descriptions, search, and cross-modal explanations.

5. Training, Pretraining, and Zero-Shot Inference

Aurora is pretrained on large-scale, cross-domain multimodal corpora (>$1$ billion points from datasets such as ERA5, UCR, Monash), with auxiliary image and text representations generated for each series. Token distillation compresses outputs from large vision/language encoders into concise, information-rich tokens. Training minimizes the prototype-guided flow-matching loss; no additional reconstruction or masked objectives are required (Wu et al., 26 Sep 2025).

At inference (zero-shot), new series and available modalities are preprocessed, encoded, distilled, and passed through the guided self-attention and decoding modules. Missing modalities can be masked, and the flow-matching inference mechanism allows flexible adaptation to new tasks and domains without task-specific retraining.

6. Empirical Results and Benchmarks

Aurora achieves state-of-the-art results on standard benchmarks:

• Classification (UEA MTS archive): Aurora reaches ~76.7% accuracy, outperforming OFA (72.2%) by +4.5 points.
• Anomaly Detection (TSB-AD-M): VUS-PR of ~0.349 (vs OFA ~0.296, +0.053).
• Forecasting (ETTh1, ECL, Traffic, Weather, Solar-Energy, TimeMMD, TSFM-Bench, ProbTS): Systematically outperforms unimodal and other multimodal baselines (e.g., VisionTS, PatchTST, TimesNet, Sundial, ROSE, CSDI, MOIRAI) in average MSE, MAE, CRPS, with MSE reduction from 15–31% over strong baselines, and first-place ranking in most experimental settings (Liu et al., 8 Oct 2025, Wu et al., 26 Sep 2025).

Ablation studies confirm the necessity of the vision branch, horizontal stacking + temporal alignment, CLIP-ViT backbones, early fusion, and temporal patch alignment for optimal performance.

Benchmark	Metric	Aurora	OFA	Competing Baseline
UEA MTS (class.)	Accuracy	76.7%	72.2%	VisionTS/GPT4MTS
TSB-AD-M (anom.)	VUS-PR	0.349	0.296	PatchTST
ProbTS (prob. fc.)	CRPS (avg)	Rank 1 (18/20)	–	CSDI, MOIRAI

Further, zero-shot/few-shot capability enables strong cross-domain generalization, and inference runtime remains competitive, with sampling efficiency scaling favorably with sample count up to $\sim$100 (Wu et al., 26 Sep 2025).

7. Open Challenges and Future Directions

Key limitations and challenges include:

• Modality selection: No universally superior rendering; the optimal modality and encoder must be selected for specific domains or tasks (Liu et al., 14 Mar 2025).
• Missing modalities: Real-world settings often exhibit missing data (e.g., absent cameras/logs at some sites). Models must handle such cases with imputation or flexible fusion (e.g., FuseMoE).
• Unseen task generalization: New forms of query require zero-shot cross-modal reasoning, as in chain-of-thought or multimodal question answering.
• Computational cost: Pretraining and inference (especially with flow-matching) remain resource-intensive (e.g., 8$\times$A800 GPUs, 30 days).
• Modal expansion and streaming: Enriching with audio, graphs, or spatial relations, and adapting to online/streaming data, are active areas for advancement (Wu et al., 26 Sep 2025).

Future work will likely focus on efficient distillation, modality-agnostic representations, scalable training and inference, and the development of richer, jointly multimodal pretraining objectives.

---

References:

• "MLLM4TS: Leveraging Vision and Multimodal LLMs for General Time-Series Analysis" (Liu et al., 8 Oct 2025)
• "How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook" (Liu et al., 14 Mar 2025)
• "Aurora: Towards Universal Generative Multimodal Time Series Forecasting" (Wu et al., 26 Sep 2025)