---
title: 'Moirai: Universal Multivariate TSFM'
url: https://www.emergentmind.com/topics/multivariate-tsfm-moirai
type: topic
---

# Moirai: Universal Multivariate TSFM

Multivariate TSFM Moirai is an encoder-based Transformer architecture specifically developed for universal multivariate time series forecasting. Distinguished by its any-variate attention mechanism, patch-based input representations, and flexible masked forecasting objective, Moirai enables parameter-efficient, zero-shot prediction across a wide spectrum of spatial–temporal regimes and multivariate target sizes. It was introduced and benchmarked in both the "Unified Training of Universal Time Series Forecasting Transformers" [2402.02592] and the systematic comparison "No One-Model-Fits-All: Uncovering Spatio-Temporal Forecasting Trade-offs..." [2511.05179], establishing itself as the canonical multivariate foundation model for time series.

## 1. Architectural Foundations: Patch Input and Any-Variate Attention

Moirai treats the historical context of a multivariate time series as a matrix $X \in \mathbb{R}^{K \times W}$, with $K$ channels (or sensors) and window length $W$. The model partitions $X$ into non-overlapping, fixed-length patches—each patch sampled from a single channel at a particular resolution (patch size $p$)—and linearly embeds each patch into a latent vector space, resulting in a flattened token sequence $E \in \mathbb{R}^{T \times d}$.

A core innovation is the "any-variate attention" scheme: every channel's patch sequence is concatenated into a single token sequence, and standard multi-head self-attention is applied. To preserve channel identity and allow cross-channel reasoning, a learned binary bias is injected into the attention scores:

\[
E_{(k,c),(\ell,d)} = (Q_{k,c})^\top [\operatorname{RoPE}(k-\ell)K_{\ell,d}] + u^{(1)}\mathbf{1}_{c=d} + u^{(2)}\mathbf{1}_{c\neq d}
\]

where $(k,c)$ and $(\ell,d)$ index (token, channel), $\operatorname{RoPE}$ denotes rotary positional encodings, and $u^{(1)},u^{(2)}$ are learned scalars. All channels are thus handled permutation-equivariantly, enabling Moirai to accommodate arbitrary sensor sets without channel-embedding tables or graph adjacency matrices.

## 2. Masked Forecasting Pretraining and Objective Function

Moirai leverages a masked-encoder strategy. During pretraining, a random portion (15–50%) of the input patches are masked—i.e., their embeddings are replaced by a shared $\langle \mathrm{MASK} \rangle$ token—and the encoder is tasked with reconstructing the masked patches as continuous values. This aligns with the "masked forecasting" paradigm:

- Mask positions corresponding to unseen future patches.
- Predict parameters $\hat \theta$ of a learnable mixture distribution for each masked patch, minimizing the negative log-likelihood:

\[
\mathcal{L} = -\frac{1}{B}\sum_{i=1}^B \log p\left(y^{(i)}_{l+1:l+h} \mid \hat\theta^{(i)}\right)
\]

The model forecasts the masked (future) window directly, with the mixture head outputting Student’s-$t$, log-normal, negative-binomial, and low-variance Gaussian components. This design is sensitive both to in-distribution uncertainty and to rare/irregular pattern variations.

## 3. Cross-Channel Dependency Learning

Unlike standard univariate TSFMs (e.g., TimesFM, Chronos), which either treat channels independently or aggregate their forecasts post hoc, Moirai’s self-attention operator natively encodes cross-sensor (cross-channel) relationships. The attention matrix $A_h[u,v]$ measures how much patch $u$ (potentially from sensor $i$) attends to (past or simultaneous) patch $v$ (potentially from sensor $j$), allowing:

\[
\alpha_{(i,t),(j,t')} = \mathrm{softmax}_{(j,t')}\left(\frac{q_{i,t} \cdot k_{j,t'}}{\sqrt{d_k}}\right)
\]

This enables joint forecasting strategies, where each forecasted token is both temporally and spatially conditioned, obviating the need for handcrafted sensor graphs (as required by STGNNs) or frequency-level conditioning.

## 4. Pretraining Dataset, Optimization, and Scalability

Pretraining is conducted on the Large-scale Open Time Series Archive (LOTSA), containing over $27 \times 10^9$ scalar observations from 100+ public datasets spanning nine domains. Inputs are subsampled and randomly grouped to form multivariate "pseudo-series" with up to 128 channels. Patch sizes are dynamically selected to match series frequency, and context lengths routinely exceed $1000$ steps.

Relevant optimization details include:

- Model depth $L \in \{6,12,24\}$, $d_{\mathrm{model}} \in \{384,768,1024\}$, $d_{\mathrm{ff}} = 4d_{\mathrm{model}}$.
- Attention heads $H \in \{6,12,16\}$, single shared input/output projections per model (no frequency tags).
- AdamW with learning rate $10^{-3}$, strong sequence packing to eliminate padding waste.

This facilitates training universal models (14M–311M params), supporting arbitrary context/horizon/channel combinations at inference.

## 5. Quantitative Performance and Comparative Evaluation

Empirical evaluations [2402.02592, 2511.05179] highlight Moirai’s strengths in multivariate, zero-shot forecasting:

- On benchmark datasets (Electricity, Solar, Weather, Traffic, Turkey Power), Moirai consistently matches or outperforms state-of-the-art full-shot and alternative TSFMs.
- Example: On Electricity (US), Moirai base attains CRPS 0.050 (best full-shot: 0.048); on Solar, CRPS 0.406 (better than full-shot baseline 0.420).
- On the IoBT spatial sensor deployment (New Mexico testbed, 25 sensors, variable rates), Moirai achieves up to 70% lower MAE than TimesFM and significantly outperforms Chronos, particularly as spatial context (number of sensors $K$) increases.
- Performance improvements with larger $K$ demonstrate Moirai’s native cross-sensor dependency modeling, with MAE dropping from 2.04 to 1.85 at 5 min frequency as $K$ increases from 8 to 25.

A summarizing table (values trace direct to [2511.05179]):

| Sampling Rate | # Sensors $K$ | Chronos (MAE/RMSE) | TimesFM (MAE/RMSE) | Moirai (MAE/RMSE) |
|:------------:|:------------:|:------------------:|:------------------:|:-----------------:|
|   5 min      |      8       |    3.84 / 4.42     |    6.39 / 8.66     |   2.04 / 2.96     |
|   5 min      |     25       |    3.44 / 4.00     |    6.96 / 9.44     |   1.85 / 2.31     |
|  60 min      |      8       |    4.04 / 4.51     | 22.38 / 25.82      |   0.93 / 1.20     |
|  60 min      |     25       |    3.92 / 4.41     | 21.59 / 25.15      |   0.88 / 1.17     |

## 6. Ablation Analysis and Model Specialization

Ablation studies suggest the following:

- Removal of multi-patch-size projections increases normalized MAE from 0.655 to 1.156.
- Excluding "any-variate" attention raises MAE to 0.904, confirming that cross-channel reasoning is central to performance.
- Substituting the flexible mixture likelihood with a single Student’s-$t$ head degrades MAE to 0.740.
- Pretraining only on smaller archives (Monash + GluonTS) results in MAE 0.809, underlining the value of large, cross-domain diversity.

Token-level specialization (as explored in Moirai-MoE [2410.10469]) and related approaches offer further efficiency and accuracy improvements, but the canonical Moirai’s masked-encoder architecture with any-variate attention forms the functional backbone of universal multivariate TSFM.

## 7. Practical Implications and Recommendations

Moirai’s architecture is robust across varying sensor counts, sampling intervals, and application domains. For new datasets, practitioners should:

- Select appropriate patch size $P$ for the series frequency.
- Tune context length to maximize information extraction.
- Normalize each channel separately using window statistics.

For spatial–temporal sensing (e.g., IoT, environmental monitoring), Moirai eliminates the need for hand-crafted spatial graphs or frequency embeddings; its cross-channel attention generalizes to unseen deployments without retraining. Resource scalability is determined primarily by the quadratic token count per context window, with no overhead imposed by per-channel tables.

Graph-based models may continue to excel for sparse, moderately sampled sensor graphs with carefully tuned adjacency matrices; however, Moirai establishes itself as the preferred zero-shot foundation model for dense, heterogeneous, and spatio-temporal multivariate forecasting settings.

Source: https://www.emergentmind.com/topics/multivariate-tsfm-moirai