Lag-Llama: Probabilistic Time Series Forecasting

• Lag-Llama is a foundation model for probabilistic time series forecasting that employs lag-based tokenization and decoder-only transformer architecture to output Student’s t distribution parameters.
• The model achieves robust zero-shot and few-shot generalization by pretraining on a heterogeneous corpus of public datasets, often outperforming established baselines in forecast accuracy.
• In some variants, integrating LSTM recurrence enhances long-range dependency modeling for both univariate and multivariate series, ensuring reliable uncertainty quantification.

Lag-Llama is a foundation model for probabilistic time series forecasting, explicitly designed to provide robust zero-shot and few-shot generalization across diverse temporal domains. Employing a decoder-only transformer architecture that incorporates lag-based tokenization and, in some variants, LSTM recurrence, Lag-Llama is trained on a large corpus of time series, delivering state-of-the-art performance for univariate and, in subsequent adaptations, multivariate forecasting tasks. The model outputs the parameters of a Student’s t distribution, enabling probabilistic forecasting and uncertainty quantification.

1. Model Family and Architectural Innovations

The foundational Lag-Llama architecture is a decoder-only transformer, in the style of LLaMA, adapted for time series forecasting (Rasul et al., 2023). Every univariate time series step at time $t$ is transformed into a “token” vector comprising:

• A lag-feature vector $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$, with lags $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$,
• Calendar/covariate features (e.g., hour-of-day, day-of-week),
• Two robust summary statistics—median and interquartile range.

These are concatenated into a context vector $c_t \in \mathbb{R}^{|\mathcal{L}|+F+2}$ and projected into a $d$-dimensional embedding $e_t = W^{(e)} c_t + b^{(e)}$ using a shared linear layer.

The transformer backbone comprises $M$ identical layers, each deploying multi-head causal self-attention with RoPE positional encoding, RMSNorm normalization, and feed-forward networks. The output for each prediction step is a triplet $(\nu, \mu, \sigma)$, with the scalar observation $x_t$ modeled by a Student’s t distribution $x_t \sim \mathrm{StudentT}(\nu,\mu,\sigma)$.

Variants of Lag-Llama also incorporate LSTM recurrence prior to or interleaved with the transformer stack, broadening the receptive field and synergizing sequence modeling mechanisms. This hybrid variant enables effective modeling of longer-range dependencies and is particularly prominent in multivariate or mixed-frequency applications (Bahelka et al., 2024).

2. Pretraining Methodology and Data Corpus

Lag-Llama is pretrained on a heterogeneous corpus of 27 public univariate datasets drawn from six domains: energy, transport, economics, nature, air-quality, and cloud operations, totaling nearly 8,000 series and approximately 352 million sliding-window tokens. The pretraining objective is to maximize the joint log-likelihood of random windows, using negative log-likelihood of predicted Student’s t densities as the loss function:

$$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$0

Data sampling is stratified by dataset size, with frequency-domain augmentations—Freq-Mask and Freq-Mix—applied during training, augmenting random windows of size $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$1. Optimization is performed with Adam ($$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$2, weight decay and dropout tuned to zero for optimal performance), and early stopping is triggered after 50 validation epochs (Rasul et al., 2023).

In multivariate and mixed-granularity settings, as for macroeconomic series, context length is set up to 2,400 daily observations, and time-step tokenization adapts to mixed-frequency alignment via suitable lagged and calendar features (Bahelka et al., 2024).

3. Zero-Shot and Few-Shot Generalization Capabilities

A core property of Lag-Llama is foundation-model generalization: once pretrained, the fixed model can be evaluated on novel downstream time series without any fine-tuning. Zero-shot prediction quality is quantified, primarily, by the continuous ranked probability score (CRPS), averaged across horizons and series.

On seven held-out datasets spanning different domains (weather, pedestrian counts, exchange rates, etc.), Lag-Llama achieves an average rank of 6.7 among 15 strong baselines, often outperforming deep learning baselines (DeepAR, N-BEATS, TFT) on several datasets and matching or exceeding classical methods (ARIMA, ETS) in zero-shot mode (Rasul et al., 2023). For example, on the "ETT-M2" dataset, Lag-Llama achieves CRPS $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$3, outperforming DeepAR ($$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$4), and on "exchange" achieves $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$5.

Fine-tuning on as little as 20% of in-domain data leads to rapid improvement: average rank improves from 6.7 (zero-shot) to 2.8 (finetuned), and, for select datasets (ETT-M2), CRPS is reduced from $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$6 to $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$7—a state-of-the-art result. Lag-Llama maintains the lowest average rank across all levels of training data, with robust performance even in low-data regimes.

4. Probabilistic Forecasting and Output Layer

At inference, Lag-Llama projects the final hidden state to the parameters $$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$8 of the Student’s t distribution:

$$k_t = [x_{t-\ell_1}, ..., x_{t-\ell_{|\mathcal{L}|}}]^\top \in \mathbb{R}^{|\mathcal{L}|}$$9

$$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$0

$$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$1

The density is:

$$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$2

This construction enables not only point forecasts (via $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$3) but also principled uncertainty quantification and full density estimation per time step. In downstream evaluation, forecast performance is quantified using MAE, MAPE, MSE, correlation, $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$4, and adjusted $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$5 metrics, enabling rigorous comparison against econometric baselines.

On euro-area inflation nowcasting (HICP), Lag-Llama achieves MAE of 0.21, MAPE of 0.40%, MSE of 0.07, correlation of 0.92, $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$6 of 0.84, and adjusted $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$7 of 0.81, outperforming a best-case, in-sample Mixed-Data Sampling (MIDAS) regression baseline on every reported metric (Bahelka et al., 2024).

5. Comparative Performance and Model Advantages

Lag-Llama is consistently superior to shallow or linear models—such as MIDAS regression—for both point and probabilistic forecasting in real-world, mixed-frequency contexts (Bahelka et al., 2024). The improvements trace to several properties:

• Nonlinear and long-memory modeling, enabled by the transformer/LSTM backbone, capturing dependencies which are inaccessible to pure OLS or shallow lags.
• Robustness to parameter inflation, since long contexts (e.g., 2,400 daily lags) are compactly modeled in a neural backbone, avoiding parameter explosion typical of linear lagged models.
• Probabilistic output, providing predictive intervals and uncertainty estimates natively.

These modeling advances translate into uniformly lower errors, higher explained variance, and superior forecast correlation for diverse macroeconomic and scientific datasets. Notably, the foundation model was not retrained on the inflation data but nevertheless surpassed the adapted MIDAS regression.

6. Limitations, Open Problems, and Future Directions

Lag-Llama’s present scope is restricted to univariate (and, per some adaptations, multivariate) time series (Rasul et al., 2023, Bahelka et al., 2024). Several open challenges and future research directions are highlighted:

• Multivariate extension: The original model is univariate; extension to conditional and joint multivariate forecasting is an open problem explicitly cited as future work.
• Context requirements: Inference requires a context at least as long as the maximal lag $$\mathcal{L} = \{\ell_1,\dots,\ell_{|\mathcal{L}|}\}$$8.
• Output distribution expressivity: The distribution head is currently limited to the (parametric) Student’s t family; more expressive flows or copula outputs are suggested as future avenues.
• Scaling: Empirical scaling laws imply that further improvements are possible with larger datasets and/or model size, motivating continued corpus expansion.

A plausible implication is that research into natural extensions—such as hierarchical models, joint distributions, and advanced distribution heads—will further embed Lag-Llama as a general foundation model for time series.

7. Applications and Research Impact

Lag-Llama establishes a new benchmark for foundation models in time series forecasting, with verified applicability to a range of forecasting settings, including energy, transportation, cloud operations, and especially economic and financial indicators. Its adaptability for zero-shot and few-shot learning reduces domain-specific retraining requirements, enabling practitioners to harness large-scale pretrained representations in settings with limited target data.

The framework’s methodological simplicity—eschewing custom seasonal decompositions or specialized attention kernels—further streamlines integration into existing pipelines. The adoption of open checkpoints and evaluation on standardized downstream benchmarks permits reproducible, rigorous comparative assessment.

Lag-Llama’s demonstrable superiority in challenging zero- and few-shot regimes positions it as a reference architecture for future research in foundation models for temporal data (Rasul et al., 2023, Bahelka et al., 2024).