TimeGrad: Autoregressive Diffusion Forecasting

• The paper's central contribution is combining autoregressive temporal conditioning with diffusion modeling to generate joint multivariate forecasts from sequential data.
• TimeGrad addresses high-dimensional probabilistic forecasting challenges by efficiently modeling complex, non-Gaussian, and multimodal distributions with state-of-the-art CRPS metrics.
• Its architecture integrates an RNN for temporal summarization with a convolutional denoising network to capture cross-series correlations and calibrate uncertainty.

TimeGrad is an autoregressive denoising diffusion model for multivariate probabilistic time series forecasting that samples from the data distribution at each forecast step by estimating its gradient (Rasul et al., 2021). It is designed for settings in which, given a $d$-dimensional history $X_{1:t} \in \mathbb{R}^d$ and covariates $C_{1:t+\tau}$, the objective is to predict future values $X_{t+1:t+\tau}$ while quantifying uncertainty jointly across all dimensions. Its central contribution is to combine autoregressive temporal conditioning with diffusion probabilistic modeling, so that each next multivariate observation is generated by a reverse denoising chain conditioned on the entire realized past rather than by a fixed parametric emission model (Rasul et al., 2021).

1. Forecasting setting and modeling rationale

TimeGrad addresses multivariate probabilistic forecasting in regimes with thousands of correlated dimensions, where predictive distributions can be non-Gaussian, multimodal, and sometimes disconnected (Rasul et al., 2021). In this setting, the target is the full conditional predictive distribution

$p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$

with outputs represented as samples and forecast intervals across all dimensions jointly rather than as marginal point forecasts.

The modeling rationale is explicitly tied to limitations of standard alternatives. Parametric multivariate Gaussians require $O(d^2)$ parameters and $O(d^3)$ computation per step, while flow-based models require structural constraints such as tractable Jacobians and may misrepresent disconnected modes (Rasul et al., 2021). Diffusion and score-based energy-based models instead learn the gradient of the log-density, or score, and sample through Langevin-like dynamics, which the original formulation presents as a way to model complex high-dimensional distributions while retaining calibrated uncertainty (Rasul et al., 2021).

A defining feature of TimeGrad is that it is not a one-shot trajectory generator. At each forecast step $k=1,\dots,\tau$, it learns the joint distribution of the next time point $X_{t+k}$ conditioned on the entire past $(X_{1:t}, X_{t+1:t+k-1})$ and covariates, then feeds the sampled $X_{1:t} \in \mathbb{R}^d$0 back into a recurrent temporal encoder to condition subsequent steps (Rasul et al., 2021). This preserves autoregressive conditioning while using diffusion to model rich multivariate emissions.

2. Diffusion formulation and conditional factorization

For a single forecast time point, TimeGrad uses a standard DDPM-style forward noising process on a clean datum $X_{1:t} \in \mathbb{R}^d$1:

$X_{1:t} \in \mathbb{R}^d$2

with $X_{1:t} \in \mathbb{R}^d$3 and $X_{1:t} \in \mathbb{R}^d$4, so that

$X_{1:t} \in \mathbb{R}^d$5

The learned reverse process is

$X_{1:t} \in \mathbb{R}^d$6

where the conditioning variable encapsulates temporal context (Rasul et al., 2021).

Training minimizes a variational bound on $X_{1:t} \in \mathbb{R}^d$7. Using the tractable Gaussian posterior

$X_{1:t} \in \mathbb{R}^d$8

TimeGrad adopts the standard DDPM noise-prediction parameterization

$X_{1:t} \in \mathbb{R}^d$9

and sets $C_{1:t+\tau}$0. This yields the simplified denoising score-matching objective

$C_{1:t+\tau}$1

In the time-series setting, the per-step conditional loss becomes

$C_{1:t+\tau}$2

where $C_{1:t+\tau}$3 is the temporal encoder state from the previous time point (Rasul et al., 2021).

The horizon-wise structure is autoregressive. The conditional factorization stated in the formulation is

$C_{1:t+\tau}$4

which TimeGrad approximates with

$C_{1:t+\tau}$5

This makes the diffusion model a conditional emission model inside an autoregressive temporal recursion rather than a standalone unconditional generator (Rasul et al., 2021).

3. Temporal conditioning and network architecture

Temporal conditioning is implemented with an autoregressive RNN summarizer,

$C_{1:t+\tau}$6

using a 2-layer LSTM with hidden size 40 (Rasul et al., 2021). Known future covariates, including calendar features, embeddings for categorical metadata, and lag features, are concatenated at each step as inputs to the RNN, so the hidden state carries both historical observations and forecast-period covariate information.

The denoising network $C_{1:t+\tau}$7 predicts per-dimension Gaussian noise at a given noise level $C_{1:t+\tau}$8, conditioned on $C_{1:t+\tau}$9. Its architecture is a conditional 1D dilated convolutional network with residual blocks inspired by WaveNet and DiffWave, operating across the spatial dimension $X_{t+1:t+\tau}$0 to capture cross-series correlations (Rasul et al., 2021). The specification given for the main model uses Fourier positional embeddings of the noise index into $X_{t+1:t+\tau}$1, 8 residual blocks, gated activations $X_{t+1:t+\tau}$2, residual channels equal to 8, alternating dilations $X_{t+1:t+\tau}$3, circular padding, Conv1x1 and Conv1d layers, and skip connections summed to produce the output (Rasul et al., 2021). Conditioning on $X_{t+1:t+\tau}$4 and the noise embedding is injected via learned affine transforms. The stated purpose of this design is to learn a flexible, non-factorized joint score over the $X_{t+1:t+\tau}$5-dimensional emissions at each time step.

This separation of responsibilities is central to the model class. The RNN captures temporal dynamics, while $X_{t+1:t+\tau}$6 captures spatial correlations among series (Rasul et al., 2021). A plausible implication is that TimeGrad decomposes forecasting into temporal state summarization and high-dimensional conditional density estimation, rather than forcing a single network to absorb both roles.

4. Training procedure, sampling, and computational profile

Training uses Adam with learning rate $X_{t+1:t+\tau}$7, batch size 64, diffusion steps $X_{t+1:t+\tau}$8, a linear variance schedule from $X_{t+1:t+\tau}$9 to $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$0, context size equal to prediction length, and early stopping on a validation split of the same size as the test split (Rasul et al., 2021). For real-valued series with heterogeneous scales, each dimension is divided by its context-window mean, or 1 if the mean is zero, and samples are rescaled back at inference; the paper states that this simple normalization strongly improves empirical performance (Rasul et al., 2021).

Inference starts from white noise and applies the DDPM reverse update

$p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$1

with $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$2 if $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$3 and $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$4 at $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$5, using $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$6 (Rasul et al., 2021). The formulation is also described through a Langevin or score-based viewpoint, where the update resembles annealed Langevin dynamics with decreasing step sizes over noise scales, although the implementation uses the DDPM parameterization with a linear $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$7 schedule (Rasul et al., 2021).

For multistep forecasting, TimeGrad performs the following autoregressive rollout: compute the final context state by warm-up on the context window, then for each horizon step draw $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$8, run the reverse chain from $p(X_{t+1:t+\tau}\mid X_{1:t}, C_{1:t+\tau}),$9 to $O(d^2)$0 to obtain $O(d^2)$1, update the RNN with $O(d^2)$2, and repeat (Rasul et al., 2021). Multiple trajectories, for example $O(d^2)$3, are generated to obtain empirical forecast intervals and CRPS estimates.

The complexity discussion in the original formulation emphasizes the contrast with full multivariate Gaussian emissions. Whereas those require $O(d^2)$4 parameters and $O(d^2)$5 per-step loss computation, TimeGrad’s per-step $O(d^2)$6 computation scales roughly as $O(d^2)$7 for $O(d^2)$8 diffusion steps, $O(d^2)$9 residual layers, and $O(d^3)$0 channels, with memory and runtime scaling linearly in $O(d^3)$1 and $O(d^3)$2 up to convolutional constants (Rasul et al., 2021). The main computational cost is sampling: each forecast step requires $O(d^3)$3 reverse passes.

5. Empirical performance and reported behavior

TimeGrad is evaluated on six real-world multivariate datasets: Exchange, Solar, Electricity, Traffic, Taxi, and Wikipedia, spanning dimensions from $O(d^3)$4 to $O(d^3)$5 and forecast lengths of 24 or 30 depending on the dataset (Rasul et al., 2021). Baselines include classical multivariate methods such as VAR, VAR-Lasso, GARCH, and VES, as well as deep probabilistic methods including KVAE, Vec-LSTM variants, GP-scaling, GP-copula, and Transformer-MAF (Rasul et al., 2021).

Using $O(d^3)$6 with lower values better, the reported results are as follows (Rasul et al., 2021):

Dataset	Setting	TimeGrad $O(d^3)$7
Exchange	$O(d^3)$8, daily, pred 30	$O(d^3)$9
Solar	$k=1,\dots,\tau$0, hourly, pred 24	$k=1,\dots,\tau$1
Electricity	$k=1,\dots,\tau$2, hourly, pred 24	$k=1,\dots,\tau$3
Traffic	$k=1,\dots,\tau$4, hourly, pred 24	$k=1,\dots,\tau$5
Taxi	$k=1,\dots,\tau$6, 30-min, pred 24	$k=1,\dots,\tau$7
Wikipedia	$k=1,\dots,\tau$8, daily, pred 30	$k=1,\dots,\tau$9

The paper characterizes these results as state of the art on five of six datasets, with Exchange described as competitive rather than dominant because several baselines tie or slightly outperform in the small-dimensional setting (Rasul et al., 2021). It also states that TimeGrad surpasses flow-based and copula baselines on large, highly correlated datasets.

The ablation on Electricity isolates the role of diffusion depth: good performance appears already at approximately $X_{t+k}$0, the optimum is around $X_{t+k}$1, and larger $X_{t+k}$2 yields diminishing returns when other hyperparameters are fixed (Rasul et al., 2021). Qualitative results on Traffic show coherent joint predictive intervals across neighboring entities that differ in scale by orders of magnitude, which the authors relate to robust handling of cross-series structure and scale normalization (Rasul et al., 2021).

6. Position in subsequent literature, applications, and limitations

Subsequent work has treated TimeGrad as a reference architecture for discrete-time conditional diffusion forecasting. ScoreGrad explicitly frames itself as a generalization from discrete DDPM-style diffusion to continuous-time SDE-based score models, preserving the autoregressive conditioning scheme while replacing discrete noise schedules with continuous SDEs and flexible predictor-corrector samplers (Yan et al., 2021). D3VAE contrasts with TimeGrad more sharply: it uses forward diffusion for augmentation and noise-scale definition, but replaces the reverse diffusion chain with a bidirectional VAE decoder and a single denoising score-matching cleaning step (Li et al., 2023).

Methodological extensions have also been proposed around the forward process rather than the backbone. A decomposable forward diffusion process was introduced as model-agnostic and explicitly compatible with existing diffusion backbones including DiffWave, TimeGrad, and CSDI; its stated purpose is to preserve seasonal structure by staging noise injection according to component energy rather than altering the TimeGrad architecture itself (Caldas et al., 29 Jan 2026). For irregular multivariate time series, LLapDiff places TimeGrad on the discrete-time side of the design space: it describes TimeGrad as operating on discrete-step sequences, typically handling irregularity through masks and temporal encodings, and notes that TimeGrad runs a full reverse chain per forecast step with generation cost $X_{t+k}$3 and average inference time around $X_{t+k}$4 ms in its comparison, versus approximately $X_{t+k}$5 ms for its own horizon-wide latent synthesis (You et al., 19 May 2026).

A recurring misconception is that TimeGrad is a continuous-time or horizon-wide diffusion model. The later comparative literature does not support that reading. TimeGrad uses a discretized grid over forecast steps, and its core mechanism is per-step autoregressive sampling rather than direct horizon-wide generation (You et al., 19 May 2026). Another misconception is that its probabilistic character implies negligible computational cost at inference; the original paper and later comparisons both identify reverse diffusion per forecast step as a material limitation (Rasul et al., 2021, You et al., 19 May 2026).

The limitations stated in the original discussion are consistent with this later positioning. TimeGrad incurs sampling cost because it requires $X_{t+k}$6 reverse steps per forecast time, is sensitive to variance schedules and conditioning design, and can accumulate errors over long autoregressive rollouts even though conditioning on realized samples helps (Rasul et al., 2021). The paper therefore proposes, as future directions, fewer-step samplers such as DDIM, stronger temporal encoders such as Transformers, graph neural networks when entity relations are known, hybrid latent-state models, adaptive schedules, and explicit handling of discrete-valued data in the energy-based setting (Rasul et al., 2021).

TimeGrad has also been used outside benchmark forecasting corpora. In generative forecasting of oil production across four multi-well sites spanning four decades, it was employed as an autoregressive diffusion model coupled with a GRU-based RNN, generating 100 samples for predictive quantiles; the reported study found that TimeGrad forecasts closely align with ground truth, particularly when oil and water were modeled jointly, while also concluding that Informer was more efficient and achieved better overall accuracy on that task (Gandhi et al., 2024). This suggests that TimeGrad’s principal strength remains probabilistic multivariate forecasting with explicit sample-based uncertainty, especially where rich joint structure matters more than one-shot inference speed.