TempoPFN: Univariate Zero-Shot Forecasting

• The paper introduces TempoPFN, a univariate forecasting model that leverages a linear RNN backbone with GatedDeltaProduct and state-weaving for zero-shot prediction.
• It achieves full-sequence parallelism through state-weaving, allowing efficient training and single-pass forecasting without conventional windowing.
• Empirical results show TempoPFN outperforms existing synthetic-only models on CRPS and MASE while ensuring reproducibility and efficient long-horizon inference.

Searching arXiv for TempoPFN and closely related PFN time-series forecasting papers to ground the article in current literature. TempoPFN is a univariate time series foundation model for zero-shot forecasting based on linear Recurrent Neural Networks (RNNs) pre-trained exclusively on synthetic data. It combines a GatedDeltaProduct architecture with state-weaving for fully parallelizable training across sequence lengths, eliminating the need for windowing or summarization techniques while maintaining robust temporal state-tracking. Within the broader literature on Prior-data Fitted Networks and synthetic pre-training for forecasting, a plausible source of nomenclatural ambiguity is that the multivariate forecasting model TimePFN has also been described parenthetically as “TempoPFN” in a separate summary; however, that system is a transformer-based architecture for multivariate time-series forecasting, whereas TempoPFN proper is a univariate linear-RNN foundation model (Moroshan et al., 29 Oct 2025, Taga et al., 22 Feb 2025).

1. Problem setting and model identity

TempoPFN addresses zero-shot time series forecasting under a synthetic-only pre-training regime. The paper positions the model against two practical constraints in foundation models for forecasting: efficient long-horizon prediction and reproducibility. Its central claim is that existing synthetic-only approaches underperform on challenging benchmarks, and that a linear-RNN backbone can be made competitive if paired with an appropriately broad synthetic training prior and a parallelizable recurrence mechanism (Moroshan et al., 29 Oct 2025).

The model is explicitly univariate. Its pre-training corpus consists of synthetic series only, and zero-shot generalization is achieved by withholding any real data during pre-training. This places TempoPFN in a distinct methodological niche relative to forecasting systems that either fine-tune extensively on target data or rely on real-world corpora during pre-training. The paper also emphasizes reproducibility: the complete data generation pipeline and training code are open-sourced, with fixed random seeds and synthetic-data random draws controlled by an RNG seed.

A common misconception is to conflate TempoPFN with multivariate PFN-style models. The most relevant adjacent system in the provided literature is TimePFN, which is designed for multivariate time-series forecasting, uses synthetic multivariate Gaussian-process priors with linear coregionalization, and relies on a channel-mixed transformer encoder rather than a linear RNN (Taga et al., 22 Feb 2025). TempoPFN, by contrast, is centered on univariate forecasting, linear recurrent state updates, and sequence-length parallelism.

2. Linear RNN backbone and GatedDeltaProduct

TempoPFN models a single layer of a linear RNN by the recurrence

$$h_t = A(x_t)\,h_{t-1} + B(x_t), \qquad \hat y_t = \mathrm{Dec}(h_t, x_t),$$

where $x_t \in \mathbb{R}^l$ is the token embedding at time $t$, $h_t \in \mathbb{R}^n$ is the hidden state, $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$ is the data-dependent state-transition matrix, and $B:\mathbb{R}^l \to \mathbb{R}^n$ injects the new token (Moroshan et al., 29 Oct 2025).

The transition operator is parameterized by DeltaProduct, in the GatedDeltaProduct form,

$$A(x_t) = \prod_{j=1}^{n_h} \bigl(I - \beta_{t,j}\,k_{t,j}k_{t,j}^{\top}\bigr),$$

with

$$k_{t,j} = \frac{\psi(W_k^{(j)}x_t)}{\|\psi(W_k^{(j)}x_t)\|_2}, \qquad \beta_{t,j} = \sigma(W_{\beta}^{(j)}x_t),$$

where $\psi(\cdot)$ is SiLU and $\sigma(\cdot)$ is sigmoid. The input term is

$x_t \in \mathbb{R}^l$0

The full recurrence is therefore

$x_t \in \mathbb{R}^l$1

After each recurrence block, the architecture applies a gated MLP formulated as

$x_t \in \mathbb{R}^l$2

where $x_t \in \mathbb{R}^l$3 is elementwise multiplication. In the reported configuration, TempoPFN is a 34.7 M-parameter model with $x_t \in \mathbb{R}^l$4, $x_t \in \mathbb{R}^l$5, and $x_t \in \mathbb{R}^l$6 heads. The design objective is not merely linear recurrence per se, but a linear recurrence whose state transition can be evaluated by associative scan primitives while preserving robust temporal state-tracking.

3. State-weaving and full-sequence parallelism

The architectural mechanism that differentiates TempoPFN from conventional causal RNNs is state-weaving. In a stack of $x_t \in \mathbb{R}^l$7 layers, each layer $x_t \in \mathbb{R}^l$8 has a learnable initial state $x_t \in \mathbb{R}^l$9. The model computes

$t$0

for $t$1 concurrently over $t$2, and then performs the weave update

$t$3

Thus, the final hidden state of layer $t$4 feeds forward into the next layer’s initial state, bypassing any strict causal ordering across layers (Moroshan et al., 29 Oct 2025).

The paper describes this as yielding full per-layer, per-sequence parallelism in $t$5 time on a prefix-sum engine or in chunked $t$6 time with constant-time recurrence per chunk. The associated computational claims are that training and inference complexity per layer is $t$7 independent of $t$8 due to full parallelism, and that memory is $t$9 per sequence, with no $h_t \in \mathbb{R}^n$0 state storage beyond input and output. For comparison, the paper states Transformer complexity as $h_t \in \mathbb{R}^n$1 and causal RNN complexity as $h_t \in \mathbb{R}^n$2 sequentially.

This design has direct implications for forecasting. Inference is a single forward pass that predicts all $h_t \in \mathbb{R}^n$3 future steps in parallel, with no autoregressive loop. The paper further reports that TempoPFN trains approximately $h_t \in \mathbb{R}^n$4 faster than causal LSTM-based TiRex and approximately $h_t \in \mathbb{R}^n$5–$h_t \in \mathbb{R}^n$6 faster than transformer baselines on long-horizon series.

4. Synthetic pre-training corpus and augmentation regime

TempoPFN is pre-trained on 10 M univariate series from ten families, uniformly interleaved. The synthetic generators span Gaussian-process priors, parametric trend-and-seasonality models, causal-kernel constructions, simple waveform families, audio-inspired procedural synthesis, and stochastic differential equations (Moroshan et al., 29 Oct 2025).

Family	Specification
Gaussian-process priors	KernelSynth/GP with sums or products of RBF, RationalQuadratic, ExpSineSquared, White
ForecastPFN-style trends+seasonality	$h_t \in \mathbb{R}^n$7
Causal-kernel	$h_t \in \mathbb{R}^n$8 with random DAG
Sawtooth	slope·$h_t \in \mathbb{R}^n$9 mod $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$0 + small_GP + seasonal
Step	piecewise constant with changepoints and smoothing around boundaries
Anomaly	base GP + random spikes of variable shape
Spikes	baseline + clusters of V/inverted-V/plateau events
SineWave	$A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$1 noise
Audio-inspired	rhythmic event arrivals, volatility bursts, network traffic bursts, multi-scale fractal noise
SDE	OU regime switch with optional fractional Brownian motion with Hurst $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$2

The Gaussian-process component uses

$A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$3

with base kernels drawn from $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$4. The ForecastPFN-style component uses a multiplicative combination of low-order trend terms and harmonic seasonality, with Weibull noise. The SDE component is an OU regime-switch process

$A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$5

where $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$6 is a two-state Markov chain and $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$7 drift via polynomials, sinusoids, logistic trends, with optional fractional Brownian motion.

Offline augmentations are extensive. They include normalization, TS-Mixup early and late, selection of 2–5 transforms from six groups with probability weights $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$8, selection by “change-score” to avoid identity transforms, and final small global scaling plus noise. The transform groups comprise reversal and sign flip; regime-change affine and shock-recovery impulse; calendar dips/spikes and amplitude modulation; Gaussian smoothing, finite differences, and integration; censoring, quantization, and resampling; and random 1D convolution filters with random kernel widths. The pipeline also injects online NaN patterns to simulate missing data. A plausible implication is that TempoPFN’s zero-shot performance is tied not only to synthetic diversity at the generator level but also to aggressive perturbation of observational regimes.

5. Training objective, benchmark protocol, and empirical results

The pre-training schedule uses AdamW in two stages: 3 M iterations at learning rate $A:\mathbb{R}^l \to \mathbb{R}^{n \times n}$9 followed by 2 M iterations at $B:\mathbb{R}^l \to \mathbb{R}^n$0. The model is trained on series of length $B:\mathbb{R}^l \to \mathbb{R}^n$1, with context length and history windows randomly sub-sampled per batch. The forecasting objective is quantile loss over quantiles $B:\mathbb{R}^l \to \mathbb{R}^n$2,

$B:\mathbb{R}^l \to \mathbb{R}^n$3

summed over all $B:\mathbb{R}^l \to \mathbb{R}^n$4 and all time-steps in the horizon. The paper states that there is no explicit $B:\mathbb{R}^l \to \mathbb{R}^n$5 regularization beyond weight decay in AdamW, and dropout appears only in MLPs to stabilize training (Moroshan et al., 29 Oct 2025).

Evaluation is conducted on the Aksu et al. 2024 GIFT-Eval suite, described as 20+ real datasets of varied domains and horizons. The reported metrics are CRPS for quantiles and MASE for point forecasting, with lower values better.

Model	CRPS	MASE
TempoPFN	0.537	0.797
TabPFN-TS	0.544	0.771
TiRex	0.566	0.824
Chronos Bolt B	0.574	0.808
TimesFM 2.0	0.550	0.758

The paper reports that TempoPFN ranks 6th by CRPS and 5th by MASE, outperforms all prior synthetic-only models, and most real-data-trained ones. Its abstract further states that TempoPFN achieves top-tier competitive performance, outperforming all existing synthetic-only approaches and surpassing the vast majority of models trained on real-world data. Robustness to missingness is another reported result: up to 90% missing, TempoPFN’s CRPS degrades by +4% versus TiRex +11%.

These results are significant because they separate three issues often conflated in zero-shot forecasting: the realism of the synthetic prior, the inductive bias of the sequence model, and the computational form of decoding. TempoPFN’s empirical profile suggests that a synthetic-only model need not rely on transformer-style quadratic attention or autoregressive rollout to remain competitive on real-world benchmarks.

6. Reproducibility, scope, and relation to adjacent PFN forecasting models

TempoPFN is presented as a reproducible foundation for future research. The release includes Python scripts for all generators and augmentations, model code in PyTorch with Triton FLA kernels, and training scripts with Accelerate. The reported key hyperparameters are 10 layers, embedding dimension 512, 4 heads per layer, convolution kernel sizes 16–32, batch size 64, total 5 M training iterations, AdamW weight decay $B:\mathbb{R}^l \to \mathbb{R}^n$6, and quantiles $B:\mathbb{R}^l \to \mathbb{R}^n$7 (Moroshan et al., 29 Oct 2025).

Its scope is specifically univariate zero-shot forecasting. That scope contrasts with TimePFN, which addresses multivariate time-series forecasting through a PFN approximation to Bayesian inference. TimePFN generates synthetic multivariate data through diverse Gaussian process kernels and the linear coregionalization method, normalizes each channel, applies 1D convolutional filtering, constructs overlapping patches, and feeds all channel tokens jointly into a multi-head self-attention encoder so that patches from different channels may attend to each other. It is evaluated in both zero-shot and few-shot settings, where fine-tuning with as few as 500 data points nearly matches full dataset training error and even 50 data points yield competitive results (Taga et al., 22 Feb 2025).

This contrast clarifies two distinct strands in PFN-style forecasting. TempoPFN is a univariate, linear-RNN, synthetic-only, zero-shot forecaster optimized for full-sequence parallelism and benchmarked on GIFT-Eval. TimePFN is a multivariate, transformer-based PFN forecaster trained on synthetic Gaussian-process priors and evaluated on nine multivariate benchmarks with zero-shot and few-shot protocols. A plausible implication is that the literature has bifurcated into at least two design programs: one emphasizing multivariate channel mixing and synthetic Bayesian priors, and another emphasizing univariate linear recurrence, efficient long-horizon inference, and end-to-end reproducibility.

At the level of misconceptions, three clarifications are especially important. First, TempoPFN is not a transformer. Second, it is not a multivariate forecasting model in the sense used by TimePFN. Third, despite its RNN backbone, it does not require an autoregressive inference loop, since all future steps are predicted in a single forward pass. These distinctions define its role within the current arXiv literature on PFN-style forecasting.