Zero-Shot Forecasting

• Zero-shot forecasting is a time series prediction approach that leverages learned universal priors to make forecasts on novel data without any in-domain fine-tuning.
• It employs prior-data fitted networks to simulate Bayesian inference in a single pass, using diverse synthetic datasets to capture trends, seasonality, and noise.
• Empirical results, as seen in ForecastPFN, show significant improvements in speed and accuracy in low-data regimes compared to traditional forecasting models.

Zero-shot forecasting is a paradigm in time series analysis wherein a forecasting model—once pre-trained, often either on vast synthetic or cross-domain data or via large-scale representation learning—can be directly applied to make predictions on a novel time series (or under new conditions) without any additional re-training or fine-tuning on that particular target series. This approach is especially powerful for applications plagued by data scarcity, domain shift, or rapidly changing environments that traditional methods are unable to handle efficiently.

1. Fundamentals and Problem Definition

Zero-shot forecasting seeks to address the challenge of making accurate predictions when historical data for a specific target series or under a new regime are too limited—or even entirely unavailable—for model adaptation. Unlike conventional frameworks that require model fitting, transfer learning, or at least some calibration to the new domain, zero-shot methods are designed to leverage strong priors or learned universal representations that generalize robustly.

The concept is best exemplified by models such as ForecastPFN, which is trained entirely on synthetic data expressly designed to cover a diversity of realistic patterns (Dooley et al., 2023). The zero-shot capability is defined as the model’s ability to deliver statistically robust forecasts on new, unseen time series with no in-domain retraining. This distinguishes zero-shot forecasting from few-shot approaches, which allow limited model adaptation to the target domain.

2. Prior-Data Fitted Networks and Bayesian Inference Approximation

A central enabling technique for zero-shot forecasting is the prior-data fitted network (PFN) (Dooley et al., 2023). The theoretical idea is that, for a given class of generative hypotheses $\Phi$ and prior $p(\phi)$, a Bayesian solution would express the posterior predictive for a new point as: $$p(y \mid T, D) \propto \int_{\Phi} p(y \mid T, \phi) p(D \mid \phi) p(\phi) d\phi\,.$$ However, direct computation or approximation of this integral at inference time is usually intractable. PFNs resolve the challenge by instead training a neural network to approximate the Bayesian posterior in a single amortized forward pass, leveraging large amounts of synthetically generated data: $$L_\theta = \mathbb{E}_{D \sim p(D)}\left[\sum_{t=l}^T (y_t - q_\theta(t, \ldots))^2 \right].$$ This approach effectively “simulates” Bayesian inference, yielding a forecast that encodes the full prior-driven uncertainty and structure, without performing explicit Bayesian update computations per new task.

3. Synthetic Data Design and Coverage

A robust zero-shot forecaster depends on a synthetic data distribution that is both diverse enough to foster adaptation to real-world time series and learnable in practice. In ForecastPFN (Dooley et al., 2023), the synthetic generator composes three core components:

• Trend: Linear and exponential growth terms, parameterized as $$\text{trend}(t) = (1 + \alpha t + \beta) \cdot (\gamma^t)$$.
• Seasonality: Multi-scale (e.g., weekly, monthly, yearly) periodicities described via truncated Fourier series:

$$\text{seasonal}(t) = 1 + m_\nu \sum_{f=1}^{\lfloor p_\nu/2 \rfloor} [c_{f,\nu} \sin(2\pi f t/p_\nu) + d_{f,\nu} \cos(2\pi f t/p_\nu)]$$

with decaying variance for higher harmonics.

• Noise: Multiplicative noise sampled from a Weibull distribution:

$$z_t = 1 + (z - \bar{z}), \; z \sim \text{Weibull}(1, k), \; \bar{z} = (\log 2)^{1/k}$$

The final observation is $y_t = \psi(t) \cdot z_t$, with $\psi(t)$ the combined trend and seasonal factor. Hyperparameters governing the diversity and relative prevalence of trends, seasonalities, and noise levels are tuned to maximize generalization without sacrificing learnability.

4. Architectural Properties and Training Paradigm

ForecastPFN utilizes a transformer encoder-based architecture, specifically designed for flexibility. Unlike sequence-to-sequence transformers that fix the prediction horizon, ForecastPFN is constructed to process queries at arbitrary time indices. Each input is tokenized as $(t, y_t)$, with $t$ enriched by various time encodings (year, month, day, etc.).

Architectural specifics:

• Tokenization of temporal inputs to incorporate calendar features
• One layer of multi-head self-attention (with four heads) is paired with two feedforward layers to maintain parameter efficiency and expressivity
• A robust scaling strategy: non-outlier normalization (2σ filtering) and 3σ clipping to prevent extreme synthetic values from corrupting model generalization

The model is trained “offline” across hundreds of thousands of independently generated synthetic series, each sliced into many sliding windows to exponentially increase the number of task instances per epoch. Crucially, the training loss is computed without the injected synthetic noise ($z_t$), allowing the network to learn the regular and seasonal structure underlying noisy observations.

5. Empirical Performance and Efficiency

Exhaustive comparative benchmarks (Dooley et al., 2023) demonstrate that ForecastPFN achieves:

• Lower mean squared error (MSE), mean absolute error (MAE), and mean squared percentage error (MSPE) than established deep learning (Informer, FEDformer, Autoformer), classical (ARIMA, Prophet), and alternative zero-shot (Meta-N-BEATS) methods on established benchmarks (ECL, ETT, Exchange, Illness, Traffic, Weather).
• Substantial speed improvements: inference takes approximately 0.2 seconds—>100x faster than transformer-based models that require in-domain fine-tuning.
• Robustness: Outperforms all baselines in extremely low-data regimes (typically 36-context points), including cases in which baselines have the advantage of access to additional in-distribution data.

A distinguishing result is that zero-shot performance (no in-domain training data) with ForecastPFN is strictly superior to Meta-N-BEATS and competitive or better than other approaches, even when those are allowed to train on hundreds of additional in-domain samples.

6. Applications, Implications, and Limitations

Zero-shot forecasting is critically important in:

• Commercial/industrial applications with start-up or rare-event series (e.g., new product launches, rare diseases, sensor rollouts with scant initial data)
• Real-time, low-latency forecasting systems where per-series retraining or fine-tuning is intolerable
• Operational regimes subject to frequent non-stationarities, where new dynamics emerge for which historical analogs do not exist

The primary limitations are tied to the sufficiency and realism of the synthetic prior. As with any amortized inference solution, performance will degrade if the prior fails to sufficiently cover the local properties of the novel series. Further, complete elimination of domain calibration risks missing local idiosyncrasies that might be present in particular applications. A plausible implication is that ongoing research will explore ways to make synthetic data generation more adaptive, or to hybridize zero-shot and few-shot methodologies for enhanced robustness.

7. Theoretical and Practical Significance

Zero-shot forecasting, as instantiated in ForecastPFN (Dooley et al., 2023), demonstrates that it is feasible to construct a general-purpose forecaster that (a) approximates Bayesian inference in a single amortized pass via pre-training on priors of sufficient diversity, and (b) delivers strong empirical gains in accuracy and computational throughput. This advances the field conceptually by reframing forecasting from being inherently coupled to the target data to being a problem of inference from learned universal priors. In practical terms, it sets a new direction for data-scarce applications, reducing the engineering and computational costs of operational forecasting, and provides a foundation for further innovations in synthetic data-driven model development.