Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts

Abstract: While AI weather models excel at short-to-medium range forecasts (up to 15 days), they frequently suffer from ill-defined "instabilities" when rolled out over longer horizons. This work addresses the lack of a formal taxonomy by categorizing these failures into three distinct regimes: blow-up, drift, and loss of seasonality, through year-long rollouts of nine state-of-the-art AI weather models. Our analysis reveals that stability hinges on the treatment of small spatio-temporal scales: unstable models amplify high-frequency energy, while stable models act as denoisers when noise is added to their inputs. Far from reducing these models to mere stochastic parrots, our findings highlight that stable models generate unique weather trajectories, conditioned on the initial state. We verify our findings through ablation studies on architectural design choices, conducted using state-of-the-art Vision Transformer (ViT) AI weather model architectures.

• The paper introduces a taxonomy of long-horizon instabilities in AI weather models, classifying failure modes such as blow-up, loss of seasonality, and spectral artifacts.
• It employs quantitative benchmarks and ablation studies to gauge the effects of denoising, model capacity, and temporal conditioning on forecast stability.
• Findings reveal a trade-off between spatial detail and seasonal fidelity, highlighting ongoing challenges in accurately reproducing extreme event statistics.

Assessing the Long-Term Predictability of AI-Based Weather Models: Taxonomy, Stability, and Design

Introduction

Recent advances in AI weather models have led to substantial improvements in short-to-medium range forecasting, but extending predictive skill to multi-week or seasonal horizons remains an unresolved challenge. This paper conducts a rigorous quantitative investigation of long-horizon stability across nine leading deep learning-based weather models, introducing a taxonomy of failure modes, defining robust metrics for instability, and systematically dissecting architecture and data contributions to rollout behavior. The findings elucidate the constraints and capabilities of current AI models for extended-range forecasting and lay out principles critical for their future development.

Taxonomy and Quantitative Characterization of Rollout Instabilities

While autoregressive rollout enables arbitrary forecast horizon, models frequently fail via catastrophic prediction errors. The authors define three primary regimes for these long-rollout instabilities:

1. Blow-up: Characterized by exponential (numerically unbounded) growth in spatial temperature extremes. For example, models such as FourCastNet and GenCast succumb to blow-up within weeks, while Aurora, SFNO, Pangu, and DLESyM sustain rollouts for at least two years without catastrophic divergence. This behavior is distinctly visualized by time series of Arctic-averaged 2-meter temperature.

   Figure 1: Rollouts of Arctic-averaged 2-meter temperature, illustrating long-term behavior and blow-up onset across nine AI weather models.

2. Loss of Seasonality: Assessed via deviation in the large-scale (wavelengths > 5000 km) energy spectra. Loss of the annual cycle signals a collapse of climatological realism. Notably, Pangu collapses to a time-invariant state after a few months for 2-meter temperature, directly attributable to the lack of time embedding. Conversely, graph-based models like AIFS and GraphCast may retain seasonality locally even as they blow up in other variables.
3. Small-Scale Spectral Artifacts: Explored via high-frequency energy content (wavelengths < 250 km), which can either be excessively suppressed (blurring), preserved at realistic amplitudes, or artificially amplified (spectral blow-up). Models such as Aurora and SFNO exhibit minor additional blurring during long rollouts, while GraphCast and AIFS display marked amplification, leading to unphysical artifacts.

Figure 3: Small-scale energy spectra deviations for SFNO (left, physically realistic) and Pangu (right, collapse of spectra and loss of seasonality).

Quantitative benchmarks for the onset and type of instability are established using log-linear regression on extremal values and spectral distances from ERA5 reference data. These metrics allow systematic comparison across models, initialization dates, and variables.

Denoising, Generalization, and Memorization Properties

By introducing noise to initial conditions, the authors probe model responses to out-of-distribution perturbations. Stable models like Aurora and SFNO display marked denoising capabilities, producing realistic fields even from heavily corrupted or synthetic (e.g., cat image) initializations, and converge on distinct, diverse weather trajectories consistent with the provided time embedding.

Figure 4: Error evolution of five models in response to additive $1\sigma$ noise. Aurora and SFNO stabilize; GraphCast amplifies error exponentially.

Absence of memorization is validated through distance-ratio tests: trajectories generated from noise or valid initializations are not closer to any single ERA5 sequence than expected by chance, confirming generative rather than retrieval-based behavior.

Figure 5: Aurora and AIFS output in East Asia under a deliberately shifted seasonal time embedding, revealing rapid convergence to seasonally consistent dynamics regardless of physical initial state.

Time embedding is shown to be a principal driver of seasonal anchoring: when inconsistent temporal information is provided, models override the physical state to align their output with the supplied temporal context, highlighting the need for careful design in temporal conditioning.

Ablation Studies: Architectural and Data Design for Autoregressive Stability

Intensive ablation experiments on the Aurora Vision Transformer family demonstrate that:

• Stability is robust to normalization (LayerNorm vs. RMSNorm), self-attention window size, and vertical atmospheric level representation.
• Time embedding is not required for numerical stability, but is essential for maintaining climatic seasonality.
• The presence of static fields (e.g., land-sea mask) is necessary only if the model is trained with them; omission at training time yields models insensitive to their presence or absence at inference.
• Reducing model capacity below a threshold (e.g., from 113M to 21M parameters) destroys seasonality but does not induce blow-up, indicating a capacity floor for climatological fidelity.
• Spatial and temporal training resolution strongly controls long-term behavior: coarser training (e.g., 24h, 1.5°) yields superior seasonality conservation at the expense of spatial detail, recapitulating findings from climate emulation literature.

Figure 6: Four-year rollouts of 2-meter temperature by Aurora variants under different spatial and temporal resolutions and ablations, highlighting the preservation of seasonality and absence of blow-up.

Seasonality RMSE between models and ERA5 consistently improves with training, but the link to short-range forecast improvements is non-monotonic, suggesting orthogonal aspects of the learning process govern weather and climate-scale fidelity.

Figure 7: RMSE convergence of monthly seasonal cycle versus ERA5 as a function of Aurora\textsubscript{S} training time, and relationship between 1-step RMSE and climatological cycle error.

Assessment of Extremes in Stable Long Rollouts

Long-term realistic weather simulation requires models to faithfully reproduce extreme event statistics. Analysis of Aurora, SFNO, and DLESyM over 10-year rollouts indicates:

• All models produce physically plausible extremes (hot/cold quantile events) in several global regions.
• Aurora and DLESyM tend to display lighter tails (underestimate extremes), while SFNO's bias is region-specific.
• Frequency of extremes for all models deviates from ERA5 at the most severe quantiles, particularly underestimating the rate for the most extreme events.

  Figure 2: QQ plots compare model and ERA5 tail quantiles of regional maxima ("hot") and minima ("cold"), highlighting systematic underestimation of extremes in Aurora and DLESyM.

  

  Figure 9: Extreme event exceedance frequencies at increasing quantile thresholds; a growing discrepancy at higher thresholds indicates lighter model tails.

Thus, while stable models generate long, physically consistent rollouts and capture general frequency and diversity of weather regimes, full statistical realism of extremes remains an open challenge.

Practical and Theoretical Implications

These results collectively suggest that:

• Model stability at long horizons is not tied to specific architectures, but to effective denoising of small-scale noise and appropriate handling of temporal conditioning.
• Stable models such as Aurora and SFNO operate far from trivial memorization, instead generalizing the underlying atmospheric dynamics.
• The ability to denoise from highly perturbed initial states while retaining trajectory diversity implies potential for generative simulation, but also risks if these 'attractors' are not physically valid under unseen long-term climate regimes.
• Achieving both spatially sharp predictions and seasonally accurate long-term dynamics may be fundamentally limited by current autoregressive architectures and training setups, with inherent tradeoffs between resolution, stability, and statistical realism.

From a climate science perspective, these findings are foundational but underscore the limits of applying AI models—trained on a non-stationary climate—to long-term forecasting or extrapolation under distribution shift (e.g., climate change scenarios). Further research into architectures, loss formulations, and explicit physical constraints is warranted. There are also implications for data assimilation strategies and hybrid modeling, where AI-based rollouts could be periodically synchronized with observations or NWP forecasts to bound drift and error accumulation.

Conclusion

This study provides a formal taxonomy and benchmarking framework for analyzing the long-range predictability of modern AI weather models. Quantitative metrics introduced here reveal that only a minority of current models maintain stability and realistic variability beyond two weeks, often by denoising and dynamically generalizing from perturbations rather than by memorization. The ability to sustain physically plausible seasonality and rare event statistics at seasonal to annual scales remains highly sensitive to model architecture, training resolution, and time conditioning.

These results point to a research agenda emphasizing hybrid designs, robust denoising mechanisms, explicit physical constraints, and careful consideration of the spatio-temporal scales present in training datasets. As AI weather and climate models are increasingly deployed for impact studies and real-world decision making, advances in these areas will be crucial for extending the reliability and applicability of deep learning in Earth system prediction.