ArchesWeather & ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting (2412.12971v1)

Abstract: Weather forecasting plays a vital role in today's society, from agriculture and logistics to predicting the output of renewable energies, and preparing for extreme weather events. Deep learning weather forecasting models trained with the next state prediction objective on ERA5 have shown great success compared to numerical global circulation models. However, for a wide range of applications, being able to provide representative samples from the distribution of possible future weather states is critical. In this paper, we propose a methodology to leverage deterministic weather models in the design of probabilistic weather models, leading to improved performance and reduced computing costs. We first introduce \textbf{ArchesWeather}, a transformer-based deterministic model that improves upon Pangu-Weather by removing overrestrictive inductive priors. We then design a probabilistic weather model called \textbf{ArchesWeatherGen} based on flow matching, a modern variant of diffusion models, that is trained to project ArchesWeather's predictions to the distribution of ERA5 weather states. ArchesWeatherGen is a true stochastic emulator of ERA5 and surpasses IFS ENS and NeuralGCM on all WeatherBench headline variables (except for NeuralGCM's geopotential). Our work also aims to democratize the use of deterministic and generative machine learning models in weather forecasting research, with academic computing resources. All models are trained at 1.5{\deg} resolution, with a training budget of $\sim$9 V100 days for ArchesWeather and $\sim$45 V100 days for ArchesWeatherGen. For inference, ArchesWeatherGen generates 15-day weather trajectories at a rate of 1 minute per ensemble member on a A100 GPU card. To make our work fully reproducible, our code and models are open source, including the complete pipeline for data preparation, training, and evaluation, at https://github.com/INRIA/geoarches .

• The paper introduces novel transformer-based deterministic and probabilistic models to significantly improve ML weather forecasting.
• ArchesWeather employs a global cross-level attention mechanism to accurately process large-scale spatial weather data with reduced computational overhead.
• ArchesWeatherGen utilizes flow matching to generate diverse probabilistic forecasts, outperforming traditional models on key meteorological metrics.

Overview of ArchesWeather and ArchesWeatherGen

In the paper "ArchesWeather {content} ArchesWeatherGen: a deterministic and generative model for efficient ML weather forecasting," the authors present innovative methodologies for improving weather forecasting through machine learning. The primary focus is on developing a transformer-based deterministic model named ArchesWeather and a probabilistic model called ArchesWeatherGen, which addresses the complexities inherent in weather prediction.

ArchesWeather: Transformer-based Deterministic Model

The deterministic model, ArchesWeather, is designed as an enhancement over existing models like Pangu-Weather by eliminating constricting inductive biases that typically hinder the flexibility of traditional models. Utilizing a vision transformer architecture adapted for processing large-scale spatial data inherent in weather statistics, ArchesWeather aims to provide robust deterministic forecasts. The model introduces a global Cross-Level Attention mechanism, thereby refining the interaction layers and improving predictive accuracy. At a 1.5º resolution, this model operates with a considerably reduced computational footprint while maintaining high accuracy, notably achieving competitive RMSE values against more resource-intensive models.

ArchesWeatherGen: Probabilistic Weather Model

The probabilistic model, ArchesWeatherGen, leverages flow matching, a modern variant of diffusion models, to generate representative samples from the distribution of potential future weather states. This stochastic emulator was trained to convert ArchesWeather's deterministic predictions into a broader distribution based on the ERA5 data. By addressing the core limitations of deterministic models—such as the tendency towards overly smooth outputs that do not capture extreme meteorological events—ArchesWeatherGen provides a more comprehensive probabilistic forecasting framework. It successfully outperforms established models like IFS ENS and NeuralGCM across various metrics, save for specific variables like geopotential height.

Computational Efficiency and Open Science

One of the significant contributions of this research is demonstrating high performance under limited computational resources. The training of ArchesWeatherGen, inclusive of both deterministic and generative phases, amounts to approximately 23 V100 days—a stark contrast to the computational demands of competing frameworks. Moreover, the authors commit to open science by making all code and models publicly accessible, thereby fostering reproducibility and broader adoption of these methodologies within the meteorological research community.

Implications and Future Directions

The results presented in the paper have substantial implications for both the theoretical understanding and practical deployment of AI-driven weather forecasting systems. By bridging deterministic and generative models, this research offers a scalable and efficient pathway toward more accurate weather predictions. The inclusion of global Cross-Level Attention marks a significant evolution in the processing of complex atmospheric data, potentially inspiring future studies to explore more generalized attention mechanisms in similar large-scale prediction tasks.

The theoretical significance of combining deterministic and generative modeling approaches lies in the potential for unlocking new insights into the probabilistic nature of weather systems. Future work could explore further fine-tuning of diffusion models to better handle the naturally heavy-tailed distribution of atmospheric phenomena and potentially integrate additional sources of data, such as real-time satellite observations, to enhance prediction accuracy and reliability.

In conclusion, the ArchesWeather and ArchesWeatherGen models represent an important step forward in machine learning applications for meteorology, showcasing the value of integrating advanced neural network architectures with domain-specific innovations to tackle complex predictive challenges efficiently.