Do AI models produce better weather forecasts than physics-based models? A quantitative evaluation case study of Storm Ciarán (2312.02658v2)

Abstract: There has been huge recent interest in the potential of making operational weather forecasts using machine learning techniques. As they become a part of the weather forecasting toolbox, there is a pressing need to understand how well current machine learning models can simulate high-impact weather events. We compare forecasts of Storm Ciar\'an, a European windstorm that caused sixteen deaths and extensive damage in Northern Europe, made by machine learning and numerical weather prediction models. The four machine learning models considered (FourCastNet, Pangu-Weather, GraphCast and FourCastNet-v2) produce forecasts that accurately capture the synoptic-scale structure of the cyclone including the position of the cloud head, shape of the warm sector and location of warm conveyor belt jet, and the large-scale dynamical drivers important for the rapid storm development such as the position of the storm relative to the upper-level jet exit. However, their ability to resolve the more detailed structures important for issuing weather warnings is more mixed. All of the machine learning models underestimate the peak amplitude of winds associated with the storm, only some machine learning models resolve the warm core seclusion and none of the machine learning models capture the sharp bent-back warm frontal gradient. Our study shows there is a great deal about the performance and properties of machine learning weather forecasts that can be derived from case studies of high-impact weather events such as Storm Ciar\'an.

• The paper demonstrates that ML models reliably capture broad synoptic features of Storm Ciarán but tend to underestimate localized wind extremes.
• It rigorously compares four AI models against traditional NWP approaches using ERA5 reanalysis data to assess forecast accuracy.
• The study underscores the potential for hybrid forecasting systems that combine ML efficiency with the precision of physics-based models.

Evaluation of AI-based Weather Forecast Models: A Study on Storm Ciarán

This paper presents a comprehensive analysis comparing the performance of ML models and numerical weather prediction (NWP) models in forecasting significant weather events, focusing specifically on Storm Ciarán, a European windstorm in November 2023. The paper provides an in-depth evaluation of the forecasts generated by four ML models—FourCastNet, Pangu-Weather, GraphCast, and FourCastNet-v2—against traditional NWP models, measuring their efficacy in predicting the synoptic-scale structures and the dynamic processes of the storm.

Objective and Research Context

The main objective is to assess the ability of ML models to forecast high-impact weather events accurately, given that these models are gaining traction in operational meteorology due to advances in computing power and AI methodologies. The authors examine whether ML models are capable of capturing critical large-scale weather patterns and phenomena necessary for timely public warnings. This paper is particularly significant as it extends beyond the typical evaluations on tropical cyclones and investigates extratropical windstorms, which are less frequently analyzed by ML weather models.

Methodology and Models Assessed

The research involves comparative analyses of four distinct ML models:

1. FourCastNet and FourCastNet-v2: Utilizes Adaptive Fourier Neural Operators and Spherical Fourier Neural Operators, respectively, to model high-resolution atmospheric data using vision transformers.
2. Pangu-Weather: Employs a 3D Earth-specific transformer to incorporate height-integrated atmospheric dynamics, which enhances the neural network's capability in understanding vertical dependencies.
3. GraphCast: Based on Graph Neural Networks, it utilizes a multi-mesh representation to propagate local and long-range information, avoiding the constraints of transformer-based models.

These models were pit against traditional NWP models, such as the IFS HRES forecast and ensemble forecasts from multiple meteorological agencies, which uses data-driven learning schemes from extensive ERA5 reanalysis datasets.

Key Findings and Results

The examination of Storm Ciarán lays bare the strengths and limitations of ML models:

• The ML models displayed competency in predicting the broad synoptic-scale features, such as the cyclone's path and overall atmospheric dynamics. They captured the storm's interactions with the jet stream, which influenced its rapid intensification.
• A significant limitation noted was the ML models' underestimation of maximum wind speeds, particularly in regions where highly localized meteorological phenomena like sting jets could have occurred. The results showed that while ML models could depict general wind patterns, they struggled with fine-scale structures and intense wind events, crucial for precise impact forecasting.
• Although ML models demonstrated a failure to simulate sharp frontal gradients and strong surface winds adequately, they did not exhibit reduced accuracy due to the resolution of training data, as they were trained on ERA5 data consistent with the resolution of NWP models.

Implications and Future Directions

The paper underscores the potential of ML models to complement traditional NWP methods in meteorological forecasting, offering efficiency gains and viable alternatives for certain forecasting tasks. However, there is a clear necessity for further development to address deficiencies in simulating detailed weather phenomena, particularly those with substantial economic and societal impacts.

Future research should prioritize the development of hybrid models that integrate ML advancements with physical-based models to leverage the strengths of both approaches. There is also a call for more extensive case studies examining a wider array of meteorological conditions and conducting robustness assessments across different ML architectures.

Conclusion

The transition towards AI-enhanced weather forecasting is nascent but promising. This paper contributes significantly to understanding the current capabilities and limitations of ML models in operational meteorology and provides a foundation for ongoing improvements. Sustained research and model refinement could lead to enhanced predictive accuracy and a deeper understanding of atmospheric dynamics through the integration of AI technologies with conventional meteorological approaches.