Robustness of AI-based weather forecasts in a changing climate

Abstract: Data-driven machine learning models for weather forecasting have made transformational progress in the last 1-2 years, with state-of-the-art ones now outperforming the best physics-based models for a wide range of skill scores. Given the strong links between weather and climate modelling, this raises the question whether machine learning models could also revolutionize climate science, for example by informing mitigation and adaptation to climate change or to generate larger ensembles for more robust uncertainty estimates. Here, we show that current state-of-the-art machine learning models trained for weather forecasting in present-day climate produce skillful forecasts across different climate states corresponding to pre-industrial, present-day, and future 2.9K warmer climates. This indicates that the dynamics shaping the weather on short timescales may not differ fundamentally in a changing climate. It also demonstrates out-of-distribution generalization capabilities of the machine learning models that are a critical prerequisite for climate applications. Nonetheless, two of the models show a global-mean cold bias in the forecasts for the future warmer climate state, i.e. they drift towards the colder present-day climate they have been trained for. A similar result is obtained for the pre-industrial case where two out of three models show a warming. We discuss possible remedies for these biases and analyze their spatial distribution, revealing complex warming and cooling patterns that are partly related to missing ocean-sea ice and land surface information in the training data. Despite these current limitations, our results suggest that data-driven machine learning models will provide powerful tools for climate science and transform established approaches by complementing conventional physics-based models.

• The paper demonstrates that AI models like GraphCast, AIFS, and Pangu-Weather show skillful 10-day forecasts in 2023 with varying biases across climate scenarios.
• It employs RMSE and spatial bias analyses to reveal significant cold biases in future warmer scenarios, especially over ocean regions.
• The findings underscore the need for enhanced model training, integration of additional environmental variables, and hybrid approaches for robust climate forecasting.

Robustness of AI-Based Weather Forecasts in a Changing Climate

This paper presents an in-depth investigation into the ability of data-driven weather forecasting models to generalize across varied climate states, representing a critical aspect of leveraging AI for climate science.

Introduction to Data-Driven Weather Forecasting

Recent advancements in data-driven machine learning models for weather forecasting have demonstrated the potential to outperform traditional physics-based models across various metrics. These models, trained on reanalysis datasets like ERA5, showcase reduced computational demands while maintaining competitive forecast accuracy. As the climate continues to change, understanding the generalization capabilities of these models across different historical and future climate scenarios, such as pre-industrial conditions, present-day, and future warmer climates, becomes vital.

Models and Methodology

The study utilizes three state-of-the-art data-driven models: AIFS (Artificial Intelligence Integrated Forecasting System), GraphCast by Google DeepMind, and Pangu-Weather by Huawei. Each model was trained using ERA5 reanalysis data, with some models undergoing additional fine-tuning with recent operational datasets. The models' abilities to perform 10-day forecasts were evaluated under two cold reference scenarios, the pre-industrial proxy year 1955 and present-day 2023, as well as a future scenario for 2049 with a projected warming of 2.9K from pre-industrial levels.

Figure 1: RMSE and bias for AIFS, GraphCast, and Pangu-Weather models across different climate states, illustrating skill and biases.

Forecast Skill and Bias Analysis

The models demonstrated varying levels of skill across different climate states. All three models provided skillful forecasts comparable to operational IFS forecasts in the warmer present-day conditions of 2023, with GraphCast and AIFS exhibiting minimal systematic bias. However, Pangu-Weather consistently exhibited a cold bias, noted across all scenarios (Figure 1).

Figure 2: Global-mean 2m temperature evolution showing forecast deviations across climate conditions, highlighting warming and cooling patterns.

In the colder climate state of 1955, both GraphCast and AIFS demonstrated a warming trend over the forecast period, aligning more closely with modern reanalysis data (Figure 2). Conversely, in the future warmer climate of 2049, AIFS and Pangu-Weather showed significant cooling trends, indicating a drift towards the colder present-day conditions, with biases relatively more pronounced over the ocean than over land. GraphCast maintained a neutral bias on average, achieved by compensating warming and cooling across different regions.

Spatial Bias and Climate Representation

Spatial analysis of forecast biases revealed that the cold biases in future conditions correlated with regions of substantial climatic change between present-day and future scenarios. Notably, the AI models showed consistent cooling over the ocean, suggesting areas where model performance may be limited by inadequate representation of oceanic processes or historical training biases (Figure 3).

Figure 3: Spatial temperature drift patterns in 2049 forecasts, illustrating regional biases and model tendencies.

Implications and Future Directions

The study concludes that while data-driven models provide promising capabilities for weather forecasting across distinct climates, enhancements are required to address systematic biases, particularly in out-of-distribution scenarios. Future research should focus on integrating additional environmental variables, such as ocean and land surface conditions, to bolster model robustness. Moreover, AI-based frameworks could offer significant support in climate sciences through efficient downscaling and scenario projection, complementing traditional models and facilitating more granular climate analyses.

Conclusion

Data-driven weather forecasting models have exhibited substantial potential for climate applications by demonstrating an ability to generalize across varying climate states. However, achieving consistently reliable forecasts across unprecedented climate conditions remains challenging. Addressing these biases in model design and training offers a pathway towards leveraging AI effectively in climate science, ultimately aiding in mitigation and adaptation strategies. This work underscores the transformative role AI could play, suggesting a future in which integrated, hybrid modeling approaches bridge current gaps in climate projection and uncertainty quantification.