Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
12 tokens/sec
GPT-4o
12 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
37 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Uncertainty quantification for data-driven weather models (2403.13458v2)

Published 20 Mar 2024 in physics.ao-ph, stat.AP, and stat.ML

Abstract: AI-based data-driven weather forecasting models have experienced rapid progress over the last years. Recent studies, with models trained on reanalysis data, achieve impressive results and demonstrate substantial improvements over state-of-the-art physics-based numerical weather prediction models across a range of variables and evaluation metrics. Beyond improved predictions, the main advantages of data-driven weather models are their substantially lower computational costs and the faster generation of forecasts, once a model has been trained. However, most efforts in data-driven weather forecasting have been limited to deterministic, point-valued predictions, making it impossible to quantify forecast uncertainties, which is crucial in research and for optimal decision making in applications. Our overarching aim is to systematically study and compare uncertainty quantification methods to generate probabilistic weather forecasts from a state-of-the-art deterministic data-driven weather model, Pangu-Weather. Specifically, we compare approaches for quantifying forecast uncertainty based on generating ensemble forecasts via perturbations to the initial conditions, with the use of statistical and machine learning methods for post-hoc uncertainty quantification. In a case study on medium-range forecasts of selected weather variables over Europe, the probabilistic forecasts obtained by using the Pangu-Weather model in concert with uncertainty quantification methods show promising results and provide improvements over ensemble forecasts from the physics-based ensemble weather model of the European Centre for Medium-Range Weather Forecasts for lead times of up to 5 days.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Preprint, available at https://arxiv.org/abs/2107.07511.
  2. ENS-10: A dataset for post-processing ensemble weather forecasts. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
  3. The quiet revolution of numerical weather prediction. Nature, 525, 47–55.
  4. The rise of data-driven weather forecasting. Preprint https://arxiv.org/abs/2307.10128.
  5. Improving medium-range ensemble weather forecasts with hierarchical ensemble transformers. Preprint, available at https://arxiv.org/abs/2303.17195.
  6. Accurate medium-range global weather forecasting with 3D neural networks. Nature, 619, 533––538.
  7. Evaluation of forecasts by a global data-driven weather model with and without probabilistic post-processing at Norwegian stations. Preprint, available at https://arxiv.org/abs/2309.01247.
  8. A Practical Probabilistic Benchmark for AI Weather Models. Preprint, available at https://arxiv.org/abs/2401.15305.
  9. Potential use of an ensemble of analyses in the ECMWF ensemble prediction system. Quarterly Journal of the Royal Meteorological Society, 134, 2051–2066.
  10. Probabilistic predictions from deterministic atmospheric river forecasts with deep learning. Monthly Weather Review, 150, 215–234.
  11. Generative machine learning methods for multivariate ensemble post-processing. Annals of Applied Statistics, 18, 159–183.
  12. FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead. Preprint, available at https://arxiv.org/abs/2304.02948.
  13. SwinRDM: Integrate SwinRNN with Diffusion Model towards High-Resolution and High-Quality Weather Forecasting. Preprint, available at https://arxiv.org/abs/2306.03110.
  14. FuXi: A cascade machine learning forecasting system for 15-day global weather forecast. npj Climate and Atmospheric Science, 6,.
  15. The Schaake shuffle: A method for reconstructing space–time variability in forecasted precipitation and temperature fields. Journal of Hydrometeorology, 5, 243–262.
  16. The EUPPBench postprocessing benchmark dataset v1.0. Earth System Science Data, 15, 2635–2653.
  17. Why should ensemble spread match the rmse of the ensemble mean? Journal of Hydrometeorology, 15, 1708–1713.
  18. Probabilistic forecasts, calibration and sharpness. Journal of the Royal Statistical Society Series B: Statistical Methodology, 69, 243–268.
  19. Probabilistic forecasting. Annual Review of Statistics and Its Application, 1, 125–151.
  20. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102, 359–378.
  21. Probabilistic solar forecasting: Benchmarks, post-processing, verification. Solar Energy, 252, 72–80.
  22. Deep learning for post-processing ensemble weather forecasts. Philosophical Transactions of the Royal Society A, 379, 20200092.
  23. Isotonic distributional regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 83, 963–993.
  24. The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 146, 1999–2049.
  25. Deep learning for post-processing global probabilistic forecasts on sub-seasonal time scales. Monthly Weather Review, 152, in press.
  26. Ensemble of data assimilations at ECMWF. ECMWF Technical Memorandum 636, available at https://doi.org/10.21957/obke4k60.
  27. Evaluating probabilistic forecasts with scoringRules. Journal of Statistical Software, 90, 1–37.
  28. Keisler, R. (2022). Forecasting global weather with graph neural networks. Preprint, available at https://arxiv.org/abs/2202.07575.
  29. Neural general circulation models. Preprint, available at https://arxiv.org/abs/2311.07222.
  30. Comparison of multivariate post-processing methods using global ECMWF ensemble forecasts. Quarterly Journal of the Royal Meteorological Society, 149, 856–877.
  31. GraphCast: Learning skillful medium-range global weather forecasting. Preprint, available at https://arxiv.org/abs/2212.12794.
  32. Simulation-based comparison of multivariate ensemble post-processing methods. Nonlinear Processes in Geophysics, 27, 349–371.
  33. Forecaster’s dilemma: Extreme events and forecast evaluation. Statistical Science, 32, 106–127.
  34. AtmoRep: A stochastic model of atmosphere dynamics using large scale representation learning. Preprint, available at https://arxiv.org/abs/2308.13280.
  35. Ensemble forecasting. Journal of Computational Physics, 227, 3515–3539.
  36. Flow-dependent versus flow-independent initial perturbations for ensemble prediction. Tellus A: Dynamic Meteorology and Oceanography, 61, 194–209.
  37. Scoring rules for continuous probability distributions. Management science, 22, 1087–1096.
  38. ClimaX: A foundation model for weather and climate. Preprint, available at https://arxiv.org/abs/2301.10343.
  39. Scaling transformer neural networks for skillful and reliable medium-range weather forecasting. Preprint, available at https://arxiv.org/abs/2312.03876.
  40. Palmer, T. (2019a). The ECMWF ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years. Quarterly Journal of the Royal Meteorological Society, 145, 12–24.
  41. Palmer, T. N. (2019b). Stochastic weather and climate models. Nature Reviews Physics, 1, 463–471.
  42. FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators. Preprint, available at https://arxiv.org/abs/2202.11214.
  43. Gencast: Diffusion-based ensemble forecasting for medium-range weather. Preprint, available at https://arxiv.org/abs/2312.15796.
  44. Comparison of Model Output Statistics and Neural Networks to Postprocess Wind Gusts. Preprint, available at https://arxiv.org/abs/2401.11896.
  45. WeatherBench 2: A benchmark for the next generation of data-driven global weather models. Preprint, available at https://arxiv.org/abs/2308.15560.
  46. Neural networks for postprocessing ensemble weather forecasts. Monthly Weather Review, 146, 3885–3900.
  47. Uncertainty quantification in complex simulation models using ensemble copula coupling. Statistical Science, 28, 616–640.
  48. Machine learning methods for postprocessing ensemble forecasts of wind gusts: A systematic comparison. Monthly Weather Review, 150, 235–257.
  49. Can artificial intelligence-based weather prediction models simulate the butterfly effect? Geophysical Research Letters, 50, e2023GL105747.
  50. Statistical postprocessing for weather forecasts: Review, challenges, and avenues in a big data world. Bulletin of the American Meteorological Society, 102, E681–E699.
  51. Easy Uncertainty Quantification (EasyUQ): Generating predictive distributions from single-valued model output. SIAM Review, 66, 91–122.
  52. Physics-based vs. data-driven 24-hour probabilistic forecasts of precipitation for northern tropical Africa. Preprint, available at https://arxiv.org/abs/2401.03746.
Citations (4)
View on Semantic Scholar

Summary

  • The paper demonstrates that integrating UQ methods in Pangu-Weather improves forecast accuracy for lead times up to 5 days.
  • It compares initial condition perturbations and post-hoc approaches to generate probabilistic forecasts from deterministic data.
  • The study shows that post-hoc methods like DRN excel at short lead times, while IC-based methods perform well for longer forecasts.

Uncertainty Quantification for Data-Driven Weather Models: A Case Study on Pangu-Weather

Introduction to Uncertainty Quantification in Weather Forecasting

The recent rise of AI-based, data-driven approaches for weather forecasting has shown significant potential in outperforming traditional numerical weather prediction (NWP) models. These data-driven models, devoid of explicit physical equations, infer atmospheric behavior directly from historical data. Despite their lower computational demands and rapid forecast generation post-training, a considerable focus has primarily been on deterministic outcomes. This approach overlooks the inherent uncertainty in weather forecasting, which is critical for research advancements and decision-making processes in practical applications. Consequently, this paper's central theme is exploring and benchmarking methods for integrating uncertainty quantification (UQ) within a deterministic data-driven weather model — specifically, the Pangu-Weather system.

Study Approach and Methodologies

The paper explores a comparative analysis of UQ methodologies to transmute deterministic forecasts from Pangu-Weather into probabilistic ones. This comparison spans two primary classes of UQ methods:

  1. Initial Condition (IC)-Based Approaches: These methods exploit the concept of generating ensemble forecasts by introducing perturbations to the model's initial conditions. Variants tested include Gaussian noise perturbations, random field perturbations, and initializing with perturbed conditions from a physics-based ensemble weather model.
  2. Post-Hoc (PH) Approaches: Contrasting the IC-based methods, PH methods post-process deterministically generated forecasts to append uncertainty post-estimations. This procedure involves statistical or machine learning techniques, leveraging historical forecast-observation pairs. Two significant methods evaluated are EasyUQ, based on isotonic distributional regression, and Distributional Regression Networks (DRNs), which are NN-based ensemble post-processing methods.

Key Findings and Observations

The paper presents a comprehensive comparison of the aforementioned UQ methods, executed through medium-range forecasts of selected weather variables over Europe. The benchmarking against ensemble outputs from a state-of-the-art physics-based model (ECMWF’s ensemble) reveals insightful findings:

  • Probabilistic forecasts generated through the application of UQ methods on Pangu-Weather demonstrate notable improvements over the ECMWF ensemble for lead times up to 5 days.
  • Among the evaluated methods, the PH approaches, especially DRN, markedly excel at shorter lead times, highlighting their potential in capturing forecast uncertainty beneficially.
  • The IC-based methods, particularly those leveraging random field perturbations, perform commendably at extended lead times, suggesting their utility in reflecting the intrinsic spread of atmospheric conditions.

Theoretical and Practical Implications

The paper’s exploration into UQ methods for data-driven weather forecasting extends significant theoretical and practical implications:

  • Theoretical: The demonstrated methodologies for integrating UQ present a foundational step towards advancing the predictability and reliability of data-driven weather models. This advancement is critical for enriching the theoretical understanding of atmospheric dynamics without relying on traditional physics-based approaches.
  • Practical: From an application standpoint, transitioning deterministic forecasts into probabilistic ones offers enhanced decision-making capabilities, particularly in sectors heavily reliant on accurate and reliable weather predictions.

Future Directions in AI and Weather Forecasting

Looking ahead, this paper paves the way for future endeavors in the field of AI-driven weather forecasting. Investigating inherently probabilistic data-driven approaches, extending the comparison to other state-of-the-art data-driven models, and exploring multivariate UQ strategies represent promising avenues for research. Moreover, the scalability of these methodologies to global, high-resolution forecasts remains an essential area for further exploration, potentially leveraging advancements in machine learning architectures and computational resources.

Concluding Remarks

The research undertaken in this paper contributes significantly to the growing body of knowledge surrounding data-driven weather forecasting. By systematically comparing and analyzing UQ methods within the context of Pangu-Weather, the paper not only benchmarks current capabilities but also outlines a roadmap for future advancements in this exciting field of paper.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown