An Observations-focused Assessment of Global AI Weather Prediction Models During the South Asian Monsoon

Abstract: Seven state-of-the-art AI weather models (FourCastNet, FourCastNet-SFNO, Pangu-Weather, GraphCast, Aurora, AIFS, and GenCast) are evaluated against observational data during the South Asian Monsoon. The models are tested on temperature, winds, global kinetic energy spectrum, regional precipitation, cloud cover, cyclone trajectory prediction, and hyperlocal predictions around extreme weather events. The models forecast large-scale dynamics with reasonable accuracy, but fall short on key metrics critical to Monsoon-time weather prediction. The models exhibit substantially higher errors when compared against ground-based weather station data than against reanalysis or conventional forecasts. The AI weather prediction models show key differences in mesoscale kinetic energy and extreme precipitation during the Monsoon, and predict markedly different Monsoon-time cyclone trajectories over the Indian subcontinent, raising questions about their readiness for operational applications. Our analysis finds that ECMWF's deterministic AIFS model offers the most reliable performance and usability, with GraphCast and GenCast being close seconds.