Dice Question Streamline Icon: https://streamlinehq.com

Attribution of Under-Predictions to Model Deficiencies vs Ground-Truth Differences

Establish whether the slight under-predictions of 2-meter temperature anomaly magnitude by GraphCast and PanguWeather during the 2021 Pacific Northwest heatwave are due to deficiencies in these ML models or are instead a consequence of discrepancies between the ERA5 and HRES-fc0 ground-truth datasets used for evaluation.

Information Square Streamline Icon: https://streamlinehq.com

Background

In the spatial analysis of the Pacific Northwest heatwave, forecasts from GraphCast and PanguWeather slightly under-predicted the temperature anomaly magnitude within the 12 K contour prior to the event, while HRES showed a smaller under-prediction.

Because the paper evaluates ML models against ERA5 and HRES against HRES-fc0, differences between ERA5 and HRES-fc0 climatologies and analyses complicate direct attribution of ML model under-predictions to model shortcomings.

References

One needs to keep in mind that the ERA5 ground truth and the HRES-fc0 ground truth don't coincide exactly, and thus it is not clear whether one can attribute this slight under-predictions of GraphCast and PanguWeather to model deficiencies.

— Validating Deep Learning Weather Forecast Models on Recent High-Impact Extreme Events (2404.17652 - Pasche et al., 26 Apr 2024) in Appendix A, Section A.2: Further Analysis on the Spatial Extent of Forecasts