A PMP-inspired Evaluation Framework for Assessing Deep-Learning Earth System Models

Abstract: In recent years, Deep-Learning Earth System Models (DL-ESMs) have emerged as promising and computationally efficient alternatives to traditional ESMs. Here, we present an evaluation framework for testing DL-ESMs from a traditional model development perspective, utilizing the PCMDI Metrics Package (PMP) standardized diagnostics. This methodology allows DL-ESMs, such as Ai2's ACE2 and Google's NeuralGCM, to be rigorously tested via multiple metrics to access their ability to simulate climatology and key modes of variability in observational reference datasets. By evaluating DL-ESMs as traditional models, we extend their application into uncharted territory and find encouraging results. This evaluation represents a critical step toward establishing trust in DL-ESMs within the scientific community, thus enhancing confidence in their potential to accelerate Earth System modeling, and guiding future model development. Our analysis sheds light on the fit-for-purpose of DL-ESMs offering insights for a wide range of Earth System science applications.