AIMIP Phase 1: systematic evaluations of AI weather and climate models

Abstract: We present the AI weather and climate model intercomparison project (AIMIP), phase 1. Drawing from the rich tradition of intercomparisons in climate model development, we specify a common experiment, output data format, and training constraints (namely, training against historical reanalysis data) for AIMIP Phase 1 models. We aim to identify differences in modeling frameworks and AI architectural choices that influence model behavior, and build trust in AI weather and climate models through open data and evaluation. AIMIP Phase 1 models must simulate the atmosphere given specified historical sea surface temperatures over 1979-2024. We evaluate the models' performance using five major evaluation criteria: biases, trends, response to El Niño-related sea surface temperature anomalies, temporal variability, and out-of-sample generalization tests. We find that the AI models are able to simulate the historical climate and response to forcing as well as a conventional physically-based model, but some AI models underestimate historical warming trends, and their predictions diverge in the out-of-sample generalization tests. We describe the AIMIP Phase 1 dataset that is publicly available for additional evaluations.

• The paper demonstrates that AI weather and climate models, evaluated with a CMIP-compliant protocol using ERA5, can reduce climatological biases compared to traditional CMIP6 models over oceanic regions.
• The paper finds that while several AI models effectively capture historical warming trends and variability, some underpredict trends and dampen daily anomaly amplitudes.
• The paper reveals that diverse AI architectures show varied generalization to +2K/ +4K SST scenarios, highlighting the need for physical constraints and hybrid approaches.

Systematic Evaluation of AI Weather and Climate Models: An Expert Perspective on AIMIP Phase 1

Introduction

The "AIMIP Phase 1: systematic evaluations of AI weather and climate models" (2605.06944) presents a rigorous, CMIP-compliant protocol for the standardized evaluation of AI weather and climate models (AIWCMs) over decadal timeframes. Drawing direct analogies to the established AMIP/CMIP frameworks, this study specifies an intercomparison experiment using ERA5 reanalysis as reference and focuses exclusively on specified SST and SIC forcing. The design directly enables comprehensive, objective assessment of AIWCMs’ performance against key climate evaluation metrics, including climatological means, variability, forced trends, ENSO response, and generalization to strongly out-of-sample conditions.

AIMIP Phase 1 Experimental Design

The AIMIP protocol strictly enforces training of AI components on ERA5 reanalysis (1979–2014), with the test regime spanning 2015–2024. Models are forced only with prescribed monthly SST/SIC and solar insolation, explicitly disallowing atmospheric CO$_2$ or other GHG concentrations as predictors—mitigating overfitting risks, such as learning to "clock" the historical sequence.

The intercomparison involves a five-member ensemble for each AIWCM, evaluating both simulated climate distributions and response to perturbed boundary conditions (i.e., $+$2K/$+$4K SST scenarios). Model outputs are harmonized according to CMIP7 conventions, greatly facilitating the downstream application of established diagnostics and analysis tools within the broader climate science ecosystem. The protocol permits a broad spectrum of AIWCM architectures, spanning pure autoregressive, hybrid, and conditional diffusion/generative models.

Participating Models

Submissions cover a representative cross-section of modern AIWCM designs, including:

• ACE2.1-ERA5: SFNO-based autoregressive emulator (1° Gaussian grid)
• ArchesWeather / ArchesWeatherGen: Vision transformer-based deterministic/probabilistic models (1° grid)
• cBottle-1.3: Conditional diffusion model employing temporally correlated latent sampling (HEALPix ~0.9°)
• DLESyM: UNet-based coupled Earth system emulator with autoregressive GRU components
• MD-1.5 v0.9: Latent autoregressive diffusion at monthly timestep (1.5° grid)
• NeuralGCM / NeuralGCM-HRD: Transformer-based GCM emulators with hierarchical, stochastic diagnostic heads (2.8°/1°)
• CMIP6 GFDL-CM4: Conventional AMIP-class physically-based model (1°) for reference

Model diversity in both structure and methodology reflects the unconstrained specification, intended to benchmark both data-centric and physically-informed architectures under controlled experimental conditions.

Key Results

Climatological Biases

A consistent finding is that AIWCMs, when trained and evaluated under identical boundary conditions, frequently achieve climatological biases (RMSB) relative to ERA5 that are lower than those from the physics-based CMIP6 GFDL-CM4—notably over oceanic domains, where SSTs are prescribed.

Figure 1: Zonal bias of 2-meter temperature and precipitation versus ERA5 for AIWCMs and GFDL-CM4 over both training and test periods.

However, systematic underprediction of warming and biases in land/ice regions persist, with model-to-model variability often exceeding ensemble spread within a given AIWCM. This demonstrates architecture- and data-handling-driven uncertainties, especially where transformations across land-sea boundaries are ill-posed.

Figure 2: Global RMSB for surface and upper air variables; ensemble ranges are typically smaller than inter-model spread.

Trend Fidelity

AIWCMs display varied capacity to reproduce historical warming trends. While some models (e.g., NeuralGCM-HRD, DLESyM, ArchesWeather) track ERA5 and GFDL-CM4 in global-mean $T_{2m}$ through both train and test periods, others (ACE2.1-ERA5, cBottle-1.3, MD-1.5) substantially underestimate both in-sample and out-of-sample trends, failing to generalize even to the relatively modest extrapolation associated with the 2015–2024 regime.

Figure 3: Global and annual mean 2-meter temperature anomalies from the training-period mean; persistent underestimation of trends visible in select AIWCMs.

This trend underestimation is linked to the protocol's explicit exclusion of atmospheric CO$_2$ as a predictor, compelling models to rely entirely on SST/SIC and seasonal insolation cycles, which are less informative regarding radiative-forced changes than in coupled physics-first schemes.

Figure 4: Annual trends in surface variables; AIWCMs generally capture trend *direction

but not always magnitude.*

ENSO and Temporal Variability

ENSO teleconnections and daily weather variability are robustly represented in several models: spatial patterns and regression coefficients for Niño3.4 reproduce observed structures, and generative models (e.g., ArchesWeatherGen, cBottle with temporally correlated latents) achieve realistic daily variability over both land and ocean.

Figure 5: ENSO coefficient maps for training period; AIWCM errors are region-specific and generally <10% of the observed signal in the tropics.

Notably, all AIWCMs tend to underestimate daily anomaly amplitude relative to ERA5, except for GFDL-CM4, which typically overestimates it—highlighting a systematic regime-dependent variance bias in purely data-driven architectures.

Figure 6: Standard deviation of daily anomalies for 2m T and surface precipitation; empirical models capture spatial gradients but with overall damped variance.

Generalization: Strongly Out-of-Sample (+2K, +4K SST)

The critical stress-test—generalization to perturbed uniform SSTs—exposes the sharpest divergences. Models such as NeuralGCM-HRD and DLESyM display physically plausible patterns: proportional oceanic and enhanced continental warming, and precipitation increases in tropical convergence zones. In contrast, others (ACE2.1-ERA5, cBottle-1.3, MD-1.5 v0.9) yield implausible cooling over land and/or attenuated ocean responses, highlighting non-robustness of their learned mappings outside the training distribution.

Figure 7: Model responses to +2K and +4K SST perturbations; only a subset of submissions align with expectations from physics-based GCMs.

This demonstrates the critical importance not only of data coverage but of architectural inductive biases, physical constraints, and the explicit treatment of missing or ill-posed boundary values (i.e., land-ice-SST handling strategies).

Practical and Theoretical Implications

From a climate science perspective, the results confirm that AIWCMs are now sufficiently mature to outperform conventional physics-first models in retrospective, boundary-forced simulation of climatologically-averaged fields—with the crucial caveat that this parity is highly scenario-dependent and does not generalize reliably to forced climate change regimes.

The findings expose several open research fronts:

• Trend extrapolation and causality: Models trained without explicit radiative forcing fail to robustly learn the relationship between SST, external forcing, and atmospheric state—limiting skill in non-stationary scenarios and precluding their use as process-emulating components in forced climate projections.
• Out-of-sample robustness: Strong generalization to unseen boundary conditions remains a bottleneck, aggravated by preprocessing decisions (e.g., how to fill or interpolate SST over land/ice), lack of physical constraints, and limited use of process-level priors in AI architectures.
• Variance representation: While generative AI yields plausible realizations for weather and low-frequency variability, systematic bias towards reduced variance suggests a need to revisit loss formulations and the integration of higher-order statistical structure in model training and architecture.
• Modular coupling: The absence of interactive land and ocean processes limits fidelity in fully coupled Earth system regimes, suggesting that hybrid, modular approaches (potentially leveraging ESM-based physical cores) may be necessary for some applications.

Future Directions

• Relaxed input protocols: Allowing anthropogenic forcing as a predictor, or including direct radiative forcing terms, is likely necessary for robust and causal trend emulation.
• Hybridization and physical constraints: Incorporating explicit process-based constraints, or hybridizing with physics-based dynamical cores, could enhance both robustness and generalization in extrapolative scenarios.
• Ensemble approaches: Increasing ensemble sizes and leveraging model diversity may offer pathways to robust uncertainty quantification for both internal and forced variability components.
• Community benchmarking and datasets: The open, CMIP-compliant dataset and evaluation codebase from AIMIP Phase 1 establishes a community resource to catalyze reproducibility and cross-comparison—a model for subsequent AIWCM development phases.

Conclusion

AIMIP Phase 1 provides a definitive, protocol-driven benchmark for current AIWCMs under common historical and out-of-sample evaluation, with strict reference to ERA5 and established CMIP methodologies. AIWCMs exhibit parity or superiority to state-of-the-art physics-based models in reproducing climatological means and variability over the training interval but demonstrate substantial limitations for trend prediction and robust extrapolation to novel states. Systematic deficiencies in generalization and forced response highlight both the progress made and the remaining challenges toward their operational deployment in climate science applications. Robust future development requires the integration of physical constraints, hybrid approaches, expanded input features, and a continued emphasis on transparent benchmarking.