AIMIP Phase 1: AI Weather–Climate Model Intercomparison

• AIMIP Phase 1 is the first coordinated experiment that standardizes AI weather–climate model evaluation through AMIP-style, atmosphere-only simulations with controlled SST/SIC forcing.
• It employs ERA5 reanalysis data and CMIP-inspired data conventions to benchmark diverse AI architectures on historical climate metrics including biases, trends, and ENSO teleconnections.
• Key findings reveal that while some AI models outperform traditional references in mean-state metrics, discrepancies appear under uniform warming tests, underscoring the impact of training choices.

AIMIP Phase 1 is the first coordinated Artificial Intelligence Model Intercomparison Project focused on weather–climate AI models performing multi-decadal, atmosphere-only “AMIP-style” simulations. It standardizes a common experiment, output conventions, and training constraints for AI weather and climate models (AIWCMs): participants train exclusively on ERA5 reanalysis, simulate the atmosphere under specified historical sea surface temperatures (SST) and sea ice concentration (SIC) from 1979 to 2024, and are evaluated against five major criteria—biases, trends, response to El Niño-related SST anomalies, temporal variability, and out-of-sample generalization tests (Henn et al., 7 May 2026). By adopting CMIP-like data standards while retaining diverse native AI architectures, AIMIP Phase 1 positions AIWCM evaluation within the methodological vocabulary of climate-model intercomparison rather than short-horizon forecast skill alone (Henn et al., 7 May 2026).

1. Purpose, scope, and relation to AMIP

AIMIP Phase 1 is explicitly framed as an AI analogue of the climate community’s longstanding intercomparison tradition. Its design borrows from AMIP and CMIP by prescribing SST/SIC boundary conditions, requiring standardized outputs, and emphasizing controlled comparisons of climate statistics and forced responses rather than only case-by-case weather prediction (Henn et al., 7 May 2026). The scope is multi-decadal and atmosphere-only: models simulate the atmosphere given specified historical SSTs, rather than coupling to an interactive ocean.

The choice of specified SST is central. In the AIMIP protocol, prescribed SST/SIC isolates atmospheric model behavior from coupled ocean feedbacks and minimizes confounding instability in nascent AI ocean components. This suggests that Phase 1 is not intended to evaluate fully coupled Earth system realism; rather, it isolates whether AI atmospheric models can recover historical climate, variability, and forced responses under controlled lower-boundary forcing (Henn et al., 7 May 2026).

A distinctive training constraint is the exclusion of CO2 and other anthropogenic radiative forcings as model input features in standard simulations. The stated rationale is to reduce overfitting risk, because AI models can otherwise use monotonic CO2 as a “clock” to memorize the timing of events. Even so, the protocol notes that AIWCMs can still learn aspects of forced change because global-mean SST trends correlate with net anthropogenic forcing trends (Henn et al., 7 May 2026). A common misconception is therefore that AIMIP Phase 1 directly tests AI climate models under full historical forcing parity with CMIP-class models; in fact, it tests SST/SIC-forced atmospheric behavior under a deliberately constrained input design.

2. Common experiment design and data conventions

The common experiment is anchored to ERA5. Training data span January 1979 through December 2014, while the standard AIMIP simulation window runs from October 1, 1978 to December 31, 2024, with a roughly three-month spinup. The designated out-of-sample period is January 2015 through December 2024 (Henn et al., 7 May 2026). ERA5 is also the sole reanalysis used for evaluation, which ensures comparability across participants but introduces the caveat that ERA5’s own imperfections, especially in precipitation and some trends, propagate into the benchmark.

The SST/SIC forcing dataset is custom-built from daily ERA5 fields at $0.25^\circ$ resolution, averaged into monthly values using centered windows to preserve annual means and avoid mid-month overshoot artifacts. The series is extended through December 2024, with values at January 1, 2025 to enable interpolation. AIMIP therefore does not use HadISST, ERSST, or OISST for Phase 1; it uses an ERA5-derived SST/SIC product for consistency with the ERA5 training and verification workflow (Henn et al., 7 May 2026).

Required outputs include three-dimensional fields on pressure levels—at minimum 1000, 850, 700, 500, 250, 100, and 50 hPa—for temperature $T$, specific humidity $q$, winds $u$ and $v$, plus 500 hPa geopotential height $Z500$. Surface diagnostics, if predicted, include surface pressure and/or mean sea-level pressure, surface temperature, 2 m air temperature and dewpoint or specific humidity, 10 m winds, and surface precipitation rate (Henn et al., 7 May 2026).

Submissions remain on native grids, but horizontal resolution must not exceed roughly 500 km. Participating models include regular latitude–longitude grids at $1^\circ$, $1.5^\circ$, and $2.8^\circ$, as well as HEALPix grids near $0.9^\circ$. Monthly means are required for the full October 1978–December 2024 period, and daily means are required for 1979 and 2024 to assess weather variability and extremes. Data are distributed in CF-compliant NetCDF, while CMIP6 reference data may use ARCO Zarr; participants are encouraged to use CMOR tooling and CMIP-like variable names and metadata (Henn et al., 7 May 2026).

Five-member ensembles are required, generated by any means such as initial-condition perturbations, stochastic sampling, or parameter perturbations. Phase 1 also specifies out-of-sample stress tests with uniform SST perturbations of $T$0 K and $T$1 K, labeled “amip-p2k” and “amip-p4k,” to probe generalization under distribution shift (Henn et al., 7 May 2026).

3. Model roster and architectural diversity

AIMIP Phase 1 compares a heterogeneous set of AIWCMs against a conventional physically based reference, NOAA GFDL CM4 with the AM4 atmosphere component. The participating AI models differ in prognostic cadence, stochasticity, handling of missing SST over land and sea ice, and diagnostic-head design (Henn et al., 7 May 2026).

Model	Institution or group	Distinguishing features
ACE2.1-ERA5	Allen Institute for AI	6-hourly autoregressive SFNO emulator; deterministic; auxiliary MLP diagnostic decoders
ArchesWeather	INRIA/ArchesWeather team	CNN encoder–decoder plus SwinTransformer with earth-specific attention; daily-averaged autoregressive training with MSE
ArchesWeatherGen	INRIA/ArchesWeather team	Probabilistic residual generative model using flow matching; ensembles via noise seeds
cBottle-1.3	NVIDIA	Conditional diffusion model on HEALPix; correlated latent sampling with AR1 half-life $T$2 h
DLESyM	University of Washington + NVIDIA	ConvNeXt + GRU U-Net; 6-hourly autoregressive forecasts with RMSE loss
MD-1.5 v0.9	University of Maryland PARETO group	Monthly latent diffusion emulator with conditional VAE and SFNO-based low-rank spectral operators
NeuralGCM	Google Research	Hybrid physics–ML atmospheric GCM; CRPS-trained diagnostic heads; GRF stochasticity
NeuralGCM-HRD	Google Research	High-resolution decoder downscaling to $T$3; scale consistency constraints
GFDL CM4 / AM4	NOAA GFDL	Physically based CMIP6 AMIP and amip-p4k reference

Several architectural choices became scientifically consequential. Models differ in how they handle missing SST over land and sea ice: some use climatological fills and masks in the loss, some interpolate SST zonally over land, and others merge SST with land/sea-ice embeddings. This detail later proves important for out-of-sample warming response (Henn et al., 7 May 2026).

The comparison also spans deterministic and probabilistic paradigms. ACE2.1-ERA5 and ArchesWeather are deterministic autoregressive emulators, while ArchesWeatherGen, cBottle-1.3, and NeuralGCM diagnostic heads explicitly introduce stochasticity through flow matching, diffusion, or Gaussian Random Fields. A plausible implication is that AIMIP Phase 1 is not merely comparing parameter counts or backbone families; it is evaluating whether different inductive biases produce distinct climatological behaviors under a common forcing protocol.

4. Evaluation methodology

AIMIP computes its principal metrics on regridded fields. Because submissions use different native meshes, model outputs are mapped to a common $T$4 grid before metric computation: conservative regridding via xESMF for most grids, and nearest-neighbor remapping for HEALPix outputs. Pressure-level analyses mask locations where levels are below ground surface using the CM4 land–orography mask (Henn et al., 7 May 2026).

Bias and root-mean-square bias are basic diagnostics. For a scalar field $T$5,

$T$6

and weighted RMSB is

$T$7

AIMIP also uses conventional MSE and RMSE definitions for field errors (Henn et al., 7 May 2026).

Trend estimation is based on ordinary least squares. For a time series $T$8,

$T$9

with

$q$0

Confidence intervals are derived from the standard OLS error model, and trends are reported on raw series rather than detrended anomalies (Henn et al., 7 May 2026).

ENSO response is quantified by regression onto the Niño3.4 index computed from the AIMIP SSTs:

$q$1

The coefficient field $q$2 is mapped spatially and compared against ERA5. Temporal variability is assessed from daily anomalies relative to monthly means, with variance

$q$3

and lag-based autocorrelation. Power spectral density and ACC are defined in the protocol as recommended extensions, but variance of daily anomalies is a core Phase 1 metric (Henn et al., 7 May 2026).

The out-of-sample generalization tests are deliberately asymmetric: the 2015–2024 holdout has an observational target in ERA5, whereas the $q$4 K and $q$5 K SST perturbation experiments do not. For the perturbed runs, plausibility is judged against physical expectations and comparison to the conventional CMIP6 reference, GFDL CM4, rather than against definitive truth (Henn et al., 7 May 2026).

5. Principal findings

The headline result is that AIWCMs can reproduce many aspects of the historical climate under specified SST/SIC as well as, and often better than, the conventional physically based reference in terms of mean-state error. Across surface and pressure-level variables, AI models generally achieve lower RMSB against ERA5 than GFDL CM4 AMIP. Biases are largest over land and sea ice, as in CM4, and smaller over oceans where SST is prescribed (Henn et al., 7 May 2026).

Trend fidelity is more heterogeneous. NeuralGCM, ArchesWeather, ArchesWeatherGen, and DLESyM reproduce the sign and general magnitude of ERA5 warming trends in both 1979–2014 and 2015–2024. By contrast, ACE2.1-ERA5, MD-1.5 v0.9, and cBottle-1.3 notably underestimate warming in both training and test periods; the underestimation appears in global 2 m temperature time series and in spatial trend maps, including weaker Arctic amplification and weaker land–ocean contrast (Henn et al., 7 May 2026). This shows that skill in historical climatology does not guarantee correct long-term forced response.

ENSO teleconnections are a relative strength. Regression maps of 2 m temperature and precipitation onto Niño3.4 show that AIWCMs capture canonical tropical Pacific teleconnections with small coefficient errors relative to ERA5 during training, and most models’ ENSO coefficient RMSEs are comparable to or smaller than CM4’s (Henn et al., 7 May 2026). The errors increase in the shorter test period, which is consistent with sampling limitations rather than necessarily with architectural failure.

Daily variability reveals a more systematic discrepancy. AIWCMs tend to underestimate daily anomaly magnitudes for many variables by roughly 2–10% of ERA5 variability in the global mean, especially over oceans where monthly forcing interpolation may smooth high-frequency variability. ArchesWeatherGen shows smaller underestimation, while CM4 often overestimates daily variability relative to ERA5. For precipitation, most AIWCMs and CM4 overestimate daily variability in wet regions; DLESyM is singled out as an exception (Henn et al., 7 May 2026).

The most striking divergence appears in the out-of-sample uniform warming experiments. NeuralGCM-HRD and DLESyM produce physically plausible warming patterns, with SST-matched warming over oceans and amplified warming over land, broadly resembling CM4 and established theory. ArchesWeather and ArchesWeatherGen warm oceans but show muted land amplification. ACE2.1-ERA5, MD-1.5 v0.9, and cBottle-1.3 underestimate ocean warming and, implausibly, predict cooling over land under uniform SST increases (Henn et al., 7 May 2026). The paper treats this as a serious generalization gap rather than an edge-case artifact.

6. Interpretation, limitations, and future directions

Several limitations shape the interpretation of AIMIP Phase 1. First, ERA5 is both the training source and the evaluation reference. This guarantees comparability, but it also means that model quality is being judged relative to a single reanalysis with known limitations, particularly for precipitation and some radiation-related trends (Henn et al., 7 May 2026). Second, the design is atmosphere-only and SST-forced; it does not test coupled air–sea feedbacks, ocean drift, or full DECK-style historical simulation.

A second common misconception is that better mean-state scores imply robust physical generalization. AIMIP Phase 1 shows the opposite: some models with strong RMSB performance diverge sharply under $q$6 K and $q$7 K SST perturbations. The handling of missing SST over land and sea ice emerges as especially consequential. The paper notes that constant fills with no perturbation over land/ice, as used in some models, risk unrealistic land responses, whereas interpolation across land or merged SST–land–ice embeddings produce more consistent inputs and better generalization in several cases (Henn et al., 7 May 2026). This suggests that preprocessing and boundary-condition encoding are not peripheral engineering choices; they are part of the model’s effective climate physics.

There are also protocol caveats. DLESyM was trained from 1983 to 2016, which slightly overlaps the AIMIP holdout by roughly 1.5 years. The paper states that its results should be interpreted accordingly (Henn et al., 7 May 2026). Formal significance testing for trends and ENSO regressions, bootstrap confidence intervals, PSD, and ACC are defined or recommended but are not all core Phase 1 deliverables.

The project’s future direction is toward broader Earth system scope and richer diagnostics. Planned extensions include coupled AI Earth system components capable of DECK-like simulations, expanded variable sets such as clouds, additional evaluation metrics including ACC, PSD, and extremes, and broader publication pathways such as ESGF (Henn et al., 7 May 2026). In a related but operationally distinct domain, WP-MIP establishes a common protocol for global deterministic intercomparison across physically based, AI, and hybrid weather prediction systems, indicating that AI intercomparison is developing simultaneously in weather and climate regimes (McTaggart-Cowan et al., 17 Apr 2026).

AIMIP Phase 1 therefore occupies a specific methodological niche. It is neither a conventional weather-forecast benchmark nor a full CMIP replacement. It is an AMIP-inspired protocol for establishing whether AIWCMs, trained under tightly controlled historical constraints, can reproduce climate means, trends, variability, ENSO teleconnections, and stress-test responses in a transparent, public, and comparable manner (Henn et al., 7 May 2026). Its chief contribution is not only the ranking of models, but the identification of which architectural and training choices materially affect climate-model behavior.