WeatherBench: ERA5 Benchmark for Forecasting

• WeatherBench is an ERA5-derived benchmark dataset providing standardized global grids and variable sets for medium-range forecasting.
• It supports both deterministic and probabilistic prediction of key variables like Z500 and T850 using latitude-weighted RMSE and other metrics.
• The framework has spurred innovations in deep learning and operational weather models, enabling direct comparisons with physical baselines.

WeatherBench is an ERA5-derived benchmark dataset and evaluation framework for data-driven global weather forecasting, introduced to make medium-range forecasting studies directly comparable through common data, variables, train/validation/test conventions, and latitude-weighted verification metrics. In its original form, it centered on deterministic global state-to-state prediction, especially of 500 hPa geopotential and 850 hPa temperature; later extensions added probabilistic verification, stronger operational baselines, and a broader benchmark infrastructure for next-generation global weather models (Rasp et al., 2020, Garg et al., 2022, Rasp et al., 2023).

1. Origins, scope, and data model

WeatherBench was introduced to address fragmentation in early machine-learning weather forecasting, where studies used different reanalyses, domains, variables, resolutions, lead times, and metrics, making inter-comparison difficult (Rasp et al., 2020). The benchmark is explicitly aimed at data-driven global medium-range forecasting, loosely the 2-day to 2-week regime, with primary emphasis on 3-day and 5-day forecasts; it is distinguished from post-processing, statistical downscaling, and nowcasting by targeting the future atmospheric state itself rather than corrections to an existing numerical forecast or very-short-range extrapolation (Rasp et al., 2020).

The source data are derived from ERA5 reanalysis, which in the original benchmark provides hourly global atmospheric fields from 1979 through 2018 on a native 0.25° latitude–longitude grid with 721 × 1440 horizontal points and 37 vertical levels (Rasp et al., 2020). To make the dataset tractable for ML workflows, WeatherBench republishes ERA5 on three regular latitude–longitude grids produced with xesmf using bilinear interpolation: 5.625° ($32 \times 64$), 2.8125° ($64 \times 128$), and 1.40525° ($128 \times 256$) (Rasp et al., 2020). For three-dimensional variables, the benchmark retains 13 pressure levels: $50, 100, 150, 200, 250, 300, 400, 500, 600, 700, 850, 925,$ and $1000\ \text{hPa}$ (Rasp et al., 2020).

The variable inventory is broad. Three-dimensional fields include geopotential, temperature, specific humidity, relative humidity, zonal wind, meridional wind, relative vorticity, and potential vorticity; two-dimensional fields include 2 m temperature, 10 m zonal wind, 10 m meridional wind, total cloud cover, total precipitation, and top-of-atmosphere incident solar radiation; and a constants file provides land-sea mask, soil type, orography, and two-dimensional latitude and longitude (Rasp et al., 2020). The processed dataset is distributed as yearly NetCDF files separated by variable and resolution, with the full 5.625° release occupying about 191 GB; the preprocessing workflow is managed with Snakemake (Rasp et al., 2020).

2. Canonical tasks, metrics, and baseline results

The original benchmark formalizes state-to-state forecasting on a global latitude–longitude grid. Its headline variables are Z500 and T850, chosen as standard medium-range synoptic verification targets, while T2M and 6-hour accumulated precipitation appear as secondary targets (Rasp et al., 2020). The recommended temporal split is 1979–2015 for training, 2016 for validation, and 2017–2018 for testing, with the first test target offset by forecast lead time to avoid leakage across train/test boundaries (Rasp et al., 2020).

WeatherBench supports both direct forecasting, where a model maps the input state at time $t$ to a field at $t+\tau$, and iterative/autoregressive forecasting, where a short-lead model—typically 6 h—is recursively applied. Verification uses latitude-weighted RMSE, MAE, and ACC, with latitude weights

$$L(j)=\frac{\cos(\operatorname{lat}(j))}{\frac{1}{N_{\text{lat}}}\sum_{j=1}^{N_{\text{lat}}}\cos(\operatorname{lat}(j))}$$

and ACC computed on anomalies relative to climatology (Rasp et al., 2020).

The original paper reported simple but influential baselines: persistence, climatology, weekly climatology, direct and iterative linear regression, a small direct and iterative CNN, and physical baselines from ECMWF IFS at T42, T63, and operational resolution (Rasp et al., 2020). The empirical picture was intentionally sobering: the direct CNN was the best ML baseline tested, but it remained clearly behind strong physical baselines, especially for Z500; the iterative CNN became unstable and diverged after roughly 1.5 days (Rasp et al., 2020).

Model	Z500 RMSE (5d)	T850 RMSE (5d)
Weekly climatology	816	3.50
Linear regression, direct	783	3.44
CNN, direct	757	3.37
IFS T42	743	3.83
IFS T63	463	2.52
Operational IFS	334	2.03

At 5 days, the iterative CNN deteriorated to 1559 RMSE for Z500 and 9.69 K for T850, whereas the direct CNN remained usable at 757 and 3.37 K respectively (Rasp et al., 2020). These results established two durable benchmark lessons: direct medium-range prediction could outperform naive iterative rollout, and good WeatherBench performance required more than a small MSE-trained CNN (Rasp et al., 2020).

3. Probabilistic and operational generalization: WeatherBench Probability and WeatherBench 2

WeatherBench Probability extends the original framework from deterministic prediction to predictive distributions by adding CRPS, spread-skill ratio, and rank histograms, along with a 50-member ECMWF IFS ensemble baseline from TIGGE (Garg et al., 2022). It evaluated three probabilistic ML families on the WeatherBench setting: Monte Carlo dropout, parametric prediction, and categorical prediction (Garg et al., 2022). The principal findings were that plain MC dropout severely underestimates uncertainty, while the parametric and categorical models are both fairly reliable and of similar quality; none of the ML methods match the operational IFS ensemble (Garg et al., 2022). The paper also exposed an important structural limitation: the parametric and categorical baselines predict pointwise distributions and therefore do not model spatially or temporally coherent forecast scenarios (Garg et al., 2022).

WeatherBench 2 recasts the benchmark as an operationally informed framework for global, medium-range (1–14 day) weather forecasting, with an open-source evaluation stack, public forecast baselines, and a continuously updated website (Rasp et al., 2023). A central design change is that all forecasts and truth are regridded to 1.5° before scoring, with explicit below-ground masking at pressure levels such as 850 hPa in mountainous regions (Rasp et al., 2023). Headline variables include Z500, T850, Q700, WV/WS850, T2M, WS10, MSLP, and TP24hr; deterministic evaluation uses RMSE and ACC, precipitation uses SEEPS, and probabilistic evaluation uses CRPS and spread-skill ratio (Rasp et al., 2023).

WB2 broadens baseline coverage to include ERA5 forecasts, IFS HRES, IFS ENS, IFS ENS mean, and leading AI systems such as Pangu-Weather, GraphCast, FuXi, and NeuralGCM (Rasp et al., 2023). Its headline conclusion is deliberately nuanced: leading data-driven deterministic models are highly competitive with IFS HRES on standard global deterministic metrics, yet the ensemble mean remains difficult to beat at longer lead times, precipitation remains challenging under operationally meaningful scores, and deterministic metrics can reward unrealistic smoothness (Rasp et al., 2023). WB2 therefore treats WeatherBench not as a one-score contest but as an evaluation framework for full forecast systems (Rasp et al., 2023).

4. WeatherBench as a methodological testbed

WeatherBench rapidly became a standard testbed for architectural, training, and probabilistic innovations. A major step beyond the original small CNN baseline was the deep fully convolutional ResNet of Rasp and Thuerey, which used a 19-block residual architecture and showed that CMIP6 climate-simulation pretraining followed by ERA5 fine-tuning improved WeatherBench performance, yielding direct pretrained scores of 268 / 523 for Z500 and 1.65 / 2.52 for T850 at 3 / 5 days; this matched IFS T63 on 3-day Z500 and 5-day T850 and outperformed earlier ML submissions (Rasp et al., 2020). In a different line of work, physics-inspired low-parameter CNNs replaced ordinary convolutions with Spherenet convolution and hemisphere-aware shared flipped filters, improving $z500$ and $t850$ up to day +10 on WeatherBench grids; at 1.40625°, the combined sphereconv_hemconv_shared model reduced day-5 Z500 RMSE from 845 to 696 and day-5 T850 RMSE from 4.11 to 3.53 relative to the plain baseline (Scher et al., 2020).

WeatherBench also served as a platform for explicitly probabilistic deep learning. A distribution-based ResNet framework discretized Z500 and T850 into 100 equal-width bins, predicted a categorical distribution at every grid point, and combined multiple subnetworks through a stacked meta-learner; on the 2017–2018 test set it achieved 375 / 627 RMSE for Z500 and 2.11 / 2.91 K for T850 at 3 / 5 days, beating persistence, climatology, and coarse IFS T42 while remaining behind the stronger deep ResNet and operational IFS baselines (Clare et al., 2021). Later generative work on WeatherBench-S combined deterministic and diffusion branches in DGDM, using single-variable 12-step-in / 12-step-out one-hourly global forecasting at $64 \times 128$0, and reported the best sample results among the compared baselines for $64 \times 128$1, humidity, and $64 \times 128$2 (Yoon et al., 2023).

More recent work increasingly treated WeatherBench as a generic spatiotemporal ML benchmark. SVQ inserted differentiable sparse soft vector quantization into SimVP-style forecasters and improved WeatherBench-S temperature MSE from 1.105 to 1.018 (Chen et al., 2023). PDR added continuous Gaussian rendering to TAU and other MetaFormer backbones on WeatherBench(T2m) with a $64 \times 128$3 setup, reaching MSE 1.071, MAE 0.6353, and RMSE 1.035 (Tang et al., 30 Jun 2026). AGCD introduced decoding-time cross-modal prior injection and improved 6-hour forecasts across ViT, CaiT, ClimaX, and Pangu-Weather at both 5.625° and 1.40625°, including strictly causal 48-hour autoregressive rollouts (Wu et al., 16 Mar 2026). Taken together, these studies show that “WeatherBench results” now span strict WB1/WB2-style leaderboard comparisons, OpenSTL-style single-variable forecasting, and broader spatiotemporal prediction regimes.

5. Repurposing, specialization, and benchmark-adjacent ecosystems

A striking feature of the WeatherBench literature is that the dataset has been repurposed far beyond the original global medium-range benchmark. One example is learned data compression: a coordinate-based neural representation compressed WeatherBench-like geopotential and temperature archives by as much as 790× in a downstream experiment, while training the standard WeatherBench baseline CNN with test weighted RMSE increases of less than 2% (Huang et al., 2022). Another is local time-series forecasting: PINT used ERA5 WeatherBench 2 m temperature not as a global field benchmark but as three single-location/city series—Seoul, Beijing, and Washington, D.C.—aggregated to daily means and forecast under a 90-day input / 30-day output setup, with iterative prediction out to two years; the study found Physics-Informed LSTM to be the strongest neural model, but also showed that a simple SHO-derived linear regression baseline remained highly competitive (Park et al., 6 Feb 2025).

Specialized regional subsets have likewise emerged. A physics-constrained hybrid study used a South Pacific WeatherBench subset from 5.625°N to 39.375°S and 174.375°E to 95.625°W at 1.40525° and 60 min cadence, reporting 8–22% RMSE reductions at 1–12 h over purely neural counterparts (Bugaev et al., 16 Jun 2026). A separate diffusion study used WeatherBench for T2M super-resolution, not lead-time forecasting, framing a 4× spatial downscaling problem from 5.625° ($64 \times 128$4) to 1.40525° ($64 \times 128$5) and finding ResDiff markedly stronger than SR3 (Martinů et al., 2024). These departures are methodologically legitimate, but they are no longer directly comparable to canonical WeatherBench forecast scores.

The benchmark ecosystem has also expanded around WeatherBench. Extreme Weather Bench is explicitly positioned as complementary to WeatherBench, targeting high-impact hazards such as heat waves, freeze events, severe convective outbreaks, tropical cyclones, and atmospheric rivers through curated case studies and impact-oriented metrics (McGovern et al., 1 May 2026). WxC-Bench is likewise described as a complement rather than a replacement, providing a multimodal “dataset of datasets” for downstream tasks such as aviation turbulence, gravity wave parameterization, weather analog search, long-range precipitation forecasting, hurricane monitoring, and natural-language forecast generation (Shinde et al., 2024). Separately, the name “WeatherBench” has also been reused for an unrelated real-world adverse weather image restoration dataset, creating a nomenclature collision with the ERA5-based forecasting benchmark (Guan et al., 15 Sep 2025).

6. Limitations, interpretive caveats, and enduring significance

The original benchmark was explicit about its limitations. It focused on deterministic forecasting, emphasized Z500 and T850 in its headline scores, evaluated at a coarse 5.625° grid, did not address long free-running climate realism, and omitted extreme-event metrics; it also noted challenges related to spherical geometry and the limited effective sample size of correlated atmospheric data (Rasp et al., 2020). WeatherBench Probability showed that pointwise probabilistic models can yield reasonable marginal calibration while still failing to represent coherent spatial-temporal scenarios (Garg et al., 2022). WB2 made the further point that ERA5 is not truth in the observational sense, initialization conventions can make comparisons unfair, one year of evaluation is insufficient for robust extreme-event statistics, and deterministic metrics can reward unrealistic smoothness (Rasp et al., 2023).

A second caveat is protocol heterogeneity. Many later papers explicitly depart from canonical WeatherBench practice through custom year splits, regional subsets, single-variable setups, local time-series extraction, compression experiments, super-resolution tasks, or nonstandard metrics (Park et al., 6 Feb 2025, Huang et al., 2022, Bugaev et al., 16 Jun 2026, Martinů et al., 2024). This suggests that the term “WeatherBench” now denotes both a strict benchmark protocol and a broader ERA5-derived experimental substrate. For expert readers, reproducing a result or comparing across papers therefore requires careful attention to resolution, variable set, forecast horizon, initialization source, evaluation metric, and whether the study is actually participating in WB1/WB2-style forecast verification.

The enduring significance of WeatherBench lies precisely in this dual role. As a benchmark, it established a common language for global ML weather forecasting and enabled direct comparison against physical baselines (Rasp et al., 2020). As an evolving framework, it absorbed probabilistic verification and operational scorekeeping in WB2 (Garg et al., 2022, Rasp et al., 2023). As a research substrate, it has supported work on pretraining, geometry-aware CNNs, distributional forecasting, diffusion models, latent quantization, learned data compression, regional hybrids, and local climate-style forecasting (Rasp et al., 2020, Scher et al., 2020, Clare et al., 2021, Yoon et al., 2023, Chen et al., 2023, Huang et al., 2022, Park et al., 6 Feb 2025). A plausible implication is that WeatherBench is strongest as broad, reanalysis-based verification infrastructure for global forecast models, while complementary resources such as WB2’s probabilistic diagnostics, Extreme Weather Bench’s hazard-specific case studies, and WxC-Bench’s downstream-task collections are needed when calibration, impacts, local extremes, or multimodal transfer are the central scientific questions (Rasp et al., 2023, McGovern et al., 1 May 2026, Shinde et al., 2024).