---
title: 'WeatherBench: ERA5 Benchmark for Forecasting'
url: https://www.emergentmind.com/topics/weatherbench
type: topic
---

# WeatherBench: ERA5 Benchmark for Forecasting

WeatherBench is an ERA5-derived benchmark dataset and evaluation framework for data-driven global weather forecasting, introduced to make medium-range forecasting studies directly comparable through common data, variables, train/validation/test conventions, and latitude-weighted verification metrics. In its original form, it centered on deterministic global state-to-state prediction, especially of 500 hPa geopotential and 850 hPa temperature; later extensions added probabilistic verification, stronger operational baselines, and a broader benchmark infrastructure for next-generation global weather models [2002.00469][2205.00865][2308.15560].

## 1. Origins, scope, and data model

WeatherBench was introduced to address fragmentation in early machine-learning weather forecasting, where studies used different reanalyses, domains, variables, resolutions, lead times, and metrics, making inter-comparison difficult [2002.00469]. The benchmark is explicitly aimed at data-driven **global medium-range forecasting**, loosely the 2-day to 2-week regime, with primary emphasis on **3-day and 5-day forecasts**; it is distinguished from post-processing, statistical downscaling, and nowcasting by targeting the future atmospheric state itself rather than corrections to an existing numerical forecast or very-short-range extrapolation [2002.00469].

The source data are derived from **ERA5 reanalysis**, which in the original benchmark provides hourly global atmospheric fields from **1979 through 2018** on a native **0.25° latitude–longitude grid** with **721 × 1440** horizontal points and **37 vertical levels** [2002.00469]. To make the dataset tractable for ML workflows, WeatherBench republishes ERA5 on three regular latitude–longitude grids produced with **xesmf** using **bilinear interpolation**: **5.625°** (\(32 \times 64\)), **2.8125°** (\(64 \times 128\)), and **1.40525°** (\(128 \times 256\)) [2002.00469]. For three-dimensional variables, the benchmark retains **13 pressure levels**: \(50, 100, 150, 200, 250, 300, 400, 500, 600, 700, 850, 925,\) and \(1000\ \text{hPa}\) [2002.00469].

The variable inventory is broad. Three-dimensional fields include geopotential, temperature, specific humidity, relative humidity, zonal wind, meridional wind, relative vorticity, and potential vorticity; two-dimensional fields include 2 m temperature, 10 m zonal wind, 10 m meridional wind, total cloud cover, total precipitation, and top-of-atmosphere incident solar radiation; and a constants file provides land-sea mask, soil type, orography, and two-dimensional latitude and longitude [2002.00469]. The processed dataset is distributed as yearly NetCDF files separated by variable and resolution, with the full **5.625°** release occupying about **191 GB**; the preprocessing workflow is managed with **Snakemake** [2002.00469].

## 2. Canonical tasks, metrics, and baseline results

The original benchmark formalizes **state-to-state forecasting** on a global latitude–longitude grid. Its headline variables are **Z500** and **T850**, chosen as standard medium-range synoptic verification targets, while **T2M** and **6-hour accumulated precipitation** appear as secondary targets [2002.00469]. The recommended temporal split is **1979–2015** for training, **2016** for validation, and **2017–2018** for testing, with the first test target offset by forecast lead time to avoid leakage across train/test boundaries [2002.00469].

WeatherBench supports both **direct forecasting**, where a model maps the input state at time \(t\) to a field at \(t+\tau\), and **iterative/autoregressive forecasting**, where a short-lead model—typically 6 h—is recursively applied. Verification uses latitude-weighted **RMSE**, **MAE**, and **ACC**, with latitude weights
$$
L(j)=\frac{\cos(\operatorname{lat}(j))}{\frac{1}{N_{\text{lat}}}\sum_{j=1}^{N_{\text{lat}}}\cos(\operatorname{lat}(j))}
$$
and ACC computed on anomalies relative to climatology [2002.00469].

The original paper reported simple but influential baselines: persistence, climatology, weekly climatology, direct and iterative linear regression, a small direct and iterative CNN, and physical baselines from ECMWF IFS at T42, T63, and operational resolution [2002.00469]. The empirical picture was intentionally sobering: the **direct CNN** was the best ML baseline tested, but it remained clearly behind strong physical baselines, especially for Z500; the **iterative CNN** became unstable and diverged after roughly 1.5 days [2002.00469].

| Model | Z500 RMSE (5d) | T850 RMSE (5d) |
|---|---:|---:|
| Weekly climatology | 816 | 3.50 |
| Linear regression, direct | 783 | 3.44 |
| CNN, direct | 757 | 3.37 |
| IFS T42 | 743 | 3.83 |
| IFS T63 | 463 | 2.52 |
| Operational IFS | 334 | 2.03 |

At 5 days, the iterative CNN deteriorated to **1559** RMSE for Z500 and **9.69 K** for T850, whereas the direct CNN remained usable at **757** and **3.37 K** respectively [2002.00469]. These results established two durable benchmark lessons: direct medium-range prediction could outperform naive iterative rollout, and good WeatherBench performance required more than a small MSE-trained CNN [2002.00469].

## 3. Probabilistic and operational generalization: WeatherBench Probability and WeatherBench 2

**WeatherBench Probability** extends the original framework from deterministic prediction to predictive distributions by adding **CRPS**, **spread-skill ratio**, and **rank histograms**, along with a **50-member ECMWF IFS ensemble** baseline from TIGGE [2205.00865]. It evaluated three probabilistic ML families on the WeatherBench setting: **Monte Carlo dropout**, **parametric prediction**, and **categorical prediction** [2205.00865]. The principal findings were that plain MC dropout **severely underestimates uncertainty**, while the parametric and categorical models are both fairly reliable and of similar quality; none of the ML methods match the operational IFS ensemble [2205.00865]. The paper also exposed an important structural limitation: the parametric and categorical baselines predict **pointwise** distributions and therefore do not model spatially or temporally coherent forecast scenarios [2205.00865].

**WeatherBench 2** recasts the benchmark as an operationally informed framework for **global, medium-range (1–14 day) weather forecasting**, with an open-source evaluation stack, public forecast baselines, and a continuously updated website [2308.15560]. A central design change is that all forecasts and truth are **regridded to 1.5°** before scoring, with explicit **below-ground masking** at pressure levels such as **850 hPa** in mountainous regions [2308.15560]. Headline variables include **Z500**, **T850**, **Q700**, **WV/WS850**, **T2M**, **WS10**, **MSLP**, and **TP24hr**; deterministic evaluation uses RMSE and ACC, precipitation uses **SEEPS**, and probabilistic evaluation uses **CRPS** and **spread-skill ratio** [2308.15560].

WB2 broadens baseline coverage to include **ERA5 forecasts**, **IFS HRES**, **IFS ENS**, **IFS ENS mean**, and leading AI systems such as **Pangu-Weather**, **GraphCast**, **FuXi**, and **NeuralGCM** [2308.15560]. Its headline conclusion is deliberately nuanced: leading data-driven deterministic models are highly competitive with **IFS HRES** on standard global deterministic metrics, yet the **ensemble mean** remains difficult to beat at longer lead times, precipitation remains challenging under operationally meaningful scores, and deterministic metrics can reward unrealistic smoothness [2308.15560]. WB2 therefore treats WeatherBench not as a one-score contest but as an evaluation framework for full forecast systems [2308.15560].

## 4. WeatherBench as a methodological testbed

WeatherBench rapidly became a standard testbed for architectural, training, and probabilistic innovations. A major step beyond the original small CNN baseline was the deep fully convolutional **ResNet** of Rasp and Thuerey, which used a **19-block** residual architecture and showed that **CMIP6 climate-simulation pretraining** followed by ERA5 fine-tuning improved WeatherBench performance, yielding direct pretrained scores of **268 / 523** for Z500 and **1.65 / 2.52** for T850 at **3 / 5 days**; this matched **IFS T63** on 3-day Z500 and 5-day T850 and outperformed earlier ML submissions [2008.08626]. In a different line of work, physics-inspired low-parameter CNNs replaced ordinary convolutions with **Spherenet convolution** and hemisphere-aware shared flipped filters, improving \(z500\) and \(t850\) up to day +10 on WeatherBench grids; at **1.40625°**, the combined **sphereconv\_hemconv\_shared** model reduced day-5 Z500 RMSE from **845** to **696** and day-5 T850 RMSE from **4.11** to **3.53** relative to the plain baseline [2008.13524].

WeatherBench also served as a platform for explicitly probabilistic deep learning. A distribution-based ResNet framework discretized Z500 and T850 into **100 equal-width bins**, predicted a categorical distribution at every grid point, and combined multiple subnetworks through a stacked meta-learner; on the 2017–2018 test set it achieved **375 / 627** RMSE for Z500 and **2.11 / 2.91 K** for T850 at **3 / 5 days**, beating persistence, climatology, and coarse IFS T42 while remaining behind the stronger deep ResNet and operational IFS baselines [2103.14430]. Later generative work on **WeatherBench-S** combined deterministic and diffusion branches in **DGDM**, using single-variable **12-step-in / 12-step-out** one-hourly global forecasting at **\(32 \times 64\)**, and reported the best sample results among the compared baselines for \(t2m\), humidity, and \(uv10\) [2312.02819].

More recent work increasingly treated WeatherBench as a generic spatiotemporal ML benchmark. **SVQ** inserted differentiable sparse soft vector quantization into SimVP-style forecasters and improved WeatherBench-S temperature MSE from **1.105** to **1.018** [2312.03406]. **PDR** added continuous Gaussian rendering to TAU and other MetaFormer backbones on **WeatherBench(T2m)** with a **\(12 \rightarrow 12\)** setup, reaching **MSE 1.071**, **MAE 0.6353**, and **RMSE 1.035** [2606.31050]. **AGCD** introduced decoding-time cross-modal prior injection and improved **6-hour** forecasts across **ViT**, **CaiT**, **ClimaX**, and **Pangu-Weather** at both **5.625°** and **1.40625°**, including strictly causal **48-hour autoregressive rollouts** [2603.15260]. Taken together, these studies show that “WeatherBench results” now span strict WB1/WB2-style leaderboard comparisons, OpenSTL-style single-variable forecasting, and broader spatiotemporal prediction regimes.

## 5. Repurposing, specialization, and benchmark-adjacent ecosystems

A striking feature of the WeatherBench literature is that the dataset has been repurposed far beyond the original global medium-range benchmark. One example is learned data compression: a coordinate-based neural representation compressed WeatherBench-like geopotential and temperature archives by as much as **790×** in a downstream experiment, while training the standard WeatherBench baseline CNN with test weighted RMSE increases of less than **2%** [2210.12538]. Another is local time-series forecasting: **PINT** used ERA5 WeatherBench **2 m temperature** not as a global field benchmark but as **three single-location/city series**—Seoul, Beijing, and Washington, D.C.—aggregated to daily means and forecast under a **90-day input / 30-day output** setup, with iterative prediction out to two years; the study found **Physics-Informed LSTM** to be the strongest neural model, but also showed that a simple **SHO-derived linear regression** baseline remained highly competitive [2502.04018].

Specialized regional subsets have likewise emerged. A physics-constrained hybrid study used a **South Pacific WeatherBench subset** from **5.625°N to 39.375°S** and **174.375°E to 95.625°W** at **1.40525°** and **60 min** cadence, reporting **8–22%** RMSE reductions at **1–12 h** over purely neural counterparts [2606.17659]. A separate diffusion study used WeatherBench for **T2M super-resolution**, not lead-time forecasting, framing a **4×** spatial downscaling problem from **5.625°** (\(32 \times 64\)) to **1.40525°** (\(128 \times 256\)) and finding **ResDiff** markedly stronger than SR3 [2406.04099]. These departures are methodologically legitimate, but they are no longer directly comparable to canonical WeatherBench forecast scores.

The benchmark ecosystem has also expanded around WeatherBench. **Extreme Weather Bench** is explicitly positioned as **complementary** to WeatherBench, targeting high-impact hazards such as heat waves, freeze events, severe convective outbreaks, tropical cyclones, and atmospheric rivers through curated case studies and impact-oriented metrics [2605.01126]. **WxC-Bench** is likewise described as a complement rather than a replacement, providing a multimodal “dataset of datasets” for downstream tasks such as aviation turbulence, gravity wave parameterization, weather analog search, long-range precipitation forecasting, hurricane monitoring, and natural-language forecast generation [2412.02780]. Separately, the name **“WeatherBench”** has also been reused for an unrelated real-world adverse weather image restoration dataset, creating a nomenclature collision with the ERA5-based forecasting benchmark [2509.11642].

## 6. Limitations, interpretive caveats, and enduring significance

The original benchmark was explicit about its limitations. It focused on **deterministic forecasting**, emphasized **Z500** and **T850** in its headline scores, evaluated at a coarse **5.625°** grid, did not address long free-running climate realism, and omitted extreme-event metrics; it also noted challenges related to spherical geometry and the limited effective sample size of correlated atmospheric data [2002.00469]. WeatherBench Probability showed that pointwise probabilistic models can yield reasonable marginal calibration while still failing to represent coherent spatial-temporal scenarios [2205.00865]. WB2 made the further point that **ERA5 is not truth in the observational sense**, initialization conventions can make comparisons unfair, one year of evaluation is insufficient for robust extreme-event statistics, and deterministic metrics can reward unrealistic smoothness [2308.15560].

A second caveat is protocol heterogeneity. Many later papers explicitly depart from canonical WeatherBench practice through custom year splits, regional subsets, single-variable setups, local time-series extraction, compression experiments, super-resolution tasks, or nonstandard metrics [2502.04018][2210.12538][2606.17659][2406.04099]. This suggests that the term “WeatherBench” now denotes both a strict benchmark protocol and a broader ERA5-derived experimental substrate. For expert readers, reproducing a result or comparing across papers therefore requires careful attention to resolution, variable set, forecast horizon, initialization source, evaluation metric, and whether the study is actually participating in WB1/WB2-style forecast verification.

The enduring significance of WeatherBench lies precisely in this dual role. As a benchmark, it established a common language for global ML weather forecasting and enabled direct comparison against physical baselines [2002.00469]. As an evolving framework, it absorbed probabilistic verification and operational scorekeeping in WB2 [2205.00865][2308.15560]. As a research substrate, it has supported work on pretraining, geometry-aware CNNs, distributional forecasting, diffusion models, latent quantization, learned data compression, regional hybrids, and local climate-style forecasting [2008.08626][2008.13524][2103.14430][2312.02819][2312.03406][2210.12538][2502.04018]. A plausible implication is that WeatherBench is strongest as broad, reanalysis-based verification infrastructure for global forecast models, while complementary resources such as WB2’s probabilistic diagnostics, Extreme Weather Bench’s hazard-specific case studies, and WxC-Bench’s downstream-task collections are needed when calibration, impacts, local extremes, or multimodal transfer are the central scientific questions [2308.15560][2605.01126][2412.02780].

Source: https://www.emergentmind.com/topics/weatherbench