Extreme Weather Bench (EWB)

• Extreme Weather Bench (EWB) is an open benchmark suite for validating high-impact weather forecasts through event-based verification.
• It standardizes evaluation with phenomenon-specific metrics and curated case studies across events like heat waves, severe convective outbreaks, and tropical cyclones.
• EWB facilitates reproducible and transparent comparisons between AI-driven models and traditional numerical weather prediction systems.

Searching arXiv for the benchmark and name ambiguity. to=arxiv_search.search 】!【json_string {"query":"(McGovern et al., 1 May 2026) Extreme Weather Bench (Quaremba et al., 4 Jul 2025) WETBench", "max_results": 5}】

Found relevant papers on - (McGovern et al., 1 May 2026): "Extreme Weather Bench: A framework and benchmark for evaluation of high-impact weather"

• (Quaremba et al., 4 Jul 2025): "WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia"

I’ll use these to disambiguate the term and ground the article. In the context of high-impact weather forecasting, “WETBench” refers informally to Extreme Weather Bench (EWB), an open, community-driven benchmark suite for validation and verification of high-impact weather forecasts from both AI-based weather prediction and traditional numerical weather prediction (NWP) models. EWB packages curated case studies, observations-based targets, event- and impact-centric metrics, and open-source evaluation code, with the stated aim of enabling apples-to-apples verification across models on hazards that matter to people around the globe (McGovern et al., 1 May 2026).

1. Identity and nomenclature

The official name used throughout the paper and repository is Extreme Weather Bench (EWB). The label “WETBench” is not defined as an official alias in that work; rather, it is described as an informal name sometimes used by readers. This distinction matters because benchmark names are not unique across fields. An unrelated benchmark titled “WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia” concerns multilingual machine-generated text detection on Wikipedia and has no meteorological scope (Quaremba et al., 4 Jul 2025).

EWB is introduced to standardize, strengthen, and make trustworthy the evaluation of high-impact weather forecasts. Its motivating critique is that global metrics such as RMSE or ACC on mid-tropospheric fields reward overly smooth solutions, do not directly reflect high-impact phenomena, and suppress physical event fidelity. At the same time, ad hoc case studies lack comparability and reproducibility. EWB is designed as a response to both limitations by defining standardized cases, common targets, phenomenon-specific metrics, and transparent code and data pipelines (McGovern et al., 1 May 2026).

A central design principle is that verification should be performed at scales relevant to societal impacts rather than only on smooth-field global summaries. This is why EWB is organized around event-centered verification on explicitly bounded spatiotemporal cases rather than generic climatological scorecards. A plausible implication is that EWB is less a replacement for global skill assessment than a complementary substrate for scrutinizing whether forecast systems preserve the structure, timing, and footprint of extreme events.

2. Hazard coverage and case-study construction

The initial EWB release includes five hazard categories: heat waves, major large-scale freeze events, severe convective outbreak days, tropical cyclones restricted to landfalling cases, and atmospheric rivers restricted to land-intersecting cases. These categories were selected to span multiple spatial and temporal scales and different parts of the weather spectrum, from short-lived, small-scale severe thunderstorms to multi-week heat waves and basin-scale tropical cyclones and atmospheric rivers (McGovern et al., 1 May 2026).

Hazard category	Core case definition elements
Heat waves	2020–2024 cases; percentile-based temperature extremes; event cubes
Major large-scale freeze events	2020–2024 cases; below-freezing cold extremes; event cubes
Severe convective outbreak days	Outbreak days plus marginal severe days
Tropical cyclones	Landfalling TCs with track-based matching
Atmospheric rivers	Land-intersecting ARs with object-based verification

Case curation is explicitly global. Event identification draws on NOAA/NCEI, WMO archives, literature, the ECMWF Severe Event Catalogue, news sources, and international collaborators. For each event, EWB defines a spatiotemporal “cube,” intended to align both with objective criteria and with social perception of the event’s footprint. The test window is 2020–2024, chosen to provide an independent test period even for AI models trained on the full ERA5 archive (McGovern et al., 1 May 2026).

A distinctive feature is the inclusion of marginal events to mitigate the forecaster’s dilemma. For temperature, EWB includes marginal days in the 16th–84th percentiles lasting at least five days over areas at least $200{,}000\ \mathrm{km}^2$. For severe convection, EWB includes marginal days such as days with zero tornado reports and limited hail reports in the United States and Canada. This structure is intended to quantify tendencies toward overforecasting extremes rather than rewarding systems only for hitting rare headline events (McGovern et al., 1 May 2026).

3. Observational targets and preprocessing

EWB prioritizes observations as ground truth wherever possible, with ERA5 used as a fallback for phenomena or regions lacking high-quality public observations. The observational targets are hazard-specific. GHCNh hourly station temperatures are used for heat and freeze verification; severe weather verification uses SPC/NCEI storm reports in the United States, the Northern Hail Project and Northern Tornadoes Project in Canada, and Australian hail and tornado datasets curated from verified sources; tropical cyclone verification uses IBTrACS best tracks at 3-hour cadence; and atmospheric rivers are verified against ERA5-derived integrated vapor transport (IVT) using a Python AR tracker (McGovern et al., 1 May 2026).

Hazard	Primary target data and defaults
Heat waves	GHCNh hourly station temperatures; $\ge 3$ consecutive days above the 85th percentile; ocean points excluded
Freezes	GHCNh hourly station temperatures; below freezing and below the 15th percentile for $\ge 3$ consecutive days
Severe convection	PPH regions from LSRs; CBSS $=$ MLCAPE $\times$ 0–6 km shear; default CBSS threshold $\ge 15{,}000$
Tropical cyclones	IBTrACS best tracks; landfall timing by linear interpolation
Atmospheric rivers	ERA5-derived IVT; IVT $\ge 400\ \mathrm{kg\ m^{-1}\ s^{-1}}$; Laplacian-of-IVT $\ge 2.5\ \mathrm{kg\ m^{-2}\ s^{-2}}$

The preprocessing stack is strongly phenomenon-specific. ERA5 2-m temperature climatologies are built from 1990–2019 so that percentile definitions remain independent of the 2020–2024 test years. Heat waves are defined as at least three consecutive days above the 85th percentile, permitting gaps up to 24 hours. Freezes require daily below-freezing conditions and below the 15th percentile for at least three consecutive days. For severe convection, EWB constructs Practically Perfect Hindcast (PPH) regions from local storm reports using Gaussian kernels, with weighting to reflect underreporting outside the United States, and uses the Craven-Brooks Significant Severe proxy $\mathrm{CBSS} = \mathrm{MLCAPE} \times$ 0–6 km shear, where shear is computed using 500 hPa and surface winds (McGovern et al., 1 May 2026).

Atmospheric river preprocessing applies explicit object-detection filters: IVT threshold at least $400\ \mathrm{kg\ m^{-1}\ s^{-1}}$, Laplacian-of-IVT at least $\ge 3$0 in an 8-gridpoint neighborhood on a $\ge 3$1 grid, at least 500 valid grid points, plus latitude and shape filters to avoid tropical cyclone misidentification. Tropical cyclone preprocessing uses a Python tracker with warm-core, pressure-gradient, and closed-contour checks, proximity to IBTrACS tracks, continuity constraints in space and time, wind-maxima search, and latitudinal bounds. Landfall timing is computed by interpolation and then filtered for spurious cases (McGovern et al., 1 May 2026).

4. Evaluation framework and verification protocol

EWB accepts forecasts from both AIWP systems and traditional NWP systems. The paper identifies AI examples including AIFS-Single, GraphCast, Pangu-Weather, and FourCastNet v2, with IFS-HRES used as the reference baseline on the NWP side. Forecasts are typically provided at 6-hour intervals on global grids, and EWB can regrid them, with CF-Conventions metadata preferred (McGovern et al., 1 May 2026).

The expected model outputs depend on the hazard. Temperature extremes and tropical cyclone intensity verification require gridded variables such as 2-m temperature, mean sea-level pressure, and near-surface winds. Severe convection requires derived CBSS. Atmospheric rivers require IVT. Tropical cyclone tracks are derived from model fields using EWB trackers. The framework also works with event-region polygons and masks, which are essential for object-based and region-based evaluation (McGovern et al., 1 May 2026).

Alignment rules are part of the benchmark definition. Temperature extremes are evaluated with a $\ge 3$2-hour relaxation to tolerate phase error. Tropical cyclone landfall uses track-based interpolation with “first” or “next” landfall options, removal of near-duplicate landfalls, and a $\ge 3$3-hour window for associating forecast and observed landfall. Atmospheric rivers use ERA5-based IVT targets, land-intersection masks, center-of-mass calculations, and IoU. Severe convective outbreaks compare PPH regions to CBSS regions, with report-based hits and misses computed within the predicted severe area. EWB recommends treating 2020–2024 as a held-out standardized test period for intercomparison, even when the upstream model training archive spans all of ERA5 (McGovern et al., 1 May 2026).

The initial release emphasizes deterministic, impact-oriented verification. Representative formulas include

$\ge 3$4

$\ge 3$5

and the object-overlap metric

$\ge 3$6

For categorical verification on binarized regions, EWB uses contingency counts $\ge 3$7, $\ge 3$8, $\ge 3$9, and $\ge 3$0, together with metrics such as

$\ge 3$1

Track-based tropical-cyclone metrics include great-circle track error, landfall spatial error, landfall temporal error, and landfall intensity errors for minimum central pressure and maximum 10 m wind speed within a 2 GCD search radius. EWB also documents ACC, Brier Score, and CRPS as future extensions rather than initial-release operational metrics (McGovern et al., 1 May 2026).

5. Impact orientation and reported benchmark findings

EWB’s metric design is explicitly impact-oriented, although the initial release uses meteorological proxies rather than direct damage or outage datasets. For heat and cold extremes, the benchmark emphasizes errors in event maxima and daily minima; the highest daily minimum temperature is singled out because nighttime minima strongly influence heat-health outcomes. For severe convective outbreaks, regional FAR and CSI against PPH are used to characterize over- and under-prediction of risk footprints relevant to warnings and resource allocation. For atmospheric rivers, landfall location error and IoU of land-intersecting IVT envelopes are treated as proxies for flood or snowstorm footprint. For tropical cyclones, landfall timing, landfall position, and landfall intensity are central because they are key drivers of impact on life and property (McGovern et al., 1 May 2026).

The paper’s examples show hazard-dependent performance differences across AI and NWP systems. For heat waves, AI models show strong RMSE performance but vary regionally. In the 2021 Pacific Northwest heatwave, AIFS outperformed HRES for event highs across all lead times, whereas none of the AI models beat HRES for event lows. On marginal temperature days, all AI models outperformed HRES. For major freezes, all models struggled with event lows at longer lead times, described as warm bias and low predictability, while AIFS corrected warm bias faster globally and in North America than other AI models (McGovern et al., 1 May 2026).

For severe convective outbreak days, AI models generally outperformed HRES in CSI and FAR when evaluated using PPH versus CBSS area metrics, although early-signal differences at long leads were small relative to HRES. The benchmark cautions that report-based hits and misses should be compared within region because reporting differs across the United States, Canada, and Australia. On marginal severe days, AI models tended to overpredict risk. This suggests that EWB’s inclusion of marginal events is not incidental but diagnostic of systematic forecast tendencies (McGovern et al., 1 May 2026).

For atmospheric rivers, AI models generally beat HRES in IoU of IVT-landfall areas, equivalent here to CSI, and showed better center-of-mass displacement in North America, with more mixed outcomes in Europe and Australia. Lead-time detection of final atmospheric-river area was longer for AI than HRES in North America. For tropical cyclones, the benchmark samples 98 landfalling TCs across hemispheres. AI models mostly outperformed HRES on minimum pressure at landfall in the Western Hemisphere, though less so at longer leads, while wind MAE at landfall in the Eastern Hemisphere was worse than HRES. Landfall timing and displacement were mixed. Example cases include GraphCast locking onto TC Beryl’s eventual track relatively early, whereas all AI models struggled early on TC Yagi (McGovern et al., 1 May 2026).

6. Reproducibility, governance, and expansion

EWB is distributed as a free, open-source system through a public repository and is also available on PyPI. The paper used version 1.0.1. Supporting artifacts include paper plotting code, a subset of model-output archives via Arraylake with free egress worldwide, and a full MLWP archive on Google Cloud Storage in Zarr and Icechunk formats. The project includes configuration files, example notebooks and scripts, unit tests, CI, and documentation aimed at reproducibility and clarity (McGovern et al., 1 May 2026).

The recommended workflow is explicit. Users install EWB from PyPI or GitHub, prepare model outputs with CF-Conventions metadata where possible, optionally regrid to $\ge 3$2, select case categories and evaluation modes, derive variables such as CBSS and IVT, run the tropical-cyclone trackers where needed, align forecasts and targets with the benchmark tolerances, and compute lead-time, event-intensity, object-based, and categorical metrics appropriate to the chosen hazard. Best practice is to use EWB’s standardized case definitions and thresholds, evaluate across multiple hazards and regions rather than cherry-picked events, include marginal-event evaluations, and document any deviations from defaults (McGovern et al., 1 May 2026).

Governance is explicitly community-driven, with worldwide participation among meteorological and forecast-verification experts and stated alignment with JWGFVR and scores package synergy. The roadmap includes droughts, derechos, floods, blizzards, ice storms, medicanes subject to track data availability, compound events such as TC tornado outbreaks and compound flooding, higher-resolution and limited-area model support, ensemble and probabilistic verification, and seasonal-to-subseasonal horizons. Additional observational targets, such as radar-derived precipitation where available and satellite global precipitation, as well as a real-time web interface and broader societal impact linkages, are also anticipated (McGovern et al., 1 May 2026).

In that sense, the significance of “WETBench” as used in the weather-forecasting community lies less in the nickname than in the benchmark’s methodological stance: observation-centered verification, event-based case construction, impact-aware metric design, and standardized cross-model intercomparison. The official name remains Extreme Weather Bench, and the benchmark’s intended role is to provide a common substrate on which AI and NWP systems can be compared transparently on the high-impact phenomena that conventional global smooth-field metrics often underresolve (McGovern et al., 1 May 2026).