Extreme Weather Bench (EWB)

• Extreme Weather Bench (EWB) is an open-source, community-driven framework defining standardized case studies for high-impact weather events.
• It evaluates AI and traditional Numerical Weather Prediction models on hazards such as heat waves, freezes, convective outbreaks, tropical cyclones, and atmospheric rivers.
• The benchmark integrates heterogeneous observational targets with impact-based, hazard-specific metrics to enhance model trustworthiness and operational relevance.

Searching arXiv for the primary benchmark paper and closely related benchmark/evaluation work to ground the article. Extreme Weather Bench (EWB) is an open-source, community-driven framework and benchmark suite for the validation and verification of high-impact weather forecasts, designed to evaluate both AI weather models and traditional Numerical Weather Prediction (NWP) systems on case studies that matter operationally and societally (McGovern et al., 1 May 2026). Rather than emphasizing only global-scale skill scores or a small number of hand-picked examples, EWB provides a standard set of case studies, observational targets, impact-based metrics, and open-source code for comparing models across multiple hazards, spatial scales, and temporal scales (McGovern et al., 1 May 2026). Its first release centers on five categories of extreme weather—heat waves, large-scale freezes, convective-outbreak days, tropical cyclones, and atmospheric rivers—using recent events from 2020–2024 together with matched marginal events to reduce distortion from the “forecaster’s dilemma” (McGovern et al., 1 May 2026).

1. Conceptual basis and design goals

EWB was introduced to address a gap in weather-model evaluation: much AI-weather verification had been performed either with global-field metrics such as RMSE of large-scale variables or with limited regional case studies, leaving high-impact phenomena insufficiently standardized for inter-model comparison (McGovern et al., 1 May 2026). The benchmark therefore emphasizes phenomenon-specific verification on a fixed, independent test set of recent events, with the stated aim of improving the trustworthiness of weather models before deployment (McGovern et al., 1 May 2026).

The design philosophy is organized around three ideas. First, EWB seeks consistency across time and across models through a fixed set of case studies. Second, it prioritizes phenomenon-specific verification, recognizing that hazards differ materially in spatial scale, temporal duration, and operational value. Third, it is open, extensible, and community-governed: the software is implemented in Python on top of Xarray and Zarr, the code and event catalog are available on GitHub, and new events, observational targets, and metrics can be contributed by the broader forecasting and verification community (McGovern et al., 1 May 2026).

The benchmark explicitly adopts an impact-based perspective. Its metrics are selected to answer operationally salient questions such as how far ahead a model can warn, whether it correctly locates an event, whether it captures intensity, and whether it overforecasts marginal cases (McGovern et al., 1 May 2026). This orientation differentiates EWB from benchmarks focused primarily on global prognostic accuracy.

2. Hazard taxonomy and event construction

The first EWB release contains five phenomena, each represented by 4–20 case studies chosen from 2020–2024 and intended to cover both global and regional impacts (McGovern et al., 1 May 2026). These categories span markedly different dynamical regimes.

Heat waves are land-only events at approximately $10^5$–$10^6\ \mathrm{km}^2$ scale and 3–7 day duration, defined where 2 m temperature exceeds its local 85th percentile for at least 3 consecutive days, with a $\pm 24$ h relaxation (McGovern et al., 1 May 2026). A cited example is the 2021 Pacific-Northwest heat dome (McGovern et al., 1 May 2026).

Freezes are also land-only, with comparable spatial extent and duration, but are defined where 2 m temperature falls below $0\,^\circ\mathrm{C}$ and below its 15th percentile for at least 3 days (McGovern et al., 1 May 2026).

Convective-outbreak days are defined over the United States, Canada, and Australia as single days with at least 50 severe-report clusters, bounded by a 2D bounding box around the 0.01-probability contour of a weighted Practically Perfect Hindcast (PPH) kernel built from tornado and hail reports (McGovern et al., 1 May 2026).

Tropical cyclones are global landfalling storms represented as first-landfall-only events, using 3 h resolution tracks, spatial scales of approximately $10^2$–$10^3\ \mathrm{km}$, and life spans up to 10 days (McGovern et al., 1 May 2026). Observed tracks are extracted from IBTrACS, while forecast tracks are obtained with a TempestExtremes-style warm-core tracker (McGovern et al., 1 May 2026).

Atmospheric rivers are global landfall IVT objects of at least 500 grid-points with IVT $> 400\ \mathrm{kg\ m^{-2}\ s^{-1}}$ and duration 1–4 days, tracked in ERA5 using filters on IVT minima, Laplacian of IVT, and object shape (McGovern et al., 1 May 2026).

A distinctive feature of the event catalog is the inclusion of marginal events for every category. Examples include days with 16th–84th percentile temperatures or days with few severe reports (McGovern et al., 1 May 2026). This is intended to mitigate the “forecaster’s dilemma,” in which forecast systems can appear skillful on extremes by overforecasting rare events at the expense of broader performance (McGovern et al., 1 May 2026).

3. Observational targets and benchmark data model

EWB organizes observational targets and forecast data by event category, with each case study carrying metadata describing its spatial bounding box, time window, and target variables (McGovern et al., 1 May 2026). The observational side is deliberately heterogeneous because each hazard requires different reference data.

For heat waves and freezes, EWB uses hourly GHCNh station data gridded to $0.25^\circ$, with ERA5 reanalysis as a fallback where station coverage is sparse (McGovern et al., 1 May 2026). For convective-outbreak days, the benchmark draws on the SPC severe-report archive for the United States, the Northern Hail and Tornado Project reports for Canada, and hand-curated Australian hail and tornado reports; PPH contours are computed by Gaussian-kernel density (McGovern et al., 1 May 2026). Tropical cyclone verification uses the IBTrACS best-track archive at 3 h resolution, including center position, maximum wind, and minimum central pressure, together with a land mask derived from a 1:50 m vector coastline shapefile (McGovern et al., 1 May 2026). Atmospheric river verification uses hourly IVT and winds from ERA5 and a Python-native tracker for object footprints and landfall intersections (McGovern et al., 1 May 2026).

Model forecasts are ingested as CF-compliant Xarray or Zarr datasets on the same $0.25^\circ$ grid and valid times; the framework automatically regrids or subsets forecasts as needed (McGovern et al., 1 May 2026). This reflects a deliberate software-level normalization of model outputs so that AI and physics-based systems can be compared under a common interface.

A concise summary of the benchmark’s first-release scope is given below.

Component	EWB specification
Phenomena	Heat waves, freezes, convective-outbreak days, tropical cyclones, atmospheric rivers
Event period	2020–2024
Forecast format	CF-compliant Xarray/Zarr on $0.25^\circ$ grid and valid times
Core observational targets	GHCNh, ERA5, SPC/NHTP/Australian severe reports, IBTrACS

This event-centric structure differs from benchmark designs centered primarily on a single data source or a single task. A plausible implication is that EWB is intended less as a leaderboard for one canonical tensor dataset than as a verification substrate for heterogeneous operational phenomena.

4. Verification methodology and metric definitions

EWB emphasizes deterministic, impact-based verification metrics tailored to hazard type, while noting that future releases will add probabilistic measures such as CRPS and ROC AUC (McGovern et al., 1 May 2026). The framework uses standard notation in which $10^6\ \mathrm{km}^2$0 denotes forecasts, $10^6\ \mathrm{km}^2$1 observations, $10^6\ \mathrm{km}^2$2 the number of valid samples, and $10^6\ \mathrm{km}^2$3, $10^6\ \mathrm{km}^2$4, $10^6\ \mathrm{km}^2$5 the contingency-table counts (McGovern et al., 1 May 2026).

For magnitude errors, EWB includes Mean Absolute Error and Root Mean Squared Error: $10^6\ \mathrm{km}^2$6 It also defines regional variants such as Regional MAE and Regional RMSE by restricting the sum to points inside the event bounding box (McGovern et al., 1 May 2026). These are intended to prevent global skill from dominating event-specific performance.

For footprint verification, EWB uses the Threat Score or Critical Success Index: $10^6\ \mathrm{km}^2$7 and notes the equivalence with Intersection-over-Union: $10^6\ \mathrm{km}^2$8 It also includes the Equitable Threat Score,

$10^6\ \mathrm{km}^2$9

and the False Alarm Ratio,

$\pm 24$0

for assessing chance-corrected performance and overforecasting behavior (McGovern et al., 1 May 2026).

Temporal utility is represented by lead time, defined as

$\pm 24$1

that is, the difference between predicted and true event onset date (McGovern et al., 1 May 2026). For point events such as tropical cyclone landfall, EWB uses spatial displacement based on great-circle distance: $\pm 24$2 This explicitly encodes geographic landfall error rather than relying on gridpointwise field discrepancies (McGovern et al., 1 May 2026).

The benchmark’s metric design is consistent with broader work on extreme-event verification. Weighted verification methods, including threshold-weighted and multivariate weighted scores, were proposed as a way to emphasize tail behavior and compound events in forecast evaluation (Allen et al., 2022). RealBench likewise adopts event-specific metrics such as POD, FAR, CSI, and tropical-cyclone track and intensity errors under operational constraints (Li et al., 24 May 2026). This situates EWB within a larger shift from broad aggregate weather skill toward hazard-conditioned evaluation.

5. Software architecture and evaluation workflow

EWB is packaged as a Python library, extremeweatherbench, and exposes both a command-line interface, ewb-run, and a modular API (McGovern et al., 1 May 2026). The codebase is organized into modules for event definitions, target acquisition, metric computation, tracking, and evaluation.

The main components are as follows:

Module	Function
core.events	Event categories, metadata, bounding boxes, time windows
core.targets	Download and preprocess observational targets
core.metrics	Implement MAE, RMSE, CSI/IOU, ETS, FAR, lead time, displacement
trackers	Python re-implementations of TempestExtremes, ARCO, and PPH construction
evaluation	Match forecasts to events, invoke metrics, summarize results

The framework also includes an extension API. New events can be defined by sub-classing EventCase, linking to new observational data, and listing metric functions; new metrics can be registered by annotating a function with @register_metric (McGovern et al., 1 May 2026). This is central to the benchmark’s claim of being extensible and community-driven.

A typical workflow consists of installing the package, fetching event metadata and observational targets, evaluating a forecast against selected events and metrics, and then inspecting JSON or CSV summaries or using built-in plotting utilities (McGovern et al., 1 May 2026). The benchmark therefore couples methodological specification with executable infrastructure, which is often absent in purely descriptive evaluation proposals.

6. Relation to adjacent benchmarks and terminological overlap

EWB was introduced in part as an advance over earlier benchmark ecosystems. WeatherBench and WeatherBench 2 are characterized as focusing on global-scale fields such as RMSE and ACC and on prognostic variables at 500 hPa, without providing high-impact case studies or observational targets for extremes (McGovern et al., 1 May 2026). ECMWF’s in-house evaluation is described as broad but not built around an openly exposed standardized test set of recent extreme events (McGovern et al., 1 May 2026). WeatherReal is noted to offer point-observation benchmarks but not event definitions and impact-based metrics (McGovern et al., 1 May 2026). In this framing, EWB’s distinct contribution lies in combining an explicit hold-out of 2020–2024 events, observational ground truth when available, hazard-tailored metrics, and open extensible code (McGovern et al., 1 May 2026).

Subsequent work broadens the landscape of extreme-weather evaluation. RealBench emphasizes low-latency operational analyses, station-based verification using more than 10,000 GHCNh stations, and a strictly out-of-distribution 2025 test year, arguing that reanalysis-based benchmarks can misestimate true operational performance on extremes (Li et al., 24 May 2026). ExEBench targets foundation models across seven extreme-event categories, including floods, wildfires, storms, tropical cyclones, extreme precipitation, heatwaves, and cold waves, and spans forecasting, segmentation, and nowcasting tasks across multiple Earth-observation modalities (Zhao et al., 13 May 2025). XWOD addresses object detection in traffic scenes under seven real-image extreme weather conditions, including flooding, tornado, and wildfire (Chen et al., 12 May 2026).

The acronym “EWB” also appears in later works with different referents. In the EWE framework, “Extreme Weather Bench” denotes a benchmark for automated diagnostic analysis of 103 high-impact events, scored with a seven-stage step-wise rubric from planning through report synthesis (Jiang et al., 26 Nov 2025). In EO-WM, the “Extreme Summer Benchmark” is described as one of two diagnostic tests in an EWB suite for weather-response fidelity in Earth observation forecasting (Luo et al., 25 Jun 2026). This suggests some terminological ambiguity in the literature, even though the 2026 forecast-verification framework remains a specific benchmark with a distinct software stack and event taxonomy (McGovern et al., 1 May 2026).

7. Scientific significance, usage guidance, and likely evolution

EWB’s stated practical recommendations are tightly coupled to its evaluation philosophy. Users are advised to evaluate on both extreme and marginal events, to prefer regional RMSE or event-region MAE over global scores when assessing event performance, to examine lead-time curves rather than only short-range behavior, and to inspect spatial skill maps or IOU trajectories for convective and atmospheric-river events in order to diagnose displacement and timing biases (McGovern et al., 1 May 2026). The benchmark also encourages extension through additional metrics and observational datasets, including possible future additions such as droughts, flash floods, radar, and satellite data (McGovern et al., 1 May 2026).

Several broader implications follow from the benchmark design, though they should be read as interpretation rather than formal specification. The first is that EWB operationalizes “trustworthiness” not as a single accuracy number but as a structured set of hazard-conditioned competencies. The second is that its event-plus-marginal construction attempts to balance sensitivity to societal extremes against resistance to pathological overforecasting. The third is that its software architecture makes benchmark evolution part of the design rather than an afterthought.

The surrounding literature points to several natural directions for expansion. Weighted scoring rules for univariate and multivariate extremes offer one path toward more explicit tail-sensitive and compound-event verification (Allen et al., 2022). RealBench’s insistence on low-latency reference data and strict out-of-distribution test periods suggests stronger operational realism as another axis of development (Li et al., 24 May 2026). The existence of task-specific extreme-weather benchmarks for autonomous driving, Earth observation, foundation models, and agentic diagnosis indicates that the benchmark ecosystem is fragmenting by modality and task (Chen et al., 12 May 2026, Zhao et al., 13 May 2025, Jiang et al., 26 Nov 2025, Luo et al., 25 Jun 2026). A plausible implication is that EWB occupies the role of a core verification framework for meteorological forecast models, while adjacent “extreme weather bench” efforts adapt the benchmark idea to other scientific and operational settings.

In that sense, EWB is best understood as a formalization of event-centric weather verification: a standardized, open, and extensible mechanism for determining whether a forecast system performs well on the specific high-impact phenomena that drive warnings, decisions, and public consequences (McGovern et al., 1 May 2026).