WeatherReal: Global In-Situ Weather Benchmark

• WeatherReal is a global in-situ observational benchmark and evaluation framework that bridges reanalysis research with real-world weather forecasting.
• It employs a fully automated multi-stage quality control system using physical bounds, clustering, and neighbor checks to ensure data reliability.
• The framework integrates real-time operational pipelines with standardized metrics for model assessment, supporting both numerical and ML-based forecasting.

WeatherReal is a global, in-situ observational benchmark, operational pipeline, and evaluation framework for weather forecast model assessment and robust adverse weather perception. It bridges the gap between reanalysis-driven research and real-world application requirements, providing ground-truth station data, rigorous quality control, standardized metrics, and real-time operational pipeline guidance for both numerical and data-driven forecasting models (Jin et al., 2024).

1. Dataset Foundations and Structure

WeatherReal encompasses two primary observational datasets:

• WeatherReal-ISD: Derived from the NCEI Integrated Surface Database (ISD), aggregating SYNOP, METAR, and national network reports. The 2023 release comprises approximately 14,000 quality-controlled, deduplicated global stations.
• WeatherReal-Synoptic: Compiled by Synoptic Data PBC, integrating over 53,000 temperature and 34,000 precipitation stations, with high density in North America and Europe due to public and private sensors (e.g., Citizen Weather Observer Program, COOP).

Key meteorological variables include: 2 m air temperature, 2 m dew point, surface and mean sea-level pressure, 10 m wind speed/direction, total cloud cover (ISD only), and 1–24 h precipitation accumulations. Spatial coverage is global, with densities mapped on a 2.5° × 2.5° grid, and native native reporting frequencies are resampled to hourly cadence via closest-to-hour selection. Data are published for a complete year (2023); subsequent releases are planned to extend this record (Jin et al., 2024).

2. Automated Quality Control and Data Curation

WeatherReal applies a multi-stage, fully automated quality-control (QC) system, with each record flagged as “normal,” “suspect,” or “erroneous.” Erroneous records are removed before public release. The QC suite comprises:

• Physical bounds: Variable-specific range checks, WMO global extremes, e.g., 0 ≤ wd ≤ 360°, 300 ≤ sp ≤ 1100 hPa.
• Distributional-gap checks: For each station-variable, differences between observation and co-located ERA5 reference are Gaussian-fitted (median/MAD). Outliers with density < 0.01 or outside k·MAD gaps are escalated.
• Cluster deviation (DBSCAN): Utilized when > 50% of station-variable deviations are outliers to preserve legitimate structural regime shifts.
• Spikes, persistence, and cross-variable consistency: Multi-hour spikes, > 48 h persistence, violations of physical relationships (e.g., supersaturation, inconsistent wind) are flagged.
• Neighbor checks: Comparison with up to 8 geographically proximate, elevation-matched stations; majority-based aggregation of anomaly flags.
• Flag refinement: Contextual downgrading (e.g., for agreed upon low-pressure troughs, diurnal temperature alignment) and final integration (e.g., suspect segments bracketed by errors are escalated, station drop if > 50% erroneous).

Only records clean or “suspect” post-refinement are retained. All datasets and QC scripts are available openly (Jin et al., 2024).

3. Benchmarking, Metrics, and Evaluation Protocol

WeatherReal serves as the backbone for diverse weather prediction evaluation tasks:

• Forecast horizons: Nowcasting (0–24 h, hourly), short-range (0–72 h, 1 h intervals), medium range (0–10 days, 6 h), and subseasonal/seasonal (weeks 1–6, probabilistic).
• Metrics:
  • Deterministic: RMSE, MAE
  • Probabilistic: CRPS
  • Binary events: Equitable Threat Score (ETS), accuracy/large-error rates.
• Data comparison: Forecast-observation pairs are matched spatiotemporally; pairs with missing/erroneous values are omitted.
• Case study integration: Detailed regional and event-specific benchmarking, e.g., European heatwaves, hurricanes, typhoons, where WeatherReal documents model-observation divergence in diurnal temperature range, precipitation maxima, surface pressure, and wind extremes.

Comparison against standard reanalysis (e.g., ERA5) reveals systematic model underestimation of extremes (e.g., diurnal temperature, intense precipitation) and highlights regional/terrain-dependent biases not captured by reanalyses (Jin et al., 2024, Li et al., 24 May 2026).

4. Operational Real-Time Forecasting and Pipeline Design

WeatherReal’s operational paradigm is influenced by the RealBench design (Li et al., 24 May 2026), embedding:

• Realtime ingestion: ECMWF operational analyses with ~1 h latency, global station feeds (buffered ±15 min for 6 h aggregation), application of reanalysis-based QC filtering thresholds.
• Strict out-of-distribution (OOD) logic: Training/validation on years ≤ 2024, full-year evaluation on 2025 for final skill; climatological thresholds are recomputed with a rolling 12-year window to reflect contemporary extremes.
• Skill assessment:
  • Grid-based RMSE, ACC, and bias versus operational analysis.
  • Station-based RMSE/ACC via local interpolation.
  • Extreme-event metrics: Probability of Detection (POD), False Alarm Ratio (FAR), Critical Success Index (CSI) for heatwaves/cold surges; Direct Position Error (DPE), intensity MAE/bias for tropical cyclones.
• Continuous dashboard: Live tracking of skill metrics, regional scorecards, extreme-event watches, and cyclone anomaly tracking. Automated alerts are triggered if CSI < 0.3 at 3-day lead.
• Model updating: Seasonal retraining on the latest year’s data, annual percentile and climatology updates, and the use of hybrid loss functions to reduce scale-mismatch.

These measures ensure zero-leakage, true operational rigor, and enforceability of real-time forecasting skill for all models benchmarked via WeatherReal (Li et al., 24 May 2026).

5. Comparative Model Performance and Case Studies

WeatherReal evaluates both advanced data-driven and operational NWP models. In medium-range (0–10 day) forecasts over 2023:

Model	t2m RMSE [K]	ws10m RMSE [m/s]	msl RMSE [hPa]	6h precip (>1 mm, ETS)
MS-Point	2.26	1.75	–	–
Aurora-9 km	2.42	2.19	2.94	–
ECMWF IFS	2.77	2.25	3.10	0.25
GFS	3.17	2.46	3.48	–

Aurora-9 km outperforms ECMWF IFS for t2m by 5–15% at all leads, but loses its edge over ECMWF during strong European heatwaves, emphasizing the criticality of in-situ evaluation under extremes (Jin et al., 2024).

WeatherReal documents that reanalysis-based benchmark skill (ERA5) overestimates operational performance by 10–20%, and that model skill in extremes degrades more rapidly in real-time evaluation versus standard benchmarks (CSI drops from ~0.5 at day 1 to <0.17 by day 10 for heatwaves) (Li et al., 24 May 2026).

6. Integration with Contemporary ML, Spatio-Temporal, and Perception Tasks

WeatherReal is cited as essential for resolving the observed domain gap between academic time-series ML and operational weather forecasting. Large-scale spatio-temporal datasets (e.g., Weather2K, WEATHER-5K) and non-stationary events challenge transformer-based TSF models, which show rapidly rising forecast errors beyond 3-day horizons and poor skill for rare extremes (SEDI ~10–12% for 99.5th percentiles at 5 days) (Han et al., 2024, Zhu et al., 2023).

WeatherReal’s evaluation framework supports hybrid physics–ML architectures, station-aware graph models, probabilistic ensemble prediction, and bias correction pipelines, and is designed to foster continuous algorithmic improvement by closing the gap between observed and reanalysis-driven skill (Li et al., 24 May 2026, Han et al., 2024, Zhu et al., 2023).

For computer vision and sensor perception in adverse weather (e.g., driving scenes), WeatherReal-style real-world augmented datasets (e.g., WeatherReal in WeatherDiffusion (Zhu et al., 9 Aug 2025)) enable benchmarking of image restoration, intrinsic decomposition, and segmentation robustness under fog, rain, and snow.

7. Broader Impact and Open Research Directions

WeatherReal delivers rigorous, scalable, and application-focused benchmarking for both traditional and AI-driven weather prediction, with strict operational quality control, real-time data delivery (via protocols such as OpenWeather (Yanes, 2011)), and open-source, reproducible evaluation. It is positioned as the empirical backbone for:

• Hyper-local extreme event nowcasting and solar/wind power integration
• Agriculture, aviation, and smart city deployment with user-driven precision
• MLWP physical consistency monitoring through frameworks such as PhysMetrics.Weather (Kasteleyn et al., 9 Jun 2026)
• Perceptual and stochastic weather synthesis for AV/robotics domain adaptation

Future extensions include integration of crowdsourced user reports, incorporation of additional observation types (e.g., cloud cover, radiation), multi-modal (satellite, radar) data fusion, and expanded hyper-local benchmarking for subseasonal, probabilistic, and extreme-weather tasks (Jin et al., 2024, Han et al., 2024).