RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges

Abstract: Accurate evaluation of weather forecasting models is critical for their reliable deployment in real-world applications. However, existing benchmarks predominantly rely on reanalysis products such as ERA5, which are generated through delayed data assimilation and do not reflect the constraints of real-time operational forecasting, thereby resulting in a systematic mismatch between benchmark performance and real-world forecasting. In this work, we introduce RealBench, a next-generation benchmark for AI weather forecasting that emphasizes realistic evaluation under operational conditions. RealBench features a strictly out-of-distribution test set spanning 2025 to eliminate data leakage and capture recent atmospheric regimes. It integrates multiple data sources, including low-latency operational analysis and a large-scale global in-situ observation dataset comprising over 10,000 stations, enabling direct evaluation against real atmospheric measurements. Beyond standard global metrics, RealBench provides a comprehensive evaluation framework for high-impact extreme events, including heatwaves, cold surges, and tropical cyclones, using event-specific metrics that better reflect real-world forecasting priorities. The evaluation results reveal substantial discrepancies between reanalysis-based metrics and real-world performance, particularly concerning extreme events. By highlighting the limitations of existing benchmarks, this work establishes a more faithful and operationally relevant evaluation paradigm, providing a rigorous foundation for advancing next-generation AI weather forecasting systems. The benchmark implementation is available at: https://github.com/lixruize-del/NWP-Benchmark.

• The paper introduces RealBench, a benchmark addressing the gap between reanalysis-based evaluations and real-world operational forecasting.
• It demonstrates significant performance drops in AI models when evaluated with real-time analysis and extensive station data, particularly for near-surface variables and extreme events.
• The study employs spatiotemporal autoregressive tasks and spectral analysis to reveal scale-dependent fidelity loss, underscoring the need for hybrid evaluation protocols.

RealBench: A Rigorous Benchmark for Operational Evaluation of AI-Driven Numerical Weather Forecasting

Motivation and Benchmark Design

Increasing reliance on AI-based numerical weather prediction (NWP) necessitates evaluation protocols that accurately mirror operational constraints and real-world forecasting needs. Most prior benchmarks, such as WeatherBench and its variants, have used retrospective reanalysis datasets (e.g., ERA5) as ground truth, neglecting the latency, assimilation realism, and observational biases inherent in operational weather forecasting. This creates a critical gap between benchmark scores and genuine operational readiness. RealBench addresses this by introducing an evaluation pipeline rooted in real-time operational analysis and dense in-situ observations, targeting both large-scale atmospheric fields and high-impact extreme events.

Figure 1: Overview of RealBench, illustrating a real-world and real-time benchmarking pipeline for evaluating AI weather models in operational forecasting settings.

The framework strictly separates evaluation from historical training data, using an out-of-distribution test period (2025) and incorporates ECMWF operational analysis—characterized by shorter assimilation windows and reduced latency—plus the WEATHER-10K station network for direct observational verification. Extreme event evaluation is systematized with percentile-based heatwave/cold-surge thresholds and event-specific metrics for tropical cyclones.

Data Sources and Analysis

RealBench leverages three main sources: ERA5 reanalysis, ECMWF operational analysis, and the WEATHER-10K station dataset. ERA5 offers spatially complete retrospective fields, while operational analysis replicates real-time forecasting constraints and observational latency. WEATHER-10K provides extensive spatial coverage from more than 10,000 globally distributed stations, though with heterogeneity in density across continents and sparse coverage over oceans and high latitudes.

Figure 2: Global and temporal RMSE analyses, station-model comparisons, and spatial bias maps under extreme events highlight variable- and region-dependent gaps between reanalysis and in-situ observations.

Comparison of ERA5 and WEATHER-10K in 2025 reveals mean RMSEs of 2.14°C (T2M), 2.52°C (D2M), and 1.97 m/s (wind speed), emphasizing spatial and seasonal modulation of error, especially under complex terrain and during extreme events. The bias structure during events such as Typhoon Ragasa further demonstrates that reanalysis products can systematically underestimate local extremes.

Figure 3: WEATHER-10K station distribution evidences strong geographic imbalance, emphasizing the need for spatially-aware evaluation protocols.

Evaluation Protocol and Baseline Comparison

Forecasting is formulated as a spatiotemporal autoregressive prediction task, with eight key atmospheric variables assessed across four spatially-weighted metrics: WRMSE, bias, ACC, and activity. Models—including AIFS, Aurora, GraphCast, Pangu-Weather, FuXi, FengWu, Stormer, and NeuralGCM—are compared using officially released checkpoints, ensuring robust architecture-level assessment.

WRMSE and ACC comparisons underscore a significant drop in performance when transitioning from ERA5-based to operational analysis and station-based evaluation, with near-surface variables and stations showing pronounced degradation and revealing a real-world gap not evident in reanalysis-only protocols.

Figure 4: Scorecard for upper-level variables in 2025, showing that operational analysis preserves relative ranking but reveals larger absolute errors compared to ERA5.

Figure 5: Surface-level variable scorecard highlights increased error and spatial variability when evaluated against WEATHER-10K station observations compared to gridded reference fields.

Figure 6: ACC comparison across three datasets (ERA5, operational analysis, and WEATHER-10K) exposes rapid performance decay for longer leads and irregular station locations.

Extreme Event Assessment

RealBench integrates rigorous evaluation of heatwaves, cold surges, and tropical cyclones using percentile-based object detection and skills metrics such as CSI, POD, and FAR. The benchmark defines temperature extremes relative to a climatological sliding window and applies IoU matching for robust event verification.

Models display marked precision-recall trade-offs. AIFS sustains highest CSI for heatwaves at short lead times (0.513/Day 1), but all models show rapid skill degradation as lead increases. Negative biases in Tmax at longer leads demonstrate systematic underestimation of heat intensity, likely due to global RMSE-driven smoothing.

Figure 7: Heatwave case analyses confirm lead-time dependent bias growth and spatial smoothing, revealing underestimation of extreme heat intensity across models.

Cold surges are forecasted with even lower skill, attributed to rarity and under-representation in training data. Positive Tmin biases at extended leads indicate models’ tendency to predict temperatures too warm, consistent with spatial smoothing effects.

Figure 8: Cold-surge event visualizations illustrate persistent warm bias during extreme cold, and increasing forecast spread with lead time.

Tropical cyclone evaluation reveals high accuracy in track prediction at short leads (errors ≈50–70 km), but models uniformly underestimate intensity, with MSLP biases up to +39.5 hPa at Day 5 and wind speed MAEs consistently above 15 m/s. Coarser resolution models (e.g., Stormer) exhibit the largest intensity error, underscoring scale limitations.

Figure 9: Mean track error curves for tropical cyclones highlight consistent lead-time dependent accuracy degradation.

Physical Realism and Spectral Fidelity

Zonal power spectra analyses indicate that AI models match ERA5 at large spatial scales but diverge increasingly at high wavenumbers as forecast lead time increases, notably for specific humidity and temperature fields. This spectral discrepancy correlates with the observed smoothing of extremes and underrepresentation of local-scale variability.

Figure 10: Zonal-mean power spectra comparisons confirm that AI models retain large-scale coherence but lose fidelity at small scales over longer forecast horizons.

Implications, Limitations, and Future Directions

RealBench demonstrates that strong ERA5 scores do not guarantee operational reliability. Models tuned to idealized, lagged reanalysis fields tend to overfit smooth global patterns, performing poorly on irregular, observation-rich ground truth and failing to capture peak intensities in extreme events. The demonstrated accuracy gaps and bias structures necessitate hybrid evaluation metrics and multi-scale training protocols.

Key limitations include geographic station imbalance, inherent spatial scale mismatch between gridded fields and pointwise stations, and reliance on a 12-year climatological baseline for event thresholds. Future work should pursue regional downscaling, probabilistic ensemble verification, and multi-source observation integration to close the scale-representativeness gap and enable uncertainty quantification in operational forecast delivery.

Conclusion

RealBench presents a rigorous, operationally relevant evaluation paradigm for AI-based weather forecasting. By incorporating real-time analysis, extensive station observations, and specialized extreme-event metrics, the benchmark exposes substantial discrepancies between reanalysis-based and real-world model performance, especially for near-surface variables and extremes. These findings establish RealBench as a robust foundation for advancing data-driven weather forecasting toward operational deployment and highlight the necessity for continuous improvement in forecast realism, spatial fidelity, and extreme event accuracy (2605.24945).