Fractions Skill Score in Forecast Verification

• Fractions Skill Score (FSS) is a spatial verification metric that quantifies agreement between observed and forecasted threshold-exceeding events, mitigating the double penalty effect.
• It is computed by comparing local fractions of hits in binary thresholded maps over moving windows, using a normalized squared difference approach.
• FSS is applied in deep learning-based weather models for lightning and extreme rainfall, providing a multi-scale assessment of forecast spatial accuracy.

The Fractions Skill Score (FSS) is a quantitative spatial verification statistic broadly adopted in earth sciences and meteorological modeling to evaluate forecast skill in predicting spatial fields of thresholded phenomena, notably precipitation and lightning. FSS quantifies the agreement between observed and modeled spatial patterns over variable scales and is especially valued for its capacity to mitigate the "double penalty" effect endemic to categorical metrics—where forecasts diverge from observations by small distances or partial coverage incur disproportionate penalties. Recent research employs FSS for skill assessment in deep learning-based parameterizations of lightning stroke density (II et al., 12 Sep 2025), extreme rainfall modeling in the Sahel (Sanogo et al., 27 Oct 2025), and precipitation nowcasting evaluation (Yan et al., 30 Oct 2024).

1. Mathematical Definition and Calculation Procedure

FSS measures the match between observed and predicted fractions of threshold-exceeding grid points within moving spatial windows and is formally defined as:

$$FSS = 1 - \frac{\sum_{i=1}^{N} (O_i - M_i)^2}{\sum_{i=1}^{N} (O_i^2 + M_i^2)}$$

where:

• $O_i$: Fraction of observed grid points exceeding a prescribed intensity threshold within window $i$
• $M_i$: Fraction of forecast grid points exceeding the threshold within window $i$
• $N$: Total number of windows in the spatial domain

The process involves:

• Thresholding the observed and predicted fields to produce binary maps (values above the threshold set to 1, others to 0)
• For each window, computing the local fraction of positives (i.e., hits)
• Aggregating the squared differences over all windows and normalizing by the sum of squares

A variant, Modified FSS (MFSS), calculates skill score within bins defined by order of magnitude (e.g., for different lightning densities), facilitating skill analysis as a function of intensity regimes (II et al., 12 Sep 2025).

2. Interpretation of FSS Values and Skillful Scale

FSS values range from 0 (no skill, total spatial disagreement) to 1 (perfect match). An FSS above ~0.5 is typically regarded as indicative of "skillful" predictive capacity at a given window size or resolution, though formal skillfulness thresholds have been proposed:

$FSS_{uniform} = 0.5 + \frac{f}{2}$

where $f$ is the domain fraction of hits. The minimal window size at which FSS exceeds $FSS_{uniform}$ defines the minimum spatial scale at which forecasts are statistically skillful (Sanogo et al., 27 Oct 2025).

High FSS indicates that a model accurately reproduces the spatial placement and coverage of high-intensity events at the tested resolution, not just the values themselves. For example, in lightning density modeling, domain-mean FSS values as high as 0.93 are reported for oceanic regions using deep learning parameterizations (II et al., 12 Sep 2025), reflecting near-perfect spatial correspondence.

3. Methodological Application in Key Research Domains

Lightning and Precipitation Modeling

• FSS is central in benchmarking the spatial skill of convolutional neural networks (CNNs) trained to predict lightning stroke density (II et al., 12 Sep 2025). It permits direct comparison against physical parameterizations (such as CAPE × Precipitation) and across different geographic regimes (land/ocean, tropics).
• In the assessment of the ICOsahedral Nonhydrostatic (ICON) meteorological model's precipitation forecasts, FSS is computed for different physical parameterization approaches (explicit vs. parameterized convection), using dynamically localized domains and empirical intensity thresholds (95th percentile) to focus evaluation on extreme events (Sanogo et al., 27 Oct 2025).

Loss Function and Deep Learning Evaluation

• FSS is routinely used to report skill in precipitation nowcasting models in deep learning literature (Yan et al., 30 Oct 2024). Its patch-wise formulation enables spatially tolerant assessment not achievable with per-pixel metrics like MSE or SSIM.

Empirical Example Table

Application Domain	FSS Range / Score	Key Context
CNN Lightning Stroke Density (Ocean) (II et al., 12 Sep 2025)	up to 0.93	CNN far outperforms CAPE×Precip parameterization
Deep Convective Rain (ICON, Sahel) (Sanogo et al., 27 Oct 2025)	FSS > 0.5 at 2.25° domain	Skillful forecast scale depends on physical regime
ConvLSTM Nowcasting (MNIST) (Yan et al., 30 Oct 2024)	0.61–0.82	FACL-trained models increase FSS over MSE-trained

4. Limitations and Extensions of FSS

Fundamental limitations of FSS arise from its reliance on binary classification of continuous intensities:

• The metric is sensitive to the choice of threshold; near-miss predictions (intensity just below threshold) are penalized identically to completely incorrect predictions.
• Granularity is lost because all above-threshold hits count equally, disregarding intensity magnitude within classes.

To address these, the Regional Histogram Divergence (RHD) has been proposed (Yan et al., 30 Oct 2024), leveraging per-patch histogram (multi-bin) comparisons via mean Kullback–Leibler divergence, thus providing a finer, threshold-free measure of local distribution matching.

5. Comparative and Diagnostic Use

FSS is frequently deployed alongside other spatial and object-based validation scores, such as the Structure-Amplitude-Location (SAL) metric (Sanogo et al., 27 Oct 2025). While FSS isolates the scale and field-wide agreement of event coverage, SAL attributes discrepancies to specific structural, amplitude, or locational errors. This enables combined diagnostic insight: FSS localizes skillful forecast scale, while SAL details the nature (size, intensity, spatial offset) of mismatch. For physically parameterized models, this has revealed case-specific superiority of explicit versus parameterized convection in reproducing extreme events.

6. Implications for Model Development and Earth System Applications

Quantitative evidence demonstrates that high FSS scores correspond to significantly reduced mean spatial bias and better representation of extremes (II et al., 12 Sep 2025). Deep learning models, evaluated with FSS, deliver major improvements in global and regional spatial fidelity over traditional parameterizations in both lightning and precipitation applications. FSS’s multi-scale analysis framework, including MFSS and synergy with object-based metrics (SAL), offers route to robust, context-appropriate verification for operational nowcasting and climate model development.

A plausible implication is that as generative and DL-based weather models become mainstream, rigorous spatial verification via FSS (and its histogram-based extensions) will remain essential for ensuring physical reliability in both research and operations.