How to derive skill from the Fractions Skill Score

Abstract: The Fractions Skill Score (FSS) is a widely used metric for assessing forecast skill, with applications ranging from precipitation to volcanic ash forecasts. By evaluating the fraction of grid squares exceeding a threshold in a neighbourhood, the intuition is that it can avoid the pitfalls of pixel-wise comparisons and identify length scales at which a forecast has skill. The FSS is typically interpreted relative to a `useful' criterion, where a forecast is considered skillful if its score exceeds a simple reference score. However, the typical reference score used is problematic, as it is not derived in a way that provides obvious meaning or that scales with neighbourhood size, and forecasts that do not exceed it can have considerable skill. We therefore provide a new method to determine forecast skill from the FSS, by deriving an expression for the FSS achieved by a random forecast, which provides a more robust and meaningful reference score to compare with. Through illustrative examples we show that this new method considerably changes the length scales at which a forecast would be regarded as skillful, and reveals subtleties in how the FSS should be interpreted.