Rugged Scores in Forecast Evaluation

• Rugged Scores are a conceptual framework for scoring rules that incorporate non-local dependencies, scale invariance, and robustness to outliers in forecast evaluation.
• They extend traditional measures like RPS and CRPS by including ordered-outcome penalties, threshold-based adjustments, and truncated kernels to mitigate the influence of extreme events.
• Empirical studies, notably in football forecasting and spatial prediction, demonstrate that rugged scores can outperform standard metrics like the ignorance and Brier scores under varying uncertainty and outlier scenarios.

A scoring rule is “a function of a probabilistic forecast and the corresponding realized outcome used to evaluate forecast performance.” The expression “Rugged Scores” is not the formal name of a single framework in the cited literature; as an Editor’s term, it can be used for scoring rules that introduce additional structure beyond a purely local dependence on the realized outcome, including ordered-outcome penalties, scale standardization, and robustness to outliers. In that sense, the relevant literature spans the Ranked Probability Score (RPS) and related non-local scores for ordered outcomes, the scaled CRPS (SCRPS) and truncated-kernel robust variants for heterogeneous uncertainty and outliers, and the FIRM family for ordered multicategorical warnings aligned with a fixed risk threshold (Wheatcroft, 2019, Bolin et al., 2019, Taggart et al., 2021).

1. Conceptual scope and defining properties

The literature distinguishes scoring rules by locality, by whether they are sensitive to distance in an ordered outcome space, by whether they are locally scale invariant, and by whether they are robust to outliers. A local score depends only on the probability at the realized outcome. A non-local score depends on more than just the realized outcome probability, using other parts of the forecast distribution as well. For ordered outcomes, a score is sensitive to distance if probability assigned to outcomes “closer” to the realized outcome is rewarded more than probability assigned to outcomes farther away. In a separate line of work, local scale invariance addresses whether average scores implicitly overweight high-uncertainty cases, while robustness addresses whether a small number of outliers can dominate average score comparisons (Wheatcroft, 2019, Bolin et al., 2019).

Score	Key property	Status in the cited literature
Ignorance score	Local; insensitive to distance	Found to outperform both the RPS and the Brier score in the football simulations
Brier score	Non-local; insensitive to distance	Often similar to RPS; sometimes better
RPS	Non-local; sensitive to distance	No evidence of superiority in the football setting
CRPS	Not locally scale invariant; not robust	Can give more importance to observations with large uncertainty
SCRPS	Locally scale invariant; not robust	A standardized, locally scale-invariant analogue of CRPS
rCRPS / rSCRPS	Robust	Derived by truncating the kernel

This grouping suggests that “ruggedness,” in the sense relevant here, is not a single mathematical property. Rather, it refers to the addition of structural constraints intended to make forecast evaluation more discriminating in settings with ordered categories, heterogeneous variance, or contamination by extreme observations.

2. Ranked probability scoring and ordered outcomes

For an ordered event with $r$ outcomes, the Ranked Probability Score is

$$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$

Here $p_j$ is the forecast probability for outcome $j$, $o_j$ is the observed outcome indicator for outcome $j$, and outcomes must be in an ordered sequence. In football forecasting, the natural ordering is

$$\text{Home win (H)},\ \text{Draw (D)},\ \text{Away win (A)}.$$

The RPS compares cumulative probabilities rather than raw probabilities, which is what makes it “sensitive to distance”: mass placed on a nearby category can be penalized less than mass placed farther away (Wheatcroft, 2019).

The same paper contrasts the RPS with two standard alternatives. The Brier score is

$\mathrm{Brier}=\sum_{i=1}^{r}(p_{i}-o_{i})^{2},$

which is non-local but does not use the ordering of outcomes. The ignorance score is

$\mathrm{IGN}=-\log_{2}(p(Y)),$

where $p(Y)$ is the probability assigned to the realized outcome $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$0. This score uses only the realized outcome probability and is therefore local (Wheatcroft, 2019).

The central critique of RPS in this setting is that the common justification for distance sensitivity is not accepted as a valid verification principle. Once a match is observed, the outcome reveals only that the realized event occurred; it does not reveal the true probabilities of the unobserved outcomes. On that basis, the paper argues that there is no principled basis for rewarding mass placed on unobserved outcomes merely because they are “close” to the realized one. The broader conclusion is that “distance sensitivity is not inherently a virtue” and that “non-locality is not a virtue in itself” in football-forecast evaluation (Wheatcroft, 2019).

3. Empirical assessment in football forecasting

Two simulation experiments were used to compare the RPS, the Brier score, and the ignorance score in terms of how efficiently they identify a perfect forecasting system from finite samples. In the first experiment, outcomes were repeatedly generated from one of two forecast distributions with equal probability $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$1, one system always issuing the true distribution and the other the alternative. In the second, a more realistic design used 39,343 football matches from bookmaker odds, with an imperfect forecast chosen to be close to the perfect forecast under

$$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$2

Across these experiments, the reported ordering was

$$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$3

with a clearer hierarchy

$$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$4

when the imperfect forecast was very close to the perfect one, such as $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$5 or $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$6 (Wheatcroft, 2019).

The match-level summaries reinforce that conclusion. In the first experiment, ignorance outperformed both RPS and Brier for almost all $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$7 in Match 5, again outperformed for larger $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$8 in Match 1, was slightly better in Match 2, tended to outperform for $$\mathrm{RPS}=\sum_{i=1}^{r-1}\sum_{j=1}^{i}(p_{j}-o_{j})^{2}.$$9 in Match 3, and in Match 4 both ignorance and Brier were noticeably better than RPS. The paper also reports a counterintuitive region in the Match-5 probability space where RPS can reward probability placed on a draw even when the draw is impossible under the data-generating distribution (Wheatcroft, 2019).

The empirical interpretation is explicit. The ignorance score “most efficiently identifies the perfect forecasting system,” RPS “does not outperform the Brier score,” and RPS “is not helped by its distance sensitivity.” The paper therefore recommends using the ignorance score rather than RPS for football forecasts and advises skepticism toward the claim that ordered outcomes automatically justify ordered-outcome scoring rules (Wheatcroft, 2019).

4. Local scale invariance and the scaled CRPS

A different problem arises when proper scoring rules are averaged over observations coming from predictive distributions with different scales. The average score

$p_j$0

can implicitly reweight observations according to their uncertainty. The cited work shows that some widely used proper scoring rules, including CRPS, can give more importance to observations with large uncertainty, which can produce unintuitive rankings in settings such as spatial prediction, stochastic volatility, and count regression (Bolin et al., 2019).

The paper formalizes this through local scale invariance. For location-scale families $p_j$1 with $p_j$2, it states:

“If $p_j$3 exists and satisfies $p_j$4, we say that $p_j$5 is locally scale invariant on $p_j$6.”

This is linked to the local expansion

$p_j$7

Under this criterion, the log score is locally scale invariant, whereas for CRPS the scale function behaves as $p_j$8, so CRPS is not locally scale invariant (Bolin et al., 2019).

The paper’s principal constructive response is the scaled CRPS (SCRPS). Starting from the usual CRPS representation, it defines

$p_j$9

Equivalently,

$j$0

Its defining feature is that the observation-dependent loss term is divided by the forecast’s own intrinsic scale. Errors are therefore penalized relative to forecast uncertainty rather than in raw units. The paper emphasizes that SCRPS remains computationally attractive for ensemble forecasts because, like CRPS, it does not require density evaluation (Bolin et al., 2019).

5. Robustness, outliers, and truncated-kernel constructions

The same literature defines robustness separately from local scale invariance. A scoring rule $j$1 on a normed space $j$2 is called robust if $j$3 is bounded as a function of $j$4 for each $j$5 in its domain. The paper also introduces a model-sensitivity index through asymptotic growth of $j$6 in $j$7, and states that a score is robust if the resulting sensitivity index is $j$8. This notion is motivated by the fact that a small number of outliers can otherwise dominate an average score and alter model rankings (Bolin et al., 2019).

Robust variants are obtained by truncating the kernel. For $j$9,

$o_j$0

This yields a robust CRPS (rCRPS), and the same idea produces a robust standardized version, rSCRPS. The truncation means that sufficiently large deviations are no longer increasingly penalized, which protects average-score comparisons against gross outliers on an absolute scale. The paper’s summary classification is explicit: CRPS is not locally scale invariant and not robust; SCRPS is locally scale invariant and not robust; rCRPS is robust and not locally scale invariant; rSCRPS is robust and not locally scale invariant; the log score and Dawid–Sebastiani score are locally scale invariant and not robust; and the Hyvärinen score is neither (Bolin et al., 2019).

The applications illustrate why these distinctions matter. In stochastic volatility simulations, SCRPS behaved almost identically to the log score and clearly outperformed CRPS and Hyvärinen in selecting the true variance parameter pre-asymptotically. In Gaussian random field prediction, CRPS could overemphasize isolated locations with large predictive variance, whereas SCRPS and the log score selected the correct range parameter more reliably. When a single outlier was added, robust scores reduced the damage from that outlier, and robust SCRPS performed especially well because it retained scale adjustment while protecting against extreme points. In negative binomial regression for pedestrian counts, average CRPS dropped sharply—about half after removing around 20 high-count observations—whereas SCRPS was much less sensitive (Bolin et al., 2019).

A caveat stated in the same work is that a truly scale-adaptive robust score remains open: fixed-$o_j$1 truncation protects against outliers in absolute terms, but not necessarily in relative terms when predictive scale varies (Bolin et al., 2019).

6. Fixed-risk multicategory scoring and tiered warnings

A related but distinct development is the FIRM framework, short for FIxed Risk Multicategory, introduced for ordered multicategorical forecasts and tiered warnings. Its premise is that the verification score should be aligned with the directive actually used in operations: the forecaster should be rewarded for issuing the category that is optimal under a fixed risk threshold, rather than under a threshold that varies from case to case. For ordered categories $o_j$2 with thresholds

$o_j$3

positive weights $o_j$4, and risk parameter $o_j$5, the scoring matrix is

$o_j$6

Entries above the diagonal are penalties for misses, entries below the diagonal are penalties for false alarms, and correct forecasts are unpenalized (Taggart et al., 2021).

The framework is consistent with the directive:

“Forecast any category $o_j$7 which contains an $o_j$8-quantile of the predictive distribution.”

Equivalently, the forecaster should issue “the highest category for which the probability of observing that category or higher exceeds $o_j$9.” In the dichotomous case, this reduces to “Warn if and only if $j$0.” The paper connects this to decision theory and the classical cost-loss model, with the ratio of the relative cost of a miss to a false alarm given by $j$1 (Taggart et al., 2021).

FIRM also includes a discounted variation that reduces penalties for near misses and close false alarms. For threshold $j$2, risk $j$3, and discounting distance $j$4,

$j$5

Its multicategory version is the weighted sum over thresholds. When $j$6 is finite, the penalty is proportional to distance from the threshold up to a cap of $j$7; when $j$8, penalties grow linearly with distance. The score is then consistent with a Huber quantile $j$9, and in the limit $$\text{Home win (H)},\ \text{Draw (D)},\ \text{Away win (A)}.$$0 it becomes an $$\text{Home win (H)},\ \text{Draw (D)},\ \text{Away win (A)}.$$1-expectile (Taggart et al., 2021).

Within the broader “rugged scores” interpretation, FIRM shows that additional structure need not take the form of non-locality alone. It can instead be built around directive consistency, fixed user risk, threshold-specific weights, and optional discounting for near-threshold errors. The paper positions this approach against equitable scores such as the Gandin and Murphy scores, including the Gerrity score, and against metrics such as POD, FAR, CSI, and EDS, arguing that those alternatives can induce threshold rules that vary with base rates or model performance rather than with the intended warning directive (Taggart et al., 2021).