Score Decompositions in Forecast Verification

• Score decompositions are defined as splitting proper scoring rules into interpretable components such as miscalibration, discrimination (or resolution), and baseline uncertainty.
• The framework extends classical decompositions like the Brier score to continuous, multivariate, and interval forecasts, enhancing clarity in forecast evaluation.
• Recent methods, including isotonic regression and the CORP approach, provide tuning-free, exact decompositions that aid in recalibration and statistical inference.

Score decompositions are representations of a score into interpretable components, most prominently in forecast verification and probabilistic machine learning. In the literature on proper scoring rules, the central objective is to partition a mean or expected score into terms attributable to calibration error, discrimination or resolution, and baseline uncertainty; in more recent work the same logic is extended to information loss under coarsening, multivariate level sets, interval forecasts, and formal statistical inference. Related but technically distinct usages also arise in score-function methods for discriminative learning, sparse PCA, and score-based diffusion models (Charpentier et al., 16 Mar 2026, Arnold et al., 2023).

1. General structure of score decompositions

For a proper scoring rule $\ell$ on a finite outcome space, a predictor $T$ measurable with respect to a sub-$\sigma$-algebra $\mathcal A$, and the conditional law $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$, the expected loss admits the one-level decomposition

$$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$

where $d_\ell$ is the proper-regret term and $\mathcal E_\ell$ is the generalized entropy or Bayes risk. This identity separates the regret of using $T$ instead of the oracle at information level $\mathcal A$ from the residual uncertainty that remains even with knowledge of $T$0 (Charpentier et al., 16 Mar 2026).

If $T$1, the same framework yields a chain rule,

$T$2

The second term measures the information gain from $T$3 to $T$4. In probabilistic classification, with features $T$5, score $T$6, $T$7, and $T$8, this becomes

$T$9

Here the first term is miscalibration, the second is a grouping term measuring information loss from $\sigma$0 to $\sigma$1, and the third is irreducible uncertainty at the feature level (Charpentier et al., 16 Mar 2026).

The framework recovers familiar special cases. For the binary Brier score,

$\sigma$2

and for log-loss,

$\sigma$3

This generalization is consequential because it makes explicit that calibration is conditional on the information retained by the predictor, not solely on the nominal forecast values (Charpentier et al., 16 Mar 2026).

2. Classical Brier-score and ranked-probability decompositions

The classical point of departure is Murphy’s binary Brier-score factorization

$\sigma$4

which writes the expected score as uncertainty minus resolution plus reliability. In this convention, uncertainty is intrinsic to the event, resolution quantifies the ability to move away from the base rate, and reliability measures calibration error (Foulley, 2021).

For three-category forecasts such as Home Win, Draw, and Away Win, the multicategory Brier score is

$\sigma$5

and the Ranked Probability Score is

$\sigma$6

The three-category score decomposes as the sum of three one-vs-all binary decompositions,

$\sigma$7

with bin-based estimators for $\sigma$8 and $\sigma$9 defined through empirical event frequencies and average forecasts within bins (Foulley, 2021).

A second family of decompositions replaces the calibration–refinement viewpoint with likelihood-base terminology. For each binary event,

$\mathcal A$0

Yates’s five-term decomposition is closely related, isolating discrimination, within-group variance, and marginal bias. These identities motivated graphical diagnostics such as reliability diagrams and discrimination diagrams, as well as numerical summaries including Reliability $\mathcal A$1, Resolution $\mathcal A$2, Discrimination $\mathcal A$3, and the Brier Skill Score $\mathcal A$4 (Foulley, 2021).

The applied implication is that mean score values alone are not sufficient for verification. In football forecasting, for example, reliability for Home/Away wins was typically small, resolution for draws was only $\mathcal A$5–$\mathcal A$6 of uncertainty, and bookmakers’ implied odds showed slightly higher resolution and discrimination than Poisson-model forecasts, even when both appeared reasonably reliable (Foulley, 2021).

3. Empirical decompositions, isotonic regression, and CORP

Classical empirical Brier decompositions depend on ad hoc binning. If forecasts are grouped into bins $\mathcal A$7 with bin weights $\mathcal A$8, bin-average forecasts $\mathcal A$9, empirical frequencies $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$0, and overall mean $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$1, then

$Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$2

The limitation is that the decomposition can be unstable under arbitrary implementation choices for the bins (Dimitriadis et al., 2020).

The CORP approach replaces fixed binning by isotonic regression. Given sorted forecasts $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$3 and outcomes $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$4, it solves

$Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$5

by the Pool-Adjacent-Violators algorithm, producing calibrated values $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$6. For the Brier score, with $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$7, $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$8, and $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$9 for $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$0, the exact decomposition is

$$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$1

The same identity extends to any strictly proper scoring rule $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$2 by replacing squared error with $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$3 (Dimitriadis et al., 2020).

CORP has three stated properties: consistency, optimality, and reproducibility. Under mild regularity, the PAV-based reliability curve converges to the population conditional event probability; in the discrete case the mean-squared error decays at rate $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$4, and in the continuous case at the Chernoff rate $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$5. The decomposition is exact, all three components are nonnegative, and the construction is tuning-free because the bins are uniquely determined by the PAV algorithm (Dimitriadis et al., 2020).

This literature also clarifies a recurring misconception: recalibrated probabilities obtained from isotonic regression are not merely a graphical aid. They define a score decomposition, a numerical measure of miscalibration, and uncertainty quantification through resampling or asymptotic theory, and they do so for arbitrary proper scoring rules rather than only for the Brier score (Dimitriadis et al., 2020).

4. CRPS, interval scores, and isotonicity-based decompositions

For real-valued outcomes, the continuous ranked probability score is

$$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$6

and the empirical target is the mean CRPS over forecast–outcome pairs. Earlier decompositions include the Candille–Talagrand decomposition, Brier-score-based and quantile-score-based decompositions, and Hersbach’s decomposition for ensembles. These differ in whether they are exact, nondegenerate, nonnegative, and applicable to general forecast types (Arnold et al., 2023).

The Candille–Talagrand decomposition is exact and has nonnegative components in population theory, but it degenerates in practice when all forecasts are distinct, yielding $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$7 and $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$8. The BS- and QS-based decompositions are nondegenerate, nonnegative, and tuning-free, but they treat thresholds or quantiles in isolation, and their calibrated forecasts need not be genuine cdfs. Hersbach’s decomposition is exact and nondegenerate for ensemble forecasts, but its discrimination term can be negative (Arnold et al., 2023).

The isotonicity-based decomposition remedies these limitations by viewing the forecast cdf $$\mathbb E[\ell(T,Y)] = \mathbb E[d_\ell(T,Q_{\mathcal A})] + \mathbb E[\mathcal E_\ell(Q_{\mathcal A})],$$9 as a covariate in the partially ordered space of cdfs under stochastic dominance. Isotonic Distributional Regression produces calibrated forecasts $d_\ell$0 minimizing average CRPS subject to isotonicity, and then defines

$d_\ell$1

so that

$d_\ell$2

Each component is nonnegative, the decomposition is exact and nondegenerate, it is trivial for static forecasts, and the uncertainty term depends only on the observations (Arnold et al., 2023).

At population level, the decomposition induces a calibration hierarchy: $d_\ell$3 Correspondingly,

$d_\ell$4

Algorithmically, stochastic-order determination is $d_\ell$5 for $d_\ell$6-point cdfs or $d_\ell$7 on a fixed grid, and the IDR step is $d_\ell$8 (Arnold et al., 2023).

An analogous program applies to interval forecasts. For a central $d_\ell$9 prediction interval $\mathcal E_\ell$0, the interval score is

$\mathcal E_\ell$1

It can be decomposed at population level into uncertainty, discrimination, and miscalibration under either auto-calibration or isotonic calibration, and the isotonic version is operationalized by applying IDR to interval covariates ordered by

$\mathcal E_\ell$2

The sample decomposition is exact, all terms are nonnegative, and the miscalibration term vanishes exactly under empirical isotonic calibration. The authors emphasize that unconditional coverage checks are relatively weak and that decomposition of the interval score provides a diagnostic for conditional calibration (Allen et al., 25 Aug 2025).

5. Multivariate distributions and level-set decompositions

For multivariate predictive distributions, a broad $\mathcal E_\ell$3 framework defines scores of the form

$\mathcal E_\ell$4

where $\mathcal E_\ell$5 is a smoothing function and $\mathcal E_\ell$6 is the density of a benchmark measure. The induced divergence is nonnegative, so the score is proper under mild regularity. This framework encompasses the quadratic score and the multivariate continuous ranked probability score (Meng et al., 2020).

The key decomposition theorem is a layer-cake representation. If $\mathcal E_\ell$7 is an upper level set, then a renormalized score can be written as

$\mathcal E_\ell$8

with

$\mathcal E_\ell$9

Each $T$0 is a consistent scoring function for the set-valued functional $T$1, and strict consistency holds under stated regularity conditions (Meng et al., 2020).

Two specializations are central. With $T$2, the decomposition recovers density level sets for the quadratic score; with $T$3, it recovers cumulative level sets for the multivariate CRPS. In dimension one, the latter reproduces the classical expression of CRPS as an integral over the quantile score. The same framework also generates scoring functions for lower-partial-moment level sets (Meng et al., 2020).

This level-set viewpoint is practically useful. A simple Monte Carlo algorithm approximates both global scores and level-set scores, with cost $T$4 per sum and $T$5 sufficient for two-dimensional examples. In simulation with $T$6 draws from a bivariate Gaussian data-generating process and $T$7 misspecified bivariate normals, the true distribution attains the lowest score in every case. In applications, the quadratic structure of the score enables forecast-combination weights to be estimated by quadratic programming, and a level-set score for the MCRPS can be used for CoVaR estimation (Meng et al., 2020).

6. Recalibration, aggregation, stagewise refinement, and inference

The information-level framework has immediate consequences for recalibration. If recalibrated predictors are restricted to the form $T$8 with $T$9 measurable with respect to $\mathcal A$0, the population-optimal recalibration is $\mathcal A$1, and for any $\mathcal A$2,

$\mathcal A$3

Thus recalibration can eliminate the miscalibration term but cannot alter the grouping term $\mathcal A$4, which is the information lost when replacing $\mathcal A$5 by $\mathcal A$6 (Charpentier et al., 16 Mar 2026).

The same framework shows that aggregation of calibrated models is delicate. Even if each component score is calibrated, the aggregated score need not be. Calibration of the ensemble requires sufficiency of the aggregate or, equivalently, existence of a map $\mathcal A$7 such that $\mathcal A$8. For stagewise or boosting constructions with a filtration $\mathcal A$9, the loss decomposes into an initial regret term, a sum of refinement gains $T$00, and residual Bayes risk. Under log-loss these increments are conditional mutual informations, and under the Brier score they reduce to the Doob–Pythagoras identity (Charpentier et al., 16 Mar 2026).

Formal inference for score decompositions has recently been developed through linear recalibration. With a strictly consistent scoring function $T$01, point forecast $T$02, and recalibration covariates $T$03, the parameter $T$04 is estimated by

$T$05

and the resulting sample decomposition yields finite-sample nonnegativity whenever the minimizations nest constants and the identity line. Under squared error,

$T$06

This establishes a direct connection to Mincer–Zarnowitz regression (Dimitriadis et al., 4 Mar 2026).

The inferential theory distinguishes interior and boundary cases. Under stationarity, mixing, moment bounds, correct linear recalibration, and smoothness conditions, the joint vector of estimated miscalibration and discrimination terms is asymptotically normal when the population components are strictly positive. If either component lies on the boundary, $T$07 or $T$08 converges to a generalized-$T$09 law. The framework supports tests for equal miscalibration and equal discrimination, and setting $T$10 recovers the classical Diebold–Mariano test for equal average scores (Dimitriadis et al., 4 Mar 2026).

Empirically, these tests reveal information that mean scores can conceal. For U.S. CPI inflation surveys, the overall score difference between SPF and Michigan forecasts is insignificant ($T$11), but professional forecasters display significantly higher discrimination ($T$12). In financial risk applications, historical simulation has low miscalibration and passes unconditional and conditional coverage backtests, but it has almost zero discrimination and a very poor score; by contrast, RV-based models fail calibration backtests yet dominate in discrimination and overall score (Dimitriadis et al., 4 Mar 2026).

7. Other technical meanings of score decomposition

The phrase “score decomposition” is not confined to forecast evaluation. In discriminative learning from unlabeled data, higher-order score functions

$T$13

yield matrix- and tensor-valued features. Their cross-moments with labels satisfy

$T$14

so second- and third-order moments can be decomposed by matrix SVD or CP tensor decomposition to recover discriminative directions. For third-order tensors, the tensor power method with deflation converges geometrically in $T$15 steps under standard conditions, and overcomplete recovery is possible up to $T$16 (Janzamin et al., 2014).

In sparse PCA, “scores” are factor scores in a matrix factorization $T$17. Because sparse loading matrices typically satisfy $T$18, the classical PCA shortcut $T$19 fails. The correct least-squares scores are

$T$20

or with the Moore–Penrose inverse in rank-deficient cases, and explained variance must be computed from $T$21 rather than $T$22. The paper’s central claim is that uncorrected scores inflate residuals and mis-allocate explained variance (Camacho et al., 2019).

In score-based diffusion modeling, the “score” is $T$23 for a Gaussian-convolved density. For a linear tilt $T$24, the score satisfies

$T$25

so a linear tilt induces a location shift and an additive constant. For a negative-quadratic tilt $T$26, the transformed score involves both a location shift $T$27 and a time or noise-level shift $T$28. These identities are presented as a way to reuse pretrained score networks under controlled tilts (McDonald, 29 Apr 2026).

Taken together, these usages show that “score decomposition” names a family of ideas rather than a single formula. In probabilistic forecast verification it refers primarily to exact or asymptotic partitions into miscalibration, discrimination or resolution, and uncertainty; in adjacent machine-learning literatures it refers to spectral decomposition of score-function moments, correction of non-orthogonal factor scores, or transport identities for diffusion-model scores.