---
title: Score Decompositions in Forecast Verification
url: https://www.emergentmind.com/topics/score-decompositions
type: topic
---

# Score Decompositions in Forecast Verification

Score decompositions are representations of a score into interpretable components, most prominently in forecast verification and probabilistic machine learning. In the literature on proper scoring rules, the central objective is to partition a mean or expected score into terms attributable to calibration error, discrimination or resolution, and baseline uncertainty; in more recent work the same logic is extended to information loss under coarsening, multivariate level sets, interval forecasts, and formal statistical inference. Related but technically distinct usages also arise in score-function methods for discriminative learning, sparse PCA, and score-based diffusion models [2603.15232], [2311.14122].

## 1. General structure of score decompositions

For a proper scoring rule $\ell$ on a finite outcome space, a predictor $T$ measurable with respect to a sub-$\sigma$-algebra $\mathcal A$, and the conditional law $Q_{\mathcal A}=\mathbb P(Y\in\cdot\mid\mathcal A)$, the expected loss admits the one-level decomposition
\[
\mathbb E[\ell(T,Y)]
=
\mathbb E[d_\ell(T,Q_{\mathcal A})]
+
\mathbb E[\mathcal E_\ell(Q_{\mathcal A})],
\]
where $d_\ell$ is the proper-regret term and $\mathcal E_\ell$ is the generalized entropy or Bayes risk. This identity separates the regret of using $T$ instead of the oracle at information level $\mathcal A$ from the residual uncertainty that remains even with knowledge of $\mathcal A$ [2603.15232].

If $\mathcal A\subseteq\mathcal B$, the same framework yields a chain rule,
\[
\mathbb E[\ell(T,Y)]
=
\mathbb E[d_\ell(T,Q_{\mathcal A})]
+
\mathbb E[d_\ell(Q_{\mathcal A},Q_{\mathcal B})]
+
\mathbb E[\mathcal E_\ell(Q_{\mathcal B})].
\]
The second term measures the information gain from $\mathcal B$ to $\mathcal A$. In probabilistic classification, with features $X$, score $S=s(X)$, $C=\mathbb P(Y\in\cdot\mid S)$, and $Q=\mathbb P(Y\in\cdot\mid X)$, this becomes
\[
\mathbb E[\ell(S,Y)]
=
\mathbb E[d_\ell(S,C)]
+
\mathbb E[d_\ell(C,Q)]
+
\mathbb E[\mathcal E_\ell(Q)].
\]
Here the first term is miscalibration, the second is a grouping term measuring information loss from $X$ to $S$, and the third is irreducible uncertainty at the feature level [2603.15232].

The framework recovers familiar special cases. For the binary Brier score,
\[
\mathbb E[(S-Y)^2]
=
\mathbb E[(S-C)^2]
+
\mathbb E[(C-Q)^2]
+
\mathbb E[Q(1-Q)],
\]
and for log-loss,
\[
\mathbb E[-\log S_Y]
=
\mathbb E[\mathrm{KL}(C\|S)]
+
I(Y;X\mid S)
+
H(Y\mid X).
\]
This generalization is consequential because it makes explicit that calibration is conditional on the information retained by the predictor, not solely on the nominal forecast values [2603.15232].

## 2. Classical Brier-score and ranked-probability decompositions

The classical point of departure is Murphy’s binary Brier-score factorization
\[
E[(P-X)^2]
=
\underbrace{\mathrm{Var}(X)}_{\mathrm{UNC}}
-
\underbrace{E_P\bigl(E[X\mid P]-E[X]\bigr)^2}_{\mathrm{RES}}
+
\underbrace{E_P\bigl(E[X\mid P]-P\bigr)^2}_{\mathrm{REL}},
\]
which writes the expected score as uncertainty minus resolution plus reliability. In this convention, uncertainty is intrinsic to the event, resolution quantifies the ability to move away from the base rate, and reliability measures calibration error [2106.14345].

For three-category forecasts such as Home Win, Draw, and Away Win, the multicategory Brier score is
\[
BS=\frac1N\sum_{i=1}^N\sum_{j\in\{H,D,A\}}(p_{i,j}-o_{i,j})^2,
\]
and the Ranked Probability Score is
\[
RPS=\frac1N\sum_{i=1}^N\sum_{k=1}^{J-1}\Bigl(\sum_{j=1}^k p_{i,j}-\sum_{j=1}^k o_{i,j}\Bigr)^2.
\]
The three-category score decomposes as the sum of three one-vs-all binary decompositions,
\[
BS=\sum_{j\in\{H,D,A\}}[\mathrm{UNC}_j-\mathrm{RES}_j+\mathrm{REL}_j]
=\mathrm{UNC}-\mathrm{RES}+\mathrm{REL},
\]
with bin-based estimators for $\mathrm{RES}_j$ and $\mathrm{REL}_j$ defined through empirical event frequencies and average forecasts within bins [2106.14345].

A second family of decompositions replaces the calibration–refinement viewpoint with likelihood-base terminology. For each binary event,
\[
E[(P-X)^2]
=
\underbrace{\mathrm{Var}(P)}_{\mathrm{REF}}
-
\underbrace{\mathrm{Var}_X(E[P\mid X])}_{\mathrm{DIS}}
+
\underbrace{E_X(E[P\mid X]-X)^2}_{CB2}.
\]
Yates’s five-term decomposition is closely related, isolating discrimination, within-group variance, and marginal bias. These identities motivated graphical diagnostics such as reliability diagrams and discrimination diagrams, as well as numerical summaries including Reliability $=100\times REL/UNC$, Resolution $=100\times RES/UNC$, Discrimination $=100\times DIS/UNC$, and the Brier Skill Score $(RES-REL)/UNC=1-BS/UNC$ [2106.14345].

The applied implication is that mean score values alone are not sufficient for verification. In football forecasting, for example, reliability for Home/Away wins was typically small, resolution for draws was only $1$–$7\%$ of uncertainty, and bookmakers’ implied odds showed slightly higher resolution and discrimination than Poisson-model forecasts, even when both appeared reasonably reliable [2106.14345].

## 3. Empirical decompositions, isotonic regression, and CORP

Classical empirical Brier decompositions depend on ad hoc binning. If forecasts are grouped into bins $k=1,\dots,K$ with bin weights $w_k$, bin-average forecasts $p_k$, empirical frequencies $o_k$, and overall mean $\bar p=\bar y$, then
\[
\overline{BS}
=
\underbrace{\bar p(1-\bar p)}_{\mathrm{Uncertainty}}
-
\underbrace{\sum_{k=1}^K w_k(p_k-\bar p)^2}_{\mathrm{Resolution}}
+
\underbrace{\sum_{k=1}^K w_k(o_k-p_k)^2}_{\mathrm{Reliability}}.
\]
The limitation is that the decomposition can be unstable under arbitrary implementation choices for the bins [2008.03033].

The CORP approach replaces fixed binning by isotonic regression. Given sorted forecasts $x_i\in[0,1]$ and outcomes $y_i\in\{0,1\}$, it solves
\[
\min_{q_1\le\cdots\le q_n}\sum_{i=1}^n(q_i-y_i)^2
\]
by the Pool-Adjacent-Violators algorithm, producing calibrated values $\hat x_i$. For the Brier score, with $\bar S_x=n^{-1}\sum_i(x_i-y_i)^2$, $\bar S_{\hat x}=n^{-1}\sum_i(\hat x_i-y_i)^2$, and $\bar S_r=n^{-1}\sum_i(r-y_i)^2$ for $r=\bar y$, the exact decomposition is
\[
\bar S_x
=
\underbrace{(\bar S_x-\bar S_{\hat x})}_{\mathrm{MCB}}
-
\underbrace{(\bar S_r-\bar S_{\hat x})}_{\mathrm{DSC}}
+
\underbrace{\bar S_r}_{\mathrm{UNC}}.
\]
The same identity extends to any strictly proper scoring rule $S$ by replacing squared error with $S(x_i,y_i)$ [2008.03033].

CORP has three stated properties: consistency, optimality, and reproducibility. Under mild regularity, the PAV-based reliability curve converges to the population conditional event probability; in the discrete case the mean-squared error decays at rate $n^{-1}$, and in the continuous case at the Chernoff rate $n^{-2/3}$. The decomposition is exact, all three components are nonnegative, and the construction is tuning-free because the bins are uniquely determined by the PAV algorithm [2008.03033].

This literature also clarifies a recurring misconception: recalibrated probabilities obtained from isotonic regression are not merely a graphical aid. They define a score decomposition, a numerical measure of miscalibration, and uncertainty quantification through resampling or asymptotic theory, and they do so for arbitrary proper scoring rules rather than only for the Brier score [2008.03033].

## 4. CRPS, interval scores, and isotonicity-based decompositions

For real-valued outcomes, the continuous ranked probability score is
\[
\mathrm{CRPS}(F,y)=\int_{-\infty}^{\infty}\bigl(F(z)-\mathbf 1\{y\le z\}\bigr)^2\,dz
=
\int_0^1 \mathrm{QS}_\alpha(F^{-1}(\alpha),y)\,d\alpha,
\]
and the empirical target is the mean CRPS over forecast–outcome pairs. Earlier decompositions include the Candille–Talagrand decomposition, Brier-score-based and quantile-score-based decompositions, and Hersbach’s decomposition for ensembles. These differ in whether they are exact, nondegenerate, nonnegative, and applicable to general forecast types [2311.14122].

The Candille–Talagrand decomposition is exact and has nonnegative components in population theory, but it degenerates in practice when all forecasts are distinct, yielding $\mathrm{MSC}=\overline{\mathrm{CRPS}}$ and $\mathrm{DSC}=0$. The BS- and QS-based decompositions are nondegenerate, nonnegative, and tuning-free, but they treat thresholds or quantiles in isolation, and their calibrated forecasts need not be genuine cdfs. Hersbach’s decomposition is exact and nondegenerate for ensemble forecasts, but its discrimination term can be negative [2311.14122].

The isotonicity-based decomposition remedies these limitations by viewing the forecast cdf $F_i$ as a covariate in the partially ordered space of cdfs under stochastic dominance. Isotonic Distributional Regression produces calibrated forecasts $\check F_i$ minimizing average CRPS subject to isotonicity, and then defines
\[
\mathrm{MSC}
=
\overline{\mathrm{CRPS}}-\overline{\mathrm{CRPS}}_{\mathrm{iso}},
\qquad
\mathrm{DSC}
=
\overline{\mathrm{CRPS}}_{\mathrm{iso}}-\overline{\mathrm{CRPS}}_\ast,
\qquad
\mathrm{UNC}
=
\overline{\mathrm{CRPS}}_\ast,
\]
so that
\[
\overline{\mathrm{CRPS}}=\mathrm{MSC}-\mathrm{DSC}+\mathrm{UNC}.
\]
Each component is nonnegative, the decomposition is exact and nondegenerate, it is trivial for static forecasts, and the uncertainty term depends only on the observations [2311.14122].

At population level, the decomposition induces a calibration hierarchy:
\[
\text{auto-calibration}
\Longrightarrow
\text{isotonic calibration}
\Longrightarrow
\text{threshold and quantile calibration}
\Longrightarrow
\text{probabilistic calibration}.
\]
Correspondingly,
\[
\mathrm{MSC}_{\mathrm{CT}}
\ge
\mathrm{MSC}_{\mathrm{ISO}}
\ge
\max\{\mathrm{MSC}_{\mathrm{BS}},\mathrm{MSC}_{\mathrm{QS}}\}.
\]
Algorithmically, stochastic-order determination is $O(n^2\cdot m)$ for $m$-point cdfs or $O(n^2)$ on a fixed grid, and the IDR step is $O(n^2)$ [2311.14122].

An analogous program applies to interval forecasts. For a central $(1-\alpha)$ prediction interval $[\ell,u]$, the interval score is
\[
S_\alpha(\ell,u;y)
=
(u-\ell)
+
\frac{2}{\alpha}(\ell-y)\mathbf 1\{y<\ell\}
+
\frac{2}{\alpha}(y-u)\mathbf 1\{y>u\}.
\]
It can be decomposed at population level into uncertainty, discrimination, and miscalibration under either auto-calibration or isotonic calibration, and the isotonic version is operationalized by applying IDR to interval covariates ordered by
\[
[\ell_i,u_i]\le[\ell_j,u_j]
\Longleftrightarrow
\ell_i\le\ell_j\ \text{and}\ u_i\le u_j.
\]
The sample decomposition is exact, all terms are nonnegative, and the miscalibration term vanishes exactly under empirical isotonic calibration. The authors emphasize that unconditional coverage checks are relatively weak and that decomposition of the interval score provides a diagnostic for conditional calibration [2508.18034].

## 5. Multivariate distributions and level-set decompositions

For multivariate predictive distributions, a broad $L^2$ framework defines scores of the form
\[
S(P,y;w,h)
=
\int_{\mathbb R^d}\bigl[(f_P*w)(z)-w(z-y)\bigr]^2 h(z)\,dz,
\]
where $w$ is a smoothing function and $h$ is the density of a benchmark measure. The induced divergence is nonnegative, so the score is proper under mild regularity. This framework encompasses the quadratic score and the multivariate continuous ranked probability score [2002.09578].

The key decomposition theorem is a layer-cake representation. If $L_\alpha(P)=\{z:(f_P*w)(z)\ge\alpha\}$ is an upper level set, then a renormalized score can be written as
\[
S'(P,y;w,h)=2\int_0^\infty s_\alpha(L_\alpha(P),y)\,d\alpha,
\]
with
\[
s_\alpha(A,y)
=
\int_{\mathbb R^d}[\alpha-w(z-y)]\,1\{z\in A\}\,h(z)\,dz.
\]
Each $s_\alpha$ is a consistent scoring function for the set-valued functional $P\mapsto L_\alpha(P)$, and strict consistency holds under stated regularity conditions [2002.09578].

Two specializations are central. With $w=\delta_0$, the decomposition recovers density level sets for the quadratic score; with $w(u)=1\{u\ge0\}$, it recovers cumulative level sets for the multivariate CRPS. In dimension one, the latter reproduces the classical expression of CRPS as an integral over the quantile score. The same framework also generates scoring functions for lower-partial-moment level sets [2002.09578].

This level-set viewpoint is practically useful. A simple Monte Carlo algorithm approximates both global scores and level-set scores, with cost $O(M)$ per sum and $M\approx10^4$ sufficient for two-dimensional examples. In simulation with $T=200{,}000$ draws from a bivariate Gaussian data-generating process and $13$ misspecified bivariate normals, the true distribution attains the lowest score in every case. In applications, the quadratic structure of the score enables forecast-combination weights to be estimated by quadratic programming, and a level-set score for the MCRPS can be used for CoVaR estimation [2002.09578].

## 6. Recalibration, aggregation, stagewise refinement, and inference

The information-level framework has immediate consequences for recalibration. If recalibrated predictors are restricted to the form $\hat S=g(S)$ with $g$ measurable with respect to $\sigma(S)$, the population-optimal recalibration is $C=\mathbb P(Y\mid S)$, and for any $g$,
\[
\mathbb E[\ell(g(S),Y)]-\mathbb E[\ell(C,Y)]
=
\mathbb E[d_\ell(g(S),C)].
\]
Thus recalibration can eliminate the miscalibration term but cannot alter the grouping term $\mathbb E[d_\ell(C,Q)]$, which is the information lost when replacing $X$ by $S$ [2603.15232].

The same framework shows that aggregation of calibrated models is delicate. Even if each component score is calibrated, the aggregated score need not be. Calibration of the ensemble requires sufficiency of the aggregate or, equivalently, existence of a map $\phi$ such that $\mathbb P(Y\mid S^{(1:M)})=\phi(S^{\mathrm{ens}})$. For stagewise or boosting constructions with a filtration $\mathcal F_0\subseteq\cdots\subseteq\mathcal F_T$, the loss decomposes into an initial regret term, a sum of refinement gains $\sum_t \mathbb E[d_\ell(Q_t,Q_{t+1})]$, and residual Bayes risk. Under log-loss these increments are conditional mutual informations, and under the Brier score they reduce to the Doob–Pythagoras identity [2603.15232].

Formal inference for score decompositions has recently been developed through linear recalibration. With a strictly consistent scoring function $S$, point forecast $X_{i_t}$, and recalibration covariates $z_{i_t}=(1,X_{i_t})^\top$, the parameter $\theta_i$ is estimated by
\[
\widehat\theta_{iT}
=
\arg\min_{\theta\in\Theta}\frac1T\sum_{t=1}^T S(z_{i_t}^\top\theta,Y_t),
\]
and the resulting sample decomposition yields finite-sample nonnegativity whenever the minimizations nest constants and the identity line. Under squared error,
\[
\widehat{MC}_{iT}
=
\widehat\theta_{i0}^2
+
\frac1T\sum_t (X_{i_t}-\bar X_i)^2(\widehat\theta_{i1}-1)^2,
\qquad
\widehat{DS}_{iT}
=
\widehat R_i^2\,\widehat\sigma_Y^2.
\]
This establishes a direct connection to Mincer–Zarnowitz regression [2603.04275].

The inferential theory distinguishes interior and boundary cases. Under stationarity, mixing, moment bounds, correct linear recalibration, and smoothness conditions, the joint vector of estimated miscalibration and discrimination terms is asymptotically normal when the population components are strictly positive. If either component lies on the boundary, $T\widehat{MC}_{iT}$ or $T\widehat{DS}_{iT}$ converges to a generalized-$\chi^2$ law. The framework supports tests for equal miscalibration and equal discrimination, and setting $(MC,DS)\to S$ recovers the classical Diebold–Mariano test for equal average scores [2603.04275].

Empirically, these tests reveal information that mean scores can conceal. For U.S. CPI inflation surveys, the overall score difference between SPF and Michigan forecasts is insignificant ($p=0.284$), but professional forecasters display significantly higher discrimination ($p=0.037$). In financial risk applications, historical simulation has low miscalibration and passes unconditional and conditional coverage backtests, but it has almost zero discrimination and a very poor score; by contrast, RV-based models fail calibration backtests yet dominate in discrimination and overall score [2603.04275].

## 7. Other technical meanings of score decomposition

The phrase “score decomposition” is not confined to forecast evaluation. In discriminative learning from unlabeled data, higher-order score functions
\[
S^{(m)}(x)=(-1)^m\frac{\nabla_x^{(m)}p(x)}{p(x)}
\]
yield matrix- and tensor-valued features. Their cross-moments with labels satisfy
\[
\mathbb E[y\otimes S^{(m)}(x)]
=
\mathbb E[\nabla_x^{(m)}G(x)],
\]
so second- and third-order moments can be decomposed by matrix SVD or CP tensor decomposition to recover discriminative directions. For third-order tensors, the tensor power method with deflation converges geometrically in $O(\log(1/\epsilon))$ steps under standard conditions, and overcomplete recovery is possible up to $r\le O(d^{3/2})$ [1412.6514].

In sparse PCA, “scores” are factor scores in a matrix factorization $X\approx TP^\top$. Because sparse loading matrices typically satisfy $P^\top P\neq I$, the classical PCA shortcut $T=XP$ fails. The correct least-squares scores are
\[
\hat T=X\hat P(\hat P^\top\hat P)^{-1},
\]
or with the Moore–Penrose inverse in rank-deficient cases, and explained variance must be computed from $\|\hat T\hat P^\top\|_F^2$ rather than $\|\hat T\|_F^2$. The paper’s central claim is that uncorrected scores inflate residuals and mis-allocate explained variance [1907.03989].

In score-based diffusion modeling, the “score” is $\nabla\log q$ for a Gaussian-convolved density. For a linear tilt $p_\ell(x)\propto e^{\ell^\top x}p_0(x)$, the score satisfies
\[
\nabla\log(p_\ell*\mathcal N)(u)
=
\ell+\nabla\log(p_0*\mathcal N)(u+\sigma^2\ell),
\]
so a linear tilt induces a location shift and an additive constant. For a negative-quadratic tilt $p_Q(x)\propto e^{-x^\top Qx/2}p_0(x)$, the transformed score involves both a location shift $u\mapsto (I+\sigma^2Q)^{-1}u$ and a time or noise-level shift $\sigma\mapsto \tau=\sigma(I+\sigma^2Q)^{-1/2}$. These identities are presented as a way to reuse pretrained score networks under controlled tilts [2604.27196].

Taken together, these usages show that “score decomposition” names a family of ideas rather than a single formula. In probabilistic forecast verification it refers primarily to exact or asymptotic partitions into miscalibration, discrimination or resolution, and uncertainty; in adjacent machine-learning literatures it refers to spectral decomposition of score-function moments, correction of non-orthogonal factor scores, or transport identities for diffusion-model scores.

Source: https://www.emergentmind.com/topics/score-decompositions