---
title: 'Probability Scale Loss: Foundations & Extensions'
url: https://www.emergentmind.com/topics/probability-scale-loss
type: topic
---

# Probability Scale Loss: Foundations & Extensions

Probability scale loss denotes a loss or scoring rule whose primary argument is a probabilistic prediction—typically a probability $p\in[0,1]$, a probability vector in the simplex $\Delta(\mathcal Y)$, or an entire predictive distribution represented by a cumulative distribution function (CDF)—rather than a single point estimate. In the cited literature, the term covers binary and multiclass proper losses such as the Brier score and log-loss, continuous-outcome scores such as the continuous ranked probability score (CRPS), and related scale-aware constructions that operate on expected probability mass or on multi-scale decompositions of probabilistic fields [2603.15232][1902.10173][2506.10868][2604.09991].

## 1. Formal definitions and scope

For finite-label prediction with $\mathcal Y=\{1,\dots,K\}$, a probabilistic predictor is a measurable map $s:\mathcal X\to\Delta(\mathcal Y)$, and the induced random prediction is $S=s(X)$. A proper probabilistic loss is written as
$$
\ell:\Delta(\mathcal Y)\times \mathcal Y\to\mathbb R,
$$
with conditional risk
$$
L(p,q):=\mathbb E_{Y\sim q}[\ell(p,Y)].
$$
Properness means that for every $q\in\Delta(\mathcal Y)$,
$$
L(q,q)\le L(p,q)\quad\text{for all }p\in\Delta(\mathcal Y),
$$
and strict propriety requires equality only when $p=q$. The associated generalized entropy and proper divergence are
$$
\mathcal E_\ell(q):=L(q,q),\qquad d_\ell(p,q):=L(p,q)-\mathcal E_\ell(q)\ge 0.
$$
These objects provide the standard decision-theoretic formalization of losses on the probability scale [2603.15232].

A binary formulation used in probabilistic prediction writes the loss as
$$
\lambda:[0,1]\times\{0,1\}\to\mathbb R\cup\{+\infty\},
$$
where the predictor outputs a probability $p\in[0,1]$ for label $1$. In the statistical evaluation literature, the related quantity of interest is often a loss $L(p,q)$ that compares the forecast probability $q$ with the true event probability $p$, rather than only with the realized Bernoulli outcome $Y$; this distinction underlies martingale-based inference for forecast evaluation [1502.06254][1202.5140].

This suggests that “probability scale loss” is best understood as a family of losses defined on probabilistic objects. The common structure is that the prediction itself lies on a probability scale—scalar, vector, or distributional—and the loss evaluates the quality of that probabilistic object directly.

## 2. Canonical proper losses on probabilities

The canonical probability-scale losses in binary and multiclass prediction are the Brier score, log-loss, and spherical loss. For multiclass prediction, the Brier score is
$$
\ell(p,y)=\sum_k (p_k-\mathbf 1_{y=k})^2,
$$
and for binary prediction it reduces to
$$
\ell(p,y)=(p-y)^2.
$$
Log-loss is
$$
\ell_{\log}(p,y)=-\log p_y.
$$
In binary form, the paper on the fundamental nature of log loss writes
$$
\lambda_{\text{Brier}}(p,y)=(y-p)^2,
$$
and
$$
\lambda_{\text{log}}(p,y)=-\left[y\ln p+(1-y)\ln(1-p)\right].
$$
Spherical loss is also strictly proper and mixable in that framework [2603.15232][1502.06254].

These losses induce different geometries on the probability scale. For the Brier score,
$$
\mathcal E_\ell(q)=\sum_k q_k(1-q_k),\qquad d_\ell(p,q)=\|p-q\|_2^2,
$$
so the divergence is Euclidean and quadratic. For log-loss,
$$
\mathcal E_\ell(q)=H(q),\qquad d_{\log}(p,q)=\mathrm{KL}(q\|p),
$$
so the divergence is information-theoretic. The decomposition framework therefore interprets Brier as variance-based and log-loss as entropy-based [2603.15232].

A central theoretical distinction concerns selectivity. In the algorithmic theory of randomness, any prediction algorithm that is optimal for a data sequence under log loss is optimal under any computable proper mixable loss; the converse fails for Brier and spherical loss. For CPMS loss functions, the fundamental losses are exactly those of degree $1$, which includes log loss but not Brier or spherical loss [1502.06254]. A common misconception is therefore that all proper losses are interchangeable. They are not: they agree on truthful reporting in expectation, but they do not induce the same notion of optimality, especially near extreme probabilities.

## 3. CRPS as a probability-scale loss for continuous outcomes

For scalar continuous outcomes, the standard probability-scale loss is CRPS. On a finite interval $[a,b]\subset\mathbb R$, for outcome $y\in\mathbb R$ and forecast CDF $F:[a,b]\to[0,1]$,
$$
\operatorname{CRPS}(F,y)=\int_a^b \bigl(F(u)-H(u-y)\bigr)^2\,du,
$$
where $H$ is the Heaviside step function. The same paper also cites a weighted version on the real line,
$$
(F,y)=\int_{-\infty}^{+\infty} \bigl(F(u)-H(u-y)\bigr)^2 \mu(u)\,du,
$$
with nonnegative density $\mu(u)$ [1902.10173].

CRPS is a probability-scale loss because its argument is the whole predictive distribution, via its CDF, and the loss is obtained by integrating over thresholds. For a realized outcome $y$, the “perfect” CDF is $H(u-y)$, which places all mass at $y$. CRPS measures the integrated squared discrepancy between the forecast CDF and this ideal CDF. Equivalently, for each threshold $u$, the event $\{Y\le u\}$ has predicted probability $F(u)$ and realized indicator $H(u-y)$, so CRPS is an integrated Brier score across thresholds. It is therefore both an $L_2$ distance on CDF space and a threshold-integrated proper score [1902.10173].

This formulation distinguishes CRPS from pointwise losses. It is defined directly on the distribution rather than on a mean, median, or quantile; it is proper and sensitive to both calibration and sharpness; and, unlike log-loss, it remains finite even when densities are not absolutely continuous or have bounded support. In the continuous-outcome setting, CRPS occupies the same role that the Brier score occupies for binary events.

## 4. Reliability, information loss, and uncertainty

A major development in the theory of probability-scale losses is the decomposition of expected proper loss into calibration and information components. For any sub-$\sigma$-algebra $\mathcal A\subseteq\mathcal F$, define the conditional law
$$
Q_{\mathcal A}:=\mathbb P(Y\in\cdot\mid\mathcal A).
$$
If $T$ is an $\mathcal A$-measurable predictor, then the one-level decomposition is
$$
\mathbb E[\ell(T,Y)]
=
\mathbb E[d_\ell(T,Q_{\mathcal A})]
+
\mathbb E[\mathcal E_\ell(Q_{\mathcal A})].
$$
For nested information levels $\mathcal A\subseteq\mathcal B$,
$$
\mathbb E[\ell(T,Y)]
=
\mathbb E[d_\ell(T,Q_{\mathcal A})]
+
\mathbb E[d_\ell(Q_{\mathcal A},Q_{\mathcal B})]
+
\mathbb E[\mathcal E_\ell(Q_{\mathcal B})].
$$
The first term is proper-regret at information level $\mathcal A$, the second is information gain from $\mathcal A$ to $\mathcal B$, and the third is residual uncertainty at the richer information level [2603.15232].

For classification with features $X$, score $S=s(X)$, and label $Y$, let
$$
Q=\mathbb P(Y\in\cdot\mid X),\qquad C=\mathbb P(Y\in\cdot\mid S).
$$
Then
$$
\mathbb E[\ell(S,Y)]
=
\mathbb E[d_\ell(S,C)]
+
\mathbb E[d_\ell(C,Q)]
+
\mathbb E[\mathcal E_\ell(Q)].
$$
This three-term identity separates miscalibration, grouping, and irreducible uncertainty. The miscalibration term $\mathbb E[d_\ell(S,C)]$ vanishes iff $S$ is perfectly calibrated; the grouping term $\mathbb E[d_\ell(C,Q)]$ measures information loss from compressing $X$ to $S$; and the final term is the Bayes risk at the feature level [2603.15232].

The specializations clarify the geometry of different losses. For binary Brier loss,
$$
\mathbb E[(S-Y)^2]
=
\mathbb E[(S-C)^2]
+
\mathbb E[(C-Q)^2]
+
\mathbb E[Q(1-Q)].
$$
For log-loss,
$$
\mathbb E[-\log S_Y]
=
\mathbb E[\mathrm{KL}(C\|S)]
+
I(Y;X\mid S)
+
H(Y\mid X).
$$
These identities show that calibration is only one component of probabilistic performance. A perfectly calibrated but coarse score can still incur large grouping loss. They also delimit the effect of post-hoc recalibration: the population-optimal recalibrated score is $C=\mathbb P(Y\in\cdot\mid S)$, which can reduce the miscalibration term but cannot change grouping or irreducible uncertainty. Even aggregation of calibrated models need not preserve calibration [2603.15232].

## 5. Mixability, regret, and online aggregation

Probability-scale losses are especially important in online learning when forecasts from several experts must be aggregated sequentially. In Vovk’s framework, a loss $\lambda(f,y)$ is $\eta$-mixable if there exists a substitution function such that
$$
e^{-\eta\lambda(f,y)}\ge \sum_{i=1}^N q_i e^{-\eta\lambda(c_i,y)}
$$
for all outcomes $y$, given expert forecasts $c_1,\dots,c_N$ and mixture weights $\mathbf q=(q_1,\dots,q_N)$. Mixability yields time-independent regret bounds of order $O(\ln N/\eta)$ [1902.10173].

For continuous outcomes on $[a,b]$, the main theoretical result is that CRPS is $\frac{2}{b-a}$-mixable. In the prediction-with-expert-advice setting, expert $i$ outputs a CDF $F_{i,t}$, the learner outputs an aggregated CDF $F_t$, and the cumulative regret is
$$
R_T
=
\sum_{t=1}^T (F_t,y_t)-\min_i \sum_{t=1}^T (F_{i,t},y_t).
$$
The resulting bound is
$$
\sum_{t=1}^T (F_t,y_t)
\le
\sum_{t=1}^T (F_{i,t},y_t)
+
\frac{b-a}{2}\ln N,
$$
for every expert $i$ and every horizon $T$. The regret is therefore time-independent and logarithmic in the number of experts [1902.10173].

The proof approximates each CDF by a piecewise-constant function on a grid, turns CRPS into a scaled sum of squared component losses, uses that square loss on $[0,1]\times\{0,1\}$ is $2$-mixable, and then lets the grid step go to zero. This places CRPS in the class of mixable proper scoring rules on distributions, closely paralleling the role of the Brier score on binary probabilities [1902.10173].

The same paper also gives practical aggregation algorithms. Vovk’s Aggregating Algorithm uses the CRPS substitution function pointwise in the threshold variable $u$ and updates expert weights with learning rate $\eta=\frac{2}{b-a}$. A simpler exponential-concave weighted average forecaster uses convex combinations of CDFs and attains the weaker bound
$$
\sum_{t=1}^T (F_t,y_t)
\le
\sum_{t=1}^T (F_{i,t},y_t)
+
2(b-a)\ln N.
$$
For specialized experts with confidence values $p_{i,t}\in[0,1]$, the discounted regret guarantee becomes
$$
\sum_{t=1}^T p_{i,t}\bigl((F_t,y_t)-(F_{i,t},y_t)\bigr)\le \frac{\ln N}{\eta}.
$$
These results show that probability-scale losses are not only evaluative devices but also algorithmic primitives for online distributional learning [1902.10173].

## 6. Scale-aware and adaptive extensions

A related line of work extends probability-scale losses to explicitly encode spatial scale or learnable loss scale. In machine-learned weather forecasting at ECMWF, AIFS-CRPS is trained by directly optimizing the almost fair CRPS,
$$
\mathrm{afCRPS}_\alpha:=\alpha\,\mathrm{fCRPS}+(1-\alpha)\,\mathrm{CRPS},\qquad 0<\alpha<1,
$$
and a multi-scale loss is formed by decomposing each field into scale components through smoothing operators $D_i$ and summing afCRPS across scales:
$$
\mathcal L_{n\text{-scale}}
=
\sum_{i=1}^n
\zeta_i\, c\int_{\mathcal M}
\mathcal S\bigl([x_{j,\text{scale}\,i}(q)],\, y_{\text{scale}\,i}(q)\bigr)\,d\mu(q).
$$
In the reported experiments, the model uses two scales with equal weights, a Gaussian filter with standard deviation equal to eight grid spacings, and exhibits nearly identical fCRPS skill to the scale-unaware baseline while better constraining small-scale variability, especially in spectra for smooth fields such as $500\,\mathrm{hPa}$ geopotential height [2506.10868].

In dense segmentation, the network output after sigmoid is a per-pixel probability map $P_{ij}\in(0,1)$, and the global predicted area
$$
A_p=\sum_{i,j} P_{ij}
$$
is interpreted as expected foreground area. The diff-based scale factor is
$$
\alpha(A_p,A_t)
=
\frac{\min(A_p,A_t)+\Delta/2}{\max(A_p,A_t)+\Delta/2},
\qquad
\Delta=|A_p-A_t|,
$$
and the final loss after warm-up is
$$
\mathcal L = 1-\alpha\cdot \mathrm{IoU}+\mathcal L_{\mathrm{loc}}.
$$
Its defining property is strictly monotonic decay as the area mismatch increases, in contrast to the non-monotonic Var-based alternative. On IRSTD-1k, the reported L1-GP-Rotated variant reaches $69.19\%$ mIoU, $94.22\%$ $P_d$, and $7.67\times 10^{-6}$ $F_a$, compared with $68.56\%$, $93.54\%$, and $11.77\times 10^{-6}$ for the Var-based counterpart [2604.09991].

Another extension treats the loss itself as a negative log-likelihood with learnable scale or shape parameters. For Gaussian regression,
$$
\ell_N(\hat y,y;\sigma)=\frac{1}{2\sigma^2}(\hat y-y)^2+\log\sigma,
$$
and for softmax classification with temperature $\tau$,
$$
\ell_{\text{softmax}}(z,y;\tau)
=
-z_y\tau+\log\left(\sum_c e^{z_c\tau}\right).
$$
Jointly optimizing model parameters with likelihood parameters such as variance, temperature, and robust-loss shape produces adaptive loss scaling for robust modeling, outlier-detection, and re-calibration; the same logic extends to $L_2$ and $L_1$ regularization through Gaussian and Laplace priors with learnable scales [2007.06059].

Taken together, these developments indicate that probability-scale loss is not a single standardized object but a coherent class of constructions. In its strictest sense, it denotes proper losses on probabilities or predictive distributions; in adjacent literatures, it also denotes scale-conditioned probabilistic objectives that preserve the central idea that the prediction is itself a probabilistic quantity, and that the loss should act on that quantity at the appropriate statistical or spatial scale.

Source: https://www.emergentmind.com/topics/probability-scale-loss